Quantum RL

Quantum Reinforcement Learning

Quantum reinforcement learning (QRL) uses quantum circuits to represent policies or value functions:

1. Quantum policy: parameterized quantum circuit maps states to actions
2. Quantum value function: quantum circuit estimates state values
3. Quantum exploration: superposition enables simultaneous exploration of actions

QRL can potentially achieve quadratic speedups in exploration and representational capacity.

Quantum Bandits

Quantum bandits use quantum superposition to explore arms simultaneously:

Classical bandit: try one arm at a time, regret Quantum bandit: try all arms in superposition, regret

The quantum advantage comes from the ability to evaluate multiple actions in superposition and use interference to amplify good actions.

Quantum Policy Gradient

Quantum policy gradient extends the REINFORCE algorithm:

where is the quantum policy.

The gradient is estimated using the parameter-shift rule:

Quantum Exploration

Quantum superposition enables efficient exploration:

1. Superposition of actions: evaluate multiple actions simultaneously
2. Interference: amplify rewards for good actions
3. Entanglement: correlate actions across time steps

The quantum exploration bonus:

where is the number of visits to state . Quantum exploration can achieve this bound with fewer samples.

Variational Quantum RL

Variational quantum RL uses parameterized quantum circuits as function approximators:

1. State encoding: encode environment state into quantum state
2. Policy network: quantum circuit with trainable parameters
3. Action selection: measure to get action
4. Reward signal: classical reward updates parameters

The quantum circuit can represent complex policies with fewer parameters than classical networks.

Applications

Quantum RL applications:

1. Game playing: quantum-enhanced game AI
2. Robotics: quantum control of robots
3. Finance: quantum trading strategies
4. Drug discovery: quantum-guided molecular design
5. Optimization: quantum combinatorial optimization

Python: Quantum Bandit

import numpy as np

def quantum_bandit_step(params, n_arms=3):
    # One step of quantum bandit algorithm.
    # Quantum circuit with rotation angles
    thetas = params[:n_arms]
    # Compute arm probabilities
    probs = np.abs(np.sin(thetas))**2
    probs = probs / np.sum(probs)
    # Sample arm
    arm = np.random.choice(n_arms, p=probs)
    return arm, probs

def quantum_bandit_episode(rewards, n_steps=100):
    # Run quantum bandit for one episode.
    n_arms = len(rewards)
    params = np.random.uniform(0, np.pi, n_arms)
    total_reward = 0

    for _ in range(n_steps):
        arm, probs = quantum_bandit_step(params)
        reward = rewards[arm] + np.random.randn() * 0.1
        total_reward += reward
        # Update params
        params[arm] += 0.1 * reward

    return total_reward

rewards = [1.0, 0.5, 0.8]  # True rewards
episode_rewards = [quantum_bandit_episode(rewards) for _ in range(10)]
print(f"Average reward: {np.mean(episode_rewards):.2f}")

Quantum RL Algorithms

Quantum Exploration Advantage

Classical exploration: regret for bandits Quantum exploration: regret for bandits

The quantum advantage comes from:

1. Superposition over actions
2. Interference to amplify rewards
3. Entanglement for correlated decisions