Reinforcement Learning: Value Functions and Policy Gradient

Module: Machine Learning | Difficulty: Advanced

Bellman Equation

Q-Learning

Policy Gradient Theorem

Advantage Function

Actor-Critic

import numpy as np

class QLearning:
    def __init__(self, n_states, n_actions, lr=0.1, gamma=0.99, epsilon=0.1):
        self.q = np.zeros((n_states, n_actions))
        self.lr = lr; self.gamma = gamma; self.epsilon = epsilon
    def choose_action(self, state):
        if np.random.random() < self.epsilon:
            return np.random.randint(self.q.shape[1])
        return np.argmax(self.q[state])
    def update(self, s, a, r, s_next):
        td_target = r + self.gamma * self.q[s_next].max()
        self.q[s,a] += self.lr * (td_target - self.q[s,a])

Research Insight: Actor-critic methods combine the low variance of value-based methods with the flexibility of policy gradient methods. The key insight is that using the advantage function reduces variance without introducing bias.