An introduction to reinforcement learning

Having taken a quick look at several overviews of reinforcement learning, I wrote a script here to conclude and take down some key concepts and points to help myself understand the reinforcement learning.

Introduction

Terminology

• Environment
• State
  • agent state
    the state that the agent can observe.
  • environment state
    the whole information that the environment contains in the state.
• Action
• Reward
• Policy
  A policy defines the learning agent’s way of behaving at a given time. It is an agent’s behavior function. It is a map from state to action.
  • Deterministic policy: $a=\pi (s)$. a: action; s: state
  • Stochastic policy: $\pi (a|s)=\mathbb{P}[A=a|S=s]$
• Value function
  Value function is a prediction of future reward. Used to evaluate how good is each state and/or action and therefore to select between actions.
• model
  how the agent senses the environment. agent’s representation of the environment. It predicts what the environment will do next. Can be model-free: not every situation needs a model.
  • Transitions model: $\mathcal{P}$ predicts the next state (i.e. dynamics)
    ${\mathcal{P}}_{s{s}^{\prime }}^a=\mathbb{P}[S^{\prime }=s^{\prime }|S=s,A=a]$
  • Rewards model: $\mathcal{R}$ predicts the next (immediate) reward, e.g.
    ${\mathcal{R}}_s^a=\mathbb{E}[R|S=s,A=a]$
• model-free
• on-policy/off_policy

Goal

To maximize the expected cumulative reward.
(Reward Hypothesis)

Classification

• two types of tasks
  • episodic
  • continuous (no terminal state). like automated stock trading.
• two ways of learning (sampling methods)
  • Monte Carlo (only update when the episodes end)
  • TD Learning Methods (update every time states switch)
• two approaches (whether MDP is known)
  • Model Free
    Policy and/or Value Function. No model.
    We don’t try to explicitly understand the environment. Don’t need to build a model to describe the environment.
    • value-based
      In value-based RL, the goal is to optimize the value function $V(s)$. The value function is a function that tells us the maximum expected future reward the agent will get at each state. The value of each state is the total amount of the reward an agent can expect to accumulate over the future, starting at that state. The agent will use this value function to select which state to choose at each step. The agent takes the state with the biggest value.
      Therefore, no policy (implicit). You don’t need a policy to sample/determine which action to choose.
    • policy-based
      The policy $\pi (s)$ is what defines the agent behavior at a given time. No value function.
      • Deterministic: a policy at a given state will always return the same action.
      • Stochastic: output a distribution probability over actions.
    • Actor Critic
      It stores both the policy and the value function at the same time.
  • Model Based
    Policy and/or Value Function. NModel.
    First build the model. Then we do actions base on the model.

Markov Decision Process

Markov Process

Markov Process (or Markov Chain) is a tuple $⟨\mathcal{S},\mathcal{P}⟩$.

• $\mathcal{S}$ is a (finite) set of states.
• $\mathcal{P}$ is a state transition probability matrix. ${\mathcal{P}}_{s{s}^{\prime }}=\mathbb{P}[S_{t+1}=s^{\prime }|S_t=s]$

Markov Reward Process

Markov Reward Process is a tuple $⟨\mathcal{S},\mathcal{P},\mathcal{R},\gamma ⟩$.

• $\mathcal{R}$ is a reward function. ${\mathcal{R}}_s=\mathbb{E}[R_{t+1}|S_t=s]$ (immediate reward)

• $\gamma$ is a discount factor. $\gamma \in [0,1]$

• The return $G_t$ is the total discounted reward from time-step t.
  $G_t=R_{t+1}+\gamma R_{t+2}+...={\sum }_{k=0}^{\infty }{\gamma }^k{R}_{t+k+1}$
  Here, no expectation is because $G_t$ is for a single sequence sample.

• The value function $v(s)$ gives the long-term value of state $s$. It is the expected return starting from state $s$.
  $v(s)=\mathbb{E}[G_t|S_t=s]$

• Bellman Equation for MRPs
  The value funxtion can be decomposed into two parts:

  • immediate reward $R_{t+1}$
  • discounted value of successor state $\gamma v(S_{t+1})$

  $v(s)=\mathbb{E}[G_t|S_t=s]=\mathbb{E}[R_{t+1}+\gamma v(S_{t+1})|S_t=s]={\mathcal{R}}_s+\gamma {\sum }_{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s{s}^{\prime }}v(s^{\prime })$
  In matrix form: $v=\mathcal{R}+\gamma \mathcal{P}v$
  Can be solved directly for small MRPs: $v=(I-\gamma \mathcal{P}{)}^{-1}\mathcal{R}$

Markov Decision Process

Markov Decision Process is a tuple $⟨\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma ⟩$

• $\mathcal{A}$ is a finite set of actions
• $\mathcal{P}$ is a state transition probability matrix
  ${\mathcal{P}}_{s{s}^{\prime }}^a=\mathbb{P}[S_{t+1}=s^{\prime }|S_t=s,A_t=a]$
• $\mathcal{R}$ is a reward function
  ${\mathcal{R}}_s^a=\mathbb{E}[R_{t+1}|S_t=s,A_t=a]$
• A policy $\pi$ is a distribution over actions given states. It fully defines the behavior of an agent.
  $\pi (a|s)=\mathbb{P}[A_t=a|S_t=s]$
• The state-value function $v_{\pi }(s)$ of an MDP is the expected return starting from state $s$, and then following policy $\pi$
  $v_{\pi }(s)={\mathbb{E}}_{\pi }[G_t|S_t=s]$
  Greedy policy improvement over $V(s)$ requires model of MDP.
• The action-value function $q_{\pi }(s,a)$ is the expected return starting from state $s$, taking action $a$, and then following policy $\pi$
  $q_{\pi }(s,a)={\mathbb{E}}_{\pi }[G_t|S_t=s,A_t=a]$
  Greedy policy improvement over $Q(s,a)$ is model-free.
• Bellman Expectation Equation (the deriving methods are very similar to the ones in MRPs)
  $v_{\pi }(s)=\mathbb{E}[R_{t+1}+\gamma v_{\pi }(S_{t+1})|S_t=s]$
  $v_{\pi }={\mathcal{R}}^{\pi }+\gamma {\mathcal{P}}^{\pi }{v}_{\pi }$
  $q_{\pi }(s,a)={\mathbb{E}}_{\pi }[R_{t+1}+\gamma q_{\pi }(S_{t+1},A_{t+1})|S_t=s,A_t=a]$
• Optimal Value Function
  The optimal state-value function $v_{\ast }(s)$ is the maximum value function over all policies
  $v_{\ast }(s)={\max }_{\pi }{v}_{\pi }(s)$
  The optimal action-value function $q_{\ast }(s,a)$ is the maximum action-value function over all policies
  $q_{\ast }(s,a)={\max }_{\pi }{q}_{\pi }(s,a)$

Dynamic Programming

Dynamic programming assumes full knowledge of the MDP.

Sampling Methods for model-free. (solutions for small MDPs)

To estimate and optimise the value function of an unknown MDP.

Monte Carlo

Monte Carlo policy evaluation uses empirical mean return instead of expected return according to $v_{\pi }(s)={\mathbb{E}}_{\pi }[G_t|S_t=s]$ using the whole episodes with terminals.

• Update value $V(S_t)$ toward actual return $G_t$.
  $V(S_t)\leftarrow V(S_t)+\alpha (G_t-V(S_t))$
• GLIE Monte-Carlo Control (On-Policy) to update action-value function through sampling.
  • Sample kth episode using $\pi :\{S_1,A_1,R_2,...,S_T\}\sim \pi$
  • For each state $S_t$ and action $A_t$ in the episode,
    $N(S_t,A_t)\leftarrow N(S_t,A_t)+1$
    $Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\frac{1}{N(S_t,A_t)}(G_t-Q(S_t,A_t))$
  • Improve policy based on new action-value function
    $ϵ\leftarrow 1/k$
    $\pi \leftarrow ϵ-greedy(Q)$

Temporal-Difference Learning (TD)

TD learning learns from incomplete episodes by bootstrapping (update involves an estimate).

• (TD/TD(0)) Update value V(St) toward estimated return $R_t+1+\gamma V(S_{t+1})$.
  $V(S_t)\leftarrow V(S_t)+\alpha (R_{t+1}+\gamma V(S_{t+1})-V(S_t))$
  SARSA (On-Policy)
  use TD instead of MC in GLIE Monte-Carlo Control
  $Q(S,A)\leftarrow Q(S,A)+\alpha (R+\gamma Q(S^{\prime },A^{\prime })-Q(S,A))$
  convergence condition
  • GLIE sequence of policies ${\pi }_t(a|s)$
  • Robbins-Monro sequence of step-sizes ${\alpha }_t$
    ${\sum }_{t=1}^{\infty }{\alpha }_t=\infty$
    ${\sum }_{t=1}^{\infty }{\alpha }_t^2<\infty$
• n-step temporal-difference learning
  $V(S_t)\leftarrow V(S_t)+\alpha (G_t^{(n)}-V(S_t))$, where $G_t^{(n)}=R_{t+1}+\gamma R_{t+2}+...+{\gamma }^{n-1}{R}_{t+n}+{\gamma }^{n}V(S_{t+n})$
  When $n=\infty$, it is MC methods.
  n-step SARSA
  $Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\alpha (q_t^{(n)}-Q(S_t,A_t))$, where $q_t^{(n)}=R_{t+1}+\gamma R_{t+2}+...+{\gamma }^{n-1}{R}_{t+n}+{\gamma }^{n}Q(S_{t+n})$
• (Forward-view TD($\lambda$)
  $V(S_t)\leftarrow V(S_t)+\alpha (G_t^{\lambda }-V(S_t))$, where $G_t^{\lambda }=(1-\lambda ){\sum }_{n=1}^{\infty }{\lambda }^{n-1}{G}_t^{(n)}$
  Like MC, can only be computed from complete episodes.
  Forward View Sarsa($\lambda$) combines all n-step Q-returns $q_t^{(n)}$
  $Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\alpha (q_t^{\lambda }-Q(S_t,A_t))$, where $q_t^{\lambda }=(1-\lambda ){\sum }_{n=1}^{\infty }{\lambda }^{n-1}{q}_t^{(n)}$
• (Backward-view TD($\lambda$))
  $V(s)\leftarrow V(s)+\alpha {\delta }_t{E}_t(s)$, where ${\delta }_t=R_{t+1}+\gamma V(S_{t+1})-V(S_t)$, $E_0(s)=0$ and $E_t(s)=\gamma \lambda E_{t-1}(s)+1(S_t=s)$

It is biased but with lower variance than the MC methods. It is more efficient but more sensitive to initial value.

Off-Policy Learning

Evaluate target policy $\pi (a|s)$ to compute $v_{\pi }(s)$ or $q_{\pi }(s,a)$, while following behaviour policy $\mu (a|s)$: $\{S_1,A_1,R_2,...,S_T\}\sim \mu$

• Importance sampling
• Q-Learning
  $Q(S,A)\leftarrow Q(S,A)+\alpha (R+\gamma {\max }_{a^{\prime }}Q(S^{\prime },a^{\prime })-Q(S,A))$

Value Function Approximation (solution for large MDPs)

Basic Idea

There are too many states and/or actions to store in memory. It is too slow to learn the value of each state individually.
So we estimate value function with function approximation

• $\hat{v}(s,w)\approx v_{\pi }(s)$
• $\hat{q}(s,a,w)\approx q_{\pi }(s,a)$
• $\hat{q}(s,a_1),...,\hat{q}(s,a_m)\approx q_{\pi }(s,a_1,w),...,q_{\pi }(s,a_m,w)$

The function approximators options (differential)

• Linear combinations of features
• Neural network

Incremental Methods

Stochastic Gradient Descent

To find parameter vector $w$ minimising mean-squared error between approximate value fn $\hat{v}(s,w)$ and true value fn $v_{\pi }(s)$ : $J(w)={\mathbb{E}}_{\pi }[(v_{\pi }(S)-\hat{v}(S,w){)}^2]$

• Gradient descent finds a local minimum
  $△w=-\frac{1}{2}\alpha {\nabla }_{w}J(w)=\alpha {\mathbb{E}}_{\pi }[(v_{\pi }(S)-\hat{v}(S,w)){\nabla }_w\hat{v}(S,w)]$
• Stochastic gradient descent samples the gradient
  $△w=\alpha (v_{\pi }(S)-\hat{v}(S,w)){\nabla }_w\hat{v}(S,w)$
• Expected update is equal to full gradient update.

Incremental Prediction Algorithms

In practice, we substitute a target for $v_{\pi }(s)$

• For MC, the target is the return $G_t$
  $∆w=\alpha (G_t-\hat{v}(St,w)){\nabla }_w\hat{v}(S_t,w)$
• For TD(0), the target is the TD target $R_{t+1}+\gamma \hat{v}(S_{t+1},w)$
  $∆w=\alpha (R_{t+1}+\gamma \hat{v}(S_{t+1},w)-\hat{v}(S_t,w)){\nabla }_w\hat{v}(S_t,w)$
• For TD($\lambda$), the target is the $\lambda$-return $G_t^{\lambda }$
  $∆w=\alpha (G_t^{\lambda }-\hat{v}(S_t,w)){\nabla }_w\hat{v}(S_t,w)$

For action-value function approximation, the derivation is similar.

Batch Methods

Least Squares Prediction

Given value function approximation $\hat{v}(s,w)\approx v_{\pi }(s)$ and experience $\mathcal{D}$ consisting of $⟨state,value⟩$ pairs $\mathcal{D}=\{⟨s_1,v_1^{\pi }⟩,⟨s_2,v_2^{\pi }⟩,...,⟨s_T,v_T^{\pi }⟩\}$, which parameters $w$ give the best fitting value fn $\hat{v}(s,w)$?
Least squares algorithms find parameter vector $w$ minimising sum-squared error between $\hat{v}(s_t,w)$ and target values $v_t^{\pi }$,
$LS(w)={\sum }_{t=1}^T(v_t^{\pi }-\hat{v}(s_t,w){)}^2={\mathbb{E}}_{\mathcal{D}}[(v^{\pi }-\hat{v}(s,w){)}^2]$

• Stochastic Gradient Descent with Experience Replay
  Value from experience $⟨s,v^{\pi }⟩\sim \mathcal{D}$ and then apply stochastic gradient descent update. It converges to least squares solution $w^{\pi }=argmi{n}_{w}LS(w)$.
  $∆w=\alpha (v^{\pi }-\hat{v}(s,w)){\nabla }_w\hat{v}(s,w)$
• Deep Q-Networks (DQN) with Experience Replay

Least Squares Control

Policy Gradient

score function ${\nabla }_{\theta }\log {\pi }_{\theta }(s,a)$
policy ${\pi }_{\theta }$ and the gradient ${\nabla }_{\theta }{\pi }_{\theta }(s,a)$
${\nabla }_{\theta }{\pi }_{\theta }(s,a)={\pi }_{\theta }(s,a)\frac{{\nabla }_{\theta }{\pi }_{\theta }(s,a)}{{\pi }_{\theta }(s,a)}={\pi }_{\theta }(s,a){\nabla }_{\theta }\log {\pi }_{\theta }(s,a)$

Monte-Carlo methods

Actor-Critic methods

Maintain two sets of parameters

• Critic: Updates action-value function parameters $w$
• Actor: Updates policy parameters $\theta$, in direction suggested by critic

(how to accelerate the gradient descent.)
Reducing Variance Using a Baseline
Subtract a baseline function $B(s)$ from the policy gradient. This can reduce variance, without changing expectation.

• ${\mathbb{E}}_{{\pi }_{\theta }}[{\nabla }_{\theta }\log {\pi }_{\theta }(s,a)B(s)]={\sum }_{s\in S}{d}^{{\pi }_{\theta }}(s){\sum }_a{\nabla }_{\theta }{\pi }_{\theta }(s,a)B(s)={\sum }_{s\in S}{d}^{{\pi }_{\theta }}B(s){\nabla }_{\theta }{\sum }_{a\in A}{\pi }_{\theta }(s,a)=0$

A good baseline is the state value function $B(s)=V^{{\pi }_{\theta }}(s)$
So we can rewrite the policy gradient using the advantage function $A^{{\pi }_{\theta }}(s,a)$

• $A^{{\pi }_{\theta }}(s,a)=Q^{{\pi }_{\theta }}(s,a)-V^{{\pi }_{\theta }}(s)$
• ${\nabla }_{\theta }J(\theta )={\mathbb{E}}_{{\pi }_{\theta }}[{\nabla }_{\theta }\log {\pi }_{\theta }(s,a)A^{{\pi }_{\theta }}(s,a)]$

Eligibility Traces (can be used online)

Deterministic gradient theorem can help improve.

Model-Based Reinforcement Learning

Model Definition

A model $\mathcal{M}$ is a representation of an MDP $⟨\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}⟩$, parametrized by $\eta$. Here, we will assume state space $\mathcal{S}$ and action space $\mathcal{A}$ are known.
So a model $\mathcal{M}=⟨{\mathcal{P}}_{\eta },{\mathcal{R}}_{\eta }⟩$ represents state transitions ${\mathcal{P}}_{\eta }\approx \mathcal{P}$ and rewards ${\mathcal{R}}_{\eta }\approx \mathcal{R}$
$S_{t+1}\sim {\mathcal{P}}_{\eta }(S_{t+1}|S_t,A_t)$
$R_{t+1}={\mathcal{R}}_{\eta }(R_{t+1}|S_t,A_t)$
Typically assume conditional independence between state transitions and rewards
$\mathbb{P}[S_{t+1},R_{t+1}|S_t,A_t]=\mathbb{P}[S_{t+1}|S_t,A_t]\mathbb{P}[R_{t+1}|S_t,A_t]$
Goal: estimate model ${\mathcal{M}}_{\eta }$ from experience $\{S_1,A_1,R_2,...,S_T\}$
After given a model $\mathcal{M}=⟨{\mathcal{P}}_{\eta },{\mathcal{R}}_{\eta }⟩$, we then solve the MDP $⟨\mathcal{S},\mathcal{A},{\mathcal{P}}_{\eta },{\mathcal{R}}_{\eta }⟩$

Integrating Learning and Planning

• Model-Free RL
  • No model
  • Learn value function (and/or policy) from real experience
• Model-Based RL (using Sample-Based Planning)
  • Learn a model from real experience
  • Plan value function (and/or policy) from simulated experience
• Dyna
  • Learn a model from real experience
  • Learn and plan value function (and/or policy) from real and simulated experience

Simulation-Based Search

combine Dyna and simulation-based search.

Ref:
https://www.freecodecamp.org/news/an-introduction-to-reinforcement-learning-4339519de419/
Online courses by David Silver