强化学习笔记_7_策略学习中的Baseline

1. Policy Gradient with Baseline

1.1 Policy Gradient

• policy network $\pi (a|s;\theta )$

• State-value function:
  $$\begin{array}{rl}{V}_{\pi }(s)& =E_{A\sim \pi }[Q_{\pi }(s,A)]\\ & =\sum _a\pi (s|s;\theta )\cdot Q_{\pi }(s,a)\end{array}$$

• Policy gradient:
  $$\frac{\partial V_{\pi }(s)}{\partial \theta }=E_{A\sim \pi }[\frac{\partial \ln \pi (A|s;\theta )}{\partial \theta }\cdot Q_{\pi }(s,A)]$$

1.2. Baseline in Policy Gradient

• Baseline 函数 $b$，不依赖于动作$A$

• 性质：如果$b$与$A$无关，则 $E_{A\sim \pi }[b\cdot \frac{\partial \ln \pi (A|s;\theta )}{\partial \theta }]=0$
  $$\begin{array}{rl}{E}_{A\sim \pi }[b\cdot \frac{\partial \ln \pi (A|s;\theta )}{\partial \theta }]& =b\cdot E_{A\sim \pi }[\frac{\partial \ln \pi (A|s;\theta )}{\partial \theta }]\\ & =b\cdot \sum _a\pi (a|s;\theta )\cdot \frac{\partial \ln \pi (a|s;\theta )}{\partial \theta }\\ & =b\cdot \sum _a\pi (a|s;\theta )\cdot \frac{1}{\pi (a|s;\theta )}\frac{\partial \pi (a|s;\theta )}{\partial \theta }\\ & =b\cdot \sum _a\frac{\partial \pi (a|s;\theta )}{\partial \theta }\\ & =b\cdot \frac{\partial {\sum }_a\pi (a|s;\theta )}{\partial \theta }\\ & =b\cdot \frac{1}{\partial \theta }\\ & =0\end{array}$$

• policy gradient
  $$\begin{array}{rl}\frac{\partial V_{\pi }(s)}{\partial \theta }& =E_{A\sim \pi }[\frac{\partial \ln (\pi (A|s;\theta ))}{\partial \theta }\cdot Q_{\pi }(s,A)]\\ & =E_{A\sim \pi }[\frac{\partial \ln (\pi (A|s;\theta ))}{\partial \theta }\cdot Q_{\pi }(s,A)]-E_{A\sim \pi }[b\cdot \frac{\partial \ln \pi (A|s;\theta )}{\partial \theta }]\\ & =E_{A\sim \pi }[\frac{\partial \ln (\pi (A|s;\theta ))}{\partial \theta }\cdot (Q_{\pi }(s,A)-b)]\end{array}$$
  即：
  $$\begin{array}{r}\frac{\partial V_{\pi }(s_t)}{\partial \theta }=E_{A\sim \pi }[\frac{\partial \ln (\pi (A_t|s_t;\theta ))}{\partial \theta }\cdot (Q_{\pi }(s_t,A_t)-b)]\end{array}$$
  在实际计算中常使用蒙特卡洛近似，$b$不会影响方差，但是会影响蒙特卡洛近似，合理取值可以减小蒙特卡洛的方差，使收敛更快。

1.3. Monte Carlo Approximation

$$\begin{array}{rl}g(A_t)& =\frac{\partial \ln (\pi (A_t|s_t;\theta ))}{\partial \theta }\cdot (Q_{\pi }(s_t,A_t)-b)\\ \frac{\partial V_{\pi }(s_t)}{\partial \theta }& =E_{A\sim \pi }[\frac{\partial \ln (\pi (A_t|s_t;\theta ))}{\partial \theta }\cdot (Q_{\pi }(s_t,A_t)-b)]\\ & =E_{A\sim \pi }[g(A_t)]\end{array}$$

以概率密度函数$a_t\sim \pi (\cdot |s_t;\theta )$抽样得到行动$a_t$，计算得到$g(a_t)$为其期望的蒙特卡洛近似，也是对策略梯度的一个无偏估计。

• Stochastic Policy Gradient（梯度上升）
  $$\begin{array}{rl}g(a_t)& =\frac{\partial \ln (\pi (a_t|s_t;\theta ))}{\partial \theta }\cdot (Q_{\pi }(s_t,a_t)-b)\\ \theta & \leftarrow \theta +\beta \cdot g(a_t)\end{array}$$
  $b$与$A_t$无关，故不会影响$g(A_t)$的期望，但是会影响其方差。如果选取的$b$很接近于$Q_{\pi }$，则方差会很小。

1.4. Choices of Baseline

• Choice 1: $b=0$，不使用Baseline

• Choice 2: $b$ is state-value, $b=V_{\pi }(s_t)$

  状态$s_t$是先于$A_t$被观测到的，于是和$A_t$无关。

  （有点像Dueling network，使用优势函数）

2. Reinforce with Baseline

2.1. Policy Gradient

使用$V_{\pi }(s_t)$作为Baseline：
$$\begin{array}{rl}g(A_t)& =\frac{\partial \ln (\pi (A_t|s_t;\theta ))}{\partial \theta }\cdot (Q_{\pi }(s_t,A_t)-V_{\pi }(s_t))\\ \frac{\partial V_{\pi }(s_t)}{\partial \theta }& =E_{A\sim \pi }[\frac{\partial \ln (\pi (A_t|s_t;\theta ))}{\partial \theta }\cdot (Q_{\pi }(s_t,A_t)-V_{\pi }(s_t))]\\ & =E_{A\sim \pi }[g(A_t)]\end{array}$$

2.2. Approximation

• 随机抽样得到行动$a_t\sim \pi (\cdot |s_t;\theta )$，得到Stochastic Policy Gradient：
  $$g(a_t)=\frac{\partial \ln (\pi (a_t|s_t;\theta ))}{\partial \theta }\cdot (Q_{\pi }(s_t,a_t)-V_{\pi }(s_t))$$

• 对$Q_{\pi }(s_t,a_t)$近似
  $$Q_{\pi }(s_t,a_t)=E[U_t|s_t,a_t]$$
  使用观测到的回报$u_t$近似$Q_{\pi }(s_t,a_t\sim u_t)$：

  • 观测到一条完整轨迹：$s_t,a_t,r_t,s_{t+1},a_{t+1},r_{t+1},\cdot \cdot \cdot ,s_n,a_n,r_n$
  • 计算回报：$u_t={\sum }_{i=t}^n{\gamma }^{i-t}\cdot r_i$
  • 使用$u_t$作为$Q_{\pi }(s_t,a_t)$的无偏估计

• 对$V_{\pi }(s_t)$近似，使用神经网络value network $v(s;w)$近似

三次Approximation后得到的结果为：
$$\frac{\partial V_{\pi }(s_t)}{\partial \theta }\approx g(a_t)\approx \frac{\partial \ln (\pi (a_t|s_t;\theta ))}{\partial \theta }\cdot (u_t-v(s_t,w))$$

2.3. Policy and Value Network

• Policy Network

• Value Network

  

• Parameter Sharing

  

2.4. Reinforce with Baseline

• Updating the policy network

  Policy gradient: $\frac{\partial V_{\pi }(s_t)}{\partial \theta }\approx \frac{\partial \ln (\pi (a_t|s_t;\theta ))}{\partial \theta }\cdot (u_t-v(s_t,w))$

  Gradient ascent: $\theta \leftarrow \theta +\beta \cdot \frac{\partial \ln (\pi (a_t|s_t;\theta ))}{\partial \theta }\cdot (u_t-v(s_t,w))$

  令
  $${\delta }_t=v(s_t;w)-u_t$$
  则Gradient ascent也可表示为：
  $$\theta \leftarrow \theta -\beta \cdot \frac{\partial \ln (\pi (a_t|s_t;\theta ))}{\partial \theta }\cdot {\delta }_t$$

• Updating the value network

  使$v(s_t;w)$接近$V_{\pi }(s_t)=E[U_t|s_t]$，使用观测值$u_t$进行拟合。

  • Prediction error: ${\delta }_t=v(s_t,w)-u_t$
  • Gradient: $\frac{\partial {\delta }_t^2/2}{\partial w}={\delta }_t\cdot \frac{\partial v(s_t;w)}{\partial w}$
  • Gradient descent: $w\leftarrow w-\alpha \cdot {\delta }_t\cdot \frac{\partial v(s_t;w)}{\partial w}$

3. Advantage Actor-Critic (A2C)

3.1. Actor and Critic

使用2.中相同结构的神经网络，但是训练方法不同。与之前Actor-Critic不同的是，这里使用状态价值而不是行动价值，状态价值只与当前状态相关，更容易训练。

3.2. Training of A2C

• Observe a transition $(s_t,a_t,r_t,s_{t+1})$

• TD target $y_t=r_t+\gamma \cdot v(s_{t+1};w)$

• TD error ${\delta }_t=v(s_t;w)-y_t$

• Update the policy network (actor) by:
  $$\theta \leftarrow \theta -\beta \cdot {\delta }_t\cdot \frac{\partial \ln \pi (a_t|s_t;\theta )}{\partial \theta }$$

• Update the value network (critic) by:
  $$w\leftarrow w-\alpha \cdot {\delta }_t\cdot \frac{\partial v(s_t;w)}{\partial w}$$

3.3.算法的数学推导

• Value functions

  在TD算法中已经得到：$Q_{\pi }(s_t,a_t)=E_{S_{t+1},A_{t+1}}[R_t+\gamma \cdot Q_{\pi }(S_{t+1},A_{t+1})]$

  由于$R_t$与$S_{t+1}$相关而与$A_{t+1}$无关，$Q_{\pi }(S_{t+1},A_{t+1})$与两者都有关，于是继续得到：
  $$\begin{array}{rl}{Q}_{\pi }(s_t,a_t)& =E_{S_{t+1}}[R_t+\gamma \cdot E_{A_{t+1}}[Q_{\pi }(S_{t+1},A_{t+1})]\\ & =E_{S_{t+1}}[R_t+\gamma \cdot V_{\pi }(S_{t+1})]\end{array}$$

  • Theorem 1: $Q_{\pi }(s_t,a_t)=E_{S_{t+1}}[R_t+\gamma \cdot V_{\pi }(S_{t+1})]$

  继续利用价值函数的定义$V_{\pi }(s_t)=E_{A_t}[Q_{\pi }(s_t,A_t)]$，得到
  $$V_{\pi }(s_t)=E_{A_t}[E_{S_{t+1}}[R_t+\gamma \cdot V_{\pi }(S_{t+1})]]$$

  • Theorem 2: $V_{\pi }(s_t)=E_{A_t,S_{t+1}}[R_t+\gamma \cdot V_{\pi }(S_{t+1})]$

• Monte Carlo approximations

  观测得到一个transition，对Theorem 1和Theorem 2进行蒙特卡洛近似：
  $$\begin{array}{rl}{Q}_{\pi }(s_t,a_t)& =r_t+\gamma \cdot V_{\pi }(s_{t+1})\\ V_{\pi }(s_t)& =r_t+\gamma \cdot V_{\pi }(s_{t+1})\end{array}$$

• Updating policy network

  带有Baseline的策略梯度下降: $g(a_t)=\frac{\partial \ln (\pi (a_t|s_t;\theta ))}{\partial \theta }\cdot (Q_{\pi }(s_t,a_t)-V_{\pi }(s_t))$

  对$Q_{\pi }(s_t,a_t)$进行蒙特卡洛近似: $g(a_t)=\frac{\partial \ln (\pi (a_t|s_t;\theta ))}{\partial \theta }\cdot r_t+\gamma \cdot V_{\pi }(s_{t+1})-V_{\pi }(s_t))$

  使用value network $v(s;w)$对$V_{\pi }(s_t)$进行拟合:
  $$g(a_t)=\frac{\partial \ln (\pi (a_t|s_t;\theta ))}{\partial \theta }\cdot (r_t+\gamma \cdot v(s_{t+1};w)-v(s_t;w))$$
  TD target: $y_t=r_t+\gamma \cdot v(s_{t+1};w)$
  $$g(a_t)=\frac{\partial \ln (\pi (a_t|s_t;\theta ))}{\partial \theta }\cdot (y_t-v(s_t;w))$$
  梯度上升:
  $$\theta \leftarrow \theta -\beta \cdot {\delta }_t\cdot \frac{\partial \ln \pi (a_t|s_t;\theta )}{\partial \theta }$$

• Updating value network

  使用value network $v(s;w)$对$V_{\pi }(s_t)$进行拟合：
  $$V(s_t;w)\approx r_t+\gamma \cdot V(s_{t+1};w)=y_t$$
  TD error: ${\delta }_t=v(s_t;w)-y_t$

  Gradient: $\frac{\partial {\delta }_t^2/2}{\partial w}={\delta }_t\frac{\partial v(s_t;w)}{\partial w}$

  Gradient descent: $w\leftarrow w-\alpha \cdot {\delta }_t\cdot \frac{\partial v(s_t;w)}{\partial w}$

4. ReinForce versus A2C

4.1 Policy and Value Networks

两种算法的网络结构相同，都包括价值网络和策略网络。Reinforce with Baseline中，价值网络仅作为Baseline以降低随机梯度造成的方差；A2C中的价值网络用于对actor进行评价（critic）。

从算法流程上看，两种算法仅在TD target和error的部分有差别。Reinforce使用了真实的观测值Return，而A2C使用了TD target，部分基于观测值，部分基于预测值。

对于Multi-Step TD target，在只计算一步的情况下，为one-step TD target；在计算所有步的情况下，则变为$u_t={\sum }_{i=t}^n{\gamma }^{i-t}\cdot r_i$，A2C算法与Reinforce相同。Reinforce是A2C算法的一个特例。

4.2 A2C with Multi-Step TD Target

one-Step TD Target:
$$y_t=r_t+\gamma \cdot v(s_{t+1};w)$$
m-Step TD Target:
$$y_t=\sum _{i=0}^{m-1}{\gamma }^i\cdot r_{t+i}+{\gamma }^m\cdot v(s_{t+m};w)$$