基于baseline的策略梯度（Reinforce算法与A2C）

1. baseline的推导

• 策略网络为：$$\pi (a|s;\theta )$$
• 状态价值函数为：$$\begin{gathered} V_{\pi }(s)=E_{A\sim \pi }[Q_{\pi }(A,s)] \\ =\sum _a\pi (a|s;\theta )\cdot Q_{\pi }(a,s) \end{gathered}$$
• 策略梯度为：$$\frac{\partial V_{\pi }(s)}{\partial \theta }=E_{A\sim \pi }[Q_{\pi }(s,a)\cdot \frac{\partial log(\pi (a|s;\theta ))}{\partial \theta }]$$
• 设b为不依赖于动作A的任何函数，则：$$\begin{gathered} E_{A\sim \pi }[b\cdot \frac{\partial log(\pi (a|s;\theta ))}{\partial \theta }] \\ =b\cdot E_{A\sim \pi }[\frac{\partial log(\pi (a|s;\theta ))}{\partial \theta }] \\ =b\cdot \sum _a\pi (a|s;\theta )\cdot \frac{\partial log(\pi (a|s;\theta ))}{\partial \theta } \\ =b\cdot \sum _a\pi (a|s;\theta )\cdot \frac{1}{\pi (a|s;\theta )}\cdot \frac{\partial \pi (a|s;\theta )}{\partial \theta } \\ =b\cdot \frac{\partial {\sum }_a\pi (a|s;\theta )}{\partial \theta } \\ =b\cdot \frac{\partial 1}{\partial \theta } \\ =0 \end{gathered}$$因此，如果b独立于动作A，则：$$E_{A\sim \pi }[b\cdot \frac{\partial log(\pi (a|s;\theta ))}{\partial \theta }]=0$$
• 则带baseline的策略梯度为：$$\begin{gathered} \frac{\partial V_{\pi }(s)}{\partial \theta }=E_{A\sim \pi }[Q_{\pi }(A,s)\cdot \frac{\partial log(\pi (A|s,\theta ))}{\partial \theta }]-E_{A\sim \pi }[b\cdot \frac{\partial log(\pi (A|s,\theta ))}{\partial \theta }] \\ =E_{A\sim \pi }[\frac{\partial log(\pi (A|s;\theta ))}{\partial \theta }\cdot (Q_{\pi }(A,s)-b)] \end{gathered}$$b不会影响期望，但合适的b会降低蒙特卡洛近似的方差，加快模型收敛。

2. 策略梯度的蒙特卡洛近似

• 基于baselin的策略梯度为：$$\begin{gathered} \frac{\partial V_{\pi }(s_t)}{\partial \theta }==E_{A_t\sim \pi }[\frac{\partial log(\pi (A_t|s_t;\theta ))}{\partial \theta }\cdot (Q_{\pi }(A_t,s_t)-b)] \\ g(A_t)=\frac{\partial log(\pi (A_t|s_t;\theta ))}{\partial \theta }\cdot (Q_{\pi }(A_t,s_t)-b) \end{gathered}$$
• 依据策略函数随机抽样得到t时刻的动作：$$a_t\sim \pi (\cdot |s_t;\theta )$$
• 则策略梯度的无偏估计为：$$g(a_t)$$
• 随机策略梯度：$$g(a_t)=(Q_{\pi }(s_t,a_t)-b)\cdot (\frac{\partial log(\pi (a_t|s_t;\theta ))}{\partial \theta })$$
• 做梯度上升：$$\theta \leftarrow \theta +\beta \cdot g(a_t)$$

3. baseline的选取

1. 标准策略梯度：$$b=0$$
2. 使用状态价值函数，因为其与动作A_t无关，且接近动作价值函数：$$\begin{gathered} b=V_{\pi }(s_t) \\ {V}_{\pi }(s_t)=E_{A_t}[Q(A_t,s_t)] \end{gathered}$$

4. Reinforce算法

4.1 基本概念

• 折扣回报：$$U_t=R_t+\gamma \cdot R_{t+1}+{\gamma }^2\cdot R_{t+2}-...$$
• 动作价值函数：$$Q_{\pi }(s_t,a_t)=E[U_t|s_t,a_t]$$
• 状态价值函数：$$V_{\pi }(s_t)=E_{A_t}[Q_{\pi }(s_t,A_t)|s_t]$$
• 带baseline的策略梯度为：$$\begin{gathered} \frac{\partial V_{\pi }(s_t)}{\partial \theta }=E_{A_t\sim \pi }[g(A_t)] \\ =E_{A_t\sim \pi }[\frac{\partial log(\pi (A_t|s_t;\theta ))}{\partial \theta }\cdot (Q_{\pi }(A_t,s_t)-V_{\pi }(s_t))] \end{gathered}$$
• 对动作进行抽样，做蒙特卡洛近似，为无偏估计：$$\begin{gathered} a_t\sim \pi (\cdot |s_t;\theta ) \\ g(a_t)=(Q_{\pi }(s_t,a_t)-b)\cdot (\frac{\partial log(\pi (a_t|s_t;\theta ))}{\partial \theta }) \end{gathered}$$
• 对动作价值函数做蒙特卡洛近似（Reinforce算法的关键）:$$\begin{gathered} Q_{\pi }(s_t,a_t)=E[U_t|s_t,a_t] \\ {Q}_{\pi }(s_t,a_t)\approx u_t \\ 观测轨迹为：s_t,a_t,r_t,s_{t+1},a_{t+1},r_{t+1},...,s_{t+n},a_{t+n},r_{t+n} \\ {u}_t=\sum _{i=t}^n{\gamma }^{i-t}{r}_i \end{gathered}$$
• 通过神经网络近似状态价值函数：$$v(s_t;W)\sim V_{\pi }(s_t)$$
• 近似后的策略梯度为：$$\frac{\partial V_{\pi }(s_t)}{\partial \theta }=\frac{\partial log(\pi (a_t|s_t;\theta ))}{\partial \theta }\cdot (u_t-v(s_t;W))$$
  上述的推导做了三次近似：

1. 动作的抽样为蒙特卡洛近似
2. 动作价值函数的近似为蒙特卡洛近似
3. 状态价值函数为神经网络近似

4.2 算法的训练流程

• 策略网络为：$$\pi (a_t|s_t;\theta )$$
  - 价值网络：$$v(s_t;W)$$

• 两个网络可以进行参数共享

1. 完成一局完整的游戏，得到一个轨迹：$$\{(s_1,a_1,r_1);(s_2,a_2,r_2);...;(s_n,a_n,r_n)\}$$
2. 计算动作价值函数的近似：$$\begin{gathered} u_t=\sum _{i=t}^n{\gamma }^{i-t}\cdot r_i \\ {\delta }_t=v(s_t;W)-u_t \end{gathered}$$
3. 依据策略梯度更新策略网络的参数：$$\theta \leftarrow \theta +\beta \cdot {\delta }_t\cdot \frac{\partial log(\pi (s_t|s_t;\theta ))}{\partial \theta }$$
4. 采用梯度下降更新价值网络的参数：$$W\leftarrow W-\alpha \cdot {\delta }_t\cdot \frac{\partial v(s_t;W)}{\partial W}$$
5. 由于轨迹的长度为n，可以对神经网络进行n次更新

5. A2C算法（Advantage Actor Critic）

5.1 网络结构及其训练过程

• 策略网络为（actor）：$$\pi (a_t|s_t;\theta )$$
  - 价值网络（critic）：$$v(s_t;W)$$

• 两个网络可以进行参数共享

1. 观测到一个transition：$$s_t,a_t,r_t,s_{t+1}$$
2. TD Target:$$r_t+v(s_{t+1};W)$$
3. TD error 为：$${\delta }_t=v(s_t;W)-y_t$$
4. 更新策略网络：$$\theta \leftarrow \theta -\beta \cdot {\delta }_t\cdot \frac{\partial log(\pi (a_t|s_t;\theta ))}{\partial \theta }$$
5. 更新价值网络：$$W\leftarrow W-\alpha \cdot {\delta }_t\cdot \frac{\partial v(s_t;W)}{\partial W}$$

5.2 数学原理推导

5.2.1 概念定义

• 折扣回报：$$U_t=R_t+\gamma R_{t+1}+{\gamma }^2{R}_{t+2}+...$$
• 动作价值函数：$$Q_{\pi }(s_t,a_t)=E[U_t|s_t,a_t]$$
• 状态价值函数：$$V_{\pi }(s_t)=E_{A_t}[Q_{\pi }(s_t,A_t)|s_t]$$

5.2.2 定理1（动作价值与状态价值之间的关系）

$$\begin{gathered} Q_{\pi }(s_t,a_t)=E[U_t|s_t,a_t] \\ =E_{A_{t+1},S_{t+1}}[R_t+\gamma \cdot Q_{\pi }(S_{t+1},A_{t+1})] \\ =E_{S_{t+1}}[R_t+\gamma \cdot E_{A_{t+1}}[Q_{\pi }(S_{t+1},A_{t+1})]] \\ =E_{S_{t+1}}[R_t+V_{\pi }(S_{t+1})] \end{gathered}$$
蒙特卡洛近似为：$$Q_{\pi }(s_t,a_t)\approx r_t+\gamma \cdot V_{\pi }(s_{t+1})$$可用于训练策略网络

5.2.3 定理2（前后时刻状态价值之间的关系）

$$\begin{gathered} V_{\pi }(s_t)=E_{A_t}[Q_{\pi }(s_t,A_t)] \\ =E_{A_t}[E_{S_{t+1}}[R_t+\gamma \cdot V_{\pi }(S_{t+1})]] \\ =E_{A_t,S_{t+1}}[R_t+V_{\pi }(S_{t+1})] \end{gathered}$$
蒙特卡洛近似为：$$V_{\pi }(s_t)\approx r_t+\gamma \cdot V_{\pi }(s_{t+1})$$可用于训练价值网络

5.2.4 策略网络的更新：

• 随机策略梯度为：$$\begin{gathered} g(a_t)=\frac{\partial log(\pi (a_t|s_t;\theta ))}{\partial \theta }\cdot (Q_{\pi }(s_t,a_t)-V_{\pi }(s_t)) \\ {Q}_{\pi }(s_t,a_t)\approx r_t+\gamma \cdot V_{\pi }(s_t)=y_t \\ v(s_t;W)\sim V_{\pi }(s_t) \\ \theta \leftarrow \theta +\beta \cdot (y_t-v(s_t;W))\cdot \frac{\partial log(\pi (a_t|s_t;\theta ))}{\partial \theta } \end{gathered}$$

5.2.5 价值网络的更新：

$$\begin{gathered} V_{\pi }(s_t)\approx r_t+\gamma \cdot V_{\pi }(s_{t+1}) \\ v(s_t;W)\approx r_t+\gamma \cdot v(s_{t+1};W)=y_t \end{gathered}$$

1. TD error为：$${\delta }_t=v(s_t;W)-y_t$$
2. 梯度为：$$\frac{\partial \frac{1}{2}\cdot {\delta }_t^2}{\partial W}={\delta }_t\cdot \frac{\partial v(s_t;W)}{\partial W}$$
3. 梯度更新：$$W\leftarrow W-\alpha \cdot {\delta }_t\cdot \frac{\partial v(s_t;W)}{\partial W}$$
   本文内容为参考B站学习视频书写的笔记！

5.3 策略梯度的理解

$$g(a_t)={(}_t+\gamma \cdot v(s_{t+1};W)-v(s_t;W))\cdot (\frac{\partial log(\pi (a_t|s_t;\theta ))}{\partial \theta })$$
两者之差反映了执行动作后的优势（回报）

6. Reinforce算法 V.S. A2C算法

6.1 A2C

6.1.1 one step TD Target

• 观测到一个transition：$$(s_t,a_t,r_t,s_{t+1})$$
• $$y_t=r_t+\gamma \cdot v(s_{t+1;W})$$

6.1.2 multi-step TD Target

• 观测到m个transition：$$\{(s_{t+i},a_{t+i},r_{t+i},s_{t+i+1}){\}}_{i=0}^{m-1}$$
• $$y_t=\sum _{i=0}^{m-1}{\gamma }^i\cdot r_{t+i}+{\gamma }^m\cdot v(s_{t+1};W)$$

6.2 Reinforce 算法

• 回报:$$u_t=\sum _{i=t}^n{\gamma }^{t-i}\cdot r_i$$
• error:$${\delta }_t=v(s_t;W)-u_t$$

6.3 Reinforce 算法为A2C的特殊形式

multi-step A2C的TD Target为： $$y_t=\sum _{i=0}^{m-1}{\gamma }^i\cdot r_{t+i}+{\gamma }^m\cdot v(s_{t+1};W)$$
当使用所有奖励时，则：$$y_t=u_t=\sum _{i=t}^n{\gamma }^{t-i}\cdot r_i$$
所以 Reinforce 算法为A2C的特例。

本文内容为参考B站学习视频书写的笔记！
by CyrusMay 2022 04 11