强化学习策略梯度定理证明

前言

好久没有更新了，最近看了Policy Gradient 的原文，里边的证明看不懂，于是又找了 Stanford University 的策略梯度定理证明的PPT，感觉写的比较清晰。

链接如下：
原文-Policy Gradient Methods for Reinforcement Learning with Function Approximation.
PPT.

策略梯度定理

策略梯度定理的意义在于：该定理建立了指标函数$J(\theta )$的梯度与策略$\pi (a|s;\theta )$的梯度的直接联系。($\theta$为策略函数中的参数，如果使用neural network (nn) 作为梯度，那么$\theta$就是nn里面的参数)。下边给出定理：
$${\nabla }_{\theta }J(\theta )={\int }_{\mathcal{S}}{\rho }^{\pi }(s){\int }_{\mathcal{A}}{\nabla }_{\theta }\pi (a|s;\theta )\cdot Q^{\pi }(s,a)\cdot da\cdot ds$$
其中${\nabla }_{\theta }J(\theta )=\frac{\partial }{\partial \theta }J(\theta )$，${\rho }^{\pi }(s)={\sum }_{t=0}^{\infty }{\gamma }^t{\int }_{s_0\in \mathcal{S}}{P}_0(s_0)P(s_0\to s_1,1,\pi )\cdot d{s}_0$。

预备公式

因本人基础比较差，所以推导的时候很多公式都不知道是什么含义，所以查阅了一下，放在这里备用。

• 概率论中的期望公式：$\mathbb{E}(X)={\int }_{x\in X}p(x)xdx$
• 强化学习状态值函数$V^{\pi }(s)$与动作值函数$Q^{\pi }(s,a)$的对应关系：
  • $V^{\pi }(s)\to Q^{\pi }(s,a)$：$V^{\pi }(s)={\int }_{a\in \mathcal{A}}\pi (a|s;\theta )Q^{\pi }(s,a)da$
  • $Q^{\pi }(s,a)\to V^{\pi }(s^{\prime })$：$Q^{\pi }(s,a)={\mathcal{R}}_s^a+{\int }_{s^{\prime }\in \mathcal{S}}{\mathcal{P}}_{s,s^{\prime }}^a{V}^{\pi }(s^{\prime })d{s}^{\prime }$

不严格地讲，这里的积分可以换成求和的形式，它们所代表的都是整个动作或者状态空间。

证明

整体的证明都是按照PPT链接中的内容进行的，不同的是在数学归纳的地方多往后推了一项，让结果看起来更加显然。

$J(\theta )$定理形式推导

$$\begin{array}{r}\begin{array}{rl}J(\theta )& ={\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t\right]\\ & =\sum _{t=0}^{\infty }{\mathbb{E}}_{\pi }{\gamma }^t{r}_t\\ & =\sum _{t=0}^{\infty }{\gamma }^t{\int }_{s\in \mathcal{S}}{r}_s^a\cdot ds\\ & =\sum _{t=0}^{\infty }{\gamma }^t{\int }_{s\in \mathcal{S}}{\mathcal{R}}_s^a{\int }_{a\in \mathcal{A}}\pi (a|s;\theta )\cdot da\cdot ds\\ & =\sum _{t=0}^{\infty }{\gamma }^t{\int }_{s\in \mathcal{S}}\cdot {\int }_{a\in \mathcal{A}}\pi (a|s;\theta ){\mathcal{R}}_s^a\cdot da\cdot ds\\ & ={\int }_{s\in \mathcal{S}}\left[\sum _{t=0}^{\infty }{\gamma }^t{\int }_{s_0\in \mathcal{S}}{P}_0(s_0)P(s_0\to s_t,t,\pi )\cdot d{s}_0\right]{\int }_{a\in \mathcal{A}}\pi (a|s;\theta ){\mathcal{R}}_s^a\cdot da\cdot ds\\ & ={\int }_{s\in \mathcal{S}}{\rho }^{\pi }(s){\int }_{a\in \mathcal{A}}\pi (a|s;\theta ){\mathcal{R}}_s^a\cdot da\cdot ds\end{array}\end{array}$$
其中，${\rho }^{\pi }(s)={\sum }_{t=0}^{\infty }{\gamma }^t{\int }_{s_0\in \mathcal{S}}{P}_0(s_0)P(s_0\to s_1,1,\pi )\cdot d{s}_0$，$P_0(s_0)$是状态的初始分布，$P(s_0\to s_1,1,\pi )$代表的是系统依照策略$\pi$从经过1步从状态$s_0$转移到$s_1$的概率(记住这个符号，最终的迭代公式会用到)。

至此，$J(\theta )$已经初步具有定理的形式了。

定理证明

式(2)给出了由初始状态值函数所定义的指标函数
$$\begin{array}{r}\begin{array}{rl}J(\theta )& ={\int }_{s_0}{P}_0(s_0)V^{\pi }(s_0)\cdot d{s}_0\\ & ={\int }_{s_0}{P}_0(s_0){\int }_{a_0}\pi (a_0|s_0;\theta )Q^{\pi }(s_0,a_0)\cdot d{a}_0\cdot d{s}_0\end{array}\end{array}$$
这里注意，为了省空间，积分变量$s_0$和$a_0$都是简写的。由(2)，可以得到指标函数对$\theta$的微分形式
$$\begin{array}{r}\begin{array}{r}{\nabla }_{\theta }J(\theta )={\nabla }_{\theta }\left[{\int }_{s_0}{P}_0(s_0){\int }_{a_0}\pi (a_0|s_0;\theta )Q^{\pi }(s_0,a_0)\cdot d{a}_0\cdot d{s}_0\right]\end{array}\end{array}$$
至此，开始了无穷无尽的推导。根据预备公式中状态值函数与动作值函数的转换公式可得
$$\begin{array}{r}\begin{array}{rl}{\nabla }_{\theta }J(\theta )& ={\int }_{s_0}{P}_0(s_0){\int }_{a_0}{\nabla }_{\theta }\pi (a_0|s_0;\theta )Q^{\pi }(s_0,a_0)\cdot d{a}_0\cdot ds\\ & +{\int }_{s_0}{P}_0(s_0){\int }_{a_0}\pi (a_0|s_0;\theta ){\nabla }_{\theta }{Q}^{\pi }(s_0,a_0)\cdot d{a}_0\cdot ds\\ & =M_0+{\int }_{s_0}{P}_0(s_0){\int }_{a_0}\pi (a_0|s_0;\theta ){\nabla }_{\theta }\left[{\mathcal{R}}_s^a+{\int }_{s_1}\gamma P_{s_0,s_1}^{a_0}{V}^{\pi }(s_1)\cdot d{s}_1\right]\cdot d{a}_0\cdot ds\end{array}\end{array}$$
其中，$M_0={\int }_{s_0}{P}_0(s_0){\int }_{a_0}{\nabla }_{\theta }\pi (a_0|s_0;\theta )Q^{\pi }(s_0,a_0)\cdot d{a}_0\cdot ds$。

由${\nabla }_{\theta }{\mathcal{R}}_s^a=0$可得，
$$\begin{array}{r}\begin{array}{rl}{\nabla }_{\theta }J(\theta )& =M_0+{\int }_{s_0}{P}_0(s_0){\int }_{a_0}\pi (a_0|s_0;\theta ){\nabla }_{\theta }{\int }_{s_1}\gamma P_{s_0,s_1}^{a_0}{V}^{\pi }(s_1)\cdot d{s}_1\cdot d{a}_0\cdot ds\\ & =M_0+{\int }_{s_1}\left[{\int }_{s_0}\gamma P_0(s_0){\int }_{a_0}\pi (a_0|s_0;\theta )P_{s_0,s_1}^{a_0}\cdot d{a}_0\cdot d{s}_0\right]{\nabla }_{\theta }{V}^{\pi }(s_1)\cdot d{s}_1\end{array}\end{array}$$
其中，红色部分恰好是
$${\int }_{a_0}\pi (a_0|s_0;\theta )P_{s_0,s_1}^{a_0}\cdot d{a}_0=P(s_0\to s_1,1,\pi )$$
因此，有
$$\begin{array}{r}\begin{array}{r}{\nabla }_{\theta }J(\theta )=M_0+{\int }_{s_1}\left[{\int }_{s_0}\gamma P_0(s_0)P(s_0\to s_1,1,\pi )\cdot d{s}_0\right]\cdot {\nabla }_{\theta }{V}^{\pi }(s_1)\cdot d{s}_1\end{array}\end{array}$$
至此，迭代公式的第一轮循环已经结束了，结束时的公式(6)中蓝色部分与公式(2)的第一行具有相似的形式。可以看出，整体公式的推导思路就是：

1. $V(s_t)\to Q(s_t,a_t)$
2. 提出一项$M_t$
3. $Q(s_t,a_t)\to V(s_{t+1})$
4. 调换积分次序
5. 得到$P(s_t\to s_{t+1},1,\pi )$
6. 剩下“一堆东西”和一个$V(s_{t+1})$

顺着这个思路继续化简(6)，将$V^{\pi }(s_1)={\int }_{a_1}\pi (a_1|s_1;\theta )Q^{\pi }(s_1,a_1)d{a}_1$可得
$$\begin{array}{r}\begin{array}{rl}{\nabla }_{\theta }J(\theta )& =M_0+{\int }_{s_1}\left[{\int }_{s_0}\gamma P_0(s_0)P(s_0\to s_1,1,\pi )\cdot d{s}_0\right]{\nabla }_{\theta }{V}^{\pi }(s_1)\cdot d{s}_1\\ & =M_0+{\int }_{s_1}\left[{\int }_{s_0}\gamma P_0(s_0)P(s_0\to s_1,1,\pi )\cdot d{s}_0\right]{\nabla }_{\theta }{\int }_{a_1}\pi (a_1|s_1;\theta )Q^{\pi }(s_1,a_1)d{a}_1\cdot d{s}_1\\ & =M_0+{\int }_{s_1}\left[{\int }_{s_0}\gamma P_0(s_0)P(s_0\to s_1,1,\pi )\cdot d{s}_0\right]\left[{\int }_{a_1}{\nabla }_{\theta }\pi (a_1|s_1;\theta )Q^{\pi }(s_1,a_1)d{a}_1+{\int }_{a_1}\pi (a_1|s_1;\theta ){\nabla }_{\theta }{Q}^{\pi }(s_1,a_1)d{a}_1\right]d{s}_1\\ & =M_0+M_1+{\int }_{s_1}\left[{\int }_{s_0}\gamma P_0(s_0)P(s_0\to s_1,1,\pi )\cdot d{s}_0\right]\left[{\int }_{a_1}\pi (a_1|s_1;\theta ){\nabla }_{\theta }{Q}^{\pi }(s_1,a_1)d{a}_1\right]d{s}_1\end{array}\end{array}$$
其中，$M_1={\int }_{s_1}\left[{\int }_{s_0}\gamma P_0(s_0)P(s_0\to s_1,1,\pi )\cdot d{s}_0\right]{\int }_{a_1}{\nabla }_{\theta }\pi (a_1|s_1;\theta )Q^{\pi }(s_1,a_1)d{a}_{1}d{s}_1$。

代入$Q^{\pi }(s_1,a_1)={\mathcal{R}}_{s_1}^{a_1}+\gamma {\int }_{s_1}{P}_{s_1,s_2}^{a_1}{V}^{\pi }(s_2)d{s}_2$，得
$$\begin{array}{r}\begin{array}{rl}{\nabla }_{\theta }J(\theta )& =M_0+M_1\\ & +{\int }_{s_1}\left[{\int }_{s_0}\gamma P_0(s_0)P(s_0\to s_1,1,\pi )\cdot d{s}_0\right]\left[{\int }_{a_1}\pi (a_1|s_1;\theta ){\nabla }_{\theta }\gamma {\int }_{s_1}{P}_{s_1,s_2}^{a_1}{V}^{\pi }(s_2)\cdot d{s}_2\cdot d{a}_1\right]d{s}_1\\ & =M_0+M_1\\ & +{\int }_{s_2}{\int }_{s_1}\left[{\int }_{s_0}{\gamma }^2{P}_0(s_0)P(s_0\to s_1,1,\pi )\cdot d{s}_0\right]{\int }_{a_1}\pi (a_1|s_1;\theta )P_{s_1,s_2}^{a_1}\cdot d{a}_1\cdot d{s}_1\cdot {\nabla }_{\theta }{V}^{\pi }(s_2)\cdot d{s}_2\end{array}\end{array}$$
同理，红色部分恰好是${\int }_{a_1}\pi (a_1|s_1;\theta )P_{s_1,s_2}^{a_1}\cdot d{a}_1=P(s_1\to s_2,1,\pi )$，进而有
$$\begin{array}{r}\begin{array}{rl}{\nabla }_{\theta }J(\theta )& =M_0+M_1\\ & +{\int }_{s_2}{\int }_{s_1}\left[{\int }_{s_0}{\gamma }^2{P}_0(s_0)P(s_0\to s_1,1,\pi )\cdot d{s}_0\right]P(s_1\to s_2,1,\pi )\cdot d{s}_1\cdot {\nabla }_{\theta }{V}^{\pi }(s_2)\cdot d{s}_2\end{array}\end{array}$$
再次调换积分次序，有
$$\begin{array}{r}\begin{array}{rl}{\nabla }_{\theta }J(\theta )& =M_0+M_1\\ & +{\int }_{s_2}{\int }_{s_1}\left[{\int }_{s_0}{\gamma }^2{P}_0(s_0)P(s_0\to s_1,1,\pi )\cdot d{s}_0\right]P(s_1\to s_2,1,\pi )\cdot d{s}_1\cdot {\nabla }_{\theta }{V}^{\pi }(s_2)\cdot d{s}_2\\ & =M_0+M_1\\ & +{\int }_{s_2}{\gamma }^2{\nabla }_{\theta }{V}^{\pi }(s_2){\int }_{s_0}{P}_0(s_0){\int }_{s_1}P(s_0\to s_1,1,\pi )P(s_1\to s_2,1,\pi )\cdot d{s}_1\cdot d{s}_0\cdot d{s}_2\end{array}\end{array}$$
可以看出，红色部分又恰好是${\int }_{s_1}P(s_0\to s_1,1,\pi )P(s_1\to s_2,1,\pi )\cdot d{s}_1=P(s_0\to s_2,2,\pi )$，进而有
$$\begin{array}{r}\begin{array}{rl}{\nabla }_{\theta }J(\theta )& =M_0+M_1\\ & +{\int }_{s_2}{\gamma }^2{\nabla }_{\theta }{V}^{\pi }(s_2){\int }_{s_0}{P}_0(s_0)P(s_0\to s_2,2,\pi )\cdot d{s}_0\cdot d{s}_2\\ & =M_0+M_1+{\int }_{s_2}\left[{\int }_{s_0}{\gamma }^2{P}_0(s_0)P(s_0\to s_2,2,\pi )\cdot d{s}_0\right]{\nabla }_{\theta }{V}^{\pi }(s_2)\cdot d{s}_2\end{array}\end{array}$$
至此，可以看出式(11)和式(6)中的$M$项和后边的积分项都具有极其相似的形式，通过找规律(归纳)，可以得出，当$t\to \infty$时，有

$$\begin{array}{r}\begin{array}{rl}{\nabla }_{\theta }J(\theta )& =\sum _{t=0}^{\infty }{\int }_{s_t}\left[{\int }_{s_0}{\gamma }^t{P}_0(s_0)P(s_0\to s_t,t,\pi )\cdot d{s}_0\right]{\int }_{a_t}{\nabla }_{\theta }\pi (s_t,a_t;\theta )Q^{\pi }(s_t,a_t)\cdot d{a}_t\cdot d{s}_t\\ & ={\int }_s{\int }_{s_0}\sum _{t=0}^{\infty }{\gamma }^t{P}_0(s_0)P(s_0\to s,t,\pi )\cdot d{s}_0{\int }_a{\nabla }_{\theta }\pi (s,a;\theta )Q^{\pi }(s,a)\cdot da\cdot ds\end{array}\end{array}$$
公式(1)的结尾得到${\rho }^{\pi }(s)={\sum }_{t=0}^{\infty }{\gamma }^t{\int }_{s_0\in \mathcal{S}}{P}_0(s_0)P(s_0\to s_1,1,\pi )\cdot d{s}_0$，故而
$${\nabla }_{\theta }J(\theta )={\int }_{\mathcal{S}}{\rho }^{\pi }(s){\int }_{\mathcal{A}}{\nabla }_{\theta }\pi (a|s;\theta )\cdot Q^{\pi }(s,a)\cdot da\cdot ds$$
证明完毕。