cs285-lec5-policy gradient

总结

按照RL的算法框架，policy gradient的整体步骤如下：

1. sample $\{s_i,a_i\}$ from ${\pi }_{\theta }(a|s)$ (run it over data)
2. 计算期望：$J(\theta )=E_{\tau \sim p_{\theta }(\tau )}[{\sum }_{t}r(s_t,a_t)]\approx \frac{1}{N}{\sum }_i{\sum }_{t}r(s_{i,t},s_{i,t})$
3. ${\nabla }_{\theta }J(\theta )\approx \frac{1}{N}{\sum }_{i=1}^N{\sum }_{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_{i,t}|s_{i,t})(({\sum }_{t^{\prime }=t}^T{\gamma }^{t^{\prime }-t}r(s_{i,t^{\prime }},a_{i,t^{\prime }}))-b)$, involving causal policy, discount factor $\gamma$, reduced baseline.
4. $\theta =\theta +\alpha {\nabla }_{\theta }J(\theta )$

Sergey把以上过程认为是一种formalization of trial-and-error

数学推导

总体来看，RL是为了学到policy，2种传统的方法，policy-gradient以及value-based方法。前者是直接对policy建模，valued-based是间接的方法

RL的优化目标为：${\theta }^{\ast }={\mathrm{argmax}}_{\theta }\underset{J(\theta )}{E_{\tau \sim p_{\theta }(\tau )}[\sum _{t}r(s_t,a_t)]}$

怎么计算$J(\theta )$？

在实际使用中，可以用多次trajectory的结果对$E$做近似，即：
$$J(\theta )=E_{\tau \sim p_{\theta }(\tau )}[\sum _{t}r(s_t,a_t)]\approx \frac{1}{N}\sum _i\sum _{t}r(s_{i,t},s_{i,t})$$

怎么优化$J(\theta )$？

在实际使用的时候，除了计算$J$，还需要优化$J$，所以需要计算$J$对$\theta$的导数

首先把$J$简写一下：$J(\theta )=E_{\tau \sim p_{\theta }(\tau )}[\underset{{\sum }_{t=1}^{T}r(s_t,a_t)}{r(\tau )}]=\int p_{\theta }(\tau )r(\tau )d\tau$

对$J$求$\theta$的导数，有：${\nabla }_{\theta }J(\theta )=\int {\nabla }_{\theta }{p}_{\theta }(\tau )r(\tau )d\tau$，这个需要求${\nabla }_{\theta }{p}_{\theta }(\tau )$，但是$p$是未知的

对$p$做一下数学变换，有：
$$p_{\theta }(\tau ){\nabla }_{\theta }\log p_{\theta }(\tau )=p_{\theta }(\tau )\frac{{\nabla }_{\theta }{p}_{\theta }(\tau )}{p_{\theta }(\tau )}={\nabla }_{\theta }{p}_{\theta }(\tau )$$
代入到${\nabla }_{\theta }J(\theta )$中，有：
$$\begin{array}{rl}{\nabla }_{\theta }J(\theta )& =\int {\nabla }_{\theta }{p}_{\theta }(\tau )r(\tau )d\tau \\ & =\int p_{\theta }(\tau ){\nabla }_{\theta }\log p_{\theta }(\tau )r(\tau )d\tau \end{array}$$
因为$p(\tau )$对某个数字的积分可以写成期望，所以进一步有：
$$\begin{array}{rl}{\nabla }_{\theta }J(\theta )& =\int {\nabla }_{\theta }{p}_{\theta }(\tau )r(\tau )d\tau \\ & =\int p_{\theta }(\tau ){\nabla }_{\theta }\log p_{\theta }(\tau )r(\tau )d\tau \\ & =E_{\tau \sim p_{\theta }(\tau )}[{\nabla }_{\theta }\log p_{\theta }(\tau )r(\tau )]\end{array}$$

现在$\log p_{\theta }(\tau )$还是没法计算，考虑到：
$$\begin{array}{rl}{p}_{\theta }(\tau )& =p_{\theta }(s_1,a_1,\dots ,s_T,a_T)\\ & =p(s_1)\prod _{t=1}^T{\pi }_{\theta }(a_t|s_t)p(s_{t+1}|s_t,a_t)\end{array}$$
公示两边同时增加log，有：
$$\begin{array}{rl}\log p_{\theta }(\tau )& =\log p(s_1)+\sum _{t=1}^T[\log {\pi }_{\theta }(a_t|s_t)+\log p(s_{t+1}|s_t,a_t)]\end{array}$$

现在可以把$\log p_{\theta }(\tau )$代入到${\nabla }_{\theta }J(\theta )$中，有：
$$\begin{array}{rl}{\nabla }_{\theta }J(\theta )& =E_{\tau \sim p_{\theta }(\tau )}[{\nabla }_{\theta }\log p_{\theta }(\tau )r(\tau )]\\ & =E_{\tau \sim p_{\theta }(\tau )}\left[{\nabla }_{\theta }\right[\underset{\text{对}\theta \text{的导数为0}}{\log p(s_1)}+\sum _{t=1}^T[\log {\pi }_{\theta }(a_t|s_t)+\underset{\text{对}\theta \text{的导数为0}}{\log p(s_{t+1}|s_t,a_t)}\left]\right]r(\tau )]\\ & =E_{\tau \sim p_{\theta }(\tau )}\left[\right(\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t))(\sum _{t=1}^{T}r(s_t,a_t))]\end{array}$$
类似求$J$的那样，用多个trajectory的和来近似$E$，即：
$$\begin{array}{rl}{\nabla }_{\theta }J(\theta )& =E_{\tau \sim p_{\theta }(\tau )}[{\nabla }_{\theta }\log p_{\theta }(\tau )r(\tau )]\\ & =E_{\tau \sim p_{\theta }(\tau )}\left[{\nabla }_{\theta }\right[\underset{\text{对}\theta \text{的导数为0}}{\log p(s_1)}+\sum _{t=1}^T[\log {\pi }_{\theta }(a_t|s_t)+\underset{\text{对}\theta \text{的导数为0}}{\log p(s_{t+1}|s_t,a_t)}\left]\right]r(\tau )]\\ & =E_{\tau \sim p_{\theta }(\tau )}\left[\right(\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t))(\sum _{t=1}^{T}r(s_t,a_t))]\\ & \approx \frac{1}{N}\sum _{i=1}^N(\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)\left)\right(\sum _{t=1}^{T}r(s_t,a_t))\end{array}$$
这样计算出来${\nabla }_{\theta }J(\theta )$后，更新$\theta$就可以用$\theta =\theta +\alpha {\nabla }_{\theta }J(\theta )$了

例如，REINFORCE算法的迭代步骤为：

1. 从${\pi }_{\theta }(a_t|s_t)$中sample ${\tau }^i$
2. ${\nabla }_{\theta }J(\theta )\approx {\sum }_i({\sum }_t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t^i|s_t^i)\left)\right({\sum }_{t}r(s_t^i,a_t^i))$
3. $\theta \leftarrow \theta +\alpha {\nabla }_{\theta }J(\theta )$

对比policy gradient和supervised learning的maximum likelihood

supervised learning的log maximum likelihood目标函数为：
$$\begin{array}{rl}\max {\nabla }_{\theta }J(\theta )& \approx \frac{1}{N}\sum _{i=1}^N(\sum _{t=1}^T{\nabla }_{\theta }\log P(X|\theta ))\\ & \approx \frac{1}{N}\sum _{i=1}^N(\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t))\end{array}$$

而policy gradient的优化目标为：
$$\max {\nabla }_{\theta }J(\theta )\approx \frac{1}{N}\sum _{i=1}^N(\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)\left)\right(\sum _{t=1}^{T}r(s_t,a_t))$$
两者的区别在于后面有没有reward

在supervised learning中，因为有label，所以可以保证$s_t,a_t$一定是好的，因此不需要reward，优化必然是要增加$\log \pi$那一部分
但reinforcement learning因为没有label，所以需要reward，在优化时可能就不是要单纯增加$\log \pi$那一部分，而可能是减小

policy gradient的核心在于$\log {\pi }_{\theta }(a_t|s_t)$的选取上

partial observability

partial observability的问题是，只能观察到obs而不是states。在RL中，只有假设states满足markov性质，但obs是不一定满足的。

但是因为policy gradient不需要markov property，所以可以用在partial observability的问题上

即，对于partial observation，目标函数同样为：
$$\max {\nabla }_{\theta }J(\theta )\approx \frac{1}{N}\sum _{i=1}^N(\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|o_t)\left)\right(\sum _{t=1}^{T}r(s_t,a_t))$$

优缺点

优点

1. 更好的收敛性，肯定能收敛
2. 可以在连续分布上选择action

缺点

1. high variance：因为很难give credit to actions that bring more future rewards
2. inefficiency：on-policy导致每次都需要sample data based on policy

reduce the variance

因果性定理

因果性定理：在t时刻做出的policy，不会影响t时刻以前的reward

RL的目标函数：
$${\nabla }_{\theta }J(\theta )\approx \frac{1}{N}\sum _{i=1}^N(\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|o_t)\left)\right(\sum _{t=1}^{T}r(s_t,a_t))$$

根据上面这个因果性定理，在${\nabla }_{\theta }J(\theta )$中将${\sum }_{t=1}^{T}r(s_t,a_t)$替换为policy所在的时刻t及之后的reward，即reward-to-go，有：
$$\begin{array}{rl}{\nabla }_{\theta }J(\theta )& \approx \frac{1}{N}\sum _{i=1}^N(\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|o_t)\left)\right(\sum _{t=1}^{T}r(s_t,a_t))\\ & \approx \frac{1}{N}\sum _{i=1}^N(\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|o_t))\underset{\text{reward-to-go},{\hat{Q}}_{i,t}}{(\sum _{t^{\prime }=t}^{T}r(s_{t^{\prime }},a_{t^{\prime }}))}\end{array}$$

替换完之后，因为sum of reward小了，所以期望也小了，进一步的variance也小了

衰减因子$\gamma$降低variance

考虑更远的未来会导致vairance过高（考虑越远，policy越会过拟合），theoretically the discount factor should start at $t=1$, that is:
$$\begin{array}{rl}{\nabla }_{\theta }J(\theta )& \approx \frac{1}{N}\sum _{i=1}^N(\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_{i,t}|s_{i,t})\left)\right(\sum _{t=1}^T{\gamma }^{t-1}r(s_{i,t},a_{i,t}))\\ & \approx \frac{1}{N}\sum _{i=1}^N\sum _{t=1}^T{\gamma }^{t-1}{\nabla }_{\theta }\log {\pi }_{\theta }(a_{i,t}|s_{i,t})(\sum _{t^{\prime }=t}^T{\gamma }^{t^{\prime }-t}r(s_{i,t^{\prime }},a_{i,t^{\prime }}))\end{array}$$

In practice we don’t use this, because it could mean later rewards and steps matter less.

So we will use:
$${\nabla }_{\theta }J(\theta )\approx \frac{1}{N}\sum _{i=1}^N\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_{i,t}|s_{i,t})(\sum _{t^{\prime }=t}^T{\gamma }^{t^{\prime }-t}r(s_{i,t^{\prime }},a_{i,t^{\prime }}))$$

减baseline方式降低variance

要降低policy gradient的variance，做法是，减掉1个baseline（减掉1个baseline，可以让reward低于平均值的action得到一个负的梯度，高于平均的得到正的梯度），也即：
$$\begin{array}{rl}{\nabla }_{\theta }J(\theta )& \approx \frac{1}{N}\sum _{i=1}^N{\nabla }_{\theta }\log p_{\theta }(\tau )[r(\tau )-b]\\ b& =\frac{1}{N}\sum _{i=1}^{N}r(\tau )\end{array}$$
减去1个常数b不会影响expection，同时还能降低variance，因为：
$$\begin{array}{rl}E[\log p_{\theta }(\tau )b]& =\int p_{\theta }(\tau ){\nabla }_{\theta }\log p_{\theta }(\tau )bd\tau \\ & =\int {\nabla }_{\theta }{p}_{\theta }(\tau )bd\tau \\ & =b{\nabla }_{\theta }\int p_{\theta }(\tau )d\tau \\ & =b{\nabla }_{\theta }1\\ & =0\end{array}$$
注：

1. baseline设置为average reward不是最优，但是效果很不错了已经
2. 上面第一步到第二步的推导，用到了：$p_{\theta }(\tau ){\nabla }_{\theta }\log p_{\theta }(\tau )=p_{\theta }(\tau )\frac{{\nabla }_{\theta }{p}_{\theta }(\tau )}{p_{\theta }(\tau )}={\nabla }_{\theta }{p}_{\theta }(\tau )$
3. $\int p_{\theta }(\tau )d\tau$是条件概率分布的积分，也即1

开始准备最优baseline（实际中不常应用，但还是可以推导一下），把variance写出来：
$$Var[x]=E[x^2]-E^2[x]$$
将$J$写成期望的形式，有：
$${\nabla }_{\theta }J(\theta )=E_{\tau \sim p_{\theta }(\tau )}[{\nabla }_{\theta }\log p_{\theta }(\tau )(r(\tau )-b)]$$
所以方差推导：
$$\begin{array}{rl}Var& =E_{\tau \sim p_{\theta }(\tau )}[(\underset{\text{用}g(\tau )\text{代替}}{{\nabla }_{\theta }\log p_{\theta }(\tau )}(r(\tau )-b){)}^2]-\underset{\text{期望与b无关, 所以可以直接写成原先的E}}{E_{\tau \sim p_{\theta }(\tau )}^2[{\nabla }_{\theta }\log p_{\theta }(\tau )(r(\tau )-b)]}\\ & =E_{\tau \sim p_{\theta }(\tau )}[g^2(\tau )(r(\tau )-b){)}^2]-E_{\tau \sim p_{\theta }(\tau )}^2[g(\tau )r(\tau )]\end{array}$$
求方差对$b$的梯度，有：
$$\begin{array}{rl}\frac{\partial Var}{\partial b}& =\frac{\partial }{\partial b}E[\underset{\text{对b的导数为0}}{g^2(\tau )(r^2(\tau )}-2br(\tau )+b^2)]\\ & =-2E[g^2(\tau )r(\tau )]+2bE[g^2(\tau )]\end{array}$$
令梯度为0，得到：
$$b=\frac{E[g^2(\tau )r(\tau )]}{E[g^2(\tau )]}$$
其实就是reward的期望/平均值，只不过做了个$E[g^2(\tau )]$的增幅，使得b和policy相关了。不同的policy parameter会导致不同的b

Considering actor-cirtic which has low variance, we could use value function as the baseline, if and only if this baseline is state-dependent:
$${\nabla }_{\theta }J(\theta )\approx \frac{1}{N}\sum _{i=1}^N\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_{i,t}|s_{i,t})((\sum _{t^{\prime }=t}^T{\gamma }^{t^{\prime }-t}r(s_{i,t^{\prime }},a_{i,t^{\prime }}))-{\hat{V}}_{\varphi }^{\pi }(s_{i,t}))$$

on-policy -> off-policy

on-policy性质

policy gradient是一种on-policy的算法，$J(\theta )=E_{\tau \sim p_{\theta }(\tau )}[{\sum }_{t}r(s_t,a_t)]$，其中$\tau \sim p_{\theta }(\tau )$表示每次都需要从当前的policy导致的样本分布中sample样本，也即每次都需要从新分布中sample样本。

由于on-policy非常inefficient，所以考虑off-policy的policy gradient

推广到off-policy

如果可以用其他分布得到samples，就可以把on-policy变成off-policy

引入一个IS/importance sample概念

importance sample，对期望做变换，有：
$$\begin{array}{rl}{E}_{x\sim p(x)}(f(x))& =\int p(x)f(x)dx\\ & =\int \frac{q(x)}{q(x)}p(x)f(x)dx\\ & =\int q(x)\frac{p(x)}{q(x)}f(x)dx\\ & =E_{x\sim q(x)}[\frac{p(x)}{q(x)}f(x)]\end{array}$$

同理代入$J(\theta )=E_{\tau \sim p_{\theta }(\tau )}[r(\tau )]$中，假设我们没有从$p_{\theta }(\tau )$中sample的$\tau$，但是有从${\bar{p}}_{\theta }(\tau )$中sample出的$\tau$，此时：
$$J(\theta )=E_{\tau \sim {\bar{p}}_{\theta }(\tau )}[\frac{p_{\theta }(\tau )}{{\bar{p}}_{\theta }(\tau )}r(\tau )]$$
对$\frac{p_{\theta }(\tau )}{{\bar{p}}_{\theta }(\tau )}$有：
$$\begin{array}{rl}\frac{p_{\theta }(\tau )}{{\bar{p}}_{\theta }(\tau )}& =\frac{p(s_1){\prod }_{t=1}^T{\pi }_{\theta }(a_t|s_t)p(s_{t+1}|s_t,a_t)}{p(s_1){\prod }_{t=1}^T{\bar{\pi }}_{\theta }(a_t|s_t)p(s_{t+1}|s_t,a_t)}\\ & =\frac{{\prod }_{t=1}^T{\pi }_{\theta }(a_t|s_t)}{{\prod }_{t=1}^T{\bar{\pi }}_{\theta }(a_t|s_t)}\end{array}$$

利用这个性质去预测新的参数${\theta }^{\prime }$，有：
$$\begin{array}{rl}J({\theta }^{\prime })& =E_{\tau \sim p_{\theta }(\tau )}[\frac{p_{{\theta }^{\prime }}(\tau )}{p_{\theta }(\tau )}r(\tau )]\\ {\nabla }_{{\theta }^{\prime }}J({\theta }^{\prime })& =E_{\tau \sim p_{\theta }(\tau )}[\frac{{\nabla }_{{\theta }^{\prime }}{p}_{{\theta }^{\prime }}(\tau )}{p_{\theta }(\tau )}r(\tau )]\\ & =E_{\tau \sim p_{\theta }(\tau )}[\frac{p_{{\theta }^{\prime }}(\tau )}{p_{\theta }(\tau )}{\nabla }_{{\theta }^{\prime }}\log p_{{\theta }^{\prime }}(\tau )r(\tau )]\end{array}$$
其中：

1. 第2步，是因为只有$p_{{\theta }^{\prime }}(\tau )$与${\theta }^{\prime }$有关，所以求导数可以直接把符号放进去
2. 第3步，利用数学变换：$p_{\theta }(\tau ){\nabla }_{\theta }\log p_{\theta }(\tau )=p_{\theta }(\tau )\frac{{\nabla }_{\theta }{p}_{\theta }(\tau )}{p_{\theta }(\tau )}={\nabla }_{\theta }{p}_{\theta }(\tau )$
3. 注意最后这个${\nabla }_{{\theta }^{\prime }}J({\theta }^{\prime })$的形式，其实和${\nabla }_{\theta }J(\theta )$几乎相同，只是前面增加了一个$\frac{p_{{\theta }^{\prime }}(\tau )}{p_{\theta }(\tau )}$

最终${\nabla }_{\theta }J(\theta )$如下

对on-policy：
$$\begin{array}{rl}{\nabla }_{\theta }J(\theta )& =E_{\tau \sim p_{\theta }(\tau )}[{\nabla }_{\theta }\log p_{\theta }(\tau )r(\tau )]\\ & \approx \frac{1}{N}\sum _{i=1}^N\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_{i,t}|s_{i,t}){\hat{Q}}_{i,t}\end{array}$$
其中：${\hat{Q}}_{i,t}={\sum }_{t^{\prime }=t}^{T}r(s_{i,t^{\prime }},a_{i,t^{\prime }})$，表示未来的reward之和

类似地，对off-policy:
$$\begin{array}{r}{\nabla }_{{\theta }^{\prime }}J({\theta }^{\prime })\approx \frac{1}{N}\sum _{i=1}^N\sum _{t=1}^T\frac{{\pi }_{{\theta }^{\prime }}(a_{i,t}|s_{i,t})}{{\pi }_{\theta }(a_{i,t}|s_{i,t})}{\nabla }_{{\theta }^{\prime }}\log {\pi }_{{\theta }^{\prime }}(a_{i,t}|s_{i,t}){\hat{Q}}_{i,t}\end{array}$$

代码实现

在实现时，计算${\nabla }_{\theta }\log {\pi }_{\theta }$效率太低了，所以做法是，找到一个loss function，使其梯度和${\nabla }_{\theta }J(\theta )$相同即可，比如：
$$\tilde{J}(\theta )\approx \frac{1}{N}\sum _{i=1}^N\sum _{t=1}^T\underset{\text{cross entropy/squared error}}{\log {\pi }_{\theta }(a_{i,t}|s_{i,t})}{\hat{Q}}_{i,t}$$

在代码中实现时，和supervised learning很相似，以下例子以离散的action为例
supervised learning：

'''
Given
actions - (N * T) * Da tensor of actions
states - (N * T) * Ds tensor of states
'''
logits = policy.predictions(states)
negative_likelihoods = tf.nn.softmax_cross_entropy_with_logits(labels=actions, logits=logits)
loss = tf.reduce_mean(negative_likelihoods)
gradients = loss.gradients(loss, variables)

policy gradient:

'''
Given
actions - (N * T) * Da tensor of actions
states - (N * T) * Ds tensor of states
q_values - (N * T) * 1 tensor of estimated state-action values
'''
logits = policy.predictions(states)
negative_likelihodds = tf.nn.softmax_cross_entropy_with_logits(labels=actions, logits=logits)
weighted_negative_likelihoods = tf.multiply(negative_likelihoods, q_values)
loss = tf.reduce_mean(weighted_negative_likelihoods)
gradients = loss.gradients(loss, variables)

注意policy gradient和supervised learning还是不一样的：

1. gradient有很大的variance
2. gradient will be noisy
3. batch_size需要很大
4. lr不好调参了（adam算法还是能用）

自动调整学习率$\alpha$

policy gradient也有学习率过大导致梯度下降不稳定、震荡的问题，因此也有自动调整的$\alpha$，可见论文：15-icml-trust region policy optimization

Questions

Why does policy gradient have high variance?

Because policy gradient optimizes on every single sample, the expected value of total rewards is calculated as:
$$J(\theta )=E_{\tau \sim p_{\theta }(\tau )}[\sum _{t}r(s_t,a_t)]\approx \frac{1}{N}\sum _i\sum _{t}r(s_{i,t},s_{i,t})$$
And to optimize $J(\theta )$, we need to calculate ${\nabla }_{\theta }J(\theta )$ as:
$${\nabla }_{\theta }J(\theta )\approx \frac{1}{N}\sum _{i=1}^N(\sum _{t=1}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)\left)\right(\sum _{t=1}^{T}r(s_t,a_t))$$

This method uses every single sample, so usually it has low bias but high variance.

How to calculate $\log {\pi }_{\theta }(a|s)$?