深度强化学习CS285 lec13-lec15 （下）

概述

• IRL的Motivation：从expert demonstration中学习reward function ，而不用人工设计

在上一篇文章提到的Soft Optimality Frameworklec13-lec15(中），对Optimal Behavior进行建模，然后使用expert demonstration进行Optimal Policy的拟合。而在IRL中，则是使用其对Optimal Behavior的建模表示，对expert demonstration进行建模，去Infer Behavior背后的Intent即$p(O_t|s_t,a_t)=exp(r(s_t,a_t))$，换个说法，就是去Learn Reward Function。

一、Inverse RL 背景

1.1 Imitation Learning 与Inverse RL

• Imitation Learning

  1. 拟合expert demonstration中的actions
  2. 不能对action的结果进行Infer and reason

• Inverse RL

  1. 拟合expert demonstration背后的intent
  2. 根据Intent 有可能采取非常不一样的action去达到Intent

1.2 Standard RL与Inverse RL

• Standard RL
  1. Reward要经过人工设计
  2. Reward在一些场景下，极其难设计，如自动驾驶中，reward要根据距离、法规、行车约定之类的设计…

• Inverse RL
  1. 通过expert demonstrations Infer出reward function
  2. 但reward function的结构、形式unknown，而且很多reward都能解释expert behavior，那怎么选？怎么衡量多个可行reward？

二、Inverse RL

2.1 Recap

上一篇文章lec13-lec15(中）对Optimal Behavior的轨迹分布(trajectory distribution)进行了建模，即

$$\begin{array}{rl}p(\tau |O_{1:T})& =p(s_{1:T},a_{1:T}|O_{1:T})\\ & =\frac{p(\tau ,O_{1:T})}{p(O_{1:T})}\\ & \propto p(\tau ,O_{1:T})\\ & =p(\tau )exp(\sum _{t=1}^{T}r(s_t,a_t))\\ & =[p(s_1)\prod _{t=1}^{T}p(s_{t+1}|s_t,a_t)\left]exp\right(\sum _{t=1}^{T}r(s_t,a_t))\end{array}$$

Soft Optimality Framework中我们关注的是Policy即$p(a_t|s_t,O_{1:T})$,通过KL divergence拉近两个分布$p(\tau )=p(s_1){\prod }_{t=1}^{T}p(s_{t+1}|s_t,a_t)exp({\sum }_{t=1}^{T}r(s_t,a_t))$与$\hat{p}(\tau )=p(s_1){\prod }_{t=1}^{T}p(s_{t+1}|s_t,a_t,O_{1:T})p(a_t|s_t,O_{1:T})$得到entropy regularised的Objective，然后套用Model-Free中的Policy Gradient、Q-Learning、Actor-Critic，经过一些改进与修改，得到其Soft的版本。

但在IRL中，我们关注的是reward function即$p(O_t|s_t,a_t)=exp(r_{\psi }(s_t,a_t))$ ，于是有

$$p(\tau |O_{1:T},\psi )\propto p(\tau )exp(\sum _{t=1}^T{r}_{\psi }(s_t,a_t))$$

2.2 Maximum Likelihood Learning

给定一些Expert Demonstration，$\{{\tau }^{(i)}\}$来自最优policy产生的轨迹分布${\pi }^{\ast }(\tau )$，已知：
$$p(\tau |O_{1:T},\psi )\propto p(\tau )exp(\sum _{t=1}^T{r}_{\psi }(s_t,a_t))=p(\tau )exp(r_{\psi }(\tau ))$$

则Maximum Likelihood 的Objective $L(\psi )$为：
$$\underset{\psi }{\max }L(\psi )=\underset{\psi }{\max }\frac{1}{N}\sum _{i=1}^{N}logp({\tau }^{(i)}|O_{1:T},\psi )=\underset{\psi }{\max }\frac{1}{N}\sum _{i=1}^N{r}_{\psi }({\tau }^{(i)})-logZ$$

其中$Z$为轨迹的归一化因子，即Partition function，因为要去掉正比$\propto$的符号，为
$$Z=\int p(\tau )exp(r_{\psi }(\tau ))d\tau$$

对Objective进行求梯度，稍微改写一下：
$$\begin{array}{rl}{\nabla }_{\psi }L(\psi )& ={\nabla }_{\psi }\left[E_{\tau \sim {\pi }^{\ast }(\tau )}\right[r_{\psi }(\tau )]-log\int p(\tau )exp(r_{\psi }(\tau )\left)d\tau \right]\\ & =E_{\tau \sim {\pi }^{\ast }(\tau )}[{\nabla }_{\psi }{r}_{\psi }(\tau )]-\underset{p(\tau |O_{1:T},\psi )}{\frac{1}{Z}\int p(\tau )exp(r_{\psi }(\tau ))}{\nabla }_{\psi }{r}_{\psi }(\tau )d\tau \\ & =E_{\tau \sim {\pi }^{\ast }(\tau )}[{\nabla }_{\psi }{r}_{\psi }(\tau )]-E_{\tau \sim p(\tau |O_{1:T},\psi )}[{\nabla }_{\psi }{r}_{\psi }(\tau )]\end{array}$$

解释一下：

• 第一项的轨迹$\tau$来自Expert Demonstration ，用样本估计，对梯度正向作用
• 第二项的轨迹$\tau$来自当前$r_{\psi }$下的soft optimal policy构成的轨迹分布$p(\tau |O_{1:T},\psi )$，估计的方式挺多，对梯度负向作用
• 因此，参数$\psi$的更新方向往第一项即expert demonstration靠近，使第二项soft optimal policy的reward提高，让两者距离缩小

现在分析一下第二项：
$$\begin{array}{rl}& E_{\tau \sim p(\tau |O_{1:T},\psi )}[{\nabla }_{\psi }{r}_{\psi }(\tau )]\\ & =E_{\tau \sim p(\tau |O_{1:T},\psi )}[{\nabla }_{\psi }\sum _{t=1}^T{r}_{\psi }(s_t,a_t)]\\ & =\sum _{t=1}^T{E}_{(s_t,a_t)\sim p(s_t,a_t|O_{1:T},\psi )}[{\nabla }_{\psi }{r}_{\psi }(s_t,a_t)]\\ & =\sum _{t=1}^T{E}_{s_t\sim p(s_t|O_{1:T},\psi ),a_t\sim p(a_t|s_t,O_{1:T},\psi )}[{\nabla }_{\psi }{r}_{\psi }(s_t,a_t)]\end{array}$$

在lec13-lec15(中）这一篇文章里，有

$$\begin{gathered} {\alpha }_t(s_t)=p(s_t|O_{1:t-1}) \\ \beta (s_t,a_t)=p(O_{t:T}|s_t,a_t) \end{gathered}$$

$$p(s_t|O_{1:T})=\frac{p(s_t,O_{1:T})}{p(O_{1:T})}=\frac{p(O_{t:T}|s_t)p(s_t,O_{1:t-1})}{p(O_{1:T})}\propto {\beta }_t(s_t){\alpha }_t(s_t)$$

$$p(a_t|s_t,O_{1:T})=\frac{{\beta }_t(s_t,a_t)}{{\beta }_t(s_t)}$$

于是固定参数$\psi$的情况下，$$\begin{array}{rl}{u}_t(s_t,a_t)=p(s_t,a_t|O_{1:T},\psi )& =p(s_t|O_{1:T},\psi )p(a_t|s_t,O_{1:T},\psi )\\ & \propto \frac{{\beta }_t(s_t,a_t)}{{\beta }_t(s_t)}{\beta }_t(s_t){\alpha }_t(s_t)\\ & ={\beta }_t(s_t,a_t){\alpha }_t(s_t)\end{array}$$

那就得到了Classical的MaxEnt RL～在2008年Ziebart的Paper中提出即2008 AAAI MEIRL

2.3 Maximum Entropy IRL

总结一下：

最麻烦的是，算法步骤4中的gradient step要去计算Forward Messages即$\alpha (s_t)$与Backward Messages即$\beta (s_t,a_t)$，再Normalize得$u_t(s_t,a_t)=p(s_t,a_t|O_{1:T},\psi )$。而前提还要是$\beta (s_t,a_t)$与$\alpha (s_t)$这些都是离散的、可exhaustive的，并且知道state transition即dynamics，这使Classical的MEIRL的适用范围限制在了Tabular Case+ Known dynamics才能进行计算。

那对于连续的、较高维的state、action以及unknown dyanmics的情况呢？
回到objective gradient的：
$${\nabla }_{\psi }L(\psi )=E_{\tau \sim {\pi }^{\ast }(\tau )}[{\nabla }_{\psi }{r}_{\psi }(\tau )]-E_{\tau \sim p(\tau |O_{1:T},\psi )}[{\nabla }_{\psi }{r}_{\psi }(\tau )]$$

Classical MEIRL是通过Backward与Forward的过程来计算Soft Optimal Policy即$p(\tau |O_{1:T},\psi )$，即PGM的Inference，其实还可以通过上一篇文章中提到的MERL Objective利用Soft算法如Soft Q-learning来得到这个Soft Optimal Policy的，具体而言：

$$J(\theta )=\sum _{t=1}^T{E}_{\pi (s_t,a_t)}[r_{\psi }(s_t,a_t)]+E_{\pi (s_t)}[H(\pi (a|s_t))]$$

从中得到Soft Optimal Policy即$p(\tau |O_{1:T},\psi )$，相当于在每一个reward的参数更新中嵌套了一个MERL，虽然通过优化Objective而不进行Inference的方式确实勉强解决了Large Continuous State/action、Unknown Dynamics的问题，但是很不切实际，一个RL算法计算量已经扛扛的，算出一个当前reward下的Soft Optimal Policy却仅仅当一个gradient step？这要算到猴年马月？

大家回想一下，在General Policy Iteration中，更新policy之后，重新Evaluate更新后Policy对应的State Value或者Q Value时，即Policy Evaluation，不需要评估到Value Converge，只需要保证一定的Policy Improvement

用这个想法来改进嵌套一个MERL获取$p(a_t|s_t,O_{1:T},\psi )$，即对每一个MERL都只跑几个gradient step来更新$p(a_t|s_t,O_{1:T},\psi )$，然后使用improved后的$p(a_t|s_t,O_{1:T},\psi )$获取Samples，估计$E_{\tau \sim p(\tau |O_{1:T},\psi )}[{\nabla }_{\psi }{r}_{\psi }(\tau )]$来更新reward。

跑几个gradient step的$p(a_t|s_t,O_{1:T},\psi )$，肯定不是当前$\psi$下的Soft Optimal Policy，由此获取的Trajectory Samples是biased的，那怎么办呢？(假定跑了几个gradient step的policy表示为$\pi (a|s)$，当前$\psi$下真正的Soft Optimal Policy）为$p(a_t|s_t,O_{1:T},\psi )$

• Important Sampling
  $$\begin{array}{rl}{\nabla }_{\psi }L(\psi )& =E_{\tau \sim {\pi }^{\ast }(\tau )}[{\nabla }_{\psi }{r}_{\psi }(\tau )]-E_{\tau \sim p(\tau |O_{1:T},\psi )}[{\nabla }_{\psi }{r}_{\psi }(\tau )]\\ & \approx \frac{1}{N}\sum _{i=1}^N{\nabla }_{\psi }{r}_{\psi }({\tau }^{(i)})-\frac{1}{{\sum }_j{w}_j}\sum _{j=1}^M{w}_j{\nabla }_{\psi }{r}_{\psi }({\tau }^{(j)})\end{array}$$

$$\begin{array}{rl}{w}_j& =\frac{p(\tau )exp(r_{\psi }({\tau }^{(j)}))}{\pi ({\tau }^{(j)})}\\ & =\frac{p(s_1){\prod }_{t}p(s_{t+1}|s_t,a_t)exp(r_{\psi }(s_t,a_t))}{p(s_1){\prod }_{t}p(s_{t+1}|s_t,a_t)\pi (a_t|s_t)}\\ & =\frac{exp({\sum }_t{r}_{\psi }(s_t,a_t))}{{\prod }_t\pi (a_t|s_t)}\end{array}$$

于是有以下Guided Cost Learning比较practical的算法

2.4 Guided Cost Learning

$$\begin{gathered} {\nabla }_{\psi }L(\psi )=\frac{1}{N}\sum _{i=1}^N{\nabla }_{\psi }{r}_{\psi }({\tau }^{(i)})-\frac{1}{{\sum }_j{w}_j}\sum _{j=1}^M{w}_j{\nabla }_{\psi }{r}_{\psi }({\tau }^{(j)}) \\ {w}_j=\frac{exp({\sum }_t{r}_{\psi }(s_t,a_t))}{{\prod }_t\pi (a_t|s_t)} \end{gathered}$$

参数化的对象有两个：${\pi }_{\theta }(a|s)、r_{\psi }(s_t,a_t)$

Guided Cost Learning的流程如上，公式过程如下：

2.5 GAIL

GAIL：GAN+Imitation Learning，关于GAN可回顾 GAN的基础知识
Guided Cost Learning整个过程有点像GAN了，policy对象${\pi }_{\theta }(a|s_t)$为生成器G，reward对象$r_{\psi }(s_t,a_t)$为判别器D。
从policy对象${\pi }_{\theta }(a|s_t)$生成的样本轨迹$\tau \sim {\pi }_{\theta }(\tau )$，然后从真实expert 的demonstration中$\tau \sim {\pi }^{\ast }(\tau )$采真实的轨迹样本，更新判别器D使其区分expert demos与policy生成的trajectory，若为expert demo输出True，否则输出False。

流程如上，公式过程如下：

值得注意的是，在G更新过程中的reward是$log{D}_{\psi }({\pi }_j)$，当轨迹${\tau }_j$越像专家数据，$D_{\psi }({\tau }_j)$越大，从而policy的gradient即$log{D}_{\psi }({\pi }_j)$也越大，使policy往expert policy方向更新。

这是一篇2016 NIPS的文章Generative Adversarial Imitation Learning

2.6 GAIRL

GAIRL实际上是下面这篇文章的，只是其过程更像是在适应IRL的结构，所以个人称为GAIRL，与GAIL的不同之处在于，GAIRL的判别器D的结构是适应IRL理论推导的结果的，而GAIL的判别器D的输出仅仅是二值变量。
A Connection Between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models

在GAN中的最优判别器形式为$$D^{\ast }(x)=\frac{p^{\ast }(x)}{p_{\theta }(x)+p^{\ast }(x)}$$

在上述IRL的推导中，本来在参数$\psi$更新过程就嵌套了一个MERL，然后这样太麻烦，就拿出MERL的Objective跑几个Policy Gradient Step。
于是在当前reward参数$\psi$给定后，本应一个MERL跑完后得到的Soft Optimal Policy为${\pi }_{\theta }(\tau )\propto p(\tau )exp(r_{\psi }(\tau ))$，将其当作$p^{\ast }(x)$，而根据MERL Objective只跑了几个Gradient Step的policy当做$p_{\theta }(\tau )$，代入判别器的最优结构形式有：

$$\begin{array}{rl}{D}_{\psi }(\tau )& =\frac{p(\tau )\frac{1}{Z}exp(r(\tau ))}{p_{\theta }(\tau )+p(\tau )\frac{1}{Z}exp(r(\tau ))}\\ & =\frac{p(\tau )\frac{1}{Z}exp(r(\tau ))}{p(\tau ){\prod }_t{\pi }_{\theta }(a_t|s_t)+p(\tau )\frac{1}{Z}exp(r(\tau ))}\\ & =\frac{\frac{1}{Z}exp(r(\tau ))}{{\prod }_t{\pi }_{\theta }(a_t|s_t)+\frac{1}{Z}exp(r(\tau ))}\end{array}$$

GAIRL算法流程变为：

后记

• 总算完成了Reframe Control As Probabilistic Inference与Inverse RL的两个lecture，信息量对我来说大得惊人，经过数天奋战，算是告一段落了。这才回溯到2016的水平，我的天！

• 贴一篇即将要看的ICLR 2018的Paper，打算复现这篇论文呢
  LEARNING ROBUST REWARDS WITH ADVERSARIAL INVERSE REINFORCEMENT LEARNING

• 参考资料依旧是CS285Lec15的PPT，还有就是PPT中提到的论文，看了两三篇，又去补了点PGM的理论知识，呜呜呜，接下来是比较轻松的章节了～

• 复现论文实验时，来重新修改一下逻辑与表述，并写篇实验文章，mark！