[RL 8] Proximal Policy Optimization Algorithms (arXiv, 1707)

Proximal Policy Optimization Algorithms (arXiv, 1707)

1.Introduction

1. room for RL
   1. scalable: support for parallel implementations, to make use of resources
   2. data efficient
   3. robust: non-sensitive to hyperparameter
2. problems
   1. A3C: poor data efficiency
   2. TRPO: complicated, not support for parameter sharing, dropout, etc
3. PPO
   1. data efficiency and reliable(of TRPO)

2.Background

1. PG
   1. problems:
      1. large policy updates
2. Trust Region Methods
   1. constrained
      • conjugate gradient algorithm (quadratic approximation)
      • complex
   2. unconstrained with KL penalty
      • SGD
      • but tuning $\beta$ is non-trivial

3.Clipped Surrogate Objective

$L^{CLIP}(\theta )={\hat{\mathbb{E}}}_t\left[\min \left(r_t(\theta ),1\pm ϵ\right){\hat{A}}_t\right]$

1. $r_t(\theta )=\frac{{\pi }_{\theta }\left(a_t\mid s_t\right)}{{\pi }_{{\theta }_{\mathrm{o}\mathrm{l}\mathrm{d}}}\left(a_t\mid s_t\right)}$
2. Motivation:
   1. objective motivated from TRPO TODO
   2. aviod large policy update
      • by set $r(\theta )$ as a constant to have $g=0$
3. Gradients
   $$\begin{array}{c}{\nabla }_{\theta }{L}_{PPO}=\{\begin{array}{ll}{\nabla }_{\theta }{L}_{\theta }& \frac{{\pi }_{\theta }(a\mid s)}{\pi (a\mid s)}\in [1-ϵ,1+ϵ]\text{ or }{L}_{\theta }^C<L_{\theta }\\ 0& \text{ otherwise }\end{array}\\ \text{ where }{L}_{\theta }:={\mathbb{E}}_{(s,a)\in \tau \sim \pi }\left[\frac{{\pi }_{\theta }(a\mid s)}{\pi (a\mid s)}{A}_{\pi }(s,a)\right]\\ \text{ and }{L}_{\theta }^C:={\mathbb{E}}_{(s,a)\in \tau \sim \pi }\left[\mathrm{clip}\left(\frac{{\pi }_{\theta }(a\mid s)}{\pi (a\mid s)},1-\epsilon ,1+\epsilon \right)A_{\pi }(s,a)\right]\end{array}$$

4.Adaptive KL Penalty Coefficient

• Objective: TRPO obj + KL penalty
  • performe worse than clipped objective

5.Algorithm

1. variance-reduced advantage-function estimators
   1. generalized advantage estimation(GAE 2015) TODO
   2. finite-horizon estimators (2016) TODO
2. parameters sharing
3. entropy bonus
4. loss function
5. parallel actors

6.Experiments

1. Cipping vs KL penalty
   1. clipping outperformes KL penalty
   2. details
      • no parameters sharing
      • no entropy bonus
      • 2 hidden layers with 64 units
      • 1M step
2. PPO vs other continous on-policy algs
   1. PPO does better (faster and higher fianl performance)
   2. A2C >= A3C
3. PPO in high-dimensional continuous control
   1. competent
4. PPO vs other dicrete on-policy algs in Aatri
   1. ACER TODO
      1. higher final performance
   2. PPO
      1. faster learning