预备知识
本文章基于ml-agents v0.7版本，因为都是阅览版，若读者使用更其他版本肯定有较大不同之处。
再看本文之前希望先对ml-agents有一个初步的认识，将项目运行起来尝试一下。
1.这里可以参考 浪尖儿大神的文章https://www.jianshu.com/p/31844f586b04。
2.除了 浪尖儿推荐的几个视频外，我还推荐李毅宏的https://www.bilibili.com/video/av24724071已经包含了ml agents所要用到的大多数算法，总长只有6个多小时，最主要的一点视频是中文的。
3.当然这里还需要了解tensorflow，起码要看的懂代码。
打开v0.7的时候我发现多了个barracude代替了原来的TFCSharp，可以使用自己的nn文件来保存模型，使得ml-agents可以在其他平台上运行。不过 barracude需要反编译才能看见源码，下次再与大家分享。
PPO
参考论文是OpenAI的Proximal Policy Optimization Algorithms
这里我们定位到ml agents项目目录下ml-agents\mlagents\trainers\ppo\models.py，其中包含了ppo和Curiosity模型的主要代码。其中create_ppo_optimizer函数包含了ppo的算法和最后的总loss。
还有一个是bc目录下的models.py这个是模仿学习用到的模型，以后再讲。
代码如下：
def create_ppo_optimizer(self, probs, old_probs, value, entropy, beta, epsilon, lr, max_step):
"""
Creates training-specific Tensorflow ops for PPO models.
:param probs: Current policy probabilities
:param old_probs: Past policy probabilities
:param value: Current value estimate
:param beta: Entropy regularization strength
:param entropy: Current policy entropy
:param epsilon: Value for policy-divergence threshold
:param lr: Learning rate
:param max_step: Total number of training steps.
"""
self.returns_holder = tf.placeholder(shape=[None], dtype=tf.float32, name='discounted_rewards')
self.advantage = tf.placeholder(shape=[None, 1], dtype=tf.float32, name='advantages')
self.learning_rate = tf.train.polynomial_decay(lr, self.global_step, max_step, 1e-10, power=1.0)
self.old_value = tf.placeholder(shape=[None], dtype=tf.float32, name='old_value_estimates')
decay_epsilon = tf.train.polynomial_decay(epsilon, self.global_step, max_step, 0.1, power=1.0)
decay_beta = tf.train.polynomial_decay(beta, self.global_step, max_step, 1e-5, power=1.0)
optimizer = tf.train.AdamOptimizer(learning_rate=self.learning_rate)
clipped_value_estimate = self.old_value + tf.clip_by_value(tf.reduce_sum(value, axis=1) - self.old_value,
- decay_epsilon, decay_epsilon)
v_opt_a = tf.squared_difference(self.returns_holder, tf.reduce_sum(value, axis=1))
v_opt_b = tf.squared_difference(self.returns_holder, clipped_value_estimate)
self.value_loss = tf.reduce_mean(tf.dynamic_partition(tf.maximum(v_opt_a, v_opt_b), self.mask, 2)[1])
# Here we calculate PPO policy loss. In continuous control this is done independently for each action gaussian
# and then averaged together. This provides significantly better performance than treating the probability
# as an average of probabilities, or as a joint probability.
#region ppo2
r_theta = tf.exp(probs - old_probs)
p_opt_a = r_theta * self.advantage
p_opt_b = tf.clip_by_value(r_theta, 1.0 - decay_epsilon, 1.0 + decay_epsilon) * self.advantage
self.policy_loss = -tf.reduce_mean(tf.dynamic_partition(tf.minimum(p_opt_a, p_opt_b), self.mask, 2)[1])
#endregion ppo2
self.loss = self.policy_loss + 0.5 * self.value_loss - decay_beta * tf.reduce_mean(
tf.dynamic_partition(entropy, self.mask, 2)[1])
#curiosity
if self.use_curiosity:
self.loss += 10 * (0.2 * self.forward_loss + 0.8 * self.inverse_loss)
self.update_batch = optimizer.minimize(self.loss)
其中的
r_theta = tf.exp(probs - old_probs)
p_opt_a = r_theta * self.advantage
p_opt_b = tf.clip_by_value(r_theta, 1.0 - decay_epsilon, 1.0 + decay_epsilon) * self.advantage
self.policy_loss = -tf.reduce_mean(tf.dynamic_partition(tf.minimum(p_opt_a, p_opt_b), self.mask, 2)[1])
就是我们的PPO算法了。对应PPO论文中的第三算法Clipped Surrogate Objective，具体公式如下：
所以显然r_theta就是,p_opt_a就是,p_opt_b就是,最后policy_loss就是。
公式的具体推导可以从前面提到的资料中找到，如李宏毅的ppo章节，还有我所写的PPO公式推导。
Curiosity
这里我们来看下上面提到过的好奇心模型，主要作用就是让机器尝试更多的方法来获得奖励。
参考论文是Curiosity-driven Exploration by Self-supervised Prediction
Curiosity
这么方法也很直观，就是在原来模型上添加ICM模块，就如从和预测出的,如果和相差越大就给越大的奖励。这就是好奇心模型的思想。ICM内部如下图：
ICM
图片和内容都来自于李宏毅的Sparse Reward章节，有兴趣的读者可以参考。
我们下面来看看ml agents的具体实现，首先在上面提到的
create_ppo_optimizer函数中如下代码：
#curiosity
if self.use_curiosity:
self.loss += 10 * (0.2 * self.forward_loss + 0.8 * self.inverse_loss)
意思就是如果使用Curiosity功能，就要添加forward_loss和inverse_loss。
forward_loss是由create_forward_model函数获得的。代码如下：
def create_forward_model(self, encoded_state, encoded_next_state):
"""
Creates forward model TensorFlow ops for Curiosity module.
Predicts encoded future state based on encoded current state and given action.
:param encoded_state: Tensor corresponding to encoded current state.
:param encoded_next_state: Tensor corresponding to encoded next state.
"""
combined_input = tf.concat([encoded_state, self.selected_actions], axis=1)
hidden = tf.layers.dense(combined_input, 256, activation=self.swish)
# We compare against the concatenation of all observation streams, hence `self.vis_obs_size + int(self.vec_obs_size > 0)`.
pred_next_state = tf.layers.dense(hidden, self.curiosity_enc_size * (self.vis_obs_size + int(self.vec_obs_size > 0)),
activation=None)
squared_difference = 0.5 * tf.reduce_sum(tf.squared_difference(pred_next_state, encoded_next_state), axis=1)
self.intrinsic_reward = tf.clip_by_value(self.curiosity_strength * squared_difference, 0, 1)
self.forward_loss = tf.reduce_mean(tf.dynamic_partition(squared_difference, self.mask, 2)[1])
create_forward_model作用是ICM图中的Network1。
Network1
inverse_loss是由create_inverse_model函数获得的。代码如下：
def create_inverse_model(self, encoded_state, encoded_next_state):
"""
Creates inverse model TensorFlow ops for Curiosity module.
Predicts action taken given current and future encoded states.
:param encoded_state: Tensor corresponding to encoded current state.
:param encoded_next_state: Tensor corresponding to encoded next state.
"""
combined_input = tf.concat([encoded_state, encoded_next_state], axis=1)
hidden = tf.layers.dense(combined_input, 256, activation=self.swish)
if self.brain.vector_action_space_type == "continuous":
pred_action = tf.layers.dense(hidden, self.act_size[0], activation=None)
squared_difference = tf.reduce_sum(tf.squared_difference(pred_action, self.selected_actions), axis=1)
self.inverse_loss = tf.reduce_mean(tf.dynamic_partition(squared_difference, self.mask, 2)[1])
else:
pred_action = tf.concat(
[tf.layers.dense(hidden, self.act_size[i], activation=tf.nn.softmax)
for i in range(len(self.act_size))], axis=1)
cross_entropy = tf.reduce_sum(-tf.log(pred_action + 1e-10) * self.selected_actions, axis=1)
self.inverse_loss = tf.reduce_mean(tf.dynamic_partition(cross_entropy, self.mask, 2)[1])
create_inverse_model作用是ICM图中的Network2。
Network2
还有
create_curiosity_encoders函数就是为了encoded_state, encoded_next_state做准备。
encoded_state就是图片中的,encoded_next_state就是;
代码如下：
def create_curiosity_encoders(self):
"""
Creates state encoders for current and future observations.
Used for implementation of Curiosity-driven Exploration by Self-supervised Prediction
See https://arxiv.org/abs/1705.05363 for more details.
:return: current and future state encoder tensors.
"""
encoded_state_list = []
encoded_next_state_list = []
if self.vis_obs_size > 0:
self.next_visual_in = []
visual_encoders = []
next_visual_encoders = []
for i in range(self.vis_obs_size):
# Create input ops for next (t+1) visual observations.
next_visual_input = self.create_visual_input(self.brain.camera_resolutions[i],
name="next_visual_observation_" + str(i))
self.next_visual_in.append(next_visual_input)
# Create the encoder ops for current and next visual input. Not that these encoders are siamese.
encoded_visual = self.create_visual_observation_encoder(self.visual_in[i], self.curiosity_enc_size,
self.swish, 1, "stream_{}_visual_obs_encoder"
.format(i), False)
encoded_next_visual = self.create_visual_observation_encoder(self.next_visual_in[i],
self.curiosity_enc_size,
self.swish, 1,
"stream_{}_visual_obs_encoder".format(i),
True)
visual_encoders.append(encoded_visual)
next_visual_encoders.append(encoded_next_visual)
hidden_visual = tf.concat(visual_encoders, axis=1)
hidden_next_visual = tf.concat(next_visual_encoders, axis=1)
encoded_state_list.append(hidden_visual)
encoded_next_state_list.append(hidden_next_visual)
if self.vec_obs_size > 0:
# Create the encoder ops for current and next vector input. Not that these encoders are siamese.
# Create input op for next (t+1) vector observation.
self.next_vector_in = tf.placeholder(shape=[None, self.vec_obs_size], dtype=tf.float32,
name='next_vector_observation')
encoded_vector_obs = self.create_vector_observation_encoder(self.vector_in,
self.curiosity_enc_size,
self.swish, 2, "vector_obs_encoder",
False)
encoded_next_vector_obs = self.create_vector_observation_encoder(self.next_vector_in,
self.curiosity_enc_size,
self.swish, 2,
"vector_obs_encoder",
True)
encoded_state_list.append(encoded_vector_obs)
encoded_next_state_list.append(encoded_next_vector_obs)
encoded_state = tf.concat(encoded_state_list, axis=1)
encoded_next_state = tf.concat(encoded_next_state_list, axis=1)
return encoded_state, encoded_next_state
其中create_visual_observation_encoder和create_vector_observation_encoder的作用就是将encode为.如图中
Feature Ext
结束语
ml agnets中的ppo\models.py就解释完了，policy.py中还有rewards的代码等到下次再说，希望读者有所收获。本文没有太多的详细的解释，因为强化学习内容丰富并不是一篇文章能说的清楚的。只是提炼出我认为重要的内容和大家分享，还有ml-agents与论文之间的联系。希望大家喜欢。
Unity 机器学习 332928260