《reinforcement learning:an introduction》第七章《Multi-step Bootstrapping》总结


由于组里新同学进来,需要带着他入门RL,选择从silver的课程开始。

对于我自己,增加一个仔细阅读《reinforcement learning:an introduction》的要求。

因为之前读的不太认真,这一次希望可以认真一点,将对应的知识点也做一个简单总结。




7.1 n-step TD Prediction

The methods that use n-step backups are still TD methods because theystill change an earlier estimate based on how it differs from a later estimate.

n-step return:


If t +n T(if the n-step return extends to or beyond termination), then all the missing terms are taken as zero ==》这个很容易理解,最后n步之内,还剩多少步就令return等于所有剩余步数的reward和。相应的,前n-1步也是没有任何更新过程,Note that no changes at all are made during the first n- 1 steps of each episode.。
《reinforcement learning:an introduction》第七章《Multi-step Bootstrapping》总结_第1张图片


n-step return的error往往更小:

《reinforcement learning:an introduction》第七章《Multi-step Bootstrapping》总结_第2张图片

Methods that involve an intermediate amount of bootstrapping are important because they will typically perform better than either extreme.





7.2 n-step SARSA 

将V改成Q即可,整个流程基本一致:

《reinforcement learning:an introduction》第七章《Multi-step Bootstrapping》总结_第3张图片


想一下为什么n-step的方法能更有效地更新Q-table:因为每一个好的(s,a)或坏的(s,a)在更新过程中都会被使用n次(看上面伪代码),这样可以有效的backup到n-step之前的相关(s,a)。

《reinforcement learning:an introduction》第七章《Multi-step Bootstrapping》总结_第4张图片




7.3 n-step Off-policy Learning by Importance Sampling

The importance sampling that we have used in this section and in Chapter 5 enables off-policy learning, but at the cost of increasing the variance of the updates. The high variance forces us to use a small step-size parameter, resulting in slow learning. It is probably inevitable that off-policy training is slower than on-policy training|after all, the data is less relevant to what you are trying to learn.

7.4 Off-policy Learning Without Importance Sampling: Then-step Tree Backup Algorithm

7.3/7.4在实际中很少用吧,跳过去了。其中12章的eligibility traces应该是n-step方法部分应该掌握的重点




silver课程的lecture 4中30页之后的内容、lecture 5中26-30关于n-step algorithm以及eligibility trace的介绍非常简单明了。

forward-view TD(λ):

《reinforcement learning:an introduction》第七章《Multi-step Bootstrapping》总结_第5张图片

《reinforcement learning:an introduction》第七章《Multi-step Bootstrapping》总结_第6张图片


backward-view TD(λ):

Forward view provides theory
Backward view provides mechanism
Update online, every step, from incomplete sequences

《reinforcement learning:an introduction》第七章《Multi-step Bootstrapping》总结_第7张图片

《reinforcement learning:an introduction》第七章《Multi-step Bootstrapping》总结_第8张图片

λ=0,TD(λ)就是之前提到的TD(0)方法;λ=1,forward-view TD(λ)就是之前提到的MC方法,backward-view 的online TD(λ)近似是之前提到的MC方法。



forward-view SARSA(λ):

《reinforcement learning:an introduction》第七章《Multi-step Bootstrapping》总结_第9张图片

backward-view SARSA(λ):

《reinforcement learning:an introduction》第七章《Multi-step Bootstrapping》总结_第10张图片

《reinforcement learning:an introduction》第七章《Multi-step Bootstrapping》总结_第11张图片



你可能感兴趣的:((深度)增强学习)