强化学习-赵世钰（三）：贝尔曼最优方程【Bellman Optimal Equation】【贝尔曼最优方程符合收缩映射理论--＞可通过迭代法求解最优State Values--＞得到最优策略】

强化学习的目的是寻找最优策略。这里学习贝尔曼最优公式需要重点关注两个概念和一个工具：

• 两个概念：optimal state value和optimal policy
• 一个基本工具：the Bellman optimality equation (BOE)

一、Motivating examples

First, we calculate the state values of the given policy. In particular, the Bellman equation of this policy is

$$\begin{array}{c}{\upsilon }_{\pi }(s_1)=-1+\gamma {\upsilon }_{\pi }(s_2),\\ {\upsilon }_{\pi }(s_2)=+1+\gamma {\upsilon }_{\pi }(s_4),\\ {\upsilon }_{\pi }(s_3)=+1+\gamma {\upsilon }_{\pi }(s_4),\\ \begin{array}{r}{\upsilon }_{\pi }(s_4)\end{array}=+1+\gamma v_{\pi }(s_4).\end{array}$$

Let γ = 0.9. It can be easily solved that

$$\begin{array}{rl}& v_{\pi }(s_4)=v_{\pi }(s_3)=v_{\pi }(s_2)=10,\\ & v_{\pi }(s_1)=8.\end{array}$$

Second, we calculate the action values for state $s_1$:

$$\begin{array}{c}{q}_{\pi }(s_1,a_1)=-1+\gamma v_{\pi }(s_1)=6.2,\\ q_{\pi }(s_1,a_2)=-1+\gamma v_{\pi }(s_2)=8,\\ q_{\pi }(s_1,a_3)=0+\gamma v_{\pi }(s_3)=9,\\ q_{\pi }(s_1,a_4)=-1+\gamma v_{\pi }(s_1)=6.2,\\ q_{\pi }(s_1,a_5)=0+\gamma v_{\pi }(s_1)=\mathrm{7.2.}\end{array}$$

It is notable that action $a_3$ has the greatest action value:

$$q_{\pi }(s_1,a_3)\ge q_{\pi }(s_1,a_i),\text{ for all }i\ne 3.$$

Therefore, we can update the policy to select $a_3$ at $s_1$.

每一次迭代的过程中，每一个state都选择action value最大的action，最终就会得到最优的Policy。

二、最优策略/optimal policy

state value可以用来衡量一个policy的好坏，对于策略 ${\pi }_1$ 和策略 ${\pi }_2$ 来说，倘若在所有的状态 $s$ 下，都存在 $v_{{\pi }_1}(s)\ge v_{{\pi }_2}(s)$ 那么可得策略 ${\pi }_1$优于策略${\pi }_2$ 。因此最优策略 ${\pi }^{\ast }$ 就是，在所有的状态 $s$ 下，均优于其他所有的策略 $\pi$。

上述定义表明，相对于所有其他策略，最优策略对于每个状态都具有最大的状态值。这个定义也引发了许多问题：

• 存在性：最优策略是否存在？
• 唯一性：最优策略是否唯一？
• 随机性：最优策略是随机的还是确定性的？
• 算法：如何获得最优策略和最优状态值？

三、贝尔曼最优公式【Bellman Optimality Equation】

1、贝尔曼公式/Bellman Equation

对于贝尔曼公式来说，求解state value时是依赖于一个给定的π；

2、贝尔曼最优公式/Bellman Optimality Equation

对于贝尔曼最优公式来说，π是不定的，是需要求解的参数；

$\begin{array}{rl}v(s)& =\underset{\pi (s)\in \Pi (s)}{\max }\sum _{a\in \mathcal{A}}\pi (a|s)\left(\sum _{r\in \mathcal{R}}p(r|s,a)r+\gamma \sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)v(s^{\prime })\right)\end{array}$ 公式的含义是：当Policy $\pi$ 取某一个最优化值时，State Value $v(s)$ 可以取到最大值。

BOE 很棘手但又优雅！

• 为什么优雅？它以一种优雅的方式描述了最优策略和最优状态值。
• 为什么棘手？右侧有一个最大化，可能不容易看出如何计算。
• 有许多问题需要回答：
  • 算法：如何解决这个方程？
  • 存在性：这个方程是否有解？
  • 唯一性：这个方程的解是否唯一？
  • 最优性：它与最优策略有什么关系？

3、压缩/收缩映射定理【Contraction mapping theorem】

不动点

压缩/收缩映射

$$||f(x_1)-f(x_2)||\le \gamma ||x_1-x_2||$$

• 其中$\gamma \in (0,1)$；
• $\gamma$必须是严格小于1，这样许多极限${\gamma }^k\to 0$当$k\to 0$成立；
• 这里$||\cdot ||$可以是任意vector norm；

示例：首先，对于标量来说

其次，对于向量来说

预备知识：Contraction mapping theorem

对于任意符合 $x=f(x)$ 格式的方程，如果$f$ 是收缩映射，那么：

• 存在性：存在一个不动点 $x^{\ast }$ 满足 $f(x^{\ast })=x^{\ast }$；
• 唯一性：这个不动点 $x^{\ast }$ 是唯一的；
• 算法：迭代式算法 $x_{k+1}=f(x_k)$，最终可以收敛到不动点处；

两个例子

• 对于标量来说：$x=0.5x$, 其中$f(x)=0.5x$ 并且 $x\in R$, $x^{\ast }=0$是一个唯一 (unique) 的fixed point, 它可以通过迭代式的方式求解

$$x_{k+1}=0.5x_k$$

• 对于向量来说，$x=Ax$,其中$f(x)=Ax$,且$x\in R^n$,$A\in R^{n\times n}$,并且$||A||<1$,$x^{\ast }=0$是唯一的不动点，它可以通过迭代式求解

$$x_{k+1}=A{x}_k$$

4、求解贝尔曼最优公式

4.1 最大化贝尔曼最优公式右侧

首先固定 $\upsilon (s^{\prime })$，因为系统模型参数 $p(r|s,a)$、$p(s^{\prime }|s,a)$都是已知的，$r$(reward)、$\gamma$(discount rate)都是给定的，所以 $q(s,a)$ 是常数，为了使得右侧取到最大，则使得右项最大时的 $\pi$ 策略也可以确定下来了。

4.2 解贝尔曼最优公式

从4.1的分析中可知，如果固定 $\upsilon (s^{\prime })$，那么贝尔曼最优公式的右侧的最大值就可以确定了。

可见右侧的最大值时取决于 $\upsilon (s^{\prime })$，也就是说右侧项是 $\upsilon (s^{\prime })$ 的函数。

上式中 $f(v)$ 是一个向量，$[f(v){]}_s$ 表示向量中对应的元素state $s$ 的值是 $\begin{array}{r}\underset{\pi }{\max }\sum _a\pi (a|s)q(s,a)\end{array}$。

4.3 应用“压缩映射定理”解贝尔曼最优公式

首先要证明贝尔曼最优方程中的 $v=f(v)$ 是一个 Contraction Mapping。

$\because$ 通过证明可以得到：$\Vert f(v_1)-f(v_2)\Vert \le \gamma \Vert v_1-v_2\Vert$，其中 $\gamma$ 是 discount rate；
$\therefore$ $v=f(v)$ 是Contraction Mapping。

由于贝尔曼最优方程符合Contraction mapping theorem，所以：

• 存在性：存在一个解 $v^{\ast }$；
• 唯一性：$v^{\ast }$是唯一的；
• 算法：State Value 可以通过迭代式算法 $\begin{array}{r}{v}_{k+1}=f(v_k)=\underset{\pi }{\max }(r_{\pi }+\gamma P_{\pi }{v}_k)\end{array}$ 最终收敛到唯一解 $v^{\ast }$ 处；

4.4 案例：求解贝尔曼最优公式

四、最优策略/Optimal Policy

贝尔曼最优公式是一个特殊的贝尔曼公式。

贝尔曼最优公式对应的策略是最优策略。

五、最优策略的决定因素

当γ比较大时，会比较远视，得到的return中远期的reward比重会相对大一些；

当γ比较小时，会比较短视，得到的return中近期的reward权重会相对大一些；







在设计reward的时候，即使将默认r设计为0，也不会绕远路，因为除了r来约束不要绕远路，γ的存在也会限制不会绕远路，因为越绕远路，得到的reward越晚，最后计算得到的return越小。

---

---

---

3.1 Motivating example: How to improve policies?

Consider the policy shown in Figure 3.2. Here, the orange and blue cells represent the forbidden and target areas, respectively. The policy here is $not$ $good$ because it selects $a_2$ ( rightward) at state $s_1$. How can we improve the given policy to obtain a better policy? The answer lies in state values and action values.

$Intuition$： It is intuitively clear that the policy can improve if it selects $a_3$(downward) instead of $a_2$(rightward) at $s_1.$ This is because moving downward enables the agent to avoid entering the forbidden area.

$Mathematics$: The above intuition can be realized based on the calculation of state values and action values.

This example illustrates that we can obtain a better policy if we update the policy to select the action with the $greatest$ $actionvalue.$ This is the basic idea of many reinforcement learning algorithms.

This example is very simple in the sense that the given policy is only not good for state $s_1.$ If the policy is also not good for the other states, will selecting the action with the greatest action value still generate a better policy? Moreover, whether there always exist optimal policies? What does an optimal policy look like? We will answer all of these questions in this chapter.

3.2 Optimal state values and optimal policies

While the ultimate goal of reinforcement learning is to obtain optimal policies, it is necessary to first define what an optimal policy is. 【强化学习的最终目标是获得最优策略，但首先需要定义什么是最优策略。】

The defnition is based on state values.

In particular, consider two given policies ${\pi }_1$ and ${\pi }_2$. If the state value of ${\pi }_1$ is greater than or equal to that of ${\pi }_2$ for any state，then ${\pi }_1$ is said to be better than ${\pi }_2$. :【考虑给定的两个策略${\pi }_1$和 ${\pi }_2$， 如果对于任何状态在策略${\pi }_1$下的状态值都大于或等于${\pi }_2$，${\pi }_1$被认为比${\pi }_2$更好。】

$$\begin{array}{r}{v}_{{\pi }_1}(s)\ge v_{{\pi }_2}(s),\text{ for all }s\in \mathcal{S},\end{array}$$

Furthermore, if a policy is better than all the other possible policies, then this policy is optimal.【如果一项政策比所有其他可能的政策都要好，那么这项政策就是最优的。】

Definition 3.1 (Optimal policy and optimal state value). $A$ $policy{\pi }^{\ast }isoptimalif$ $v_{{\pi }^{\ast }}(s)\ge v_{\pi }(s)foralls\in \mathcal{S}andforanyotherpolicy\pi .Thestatevaluesof{\pi }^{\ast }arethe$ $optimal$ $state$ $values.$

The above definition indicates that an optimal policy has the greatest state value for every state compared to all the other policies. This definition also leads to many questions:

• Existence: Does the optimal policy exist?
• Uniqueness: Is the optimal policy unique?
• Stochasticity: Is the optimal policy stochastic or deterministic?
• Algorithm: How to obtain the optimal policy and the optimal state values?

These fundamental questions must be clearly answered to thoroughly understand optimal policies.

For example, regarding the existence of optimal policies, if optimal policies do not exist, then we do not need to bother to design algorithms to find them.

3.3 Bellman optimality equation【贝尔曼最优方程】

The tool for analyzing optimal policies and optimal state values is the Bellman optimality equation (BOE).

By solving this equation, we can obtain optimal policies and optimal state values. 【通过解决这个方程，我们可以得到最优策略和最优状态值。】

We next present the expression of the BOE and then analyze it in detail.

贝尔曼方程：
$$\begin{array}{rl}{v}_{\pi }(s)& =\mathbb{E}[R_{t+1}|S_t=s]+\gamma \mathbb{E}[G_{t+1}|S_t=s]\\ & =\underset{\text{mean of immediate rewards}}{\sum _{a\in \mathcal{A}}\pi (a|s)\sum _{r\in \mathcal{R}}p(r|s,a)r}+\underset{\text{mean of future rewards}}{\gamma \sum _{a\in \mathcal{A}}\pi (a|s)\sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)v_{\pi }(s^{\prime }),}\\ & =\sum _{a\in \mathcal{A}}\pi (a|s)\left[\sum _{r\in \mathcal{R}}p(r|s,a)r+\gamma \sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)v_{\pi }(s^{\prime })\right],\text{for all }s\in \mathcal{S}\end{array}$$

For every $s\in \mathcal{S}$, the elementwise expression of the BOE（贝尔曼最优方程） is

$$\begin{array}{rl}\upsilon (s)& \begin{array}{r}=\underset{\pi (s)\in \Pi (s)}{\max }\sum _{a\in \mathcal{A}}\pi (a|s)\left(\sum _{r\in \mathcal{R}}p(r|s,a)r+\gamma \sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)v(s^{\prime })\right)\end{array}\\ & =\underset{\pi (s)\in \Pi (s)}{\max }\sum _{a\in \mathcal{A}}\pi (a|s)q(s,a),\end{array}$$

where $v(s),v(s^{\prime })$ are unknown variables to be solved and

$$q(s,a)\doteq \sum _{r\in \mathcal{R}}p(r|s,a)r+\gamma \sum _{s^{\prime }\in \mathcal{S}}p(s^{\prime }|s,a)v(s^{\prime }).$$

Here, $\pi (s)$ denotes a policy for state $s$, and $\Pi (s)$ is the set of all possible policies for $s$.

The BOE（贝尔曼最优方程）is an elegant and powerful tool for analyzing optimal policies.

However, it may be nontrivial to understand this equation. For example, this equation has two unknown variables $v(s)$ and $\pi (a|s)$.

It may be confusing to beginners how to solve two unknown variables from one equation.

Moreover, the BOE is actually a special Bellman equation.

However, it is nontrivial to see that since its expression is quite different from that of the Bellman equation.

We also need to answer the following fundamental questions about the BOE.

• Existence: Does this equation have a solution?
• Uniqueness: Is the solution unique?
• Algorithm: How to solve this equation?
• Optimality: How is the solution related to optimal policies?

Once we can answer these questions, we will clearly understand optimal state values and optimal policies.

3.3.1 Maximization of the right-hand side of the BOE

We next clarify how to solve the maximization problem on the right-hand side of the BOE.

At first glance, it may be confusing to beginners how to solve $two$ unknown variables $v(s)$ and $\pi (a|s)$ from $one$ equation.

In fact, these two unknown variables can be solved one by one.

This idea is illustrated by the following example.

Example $\mathrm{3.1.}Considertwounknownvariablesx,y\in \mathbb{R}thatsatisfy$

$$\begin{array}{rl}x& =\underset{y\in \mathbb{R}}{\max }(2x-1-y^2).\end{array}$$

$Thefirststepistosolveyontheright-handsideoftheequation.Regardlessofthevalue$ $ofx,wealwayshave{\max }_y(2x-1-y^2)=2x-1,wherethemaximumisachievedwhen$ $y=0.Thesecondstepistosolvex.Wheny=0,theequationbecomesx=2x-1$, $whichleadstox=1.Therefore,y=0andx=1arethesolutionsoftheequation.$

We now turn to the maximization problem on the right-hand side of the BOE. The BOE in (3.1) can be written concisely as

$$v(s)=\underset{\pi (s)\in \Pi (s)}{\max }\sum _{a\in \mathcal{A}}\pi (a|s)q(s,a),s\in \mathcal{S}.$$

Inspired by Example 3.1, we can first solve the optimal $\pi$ on the right-hand side. How to do that? The following example demonstrates its basic idea.
Example $\mathrm{3.2.}Given{q}_1,q_2,q_3\in \mathbb{R},wewouldliketofindtheoptimalvaluesof{c}_1,c_2,c_3$
$to$ $maximize$

$$\begin{array}{rl}\sum _{i=1}^3{c}_i{q}_i& =c_1{q}_1+c_2{q}_2+c_3{q}_3,\end{array}$$

$\begin{array}{r}where{c}_1+c_2+c_3=1and{c}_1,c_2,c_3\ge 0.\end{array}$

$Withoutlossofgenerality,supposethat{q}_3\ge q_1,q_2.Then,theoptimalsolutionis$ $\begin{array}{r}{c}_3^{\ast }=1and{c}_1^{\ast }=c_2^{\ast }=0.Thisisbecause\end{array}$

$$q_3=(c_1+c_2+c_3)q_3=c_1{q}_3+c_2{q}_3+c_3{q}_3\ge c_1{q}_1+c_2{q}_2+c_3{q}_3$$

$\begin{array}{r}forany{c}_1,c_2,c_3.\end{array}$

Inspired by the above example, since ${\sum }_a\pi (a|s)=1$, we have

$$\sum _{a\in \mathcal{A}}\pi (a|s)q(s,a)\le \sum _{a\in \mathcal{A}}\pi (a|s)\underset{a\in \mathcal{A}}{\max }q(s,a)=\underset{a\in \mathcal{A}}{\max }q(s,a),$$

where equality is achieved when

$$\pi (a|s)=\{\begin{array}{ll}1,& a=a^{\ast },\\ 0,& a\ne a^{\ast }.\end{array}$$

Here, $a^{\ast }=\arg {\max }_{a}q(s,a).$ In summary, the optimal policy $\pi (s)$ is the one that selects the action that has the greatest value of $q(s,a).$

3.3.2 Matrix-vector form of the BOE

The BOE refers to a set of equations defined for all states. If we combine these equations, we can obtain a concise matrix-vector form, which will be extensively used in this chapter.

The matrix-vector form of the BOE is

$$\upsilon =\underset{\pi \in \Pi }{\max }(r_{\pi }+\gamma P_{\pi }\upsilon ),$$

where $v\in {\mathbb{R}}^{|\mathcal{S}|}$ and max is performed in an elementwise manner. The structures of $r_{\pi }$ and $P_{\pi }$ are the same as those in the matrix-vector form of the normal Bellman equationı

$$[r_{\pi }{]}_s\doteq \sum _{\alpha \in \mathcal{A}}\pi (a|s)\sum _{r\in \mathcal{R}}p(r|s,a)r,[P_{\pi }{]}_{s,s^{\prime }}=p(s^{\prime }|s)\doteq \sum _{\alpha \in \mathcal{A}}\pi (a|s)p(s^{\prime }|s,a).$$

Since the optimal value of $\pi$ is determined by $v$, the right-hand side of (3.2) is a function of $v, $ denoted as

$$f(v)\doteq \underset{\pi \in \Pi }{\max }(r_{\pi }+\gamma P_{\pi }v).$$
Then, the BOE can be expressed in a concise form as

$$\upsilon =f(\upsilon )$$

3.3.3 Contraction mapping theorem

Since the BOE can be expressed as a nonlinear equation $v=f(v)$, we next introduce the contraction mapping theorem [6] to analyze it. The contraction mapping theorem is a powerful tool for analyzing general nonlinear equations. It is also known as the fixedpoint theorem. Readers who already know this theorem can skip this part. Otherwise the reader is advised to be familiar with this theorem since it is the key to analyzing the BOE.

Consider a function $f( x) , $where $x\in {\mathbb{R}}^d$ and $f:{\mathbb{R}}^d\to {\mathbb{R}}^d.$

A point $x^{\ast }$ is called a fxed point if

$$f(x^{\ast })=x^{\ast }.$$

The interpretation of the above equation is that the map of $x^{\ast }$ is itself. This is the reason why $x^{\ast }$ is called “fixed”. The function $f$ is a $contraction$ $mapping$ (or contractive function) if there exists $\gamma\in ( 0, 1) $ such that

$$\Vert f(x_1)-f(x_2)\Vert \le \gamma \Vert x_1-x_2\Vert$$

for any $x_1,x_2\in {\mathbb{R}}^d.$ In this book, $\Vert \cdot \Vert$denotes a vector or matrix norm.

---

---

---

【强化学习的数学原理】课程：从零开始到透彻理解（完结）
MathFoundationRL/Book-Mathmatical-Foundation-of-Reinforcement-Learning
学习笔记-强化学习4-用Banach不动点定理证明Value-based RL收敛性
【强化学习】强化学习数学基础：贝尔曼最优公式
学习心得-强化学习【贝尔曼最优公式】