第2章 马尔可夫决策过程

2.1 马尔可夫决策过程（上）

Markov Decision Process（MDP）

1. Markov Decision Process can model a lot of real-world problem. It formally describes the framework of reinforcement learning
2. Under MDP, the environment is fully observable.
   1. Optimal control primarily deals with continuous MDPs
   2. Partially observable problems can be converted into MDPs

Markov Property

1. The history of states: $h_t=\left\{s_1,s_2,s_3,...,s_t\right\}$

2. State $s_t$ is Markovian if and only if:
   $$p(s_{t+1}|s_t)=p(s_{t+1}|h_t)$$

   $$p(s_{t+1}|s_t,a_t)=p(s_{t+1}|h_t,a_t)$$

3. “The future is independent of the past given the present”

Markov Process/Markov Chain

1. State transition matrix P specifies $p(s_{t+1}=s^{\prime }|s_t=s)$
   P = [ P ( s 1 ∣ s 1 ) P ( s 2 ∣ s 1 ) . . . P ( s N ∣ s 1 ) P ( s 1 ∣ s 2 ) P ( s 2 ∣ s 2 ) . . . P ( s N ∣ s 2 ) . . . . . . ⋱ . . . P ( s 1 ∣ s N ) P ( s 2 ∣ s N ) . . . P ( s N ∣ s N ) ] P=\begin{bmatrix} P(s_{1}|s_{1}) & P(s_{2}|s_{1}) & ... & P(s_{N}|s_{1})\\ P(s_{1}|s_{2}) & P(s_{2}|s_{2}) & ... & P(s_{N}|s_{2})\\ ... & ... & \ddots & ...\\ P(s_{1}|s_{N}) & P(s_{2}|s_{N}) & ... & P(s_{N}|s_{N}) \end{bmatrix} P= ​P(s1​∣s1​)P(s1​∣s2​)...P(s1​∣sN​)​P(s2​∣s1​)P(s2​∣s2​)...P(s2​∣sN​)​......⋱...​P(sN​∣s1​)P(sN​∣s2​)...P(sN​∣sN​)​ ​

Example of MP

1. Sample episodes starting from $s_3$
   1. $s_3,s_4,s_5,s_6,s_6$
   2. $s_3,s_2,s_3,s_2,s_1$
   3. $s_3,s_4,s_4,s_5,s_5$

Markov Reward Process (MRP)

1. Markov Reward Process is a Markov Chain + reward
2. Definition of Markov Reward Process (MRP)
   1. S is a (finite) set of states (s ∈ S)
   2. P is dynamics/transition model that specifies $P(S_{t+1}=s^{\prime }|s_t=s)$
   3. R is a reward function $R(s_t=s)=E[r_t|s_t=s]$
   4. Discount factor $\gamma \in [0,1]$
3. If finite number of states, R can be a vector

Example of MRP

Reward: +5 in $s_1$, +10 in $s_7$, 0 in all other states. So that we can represent R = [5, 0, 0, 0, 0, 0, 10]

Return and Value function

1. Definition of Horizon

   1. Number of maximum time steps in each episode
   2. Can be infinite, otherwise called finite Markov (reward) Process

2. Definition of Return

   1. Discounted sum of rewards from time step t to horizon
      $$G_t=R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+{\gamma }^3{R}_{t+4}+...+{\gamma }^{T-t-1}{R}_T$$

3. Definition of state value function $V_t(s)$ for a MRP

   1. Expected return from t in state s
      $$V_t(s)=E[G_t|s_t=s]=E[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+{\gamma }^3{R}_{t+4}+...+{\gamma }^{T-t-1}{R}_T|s_t=s]$$

   2. Present value of future rewards

Why Discount Factor γ

1. Avoid infinite returns in cycle Markov processes
2. Uncertainly about the future may not be fully represented
3. If the reward is financial, immediate rewards may earn more interest than delayed rewards
4. Animal/human behaviour shows preference for immediate reward
5. It is sometimes possible to use undiscounted Markov reward processes (i.e. γ = 1), e.g if all sequences terminate.
   1. γ = 0: Only care about the immediate reward
   2. γ = 1: Future reward is equal to the immediate reward.

Example of MRP

1. Reward: +5 in $s_1$, +10 in $s_7$, 0 in all other states. So that we can represent R = [5, 0, 0, 0, 0, 0, 10]
2. Sample returns G for a 4-step episodes with γ = 1/2
   1. return for $s_4,s_5,s_6,s_7$ : $0+\frac{1}{2}\times 0+\frac{1}{4}\times 0+\frac{1}{8}\times 10=1.25$
   2. return for $s_4,s_3,s_2,s_1$ : $0+\frac{1}{2}\times 0+\frac{1}{4}\times 0+\frac{1}{8}\times 5=0.625$
   3. return for $s_4,s_5,s_6,s_6$ : = 0
3. How to compute the value function? For example, the value of state $s_4$ as $V(s_4)$

Compute the Value of a Markov Reward Process

1. Value function: expected return from starting in state s
   $$V(s)=E[G_t|s_t=s]=E[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+{\gamma }^3{R}_{t+4}+...+{\gamma }^{T-t-1}{R}_T|s_t=s]$$

2. MRP value function satisfies the following Bellman equation:
   $$V(s)=\underset{Immediatereward}{R(s)}+\underset{Discountedsumoffuturereward}{\gamma \sum _{s\in S^{\prime }}P(s^{\prime }|s)V(s^{\prime })}$$

3. Practice: To derive the Bellman equation for V(s)

   1. Hint: $V(s)=E[R_{t+1}+\gamma E[R_{t+2}+{\gamma }^2{R}_{t+3}+...]|s_t=s]$

Understanding Bellman equation

1. Bellman equation describes the iterative relations of states
   $$V(s)=R(s)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s)V(s^{\prime })$$

Matrix Form of Bellman Equation for MRP

Therefore, we can express V(s) using the matrix form:

1. Analytic solution for value of MRP: $V=(I-\gamma P{)}^{-1}R$
   1. Matrix inverse takes the complexity $O(N^3)$ for N states
   2. Only possible for a small MRPs

Iterative Algorithm for Computing Value of a MRP

1. Iterative methods for large MRPs:
   1. Dynamic Programming
   2. Monte-Carlo evaluation
   3. Temporal-Difference learning

Monte Carlo Algorithm for Computing Value of a MRP

Algorithm 1 Monte Carlo simulation to calculate MRP value function

1. $i\leftarrow 0,G_t\leftarrow 0$

2. while $i\ne N$ do

3. ​ generate an episode, starting from state s and time t

4. ​ Using the generated episode, calculate return $g={\sum }_{i=t}^{H-1}{\gamma }^{i-t}{r}_i$

5. ​ $G_t\leftarrow G_t+g,i\leftarrow i+1$

6. end while

7. $V_t(s)\leftarrow G_t/N$

8. For example: to calculate $V(s_4)$ we can generate a lot of trajectories then take the average of the returns:

   1. return for $s_4,s_5,s_6,s_7$​ : $0+\frac{1}{2}\times 0+\frac{1}{4}\times 0+\frac{1}{8}\times 10=1.25$
   2. return for $s_4,s_3,s_2,s_1$ : $0+\frac{1}{2}\times 0+\frac{1}{4}\times 0+\frac{1}{8}\times 5=0.625$
   3. return for $s_4,s_5,s_6,s_6$ : = 0
   4. more trajectories

Iterative Algorithm for Computing Value of a MRP

Algorithm 1 Iterative Algorithm to calculate MRP value function

1. for all states $s\in S,V^{\prime }(s)\leftarrow 0,V(s)\leftarrow \infty$
2. while $||V-V^{\prime }||>ϵ$ do
3. ​ $V\leftarrow V^{\prime }$
4. ​ For all states $s\in S,V^{\prime }(s)=R(s)+\gamma {\sum }_{s^{\prime }\in S}^{}P(s^{\prime }|s)V(s^{\prime })$
5. end while
6. return $V^{\prime }(s)$ for all $s\in S$

Markov Decision Process (MDP)

1. Markov Decision Process is Markov Reward Process with decisions.

2. Definition of MDP

   1. S is a finite set of states

   2. A is a finite set of actions

   3. $P^a$ is dynamics/transition model for each action

      $P(s_{t+1}=s^{\prime }|s_t=s,a_t=a)$

   4. R is a reward function $R(s_t=s,a_t=a)=E[r_t|s_t=s,a_t=a]$

   5. Discount factor $\gamma \in [0,1]$

3. MDP is a turple: $(S,A,P,R,\gamma )$

Policy in MDP

1. Policy specifies what action to take in each state

2. Give a state, specify a distribution over actions

3. Policy: $\pi (a|s)=P(a_t=a|s_t=s)$

4. Policies are stationary (time-independent), $A_t\sim \pi (a|s)$ for any t > 0

5. Given an MDP $(S,A,P,R,\gamma )$ and a policy $\pi$

6. The state sequence $S_1,S_2,...$ is a Markov process $(S,P^{\pi })$

7. The state and reward sequence $S_1,R_1,S_2,R_2,...$ is a Markov reward process $(S,P^{\pi },R^{\pi },\gamma )$ where,
   $$P^{\pi }(s^{\prime }|s)=\sum _{a\in A}\pi (a|s)P(s^{\prime }|s,a)$$

$$R^{\pi }(s)=\sum _{a\in A}\pi (a|s)P(s,a)$$

Comparison of MP/MRP and MDP

Value function for MDP

1. The state-value function $v^{\pi }(s)$ of an MDP is the expected return starting from state s, and following policy $\pi$
   $$v^{\pi }(s)=E\pi [G_t|s_t=s]$$

2. The action-value function $q^{\pi }(s,a)$ is the expected return starting from state s, taking action a, and following policy $\pi$
   $$q^{\pi }(s,a)=E\pi [G_t|s_t=s,A_t=a]$$

3. We have the relation between $v^{\pi }(s)$ and $q^{\pi }(s,a)$
   $$v^{\pi }(s)=\sum _{a\in A}\pi (a|s)q^{\pi }(s,a)$$

Bellman Expection Equation

1. The state-value function can be decomposed into immediate reward plus discounted value of successor state,
   $$v^{\pi }(s)=E_{\pi }[R_{t+1}+\gamma v^{\pi }(s_{t+1})|s_t=s]$$

2. The action-value function can similarly be decomposed
   $$q^{\pi }(s,a)=E_{\pi }[R_{t+1}+\gamma q^{\pi }(s_{t+1},A_{t+1})|s_t=s,A_t=a]$$

Bellman Expection Equation for $V^{\pi }$ and $Q^{\pi }$

$$v^{\pi }(s)=\sum _{a\in A}\pi (a|s)q^{\pi }(s,a)$$

$$q^{\pi }(s,a)=R_s^a+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)v^{\pi }(s^{\prime })$$

Thus
$$v^{\pi }(s)=\sum _{a\in A}\pi (a|s)(R(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)v^{\pi }(s^{\prime }))$$

$$q^{\pi }(s,a)=R(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)\sum _{a^{\prime }\in A}\pi (a^{\prime }|s^{\prime })q^{\pi }(s^{\prime },a^{\prime })$$

Backup Diagram for $V^{\pi }$

$$v^{\pi }(s)=\sum _{a\in A}\pi (a|s)(R(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)v^{\pi }(s^{\prime }))$$

Backup Diagram for $Q^{\pi }$

$$q^{\pi }(s,a)=R(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)\sum _{a^{\prime }\in A}\pi (a^{\prime }|s^{\prime })q^{\pi }(s^{\prime },a^{\prime })$$

Policy Evaluation

1. Evaluate the value of state given a policy $\pi$: compute $v^{\pi }(s)$
2. Also called as (value) prediction

Example: Navigate the boat

Example: Policy Evaluation

1. Two actions: Left and Right

2. For all actions, reward: +5 in $s_1$, +10 in $s_7$, 0 in all other states. So that we can represent R = [5, 0, 0, 0, 0, 0, 10]

3. Let’s have a deterministic policy $\pi (s)=$Left and $\gamma =0$ for any state s, then what is the value of the policy?

   1. $v^{\pi }=[5,0,0,0,0,0,10]$

4. Iteration: $v_k^{\pi }(s)=r(s,\pi (s))+\gamma {\sum }_{s^{\prime }\in S}P(s^{\prime }|s,\pi (s))v_{k-1}^{\pi }(s^{\prime })$

5. $R=[5,0,0,0,0,0,10]$

6. Practice 1: Deterministic policy $\pi (s)=$Left and $\gamma =0.5$ for any state s, then what are the states values under the policy?

7. Practice 2: Stochastic policy $P(\pi (s)=Left)=0.5$ and $P(\pi (s)=Right)=0.5$ and $\gamma =0.5$ for any state s, then what are the states values under the policy?

8. Iteration: $v_k^{\pi }(s)=r(s,\pi (s))+\gamma {\sum }_{s^{\prime }\in S}P(s^{\prime }|s,\pi (s))v_{k-1}^{\pi }(s^{\prime })$

2.2 马尔可夫决策过程（下）

Decison Making in Markov Decision Process（MDP）

1. Prediction (evaluate a given policy):
   1. Input: MDP $<S,A,P,R,\gamma >$ and policy $\pi$ or MRP $<S,P^{\pi },R^{\pi },\gamma >$
   2. Output: value function $v^{\pi }$
2. Control (search the optimal policy):
   1. Input: MDP $<S,A,P,R,\gamma >$
   2. Output: optimal value function $v^{\ast }$ and optimal policy ${\pi }^{\ast }$
3. Prediction and control in MDP can be solved by dynamic programming.

Dynamic programming

Dynamic programming is a very general solution method for problems which have two properties:

1. Optimal substructure
   1. Principle of optimality applies
   2. Optimal solution can be decomposed into subproblems
2. Overlapping subproblems
   1. Subproblems recur many times
   2. Solutions can be cached and reused

Markov decision processes satisfy both properties

1. Bellman equation gives recursive decomposition
2. Value function stores and reuses solutions

Policy evaluation on MDP

1. Objective: Evaluate a given policy $\pi$ for a MDP

2. Output: the value function under policy $v^{\pi }$

3. Solution: iteration on Bellman expectation backup

4. Algorithm: Synchronous backup

   1. At each iteration t+1

      update $v_{t+1}(s)$ from $v_t(s^{\prime })$ for all states $s\in S$ where s’ is a successor state of s
      $$v_{t+1}(s)=\sum _{a\in A}\pi (a|s)(R(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)v_t(s^{\prime }))$$

5. Convergence: $v_1\to v_2\to ...\to v^{\pi }$

Policy evaluation: Iteration on Bellman expectation backup

Bellman expectation backup for a particular policy
$$v_{t+1}(s)=\sum _{a\in A}\pi (a|s)(R(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)v_t(s^{\prime }))$$
Or if in the form of MRP $<S,P^{\pi },R^{\pi },\gamma >$
$$v_{t+1}(s)=R^{\pi }(s)+\gamma P^{\pi }(s^{\prime }|s)V_t(s^{\prime })$$

Evaluating a Random Policy in the Small Gridworld

Example 4.1 in the Sutton RL textbook

1. Undiscounted episodic MDP ($\gamma =1$)
2. Nonterminal states 1, …, 14
3. Two terminal states (two shaded squares)
4. Action leading out of grid leaves state unchanged, $P(7|7,right)=1$
5. Reward is -1 until the terminal state is reach
6. Transition is deterministic given the action, e.g., $P(6|5,right)=1$
7. Uniform random policy $\pi (l|.)=\pi (r|.)=\pi (u|.)=\pi (d|.)=0.25$

A live demo on policy evaluation

$$v^{\pi }(s)=\sum _{a\in A}\pi (a|s)(R(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)v^{\pi }(s^{\prime }))$$

1. https://cs.stanford.edu/people/karpathy/reinforcejs/gridworld_dp.html

Optimal Value Function

1. The optimal state-value function $v^{\ast }(s)$ is the maximum value function over all policies
   $$v^{\ast }(s)=\underset{\pi }{\max }{v}^{\pi }(s)$$

2. The optimal policy
   $${\pi }^{\ast }(s)=arg\underset{\pi }{\max }{v}^{\pi }(s)$$

3. An MDP is “solved” when we know the optimal value

4. There exists a unique optimal value function, but could be multiple optimal policies (two actions that have the same optimal value function)

Finding Optimal Policy

1. An optimal policy can be found by maximizing over $q^{\ast }(s,a)$,
   $${\pi }^{\ast }(a|s)\{\begin{array}{ll}1,& \text{ if }a=argma{x}_{a\in A}{q}^{\ast }(s,a)\\ 0,& \text{ otherwise }\end{array}$$

2. There is always a deterministic optimal policy for any MDP

3. If we know $q^{\ast }(s,a)$, we immediately have the optimal policy

Policy Search

1. One option is to enumerate search the best policy
2. Number of deterministic policies is $|A{|}^{|S|}$
3. Other approaches such as policy iteration and value iteration are more efficient

MDP Control

1. Compute the optimal policy
   $${\pi }^{\ast }(s)=arg\underset{\pi }{\max }{v}^{\pi }(s)$$

2. Optimal policy for a MDP in an infinite horizon problem (agent acts forever) is

   1. Deterministic
   2. Stationary (does not depend on time step)
   3. Unique? Not necessarily, may have state-actions with identical optimal values

Improving a Policy through Policy Iteration

1. Iterate through the two steps:

   1. Evaluate the policy $\pi$ (computing v given current $\pi$)

   2. Improve the policy by acting greedily with respect to $v^{\pi }$
      $${\pi }^{\prime }=greedy(v^{\pi })$$

Policy Improvement

1. Compute the state-action value of a policy $\pi$:
   $$q^{{\pi }_i}(s,a)=R(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)v^{{\pi }_i}(s^{\prime })$$

2. Compute new policy ${\pi }_{i+1}$ for all $s\in S$ following
   $${\pi }_{i+1}(s)=arg\underset{a}{\max }{q}^{{\pi }_i}(s,a)$$

Monotonic Improvement in Policy

1. Consider a deterministic policy $a=\pi (s)$

2. We improve the policy through
   $${\pi }^{\prime }(s)=arg\underset{a}{\max }{q}^{\pi }(s,a)$$

3. This improves the value from any state s over one step,
   $$q^{\pi }(s,{\pi }^{\prime }(s))=\underset{a\in A}{\max }{q}^{\pi }(s,a)\ge q^{\pi }(s,\pi (s))=v^{\pi }(s)$$

4. It therefore improves the value function, $v_{{\pi }^{\prime }(s)}\ge v^{\pi }(s)$

   $v^{\pi }(s)\le q^{\pi }(s,{\pi }^{\prime }(s))=E_{{\pi }^{\prime }}[R_{t+1}+\gamma v^{\pi }(S_{t+1}|S_t=s)]$

   ​ $\le E_{{\pi }^{\prime }}[R_{t+1}+\gamma q^{\pi }(S_{t+1},{\pi }^{\prime }(S_{t+1}))|S_t=s)]$

   ​ $\le E_{{\pi }^{\prime }}[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{q}^{\pi }(S_{t+2},{\pi }^{\prime }(S_{t+2}))|S_t=s)]$

   ​ $\le E_{{\pi }^{\prime }}[R_{t+1}+\gamma R_{t+2}+...|S_t=s)]=v_{{\pi }^{\prime }}(s)$

5. If iImprovements stop,
   $$q^{\pi }(s,{\pi }^{\prime }(s))=\underset{a\in A}{\max }{q}^{\pi }(s,a)\ge q^{\pi }(s,\pi (s))=v^{\pi }(s)$$

6. Thus the Bellman optimality equation has been satisified

   $$v^{\pi }(s)=\underset{a\in A}{\max }{q}^{\pi }(s,a)$$

7. Therefore $v^{\pi }(s)=v^{\ast }(s)$ for all $s\in S$, so $\pi$ is an optimal policy

Bellman Optimality Equation

1️⃣The optimal value functions are reached by the Bellman optimality equations:
$$v^{\ast }(s)=\underset{a}{\max }{q}^{\pi }(s,a)$$

$$q^{\ast }(s,a)=R(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)v^{\ast }(s^{\prime })$$

thus
$$v^{\ast }(s)=\underset{a}{\max }R(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)v^{\ast }(s^{\prime })$$

$$q^{\ast }(s,a)=R(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)\underset{a^{\prime }}{\max }{q}^{\ast }(s^{\prime },a^{\prime })$$

Value Iteration by turning the Bellman Optimality Equation as update rule

1️⃣If we know the solution to subproblem $v^{\ast }(s^{\prime })$, which is optimal.

2️⃣Then the solution for the optimal $v^{\ast }(s)$ can be found by iteration over the following Bellman Optimality backup rule,
$$v(s)\leftarrow \underset{a\in A}{\max }(R(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)v^{\ast }(s^{\prime }))$$

3️⃣The idea of value iteration is to apply these updates iteratively

Algorithm of Value Iteration

1. Objective: find the optimal policy $\pi$

2. Solution: iteration on the Bellman optimality backup

3. Value Iteration algorithm:

   1. initialize $k=1$ and $v_0(s)=0$ for all states s

   2. For $k=1:H$

      1. for each states s
         $$q_{k+1}(s,a)=R(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)v_k(s^{\prime })$$

         $$v_{k+1}(s)=\underset{a}{\max }{q}_{k+1}(s,a)$$

      2. $k\leftarrow k+1$

   3. To retrieve the optimal policy after value iteration:
      $$\pi (s)=arg\underset{a}{\max }R(s,a)+\gamma \sum _{s^{\prime }\in S}P(s^{\prime }|s,a)v_{k+1}(s^{\prime })$$

Example: Shortest Path

After the optimal values are reached, we run policy extraction to retrieve the optimal policy.

Demo of policy iteration and value Iteration

1️⃣Policy iteration: Iteration of policy evaluation and policy improvement(update)

2️⃣Value iteration

3️⃣https://cs.stanford.edu/people/karpathy/reinforcejs/gridworld_dp.html

Policy iteration and value iteration on FrozenLake

1️⃣https://github.com/cuhkrlcourse/RLexample/tree/master/MDP

Different between Policy Iteration and Value Iteration

1️⃣Policy iteration includes: policy evaluation + policy improvement, and the two are repeated iteratively until policy converges.

2️⃣Value Iteration includes: finding optimal value function + one policy extraction.There is no repeat of the two because once the value function is optimal, then the policy out of it should also be optimal (i.e. converged).

3️⃣Finding optimal value function can also be seen as a combination of policy improvement (due to max) and truncated policy evaluation (the reassignment of v(s) after just one sweep of all states regardless of convergence).

Summary for Prediction and Control in MDP

End

1️⃣Optional Homework 1 is available at https://github.com/cuhkrlcourse/ierg6130-assignment