[CMU16-745] Lecture 6 Deterministic Optimal Control Introduction

Source: CMU 16-745 Study Notes, taught by Prof. Zac Manchester

Lecture 5 Optimization Part 3

Review

Constrained Optimization

• Augmented Lagrangian
• Merit Functions/Line Search

Deterministic Optimal Control Introduction

Deterministic Optimal Control

(1) Continuous-Time Formulation

$$\underset{x(t),u(t)}{\min }J(x(t),u(t))={\int }_{t_0}^{t_f}\ell (x(t),u(t))dt+{\ell }_F(x(t_f))\text{s.t.}\dot{x}(t)=f(x(t),u(t))$$

• State: $x(t)$
• Control input: $u(t)$
• Cost function: $J(x(t),u(t))$
• Stage cost: $\ell (t,x(t),u(t))$
• Terminal cost: ${\ell }_F(x(t_f))$
• Dynamics constraints: $f(x(t),u(t))$

This is an infinite-dimensional optimization problem in the following sense:
$$u(t)=\underset{N\to \infty }{\lim }{u}_{1:N}$$

Solutions are open-loop trajectories.
We will focus on the discrete-time setting which leads to tractable algorithms.

(2) Discrete-Time Formulation

$$\underset{x_{1:N},u_{1:N-1}}{\min }J(x_{1:N},u_{1:N-1})=\sum _{k=1}^{N-1}\ell (x_k,u_k)+{\ell }_F(x_N)\text{s.t.}{x}_{k+1}=f(x_k,u_k)$$

• Torque limits: $u_{\min }\le u_k\le u_{\max }$
• Obstacle/safety constraints: $c(x_k)\le 0$

This is a finite-dimensional optimization problem.

• Knot points: Samples $x_k$, $u_k$
• Continuous to Discrete: Use integration (e.g., Runge-Kutta)
• Discrete to Continuous: Use interpolation (e.g., cubic splines)

Pontryagin’s Minimum Principle

Also known as the maximum principle if you maximize a reward.
First-order necessary conditions for deterministic optimal control problems.
We can form the Lagrangian:
$$\mathcal{L}=\sum _{k=1}^{N-1}\left[\ell (x_k,u_k)+{\lambda }_{k+1}^{\top }(f(x_k,u_k)-x_{k+1})\right]+{\ell }_F(x_N)$$

The Hamiltonian:
$$\mathcal{H}(x,u,\lambda )=\ell (x,u)+{\lambda }^{\top }f(x,u)$$

Plugging into $\mathcal{L}$:
$$\mathcal{L}=\mathcal{H}(x_1,u_1,{\lambda }_2)+\left[\sum _{k=2}^{N-1}\mathcal{H}(x_k,u_k,{\lambda }_{k+1})-{\lambda }_k^{\top }{x}_k\right]+{\ell }_F(x_N)-{\lambda }_N^{\top }{x}_N$$

Taking derivatives with respect to $x_k$ and ${\lambda }_k$:

1. For ${\lambda }_k$:
   $$\frac{\partial \mathcal{L}}{\partial {\lambda }_k}=\frac{\partial \mathcal{H}}{\partial {\lambda }_k}-x_k=f(x_{k-1},u_{k-1})-x_k=0$$

2. For $x_k$:
   $$\frac{\partial \mathcal{L}}{\partial x_k}=\frac{\partial \mathcal{H}}{\partial x_k}-{\lambda }_k=\frac{\partial \ell }{\partial x_k}+{\lambda }_{k+1}^{\top }\frac{\partial f}{\partial x_k}-{\lambda }_k^{\top }=0$$

For the $N$-th state:
$$\frac{\partial \mathcal{L}}{\partial x_N}=\frac{\partial {\ell }_F}{\partial x_N}-{\lambda }_N^{\top }=0$$

For $u_k$:
$$u_k=\arg \underset{u}{\min }\mathcal{H}(x_k,u,{\lambda }_{k+1})\text{s.t.}u\in U$$

Summary

• $x_{k+1}={\nabla }_{\lambda }\mathcal{H}(x_k,u_k,{\lambda }_{k+1})=f(x_k,u_k)$
• ${\lambda }_k={\nabla }_x\mathcal{H}(x_k,u_k,{\lambda }_{k+1})={\nabla }_x\ell (x_k,u_k)+{\left(\frac{\partial f}{\partial x_k}\right)}^{\top }{\lambda }_{k+1}$
• $u_k=\arg {\min }_u\mathcal{H}(x_k,u,{\lambda }_{k+1})\text{s.t.}u\in U$
• ${\lambda }_N=\frac{\partial {\ell }_F}{\partial x_N}$

This is where the shooting method comes from.

Continuous-time version

$$\dot{x}(t)={\nabla }_{\lambda }H(x(t),u(t),\lambda (t))=f(x(t),u(t))$$$$-\dot{\lambda }(t)={\nabla }_{x}H(x(t),u(t),\lambda (t))={\nabla }_x\ell (x(t),u(t))+{\left(\frac{\partial f}{\partial x}\right)}^{\top }\lambda (t)$$$$u(t)=\arg \underset{u}{\min }H(x(t),u,\lambda (t))\text{s.t.}u\in U$$$$\lambda (t_f)=\frac{\partial {\ell }_F}{\partial x}(t_f)$$

Some Notes:

• Historically, many algorithms were based on integrating the continuous ODEs, forward/backward to do gradient descent on $u(t)$.
• These methods are called indirect methods or shooting methods.
• In continuous-time, $\lambda (t)$ is called the co-state trajectory.
• These methods have largely fallen out of favor as computers have improved.

References

[1] Lecture 6 确定性最优控制（Deterministic Optimal Control
[2] 【Optimal Control (CMU 16-745)】Lecture 6 Deterministic Optimal Control Introduction
[3] CMU Optimal Control 16-745 Video From Bilibili
[4] CMU Optimal Control 16-745

感谢大家阅读这篇学习笔记！记录学习的内容，如果有问题或者不准确的地方，欢迎随时联系我改正，与大家共同进步！