人工智能教程 - 数学基础课程1.7 - 最优化方法4-7 最优化思路第二步核心,约束条件,KKT

最优化思路第二步核心

Determination of a search direction $d_k$:

• Basis of the methods: Taylor series of f(x) in a small vicinity surrounding point $x_k$:
  $f(x_k+\delta )=f(x_k)+{▽}^{T}f(x_k).\delta +\frac{1}{2}{\delta }^T.{▽}^T.{▽}^{2}f(x_k).\delta +O(||\delta |{|}^3)$
• steepest descent method: $d_k=-▽f(x_{k}1.i.e.,d_k=-g(x_k))$
• Newton method :$d_k=-H^{-1}(x_k)g(x_k)$

约束条件，KKT

Constraint Optimization: The Karush-Kuhn-Tucker(KKT) Conditions

很多情况下，什么也没有，无参考/有约束条件是没有意义的。

• The constrained optimization problem
  minimize f(x)
  subject to: $a_i(x)=0$ for i = 1,2, …p
  $c_j(x)\ge 0$ for j = 1,2, …p

• Feasible region{$x:a_i(x)=0$ for i = 1,2, …p
  ,and $c_j(x)\ge 0$ for j = 1,2, …p}

• A point x is said to be feasible if it is in the feasible region (inside or on the boundary). We say an
  inequality constraint $c_j(x)\ge 0$ is active at a feasible point x, if $c_j(x)=0$. And inequality constraint $c_j(x)\ge 0$ is said to be inactive at a feasible point x if $c_j(x)>0$
  图
  但求一维图的最小值是没有意义的，但是如果可以约束为[a,b]区间内的最小值，则有意义。

• KKT conditions (first-order necessary conditions for point $x^{\ast }$ to be a minimum point):
  If $x^{\ast }$ is a local minimizer of the constrained problem, then
  (a) $a_i(x^{\ast })=0$ for 1,2,…,p
  (b) $a_j(x^{\ast })=0$ for 1, 2,…, q
  (c ) there exist Lagrange multipliers${\lambda }_i^{\ast }$ for$1\le i\le p$ and ${\mu }_i^{\ast }$ for $1\le j\le q$ such that
  $▽f(x^{\ast })={\sum }_{i=1}^p{\lambda }_i^{\ast }▽a_i(x^{\ast })+{\sum }_{j=1}^q{\mu }_j^{\ast }▽c_i(x^{\ast })$
  (d) ${\mu }_j^{\ast }{c}_j(x^{\ast })=0$ for 1, 2,…,q
  (e) ${\mu }_j^{\ast }\ge 0$ for 1, 2,…, j μ j q
  Ex:
  
  $f(x_1,x_2)=-x_1+x_2$ 无意义
  加上一下三个约束条件，则有意义了：

1. $c_1(x)=x_1\ge 0$
2. $c_2(x)=x_2\ge 0$ (active)
3. $x_2=-x_1+1$
   $c_3(x)=-x_1-x_2+1\ge 0$ (active)

$x^{\ast }=\left[\begin{array}{c}1\\ 0\end{array}\right]$

$▽f=\left[\begin{array}{c}-1\\ 0\end{array}\right]=0\left[\begin{array}{c}1\\ 0\end{array}\right]+2\left[\begin{array}{c}0\\ 1\end{array}\right]+1\left[\begin{array}{c}-1\\ -1\end{array}\right]$