[笔记] Convex Optimization 2015.11.04

Schur Complement: Consider \begin{bmatrix} A & B \\ B^T & C \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} u \\ v \end{bmatrix} with A invertible, C symmetric.
Have Ax + By = u \implies x = A^{-1} (u - By)
also B^Tx + Cy = v \implies \begin{align*} v = & B^T A^{-1} (u - By) + Cy \\ = & B^T A^{-1} u + (C - B^T A^{-1}B)y \\ = & B^T A^{-1} u + Sy \end{align*}
where S := C - B^T A^{-1}B is the Schur Complement of C in the matrix X := \begin{bmatrix} A & B \\ B^T & C \end{bmatrix}

• Corollary: If X is invertible, then S is invertible.
  And y = S^{-1}(v - B^T A^{-1} u) = S^{-1}v - S^{-1} B^T A^{-1} u
  also x = A^{-1} (u - B S^{-1} v + B S^{-1} B^T A^{-1} u) = (A^{-1} + A^{-1} B S^{-1} B^T A^{-1}) u - A^{-1} B S^{-1} v
  Since \begin{bmatrix} x \\ y \end{bmatrix} = X^{-1} \begin{bmatrix} u \\ v \end{bmatrix} this implies X^{-1} = \begin{bmatrix} A^{-1} + A^{-1} B S^{-1} B^T A^{-1} & -A^{-1} B S^{-1} \\ -S^{-1} B^T A^{-1} & S^{-1} \end{bmatrix}

Consider \inf _u \begin{bmatrix} u^T & v^T \end{bmatrix} \begin{bmatrix} A & B \\ B^T & C \end{bmatrix} \begin{bmatrix} u \\ v \end{bmatrix} = \inf _u u^T A u + 2v^T B u + v^T C v
for fixed v, A, C symmetric
gradient = 2Au + (2v^T B)^T, optimum achieved when Au = -B^T v.
Assume A^{-1} exists, optimum achieved at u = -A^{-1} B^T v
optimum value of the \inf: v^T B A^{-1} A A^{-1} B^T v - 2v^T B A^{-1} B^T v + v^T C v = -v^T B A^{-1} B^T v + v^T C v = v^T(C - B A^{-1} B^T)v = v^T S v.

• Lessons: If S \not\succeq 0, then X = \begin{bmatrix} A & B \\ B^T & C \end{bmatrix} \not\succeq 0.
  And S \succeq 0 \implies X \succeq 0.
  Conclusion: A \succ 0 \implies (X \succeq 0 \iff S \succeq 0)

• Example: Consider f(x, Y) = x^T Y^{-1} x \;\; dom \, f = \mathbb{R}^n \times S_{++}^n
  Claim: f is convex (meaning: \forall (x_1, Y_1), (x_2, Y_2) \in dom \, f and \forall \theta \in [0, 1], f(\theta x_1 + (1 - \theta) x_2, \theta Y_1 + (1 - \theta) Y_2) \le \theta f(x_1, Y_1) + (1 - \theta) f(x_2, Y_2)

• Proof: Show that epi(f) is a convex set.
  \begin{align*} epi(f) = & \{ (x, Y, t) \in \mathbb{R} \times S_{++}^n \times \mathbb{R} : f(x, Y) \le t \} \\ = & \{ (x, Y, t) : x^T Y^{-1} x \le t \} \\ = & \{ (x, Y, t) : t - x^T Y^{-1} x \ge 0 \} \\ = & \{ (x, Y, t) : \begin{bmatrix} Y & x \\ x^T & t \end{bmatrix} \succeq 0 \} & \text{by Schur Complement} \\ = & S_+^{n + 1} \text{ is convex} \end{align*}
  (General version of x^2 / y.)

Jensen’s Inequality:
1. If f is a convex function then f(\theta _1 x_1 + \cdots + \theta _n x_n) \le \theta _1 f(x_1) + \cdots \theta _n f(x_n) for all x_1, \cdots, x_n \in dom \, f, \theta _1 + \cdots + \theta _n = 1, \theta _1, \cdots, \theta _n \ge 0
2. f(E[x]) \le E[f(x)] for all row vector x s.t. x \in dom \, f with probability 1.

If x_1, x_2, \cdots \in C, then \sum_{i = 1}^n \theta _i x_i \in C for \theta _1 + \theta _ 2 + \cdots = 1.
Say x is a row vector s.t. x = x_i with probability \theta _i.
Let y_i = f(x_i), epi(f) = \{(x, t) : f(x) \le t\}
((\sum \theta _i x_i, \sum \theta_i y_i) \in epi(f) ?)
f(\sum \theta _i x_i) \le \sum \theta_i f(x_i)

• Example: Let a, b \ge 0, then \sqrt{ab} \ge \frac{a + b}{2}.

• Proof: By f(x) = x^2 is convex,
  \begin{align*} f(E[x]) = & f(\frac{\sqrt{a} + \sqrt{b}}{2}) \\ = & \frac{a + b + 2\sqrt{ab}}{4} \\ \le & E[f(x)] \\ = & \frac{f(\sqrt{a}) + f(\sqrt{b})}{2} \\ = & \frac{a + b}{2} \end{align*}

• Example: Let x_1, \cdots, x_n \gt 0, and \theta _1, \cdots, \theta _n “the used”,
  then x_1^{\theta _1} x_2^{\theta _2} \cdots x_n^{\theta _n} \le \theta _1 x_1 + \theta _2 x_2 + \cdots + \theta _n x_n

• Example: Let a, b \gt 0, p, q \gt 1, \frac{1}{p} + \frac{1}{q} = 1, a^{\frac{1}{p}} + b^{\frac{1}{q}} \le \frac{1}{p} a + \frac{1}{q} b

Hölder’s inequality: x^T y \le \lVert x \rVert _p \cdot \lVert y \rVert _q for \frac{1}{p} + \frac{1}{q} = 1, p, q \gt 1.
1. can normalize: \lVert x \rVert _p = 1, \lVert y \rVert _q = 1.
2. x^T y = \sum_{i = 1}^n x_i y_i = \sum_{i = 1}^n (x_i^p)^{\frac{1}{p}} (y_i^q)^{\frac{1}{q}} \le \sum_{i = 1}^n \left( \frac{1}{p} x_i^p + \frac{1}{q} y_i^q \right) = 1 = \lVert x \rVert _p \cdot \lVert y \rVert _q

a^\theta b^{1 - \theta} \le \theta a + (1 - \theta) b
\begin{align*} xy \le & \theta x^{\frac{1}{\theta}} + (1 - \theta) y^{\frac{1}{1 - \theta}} \\ = & \int _0^x t^{\frac{1}{\theta} - 1} dt + \int _0^y t^{\frac{1}{1 - \theta} - 1} dt \\ = & \int _0^x t^{\frac{1 - \theta}{\theta}} dt + \int _0^y t^{\frac{\theta}{1 - \theta}} dt \end{align*}

Let \mathcal{F} be some collection of convex functions.
Let g(x) = \sup _{f \in \mathcal{F}} f(x).
g(\theta x + (1 - \theta) y) = \sup _{f \in \mathcal{F}} f(\theta x + (1 - \theta) y) \le \sup _{f \in \mathcal{F}} \theta f(x) + (1 - \theta) f(y)
g(f(\theta x + (1 - \theta) y)) \le g(\theta f(x) + (1 - \theta) f(y)) \le \theta g(f(x)) + (1 - \theta) g(f(y))