[笔记] Convex Optimization 2015.11.25

∥y∥∗=sup{xTy:∥x∥≤1}⟹xTy≤∥x∥⋅∥y∥∗
(because xT∥x∥y≤∥y∥∗ )
Want inequality of type: xTy≤f(x)+"f∗(y)" for “general” f (Fenchel’s Inequality)

• Definition: For f:Rn→R , the conjugate f∗ of f is defined by f∗(y)=supx(xTy−f(x))
  with domf∗= set of y ’s for which sup is <∞ .

• Example:

  1. f(x)=aTx+b(x∈Rn)
     f∗(y)=supxxTy−aTx−b={∞−bif y≠aif y=a
  2. f(x)=−logx(x>0)
     (xy+logx)′=y+1x=0⟹x=−1y
     f∗(y)=supx>0xTy+logx={∞−log(−y)−1if y≥0if y<0
  3. f(x)=ex(x∈R)
     (xy−ex)′=y−ex=0⟹x=logy
     f∗(y)=supxxTy−ex={∞ylogy−yif y<0if y≥0
  4. f(x)=xlogx(x≥0)
     (xy−xlogx)′=y−logx−1=0⟹x=ey−1
     f∗(y)=supx≥0xTy−xlogx=yey−1−(y−1)ey−1=ey−1
  5. f(x)=12xTQx with Q∈Sn++
     f∗(y)=supxxTy−12xTQx=yTQ−1y−12yTQ−1y=12yTQ−1y
     ( infxxTAx+xTb⟹bestx=−12A−1b )
     So x=Q−1y
     ⟹xTy≤12xTQx+12yTQ−1y , for all Q≻0
  6. f(x)=log(∑ni=1exi)
     f∗(y)=supxxTy−log(∑ni=1exi)
     (xy−log(∑ni=1exi))′=y−exi∑ni=1exi=0
     ⟹yi=exi∑ni=1exi,y⪰0,1Ty=1
     assume for simplicity, y≻0
     put xi=log(yi) , then ∑exi=1Ty=1 and optimality conditions hold
     then f∗(y)=∑ni=1yilog(yi)−log(1Ty)=∑ni=1yilog(yi)
  7. f(x)=∥x∥
     f∗(y)=supxxTy−∥x∥={0∞if ∥y∥∗≤1if ∥y∥∗>1
     xTy−∥x∥≤∥x∥⋅∥y∥∗−∥x∥=∥x∥(∥y∥∗−1)≤0 if ∥y∥∗−1≤0
  8. f(x)=12∥x∥2
     f∗(y)=supxxTy−12∥x∥2=12∥y∥2∗
     xTy−12∥x∥2≤∥x∥⋅∥y∥∗−12∥x∥2≤12∥y∥2∗ ( ∥x∥=∥y∥∗ )
     ⟹xTy≤12∥x∥2+12∥y∥2∗

• Proof of general hyperplane seperation:
  Let C⊆Rn be a convex set, H⊆R be the affine subspace of smallest dimention containing C , we write Cε={x:Bε(x)⋂H⊆C}
  then Cε⊆"relint(C)"=⋃ε>0Cε . (relint: relative interior)
  ( C⊆relint(C)¯¯¯¯¯¯¯¯¯¯¯¯¯ , C is a subset of closure of relint(C) )
  Let C,D be disjoint convex sets. Then for every ε>0 the sets Aε=Cε¯¯¯¯⋂B1ε(0) , D¯¯¯ are closed disjoint convex sets with Cε¯¯¯¯⋂B1ε(0) bounded, and dist(Aε,D¯¯¯)≥ε>0 .
  So ∃Aε∈Rn , aε≠0 , bε∈R s.t. (aε,bε) define a seperating hyperplane for Aε,D¯¯¯ .
  aTεx≤bε∀x∈Aε , aTεx≥bε∀x∈D¯¯¯
  WLOG ∥aε∥=1
  The sequence (a⃗ 1n)∞n=1 is a sequence of unit vectors and so has a convergent subsequence, say WLOG convergent to a0∈Rn .
  can assume sequence b1n is bonded (or else one of the sets C,D is empty)
  and so also convergent to some value b0∈R .
  Want to show (a0,b0) is SH for C,D , i.e., that
  aT0x≤b0∀x∈C,aT0x≥b0∀x∈D
  (Assume C is not a point, proof like above; then assume D is not a point, switch C,D .
  If C,D are points, obious true.)

Log-convexity and log-concavity
- Definition: f:Rn→R>0 is log-convex (log-concave) if log(f) is convex (concave).
- Convexity:
log(f(θx+(1−θ)y))≤θlog(f(x))+(1−θ)log(f(y))=log(f(x)θf(y)1−θ)
⟺f(θx+(1−θ)y)≤f(x)θf(y)1−θ
- Remark 2: log-convex ⟹ convex, f(x)=elogf(x) , (composition function, QED)
concave ⟹ log-concave