【Machine Learning】Generalization Theory

本笔记基于清华大学《机器学习》的课程讲义中泛化理论相关部分，基本为笔者在考试前一两天所作的Cheat Sheet。内容较多，并不详细，主要作为复习和记忆的资料。

No free lunch

• For algroithm $A^{\prime }$, exsits $f$ that is perfect answer of $D\in C\times \{0,1\}$, such that $L_D(f)=0$ and

$$E_{S\sim D^m}[L_D(A^{\prime }(S))]\ge \frac{1}{4}$$

• Then $\mathrm{Pr}[L_D(A^{\prime }(S))\ge \frac{1}{8}]\ge \frac{1}{7}$

• Proof:
  $$\begin{array}{rl}\underset{i}{\max }{E}_{S\sim D_i^m}[L_D(A^{\prime }(S))]& =\underset{i}{\max }\frac{1}{k}\sum _{i=1}^k{L}_{D_i}(A^{\prime }(S_i))\\ & \ge \frac{1}{T}\sum _{j=1}^T\frac{1}{k}\sum _{i=1}^k{L}_{D_j}(A^{\prime }(S_i))\\ & \ge \frac{1}{k}\sum _{i=1}^k\frac{1}{T}\sum _{j=1}^T{L}_{D_j}(A^{\prime }(S_i))\\ & \ge \underset{S}{\min }\frac{1}{T}\sum _{j=1}^T{L}_{D_j}(A^{\prime }(S))\\ & \ge \underset{S}{\min }\frac{1}{T}\sum _{j=1}^T\frac{1}{2m}\sum _{i=1}^p{1}_{A^{\prime }\text{ wrong at }{v}_i}\\ & \ge \underset{S}{\min }\frac{1}{T}\sum _{j=1}^T\frac{1}{2p}\sum _{i=1}^p{1}_{A^{\prime }\text{ wrong at }{v}_i}\\ & \ge \frac{1}{2}\underset{S}{\min }\frac{1}{T}\sum _{j=1}^T\underset{i}{\min }{1}_{A^{\prime }\text{ wrong at }{v}_i}\\ & \ge \frac{1}{4}\end{array}$$

  • The last inequality is beause divide $T$ into $2$ parts. One pair $f_i,f_{i^{\prime }}$only differs at $v_i$.

ERM

• With realizable assumption, the hypothesis class found by ERM is good enough with at least some samples

  • Consider the probability of bad samples $L_S(h_S)=L_S(h^{\ast })=0$ but $L_{D,f}(h_S)>ϵ$. Then we need $S$ to be the union(apply union bound) of misleading set $L_S(h_S)=0$, each sample has probability $\le 1-ϵ$. Then probability is $|H_B|(1-ϵ{)}^m$

• PAC learnable: As sample number $m\ge m(ϵ,\delta )$, w.p. $1-\delta$ we can find a $h$ such that $L_{D,f}(h)\le ϵ$.

  • Agnostic PAC learnable: $L_D(h)\le L_D(h^{\ast })+ϵ$

• VC dimension

Rademacher

• Generalization:
  $$L_D(h)-L_S(h)\le 2E_{S^{\prime }\sim D^m}R(l\circ H\circ S^{\prime })+c\sqrt{\frac{2\ln \frac{2}{\delta }}{m}}$$

• Massart Lemma:
  $$R(A)\le \underset{a\in A}{\max }|a-\bar{a}|\frac{\sqrt{2\log N}}{m}$$

• Contraction Lemma: If $\varphi$ is $\rho$-lipschitz, then
  $$R(\varphi \circ A)\le \rho R(A)$$