《统计学习方法》第7章习题答案

7.2

$$\begin{gathered} \min \frac{1}{2}{\parallel w\parallel }^2 \\ s.t.y_i(w\cdot x_i+b)\ge 1,i=1,2,\dots ,5 \end{gathered}$$

将$(x_1,y_1),\dots ,(x_5,y_5)$代入，得

$$\begin{gathered} \min \frac{1}{2}(w_1^2+w_2^2) \\ s.t.w_1+2w_2+b\ge 1① \\ 2w_1+3w_2+b\ge 1② \\ 3w_1+3w_2+b\ge 1③ \\ -2w_1-w_2-b\ge 1④ \\ -3w_1-2w_2-b\ge 1⑤ \end{gathered}$$

$$\begin{gathered} ①+④,得-w_1+w_2\ge 2 \\ ①+⑤,得-2w_2\ge 2 \\ ②+④,得2w_2\ge 2 \\ ②+⑤,得-w_1+w_2\ge 2 \\ ③+④,得w_1+2w_2\ge 2 \\ ③+⑤,得w_2\ge 2 \end{gathered}$$

由上述可知

$$\begin{gathered} -w_1+w_2\ge 2 \\ {w}_1\le -1 \\ {w}_2\ge 2 \end{gathered}$$

为使$w_1^2+w_1^2$最小，令$w_1=-1,w_2=2$

代入$①-⑤$得，$b=-2$

$y_1(w\cdot x_1+b)=y_3(w\cdot x_3+b)=y_5(w\cdot x_5+b)=1$

支持向量为$x_1=(1,2{)}^T,x_3=(3,3{)}^T,x_5=(3,2{)}^T$

最大间隔分离超平面为$-x_1+2x_2-2=0$

分类决策函数为$f(x)=sign(-x_1+2x_2-2)$

7.3

$$\begin{gathered} \underset{w,b,\xi }{\min }\frac{1}{2}{\parallel w\parallel }^2+C\sum _{i=1}^N{{\xi }_i}^2 \\ s.t.y_i(w\cdot x_i+b)\ge 1-{\xi }_i,i=1,2,\dots ,N \\ {\xi }_i\ge 0,i=1,2,\dots ,N \end{gathered}$$

对应的拉格朗日函数是$L(w,b,\xi ,\alpha ,\gamma )=\frac{1}{2}{\parallel w\parallel }^2+C{\sum }_{i=1}^N{{\xi }_i}^2+{\sum }_{i=1}^N{\alpha }_i(1-{\xi }_i-y_i(w\cdot x_i+b))-{\sum }_{i=1}^N{\gamma }_i{\xi }_i$

使用KKT条件得到

$$\begin{gathered} \frac{\partial L}{\partial w}=w-\sum _{i=1}^N{\alpha }_i{y}_i{x}_i=0 \\ \frac{\partial L}{\partial b}=-\sum _{i=1}^N{\alpha }_i{y}_i=0 \\ \frac{\partial L}{\partial {\xi }_i}=2C{\xi }_i-{\alpha }_i-{\gamma }_i=0 \end{gathered}$$

因此

$$\begin{gathered} w=\sum _{i=1}^N{\alpha }_i{y}_i{x}_i \\ \sum _{i=1}^N{\alpha }_i{y}_i=0 \\ 2C{\xi }_i={\alpha }_i+{\gamma }_i \end{gathered}$$

代入拉格朗日函数可得

$$\begin{gathered} \underset{w,b,\xi }{\min }L(w,b,\xi ,\alpha ,\gamma )=\frac{1}{2}{\parallel w\parallel }^2+C\sum _{i=1}^N{{\xi }_i}^2+\sum _{i=1}^N{\alpha }_i-\sum _{i=1}^N({\alpha }_i+{\gamma }_i){\xi }_i-\sum _{i=1}^N{\alpha }_i{y}_{i}w\cdot x_i-\sum _{i=1}^N{\alpha }_i{y}_{i}b \\ =-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i{\alpha }_j{y}_i{y}_j{x}_i{x}_j+\sum _{i=1}^N{\alpha }_i-\frac{1}{2}\sum _{i=1}^N({\alpha }_i+{\gamma }_i){\xi }_i \\ =-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i{\alpha }_j{y}_i{y}_j{x}_i{x}_j+\sum _{i=1}^N{\alpha }_i-\frac{1}{2}\sum _{i=1}^N({\alpha }_i+{\gamma }_i)\frac{{\alpha }_i+{\gamma }_i}{2C} \\ =-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i{\alpha }_j{y}_i{y}_j{x}_i{x}_j+\sum _{i=1}^N{\alpha }_i-\frac{1}{4C}\sum _{i=1}^N({\alpha }_i+{\gamma }_i{)}^2 \end{gathered}$$

对偶问题为
$$\begin{gathered} \underset{\alpha }{\max }W(\alpha )=-\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i{\alpha }_j{y}_i{y}_j{x}_i{x}_j+\sum _{i=1}^N{\alpha }_i-\frac{1}{4C}\sum _{i=1}^N({\alpha }_i+{\gamma }_i{)}^2 \\ s.t.\sum _{i=1}^N{\alpha }_i{y}_i=0 \\ {\alpha }_i\ge 0,{\gamma }_i\ge 0,i=1,2,\dots ,N \end{gathered}$$

7.4

对$p$进行数学归纳。

当$p=1$时，$K(x,z)=x\cdot z$， 则$\varphi (x)=x$

假设$p=k$时，$K(x,z)=(x\cdot z{)}^k={\varphi }_k(x)\cdot {\varphi }_k(z)$

当$p=k$时，$K(x,z)=(x\cdot z{)}^{k+1}=(x\cdot z{)}^k(x\cdot z)={\varphi }_k(x)\cdot {\varphi }_k(z)(x\cdot z)$

不妨设${\varphi }_k(x)=(f_1(x),f_2(x),\dots ,f_m(x){)}^T,x=(x_1,x_2,\dots ,x_n{)}^T$

则$$\begin{gathered} K(x,z)=(f_1(x)f_1(z)+f_2(x)f_2(z)+\dots +f_m(x)f_m(z))(x_1{z}_1+x_2{z}_2+\dots +x_n{z}_n) \\ =f_1(x)f_1(z)(x_1{z}_1+x_2{z}_2+\dots +x_n{z}_n)+f_2(x)f_2(z)(x_1{z}_1+x_2{z}_2+\dots +x_n{z}_n)+\dots +f_m(x)f_m(z)(x_1{z}_1+x_2{z}_2+\dots +x_n{z}_n) \\ =(f_1(x)x_1)(f_1(z)z_1)+(f_1(x)x_2)(f_1(z)z_2)+\dots +(f_1(x)x_n)(f_1(z)z_n)+(f_2(x)x_1)(f_2(z)z_1)+\dots \\ +(f_2(x)x_n)(f_2(z)z_n)+(f_m(x)x_1)(f_m(z)z_1)+\dots +(f_m(x)x_n)(f_m(z)z_n) \\ :={\varphi }_{k+1}(x)\cdot {\varphi }_{k+1}(z) \end{gathered}$$

其中${\varphi }_{k+1}(x)=(f_1(x)x_1,f_1(x)x_2,\dots ,f_1(x)x_n,f_2(x)x_1,\dots ,f_2(x)x_n,\dots ,f_m(x)x_1,\dots ,f_m(x)x_n{)}^T$

因此$K(x,z)=(x\cdot z{)}^p$是正定核。