机器学习-白板推导 P5_2 (降维基础知识)

降维基础知识

$$X={\left[\begin{array}{cccc}{x}_1& x_2& ...& x_N\end{array}\right]}^T=\left[\begin{array}{c}{x}_1^T\\ x_2^T\\ ⋮\\ x_N^T\end{array}\right]={\left[\begin{array}{cccc}{x}_{11}& x_{12}& ...& x_{1p}\\ x_{21}& x_{22}& ...& x_{2p}\\ ⋮& ⋮& \ddots & ⋮\\ x_{N1}& x_{N2}& ...& x_{Np}\end{array}\right]}_{N\ast p}$$
$$1_N=\left[\begin{array}{c}1\\ 1\\ ⋮\\ 1\end{array}\right]$$
$x_i\in R^p,y_i\in R,i=1,2...N$

$SampleMean:{\overline{X}}_{p\ast 1}=\frac{1}{N}{\sum }_{i=1}^N{x}_i$

$SampleCovariance:S_{p\ast p}=\frac{1}{N}{\sum }_{i=1}^N(x_i-\overline{X})(x_i-\overline{X}{)}^T$

$$\overline{X}=\frac{1}{N}\sum _{i=1}^N{x}_i=\frac{1}{N}\left[\begin{array}{cccc}{x}_1& x_2& ...& x_N\end{array}\right]\left[\begin{array}{c}1\\ 1\\ ⋮\\ 1\end{array}\right]=\frac{1}{N}{X}^T{1}_N$$
$$\begin{array}{rl}S& =\frac{1}{N}\sum _{i=1}^N(x_i-\overline{X})(x_i-\overline{X}{)}^T\\ & =\frac{1}{N}\left[\begin{array}{cccc}{x}_1-\overline{X}& x_2-\overline{X}& ...& x_N-\overline{X}\end{array}\right]\left[\begin{array}{c}(x_1-\overline{X}{)}^T\\ (x_2-\overline{X}{)}^T\\ ⋮\\ (x_N-\overline{X}{)}^T\end{array}\right]\\ & =\frac{1}{N}(X^T-\overline{X}{1_N}^T)(X^T-\overline{X}{1_N}^T{)}^T\\ & =\frac{1}{N}(X^T-\frac{1}{N}{X}^T{1}_N{1_N}^T)(X^T-\frac{1}{N}{X}^T{1}_N{1_N}^T{)}^T\\ & =\frac{1}{N}{X}^T(I_N-\frac{1}{N}{1}_N{1_N}^T)(I_N-\frac{1}{N}{1}_N{1_N}^T{)}^{T}X\end{array}$$
定义:$H_N=I_N-\frac{1}{N}{1}_N{1_N}^T\to centeringmatrix$

$H_N$每个样本减去均值，会使图像向中心移动
$$\begin{array}{rl}S& =\frac{1}{N}{X}^{T}H{H}^{T}X\end{array}$$

$H_N=I_N-\frac{1}{N}{1}_N{1_N}^T$

$H_N^T=I_N-\frac{1}{N}{1}_N{1_N}^T=H_N$

$H^2=H.H=(I_N-\frac{1}{N}{1}_N{1_N}^T)(I_N-\frac{1}{N}{1}_N{1_N}^T)=I_N-\frac{2}{N}{1}_N{1_N}^T+\frac{1}{N^2}{1}_N{1_N}^T{1}_N{1_N}^T=I_N-\frac{1}{N}{1}_N{1_N}^T=H_N$

$H^N=H$

所以：
$$\begin{array}{rl}S& =\frac{1}{N}{X}^{T}H{H}^{T}X=\frac{1}{N}{X}^{T}HX\end{array}$$