谱聚类的原理全网最详细的推导过程！！

谱聚类

谱聚类思想

​ 谱聚类的思想来源于图论，它把待聚类的数据集中的每一个样本看做是图中一个顶点，这些顶点连接在一起，连接的这些边上有权重，权重的大小表示这些样本之间的相似程度。同一类的顶点它们的相似程度很高，在图论中体现为同一类的顶点中连接它们的边的权重很大，不在同一类的顶点连接它们的边的权重很小。于是谱聚类的最终目标就是找到一种切割图的方法，使得切割之后的各个子图内的权重很大，子图之间的权重很小。

---------------------------------------------------------------------------------------------------------------------------------------

输入：样本集合 $X=\{x_1,x_2,...,x_N\}$、聚类数量K

输出：样本集合的聚类C*

优化问题：
m i n   { A k } k = 1 K N c u t ( V ) = m i n   { A k } k = 1 K ∑ k = 1 K W ( A k , A ‾ k ) ∑ i ∈ A k d i → { A k } k = 1 K = a r g m i n   { A k } k = 1 K N c u t ( V ) = a r g m i n   { A k } k = 1 K ∑ k = 1 K W ( A k , A ‾ k ) ∑ i ∈ A k d i → Y ^ = a r g   m i n Y ∑ k = 1 K y k T L y k y k T D y k 这里 y k 是 Y 的第 k 列 Y ∈ R N × K , b e   t h e   c l u s t e r   i n d i c a t o r   m a t r i x , i n   w h i c h   y i l = 1   i n d i c a t e s   t h a t   x i   i s   a s s i g n e d   t o   t h e   l t h   c l u s t e r . { y i ∈ { 0 , 1 } K ∑ j = 1 K y i j = 1           y i = [ y i 1 y i 2 . . . y i K ] \underset{\{A_k\}_{k=1}^K}{min\ }Ncut(V)=\underset{\{A_k\}_{k=1}^K}{min\ }\sum_{k=1}^K\frac{W(A_k,\overline A_k)}{\sum_{i∈A_k}d_i}\\ \\ →\{A_k\}_{k=1}^K=arg \underset{\{A_k\}_{k=1}^K}{min\ }Ncut(V)=arg\underset{\{A_k\}_{k=1}^K}{min\ }\sum_{k=1}^K\frac{W(A_k,\overline A_k)}{\sum_{i∈A_k}d_i} \\ \\ →\hat Y=arg\ \underset{Y}{min}\sum_{k=1}^K\frac{y_k^TLy_k}{y_k^TDy_k}\\ \\ 这里y_k是Y的第k列\\ \\ Y∈R^{N×K},be\ the\ cluster\ indicator\ matrix,\\ in\ which\ y_{il}=1\ indicates\ that\ x_i\ is\ assigned\ to\ the\ l_{th}\ cluster.\\ \begin{cases} y_i ∈\{0,1\}^K \\ \\ \sum_{j=1}^Ky_{ij}=1 \\ \end{cases}\ \ \ \ \ \ \ \ \ y_i= \begin{bmatrix} y_{i1}\\ y_{i2}\\ ... \\ y_{iK} \end{bmatrix} {Ak​}k=1K​min ​Ncut(V)={Ak​}k=1K​min ​k=1∑K​∑i∈Ak​​di​W(Ak​,Ak​)​→{Ak​}k=1K​=arg{Ak​}k=1K​min ​Ncut(V)=arg{Ak​}k=1K​min ​k=1∑K​∑i∈Ak​​di​W(Ak​,Ak​)​→Y^=arg Ymin​k=1∑K​ykT​Dyk​ykT​Lyk​​这里yk​是Y的第k列Y∈RN×K,be the cluster indicator matrix,in which yil​=1 indicates that xi​ is assigned to the lth​ cluster.⎩ ⎨ ⎧​yi​∈{0,1}K∑j=1K​yij​=1​         yi​= ​yi1​yi2​...yiK​​ ​

$$\begin{gathered} \underset{Y}{\min }Tr(Y^{T}LY(Y^{T}DY{)}^{-1}) \\ \hat{Y}=arg\underset{Y}{\min }Tr(Y^{T}LY(Y^{T}DY{)}^{-1}) \end{gathered}$$

这里L=D-W是拉普拉斯矩阵

---------------------------------------------------------------------------------------------------------------------------------------

推导过程

符号说明：Graph-based（带权重的无向图）

样本数据：$X=(x_1,...,x_N{)}^{\top }$
无向图： $G=\{V,E\}$
顶点集： $V=\{1,2,...,N\}\iff X$
边集： $E:similaritymatrix(affimtymatrix)$

权重矩阵：W
W = [ w 11 w 12 . . . w 1 N w 21 w 22 . . . w 2 N . . . . . . . . . . . . w N 1 w N 2 . . . w N N ] = [ w i j ] , 1 ≤ i , j ≤ N 其中 w i j = { K ( x i , x j ) = e x p { − ∣ ∣ x i − x j ∣ ∣ 2 2 2 θ 2 } if  ( i , j ) ∈ E 0 if  ( i , j ) ∉ E W= \begin{bmatrix} w_{11} & w_{12} & ... & w_{1N} \\ w_{21} & w_{22} & ... & w_{2N}\\ ... & ... & ... & ...\\ w_{N1} & w_{N2} & ... & w_{NN} \end{bmatrix} =[w_{ij}],1≤i,j≤N\\ \\ 其中w_{ij}= \begin{cases} K(x_i,x_j)=exp\{-\frac{||x_i-x_j||_2^2}{2\theta ^2} \} & \text{if } (i,j)∈E \\ \\ 0 & \text{if } (i,j)∉E \\ \end{cases}\\ \\ W= ​w11​w21​...wN1​​w12​w22​...wN2​​............​w1N​w2N​...wNN​​ ​=[wij​],1≤i,j≤N其中wij​=⎩ ⎨ ⎧​K(xi​,xj​)=exp{−2θ2∣∣xi​−xj​∣∣22​​}0​if (i,j)∈Eif (i,j)∈/E​

顶点i的度: $d_i={\sum }_{j=1}^N{w}_{ij}$

度矩阵:$D=diag(W\cdot 1_N)$
D = d i a g ( W ⋅ 1 N ) = [ d 1 0 . . . 0 0 d 2 . . . 0 . . . . . . . . . . . . 0 0 . . . d N ] = [ ∑ j = 1 N w 1 j 0 . . . 0 0 ∑ j = 1 N w 2 j . . . 0 . . . . . . . . . . . . 0 0 . . . ∑ j = 1 N w N j ] D=diag(W⋅\mathbf{1}_N)= \begin{bmatrix} d_1 & 0 & ... & 0 \\ 0 & d_2 & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & d_N \end{bmatrix}= \begin{bmatrix} \sum_{j=1}^Nw_{1j} & 0 & ... & 0 \\ 0 & \sum_{j=1}^Nw_{2j} & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & \sum_{j=1}^Nw_{Nj} \end{bmatrix}\\ \\ D=diag(W⋅1N​)= ​d1​0...0​0d2​...0​............​00...dN​​ ​= ​∑j=1N​w1j​0...0​0∑j=1N​w2j​...0​............​00...∑j=1N​wNj​​ ​
Laplacian Matrix:$L=D-W$

相关定义：

​ 对于集合A和集合B,A和B是顶点集V的子集且A和B的交集为空,即:
$$\begin{gathered} A\subset V,B\subset V,A\cap B=\varnothing \\ \to W(A,B)=\sum _{i\in A,j\in B}{w}_{ij} \end{gathered}$$
这个就是说对于A和B两个类别，计算一个类的节点到另一个类的节点所有边的权重和。

​ 更严格的切图的定义如下：假如一共K个类别也就是将顶点集合分为K个子集合，即：

​ $V={\cup }_{k=1}^K{A}_k=A_1\cup A_2\cup A_3\cup ...\cup A_K$
$$Cut(V)=Cut(A_1,...,A_K)=\sum _{k=1}^{K}W(A_k,{\overline{A}}_k)=\sum _{k=1}^{K}W(A_k,V)-\sum _{k=1}^{K}W(A_k,A_k)$$
我们的目标是$\underset{\{A_k{\}}_{k=1}^K}{\min }Cut(V)$（切割之后的各个子图内的权重很大，子图之间的权重很小。），但是直接拿Cut作为目标函数会有问题，如下图我们选择一个权重最小的边缘的点，比如C和H之间进行cut，这样可以最小化$Cut(A_1,A_2,...A_k)$, 但是却不是最优的切图。

对Cut做一个normalize。

Ncut的定义：

除以度来做一个Normalize，度用$degree(A_k)$表示。
$$\begin{gathered} cut(V)=\sum _{k=1}^{K}W(A_k,{\overline{A}}_k) \\ \to Ncut=\sum _{k=1}^K\frac{W(A_k,{\overline{A}}_k)}{\Delta } \\ \Delta =degree(A_k)=\sum _{i\in A_k}{d}_i{d}_i=\sum _{j=1}^N{w}_{ij} \\ \to Ncut=\sum _{k=1}^K\frac{W(A_k,{\overline{A}}_k)}{{\sum }_{i\in A_k}{d}_i}{d}_i=\sum _{j=1}^N{w}_{ij} \\ =\sum _{k=1}^K\frac{W(A_k,V)-W(A_k,A_k)}{{\sum }_{i\in A_k}{d}_i} \\ =\sum _{k=1}^K\frac{W(A_k,V)-W(A_k,A_k)}{{\sum }_{i\in A_k}{\sum }_{j=1}^N{w}_{ij}} \end{gathered}$$

优化目标：
$$\underset{\{A_k{\}}_{k=1}^K}{\min }Ncut(V)$$

Model
$$\begin{gathered} \underset{\{A_k{\}}_{k=1}^K}{\min }Ncut(V)=\underset{\{A_k{\}}_{k=1}^K}{\min }\sum _{k=1}^K\frac{W(A_k,{\overline{A}}_k)}{{\sum }_{i\in A_k}{d}_i} \\ \to \{A_k{\}}_{k=1}^K=arg\underset{\{A_k{\}}_{k=1}^K}{\min }Ncut(V)=arg\underset{\{A_k{\}}_{k=1}^K}{\min }\sum _{k=1}^K\frac{W(A_k,{\overline{A}}_k)}{{\sum }_{i\in A_k}{d}_i} \end{gathered}$$
引入指示向量indicator vector：
{ y i ∈ { 0 , 1 } K ∑ j = 1 K y i j = 1           y i = [ y i 1 y i 2 . . . y i K ]          y i j = 1 ⇔ 第  i 个样本属于第  j 个类别 \begin{cases} y_i ∈\{0,1\}^K \\ \\ \sum_{j=1}^Ky_{ij}=1 \\ \end{cases}\ \ \ \ \ \ \ \ \ y_i= \begin{bmatrix} y_{i1}\\ y_{i2}\\ ... \\ y_{iK} \end{bmatrix} \ \ \ \ \ \ \ \ \\ y_{ij}=1⇔第\ i个样本属于第\ j个类别 ⎩ ⎨ ⎧​yi​∈{0,1}K∑j=1K​yij​=1​         yi​= ​yi1​yi2​...yiK​​ ​        yij​=1⇔第 i个样本属于第 j个类别

$$\begin{gathered} Y=[y_1,...y_K{]}_{N\times K}^{\top } \\ 将问题模型转换:\hat{Y}=arg\underset{\hat{Y}}{\min }Ncut(V) \end{gathered}$$

​ 目的是将问题模型转换: $\hat{Y}=arg\underset{\hat{Y}}{\min }Ncut(V)$

将Ncut转换成矩阵的形式：
N c u t = ∑ k = 1 K W ( A k , A ‾ k ) ∑ i ∈ A k d i = T r [ W ( A 1 , A ‾ 1 ) ∑ i ∈ A 1 d i 0 . . . 0 0 W ( A 2 , A ‾ 2 ) ∑ i ∈ A 2 d i . . . 0 . . . . . . . . . . . . 0 0 . . . W ( A K , A ‾ K ) ∑ i ∈ A K d i ] = T r [ W ( A 1 , A ‾ 1 ) 0 . . . 0 0 W ( A 2 , A ‾ 2 ) . . . 0 . . . . . . . . . . . . 0 0 . . . W ( A K , A ‾ K ) ] [ ∑ i ∈ A 1 d i 0 . . . 0 0 ∑ i ∈ A 2 d i . . . 0 . . . . . . . . . . . . 0 0 . . . ∑ i ∈ A K d i ] − 1 Ncut=\sum_{k=1}^K\frac{W(A_k,\overline A_k)}{\sum_{i∈A_k}d_i}\\ \\ \\ =Tr \begin{bmatrix} \frac{W(A_1,\overline A_1)}{\sum_{i∈A_1}d_i} & 0 & ... & 0 \\ 0 & \frac{W(A_2,\overline A_2)}{\sum_{i∈A_2}d_i} & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & \frac{W(A_K,\overline A_K)}{\sum_{i∈A_K}d_i} \end{bmatrix}\\ \\ \\ =Tr \begin{bmatrix} W(A_1,\overline A_1) & 0 & ... & 0 \\ 0 & W(A_2,\overline A_2) & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & W(A_K,\overline A_K) \end{bmatrix} \begin{bmatrix} \sum_{i∈A_1}d_i & 0 & ... & 0 \\ 0 & \sum_{i∈A_2}d_i & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & \sum_{i∈A_K}d_i \end{bmatrix}^{-1} Ncut=k=1∑K​∑i∈Ak​​di​W(Ak​,Ak​)​=Tr ​∑i∈A1​​di​W(A1​,A1​)​0...0​0∑i∈A2​​di​W(A2​,A2​)​...0​............​00...∑i∈AK​​di​W(AK​,AK​)​​ ​=Tr ​W(A1​,A1​)0...0​0W(A2​,A2​)...0​............​00...W(AK​,AK​)​ ​ ​∑i∈A1​​di​0...0​0∑i∈A2​​di​...0​............​00...∑i∈AK​​di​​ ​−1

记 O K × K = [ W ( A 1 , A ‾ 1 ) 0 . . . 0 0 W ( A 2 , A ‾ 2 ) . . . 0 . . . . . . . . . . . . 0 0 . . . W ( A K , A ‾ K ) ] P K × K = [ ∑ i ∈ A 1 d i 0 . . . 0 0 ∑ i ∈ A 2 d i . . . 0 . . . . . . . . . . . . 0 0 . . . ∑ i ∈ A K d i ] 记O_{K×K}= \begin{bmatrix} W(A_1,\overline A_1) & 0 & ... & 0 \\ 0 & W(A_2,\overline A_2) & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & W(A_K,\overline A_K) \end{bmatrix}\\ \\ \\ P_{K×K}= \begin{bmatrix} \sum_{i∈A_1}d_i & 0 & ... & 0 \\ 0 & \sum_{i∈A_2}d_i & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & \sum_{i∈A_K}d_i \end{bmatrix} 记OK×K​= ​W(A1​,A1​)0...0​0W(A2​,A2​)...0​............​00...W(AK​,AK​)​ ​PK×K​= ​∑i∈A1​​di​0...0​0∑i∈A2​​di​...0​............​00...∑i∈AK​​di​​ ​

现在问题转化为：$\underset{\{A_k{\}}_{k=1}^K}{\min }Ncut(V)=\underset{\{A_k{\}}_{k=1}^K}{\min }Tr(O{P}^{-1})$

已知W、Y，求O、P，我们要将O和P用Y和W表示：

先求解P：
Y ⊤ Y = [ y 1 , . . . y N ] [ y 1 T y 2 T . . . y N T ] Y^\top Y=[y_1,...y_N] \begin{bmatrix} y_{1}^T\\ y_{2}^T\\ ... \\ y_{N}^T \end{bmatrix} Y⊤Y=[y1​,...yN​] ​y1T​y2T​...yNT​​ ​
= ∑ i = 1 N y i y i T = [ N 1 0 . . . 0 0 N 2 . . . 0 . . . . . . . . . . . . 0 0 . . . N K ] K × K =\sum_{i=1}^Ny_iy_i^T= \begin{bmatrix} N_1 & 0 & ... & 0 \\ 0 & N_2 & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & N_K \end{bmatrix}_{K×K} =i=1∑N​yi​yiT​= ​N1​0...0​0N2​...0​............​00...NK​​ ​K×K​
= [ ∑ i ∈ A 1 1 0 . . . 0 0 ∑ i ∈ A 2 1 . . . 0 . . . . . . . . . . . . 0 0 . . . ∑ i ∈ A K 1 ] K × K =\begin{bmatrix} \sum_{i∈A_1}1 & 0 & ... & 0 \\ 0 & \sum_{i∈A_2}1 & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & \sum_{i∈A_K}1 \end{bmatrix}_{K×K} = ​∑i∈A1​​10...0​0∑i∈A2​​1...0​............​00...∑i∈AK​​1​ ​K×K​

$N_k$ 的含义：在N个样本中，属于类别k的样本个数。${\sum }_{k=1}^N{N}_k=N,N_k=|A_k|={\sum }_{i\in A_k}1$

∑ i = 1 N y i d i y i T = y 1 d 1 y 1 T + y 2 d 2 y 2 T . . . + y N d N y N T = Y T D Y P K × K = [ ∑ i ∈ A 1 d i 0 . . . 0 0 ∑ i ∈ A 2 d i . . . 0 . . . . . . . . . . . . 0 0 . . . ∑ i ∈ A K d i ] = Y T D Y 其中 , D = [ d 1 0 . . . 0 0 d 2 . . . 0 . . . . . . . . . . . . 0 0 . . . d N ] = d i a g ( W ⋅ 1 N ) = [ ∑ j = 1 N w 1 j 0 . . . 0 0 ∑ j = 1 N w 2 j . . . 0 . . . . . . . . . . . . 0 0 . . . ∑ j = 1 N w N j ] \sum_{i=1}^Ny_id_iy_i^T=y_1d_1y_1^T+y_2d_2y_2^T...+y_Nd_Ny_N^T=Y^TDY \\ \\ \\ P_{K×K}= \begin{bmatrix} \sum_{i∈A_1}d_i & 0 & ... & 0 \\ 0 & \sum_{i∈A_2}d_i & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & \sum_{i∈A_K}d_i \end{bmatrix}=Y^TDY\\ \\ \\ 其中,D= \begin{bmatrix} d_1 & 0 & ... & 0 \\ 0 & d_2 & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & d_N \end{bmatrix}=diag(W·\mathbf{1}_N)= \begin{bmatrix} \sum_{j=1}^Nw_{1j} & 0 & ... & 0 \\ 0 & \sum_{j=1}^Nw_{2j} & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & \sum_{j=1}^Nw_{Nj} \end{bmatrix}\\ \\ \\ i=1∑N​yi​di​yiT​=y1​d1​y1T​+y2​d2​y2T​...+yN​dN​yNT​=YTDYPK×K​= ​∑i∈A1​​di​0...0​0∑i∈A2​​di​...0​............​00...∑i∈AK​​di​​ ​=YTDY其中,D= ​d1​0...0​0d2​...0​............​00...dN​​ ​=diag(W⋅1N​)= ​∑j=1N​w1j​0...0​0∑j=1N​w2j​...0​............​00...∑j=1N​wNj​​ ​
所以我们求解的P为：

$$\begin{gathered} P=Y^{T}DY \\ 其中,D=diag(W\cdot 1_N) \end{gathered}$$
再求解O：

O K × K = [ W ( A 1 , A ‾ 1 ) 0 . . . 0 0 W ( A 2 , A ‾ 2 ) . . . 0 . . . . . . . . . . . . 0 0 . . . W ( A K , A ‾ K ) ] W ( A k , A k ‾ ) = W ( A k , V ) ⏟ ∑ i ∈ A k d i − W ( A k , A k ) ⏟ ∑ i ∈ A k ∑ j ∈ A k w i j → O = [ ∑ i ∈ A 1 d i 0 . . . 0 0 ∑ i ∈ A 2 d i . . . 0 . . . . . . . . . . . . 0 0 . . . ∑ i ∈ A K d i ] − [ W ( A 1 , A 1 ) 0 . . . 0 0 W ( A 2 , A 2 ) . . . 0 . . . . . . . . . . . . 0 0 . . . W ( A K , A K ) ] O_{K×K}= \begin{bmatrix} W(A_1,\overline A_1) & 0 & ... & 0 \\ 0 & W(A_2,\overline A_2) & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & W(A_K,\overline A_K) \end{bmatrix}\\ \\ \\ W(A_k,\overline{A_k})=\underbrace{W(A_k,V)}_{\sum_{i∈A_k}d_i}-\underbrace{W(A_k,A_k)}_{\sum_{i∈A_k}\sum_{j∈A_k}w_{ij}}\\ \\ \\ →O= \begin{bmatrix} \sum_{i∈A_1}d_i & 0 & ... & 0 \\ 0 & \sum_{i∈A_2}d_i & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & \sum_{i∈A_K}d_i \end{bmatrix}- \begin{bmatrix} W(A_1,A_1) & 0 & ... & 0 \\ 0 & W(A_2, A_2) & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & W(A_K, A_K) \end{bmatrix}\\ \\ \\ OK×K​= ​W(A1​,A1​)0...0​0W(A2​,A2​)...0​............​00...W(AK​,AK​)​ ​W(Ak​,Ak​​)=∑i∈Ak​​di​ W(Ak​,V)​​−∑i∈Ak​​∑j∈Ak​​wij​ W(Ak​,Ak​)​​→O= ​∑i∈A1​​di​0...0​0∑i∈A2​​di​...0​............​00...∑i∈AK​​di​​ ​− ​W(A1​,A1​)0...0​0W(A2​,A2​)...0​............​00...W(AK​,AK​)​ ​
前面的矩阵我们知道:$Y^{T}DY$, 再来看后面部分:

[ W ( A 1 , A 1 ) 0 . . . 0 0 W ( A 2 , A 2 ) . . . 0 . . . . . . . . . . . . 0 0 . . . W ( A K , A K ) ] \begin{bmatrix} W(A_1,A_1) & 0 & ... & 0 \\ 0 & W(A_2, A_2) & ... & 0\\ ... & ... & ... & ...\\ 0 & 0 & ... & W(A_K, A_K) \end{bmatrix} ​W(A1​,A1​)0...0​0W(A2​,A2​)...0​............​00...W(AK​,AK​)​ ​

= [ ∑ i ∈ A 1 ∑ j ∈ A 1 w i j 0 . . . 0 0 ∑ i ∈ A 2 ∑ j ∈ A 2 w i j . . . 0 . . . . . . . . . . . . 0 . . . . . . ∑ i ∈ A K ∑ j ∈ A K w i j ] =\begin{bmatrix} \sum_{i∈A_1}\sum_{j∈A_1}w_{ij} & 0 & ... & 0 \\ 0 & \sum_{i∈A_2}\sum_{j∈A_2}w_{ij} & ... & 0\\ ... & ... & ... & ...\\ 0 & ... & ...& \sum_{i∈A_K}\sum_{j∈A_K}w_{ij} \end{bmatrix} = ​∑i∈A1​​∑j∈A1​​wij​0...0​0∑i∈A2​​∑j∈A2​​wij​......​............​00...∑i∈AK​​∑j∈AK​​wij​​ ​

猜想后半部分是否等于$Y^{T}WY$ 验证一下：
Y T W Y 维度是 K × K 维 Y T W Y = [ y 1 , . . . y N ] [ w 11 w 12 . . . w 1 N w 21 w 22 . . . w 2 N . . . . . . . . . . . . w N 1 w N 2 . . . w N N ] [ y 1 T y 2 T . . . y N T ] = [ ∑ i = 1 N y i w i 1 , . . . , ∑ i = 1 N y i w N i ] [ y 1 T y 2 T . . . y N T ] = ∑ i = 1 N ∑ j = 1 N y i w i j y i T = ∑ i = 1 N ∑ j = 1 N y i y i T w i j = [ ∑ i ∈ A 1 ∑ j ∈ A 1 w i j ∑ i ∈ A 1 ∑ j ∈ A 2 w i j . . . ∑ i ∈ A 1 ∑ j ∈ A K w i j ∑ i ∈ A 2 ∑ j ∈ A 1 w i j ∑ i ∈ A 2 ∑ j ∈ A 2 w i j . . . ∑ i ∈ A 2 ∑ j ∈ A K w i j . . . . . . . . . . . . ∑ i ∈ A K ∑ j ∈ A 1 w i j ∑ i ∈ A K ∑ j ∈ A 2 w i j . . . ∑ i ∈ A K ∑ j ∈ A K w i j ] Y^TWY维度是K×K维\\ \\ \\ Y^TWY=[y_1,...y_N] \begin{bmatrix} w_{11} & w_{12} & ... & w_{1N} \\ w_{21} & w_{22} & ... & w_{2N}\\ ... & ... & ... & ...\\ w_{N1} & w_{N2} & ... & w_{NN} \end{bmatrix} \begin{bmatrix} y_{1}^T\\ y_{2}^T\\ ... \\ y_{N}^T \end{bmatrix}\\ \\ =[\sum_{i=1}^Ny_iw_{i1},...,\sum_{i=1}^Ny_iw_{Ni}] \begin{bmatrix} y_{1}^T\\ y_{2}^T\\ ... \\ y_{N}^T \end{bmatrix}\\ \\ =\sum_{i=1}^N\sum_{j=1}^Ny_iw_{ij}y_i^T=\sum_{i=1}^N\sum_{j=1}^Ny_iy_i^Tw_{ij}\\ \\ =\begin{bmatrix} \sum_{i∈A_1}\sum_{j∈A_1}w_{ij} & \sum_{i∈A_1}\sum_{j∈A_2}w_{ij} & ... & \sum_{i∈A_1}\sum_{j∈A_K}w_{ij} \\ \sum_{i∈A_2}\sum_{j∈A_1}w_{ij} & \sum_{i∈A_2}\sum_{j∈A_2}w_{ij} & ... & \sum_{i∈A_2}\sum_{j∈A_K}w_{ij}\\ ... & ... & ... & ...\\ \sum_{i∈A_K}\sum_{j∈A_1}w_{ij} & \sum_{i∈A_K}\sum_{j∈A_2}w_{ij} & ... & \sum_{i∈A_K}\sum_{j∈A_K}w_{ij} \end{bmatrix} YTWY维度是K×K维YTWY=[y1​,...yN​] ​w11​w21​...wN1​​w12​w22​...wN2​​............​w1N​w2N​...wNN​​ ​ ​y1T​y2T​...yNT​​ ​=[i=1∑N​yi​wi1​,...,i=1∑N​yi​wNi​] ​y1T​y2T​...yNT​​ ​=i=1∑N​j=1∑N​yi​wij​yiT​=i=1∑N​j=1∑N​yi​yiT​wij​= ​∑i∈A1​​∑j∈A1​​wij​∑i∈A2​​∑j∈A1​​wij​...∑i∈AK​​∑j∈A1​​wij​​∑i∈A1​​∑j∈A2​​wij​∑i∈A2​​∑j∈A2​​wij​...∑i∈AK​​∑j∈A2​​wij​​............​∑i∈A1​​∑j∈AK​​wij​∑i∈A2​​∑j∈AK​​wij​...∑i∈AK​​∑j∈AK​​wij​​ ​
观察上式和O的后半部分：
[ ∑ i ∈ A 1 ∑ j ∈ A 1 w i j 0 . . . 0 0 ∑ i ∈ A 2 ∑ j ∈ A 2 w i j . . . 0 . . . . . . . . . . . . 0 . . . . . . ∑ i ∈ A K ∑ j ∈ A K w i j ] \begin{bmatrix} \sum_{i∈A_1}\sum_{j∈A_1}w_{ij} & 0 & ... & 0 \\ 0 & \sum_{i∈A_2}\sum_{j∈A_2}w_{ij} & ... & 0\\ ... & ... & ... & ...\\ 0 & ... & ...& \sum_{i∈A_K}\sum_{j∈A_K}w_{ij} \end{bmatrix} \\ \\ \\ ​∑i∈A1​​∑j∈A1​​wij​0...0​0∑i∈A2​​∑j∈A2​​wij​......​............​00...∑i∈AK​​∑j∈AK​​wij​​ ​

[ ∑ i ∈ A 1 ∑ j ∈ A 1 w i j ∑ i ∈ A 1 ∑ j ∈ A 2 w i j . . . ∑ i ∈ A 1 ∑ j ∈ A K w i j ∑ i ∈ A 2 ∑ j ∈ A 1 w i j ∑ i ∈ A 2 ∑ j ∈ A 2 w i j . . . ∑ i ∈ A 2 ∑ j ∈ A K w i j . . . . . . . . . . . . ∑ i ∈ A K ∑ j ∈ A 1 w i j ∑ i ∈ A K ∑ j ∈ A 2 w i j . . . ∑ i ∈ A K ∑ j ∈ A K w i j ] \begin{bmatrix} \sum_{i∈A_1}\sum_{j∈A_1}w_{ij} & \sum_{i∈A_1}\sum_{j∈A_2}w_{ij} & ... & \sum_{i∈A_1}\sum_{j∈A_K}w_{ij} \\ \sum_{i∈A_2}\sum_{j∈A_1}w_{ij} & \sum_{i∈A_2}\sum_{j∈A_2}w_{ij} & ... & \sum_{i∈A_2}\sum_{j∈A_K}w_{ij}\\ ... & ... & ... & ...\\ \sum_{i∈A_K}\sum_{j∈A_1}w_{ij} & \sum_{i∈A_K}\sum_{j∈A_2}w_{ij} & ... & \sum_{i∈A_K}\sum_{j∈A_K}w_{ij} \end{bmatrix} ​∑i∈A1​​∑j∈A1​​wij​∑i∈A2​​∑j∈A1​​wij​...∑i∈AK​​∑j∈A1​​wij​​∑i∈A1​​∑j∈A2​​wij​∑i∈A2​​∑j∈A2​​wij​...∑i∈AK​​∑j∈A2​​wij​​............​∑i∈A1​​∑j∈AK​​wij​∑i∈A2​​∑j∈AK​​wij​...∑i∈AK​​∑j∈AK​​wij​​ ​

我们发现，这两个矩阵对角线元素是相同的，又因为我们是对迹求最小，即只考虑对角线的元素。所以将O的后半部分换成$Y^TWY $并不影响我们的结果

记 $O^{\prime }=Y^{T}DY-Y^{T}WY$ 那么$O^{\prime }P$相当于对$O^{\prime }$的对角线做一些变化。那么就有$Tr(OP)=Tr(O^{\prime }P)$。

至此我们解出了$O$ ，并且提出了用$O^{\prime }$ 代替$O$ 可以达到同样的目的。
$$O^{\prime }=Y^{T}DY-Y^{T}WY$$

我们最终的优化问题变为：
$$\begin{gathered} \hat{Y}=arg\underset{\hat{Y}}{\min }Tr(Y^T(D-W)Y(Y^{T}DY{)}^{-1}) \\ =\hat{Y}=arg\underset{\hat{Y}}{\min }Tr(Y^{T}LY(Y^{T}DY{)}^{-1}) \\ 这里L=D-W是拉普拉斯矩阵 \end{gathered}$$

To minimaze $Tr(Y^{T}LY(Y^{T}DY{)}^{-1})$

$Tr(Y^{T}LY(Y^{T}DY{)}^{-1})$

其中$Y\in R^{N\times K}$ ，每一行是ONE-HOT，表示第i行属于哪一类。$Y$ 形如：
[ 0 . . . 0 1 0 . . . 0 0 . . . 1 0 0 . . . 0 . . . . . . . . . . . . . . . . . . . . . 0 . . . 0 0 0 . . . 1 1 . . . 0 1 0 . . . 0 ] \begin{bmatrix} 0 & ...& 0 & 1 & 0 &... & 0 \\ 0 & ...&1 & 0 & 0 &... & 0\\ ... & ...& ... & ... & ... & ... & ...\\ 0 & ...& 0 & 0 & 0 &... & 1\\ 1 & ...&0 & 1 & 0 &... & 0 \end{bmatrix} ​00...01​...............​01...00​10...01​00...00​...............​00...10​ ​
记：
$$\begin{gathered} P=Y^{T}DY=diag(\sum _{i\in A_1}{d}_i,\sum _{i\in A_2}{d}_i,...,\sum _{i\in A_K}{d}_i)=diag(p_1,p_2,...,p_k) \\ 原式=Tr(Y^{T}LY{P}^{-1})=Tr(Y^{T}LY{P}^{-\frac{1}{2}}{P}^{-\frac{1}{2}})=Tr(P^{-\frac{1}{2}}{Y}^{T}LY{P}^{-\frac{1}{2}}) \end{gathered}$$
记：
$$\begin{gathered} H=Y{P}^{-\frac{1}{2}},H^T=P^{-\frac{1}{2}}{Y}^T \\ {H}^{T}H=P^{-\frac{1}{2}}{Y}^{T}Y{P}^{-\frac{1}{2}}=P^{-\frac{1}{2}}I{P}^{-\frac{1}{2}}=P^{-1} \end{gathered}$$

$$原式=Tr(H^{T}LH)$$
定理1：

对于半正定矩阵L， 特征值（eigenvalue）：$0\le {\lambda }_1\le {\lambda }_2\le ...\le {\lambda }_n$

​ 特征基（eigbasis）：$\{{\overline{v}}_1,{\overline{v}}_2,...,{\overline{v}}_n\}$ →Orthonormal，标准正交化之后的特征向量

当$\mathbf{x}\in R^N,and{\mathbf{x}}^T\mathbf{x}=1$ 时，${\mathbf{x}}^{T}L\mathbf{x}$ 的最小值在$\mathbf{x}={\overline{v}}_1$ 时取到。

proof：
$$\begin{gathered} \mathbf{x}可以用eigbasis表示,因为eigbasis是orthonormal \\ \mathbf{x}=c_1{\overline{v}}_1+c_2{\overline{v}}_2+...+c_n{\overline{v}}_n \\ L\mathbf{x}=\lambda \mathbf{x}=c_1{\lambda }_1{\overline{v}}_1+c_2{\lambda }_2{\overline{v}}_2+...+c_n{\lambda }_n{\overline{v}}_n \\ \to {\mathbf{x}}^{T}L\mathbf{x}=c_1^2{\lambda }_1+c_2^2{\lambda }_2+...+c_n^2{\lambda }_n \\ 因为{\mathbf{x}}^T\mathbf{x}=1\to c_1^2+c_2^2+...+c_n^2 \\ \to {\mathbf{x}}^{T}L\mathbf{x}=c_1^2{\lambda }_1+c_2^2{\lambda }_2+...+c_n^2{\lambda }_n\ge {\lambda }_1 \\ 当c_1^2=1,c_i=0,i\ne 1时等号成立\iff \mathbf{x}={\overline{v}}_{1}or\mathbf{x}=-{\overline{v}}_1 \end{gathered}$$
定理2：

对于半正定矩阵L， 特征值（eigenvalue）：$0\le {\lambda }_1\le {\lambda }_2\le ...\le {\lambda }_n$

​ 特征基（eigbasis）：$\{{\overline{v}}_1,{\overline{v}}_2,...,{\overline{v}}_n\}$ →Orthonormal，标准正交化之后的特征向量

当$F\in R^{N\times K},and{F}^{T}F=I$ 时，$Tr(F^{T}LF)$ 的最小值在$F=[{\overline{v}}_1,{\overline{v}}_2,...,{\overline{v}}_K]$ 时取到

proof:
$$DenoteF=[f_1,$$