西瓜书公式（10.31）的推导

西瓜书 10.5.2 节 局部线性嵌入

与 Isomap 试图保持近邻样本之间的距离不同，局部线性嵌入（Locally Linear Embedding, 简称 LLE）试图保持邻域内样本之间的线性关系。

LLE 先为每个样本 $x_i$ 找到其近邻下标集合 $Q_i$, 然后计算出基于 $Q_i$ 中的样本点对 $x_i$ 进行线性重构的系数 $w_i$

LLE 在低维空间（维度为 $d^{\prime }$）中保持 $w_i$ 不变，于是 $x_i$ 对应的低维空间坐标 $z_i$ 可通过下式求解：
min ⁡ z 1 , z 2 , . . . , z m ∑ i = 1 m ∥ z i − ∑ j ∈ Q i w i j z j ∥ 2 2 (10.29) \min_{z_1,z_2,...,z_m}\sum_{i=1}^m \left \lVert z_i-\sum_{j\in Q_i} w_{ij} z_j\right \rVert_2^2 \tag{10.29} z1​,z2​,...,zm​min​i=1∑m​ ​zi​−j∈Qi​∑​wij​zj​ ​22​(10.29)

令 $Z=\left(z_1,z_2,...,z_m\right)\in {\mathbb{R}}^{d^{\prime }\times m}$, $(W{)}_{i,j}=w_{ij}$,
$$M=(I-W{)}^T(I-W)\text{\hspace{1em}(10.30)}$$
则式 (10.29) 可重写为
$$\begin{gathered} \underset{Z}{\min }tr(ZM{Z}^T), \\ s.t.Z{Z}^T=I.\text{\hspace{1em}(10.31)} \end{gathered}$$
其中 $Z^T\in {\mathbb{R}}^{m\times d^{\prime }}$ 中的 $d^{\prime }$ 个列向量是对 $M$ 进行特征值分解后最小的 $d^{\prime }$ 个特征值对应的特征向量（$Z{Z}^T=I$ 表示要求特征向量是单位向量）。

下面我们对式 (10.31) 进行推导：
t r ( Z M Z T ) = t r ( Z ( I − W ) T ( I − W ) Z T ) = t r ( Z Z T − Z W T Z T − Z W Z T + Z W T W Z T ) = t r ( Z Z T ) − t r ( Z W T Z T ) − t r ( Z W Z T ) + t r ( Z W T W Z T ) = t r ( Z Z T ) − 2 ⋅ t r ( Z W Z T ) + t r ( Z W T W Z T ) = ∑ j = 1 d ′ ( ( Z Z T ) j j − 2 ⋅ ( Z W Z T ) j j + ( Z W T W Z T ) j j ) = ∑ j = 1 d ′ [ ∑ i = 1 m Z j i ( Z T ) i j − 2 ⋅ ∑ i = 1 m Z j i ( W Z T ) i , j + ∑ i = 1 m ( Z W T ) j i ( W Z T ) i j ] = ∑ i = 1 m [ ∑ j = 1 d ′ Z j i ( Z T ) i j − 2 ⋅ ∑ j = 1 d ′ Z j i ( W Z T ) i , j + ∑ j = 1 d ′ ( Z W T ) j i ( W Z T ) i j ] = ∑ i = 1 m [ ∑ j = 1 d ′ ( Z T ) i j Z j i − 2 ⋅ ∑ j = 1 d ′ ( W Z T ) i j Z j i + ∑ j = 1 d ′ ( W Z T ) i j ( Z W T ) j i ] = ∑ i = 1 m [ ( Z T ) i ( Z T ) i T − 2 ( W Z T ) i ( Z T ) i T + ( W Z T ) i ( W Z T ) i T ] = ∑ i = 1 m [ z i T z i − 2 ( W Z T ) i z i + ( W Z T ) i ( W Z T ) i T ] \begin{aligned} tr(ZMZ^T) &=tr\left(Z(I - W)^T(I-W)Z^T\right) \\ &= tr(ZZ^T-ZW^TZ^T-ZWZ^T+ZW^TWZ^T) \\ &=tr(ZZ^T)-tr(ZW^TZ^T)-tr(ZWZ^T)+tr(ZW^TWZ^T) \\ &=tr(ZZ^T)-2\cdot tr(ZWZ^T)+tr(ZW^TWZ^T) \\ &=\sum_{j=1}^{d'}\left((ZZ^T)_{jj}-2\cdot(ZWZ^T)_{jj}+ (ZW^TWZ^T)_{jj}\right) \\ &=\sum_{j=1}^{d'}\left[ \sum_{i=1}^{m}Z_{ji}(Z^T)_{ij} - 2 \cdot \sum_{i=1}^{m}Z_{ji}(WZ^T)_{i,j} + \sum_{i=1}^{m}(ZW^T)_{ji} (WZ^T)_{ij} \right]\\ &=\sum_{i=1}^m\left[ \sum_{j=1}^{d'}Z_{ji}(Z^T)_{ij} - 2 \cdot \sum_{j=1}^{d'}Z_{ji}(WZ^T)_{i,j} + \sum_{j=1}^{d'}(ZW^T)_{ji} (WZ^T)_{ij} \right]\\ &=\sum_{i=1}^m\left[ \sum_{j=1}^{d'}(Z^T)_{ij}Z_{ji} - 2 \cdot \sum_{j=1}^{d'}(WZ^T)_{ij}Z_{ji} + \sum_{j=1}^{d'}(WZ^T)_{ij}(ZW^T)_{ji} \right]\\ &=\sum_{i=1}^m\left[ (Z^T)_{i}(Z^T)_{i}^T - 2 (WZ^T)_{i}(Z^T)_{i}^T + (WZ^T)_{i}(WZ^T)_{i}^T \right]\\ &=\sum_{i=1}^m\left[ z_{i}^T z_{i}- 2 (WZ^T)_{i}z_{i} + (WZ^T)_{i}(WZ^T)_{i}^T \right]\\ \end{aligned} tr(ZMZT)​=tr(Z(I−W)T(I−W)ZT)=tr(ZZT−ZWTZT−ZWZT+ZWTWZT)=tr(ZZT)−tr(ZWTZT)−tr(ZWZT)+tr(ZWTWZT)=tr(ZZT)−2⋅tr(ZWZT)+tr(ZWTWZT)=j=1∑d′​((ZZT)jj​−2⋅(ZWZT)jj​+(ZWTWZT)jj​)=j=1∑d′​[i=1∑m​Zji​(ZT)ij​−2⋅i=1∑m​Zji​(WZT)i,j​+i=1∑m​(ZWT)ji​(WZT)ij​]=i=1∑m​ ​j=1∑d′​Zji​(ZT)ij​−2⋅j=1∑d′​Zji​(WZT)i,j​+j=1∑d′​(ZWT)ji​(WZT)ij​ ​=i=1∑m​ ​j=1∑d′​(ZT)ij​Zji​−2⋅j=1∑d′​(WZT)ij​Zji​+j=1∑d′​(WZT)ij​(ZWT)ji​ ​=i=1∑m​[(ZT)i​(ZT)iT​−2(WZT)i​(ZT)iT​+(WZT)i​(WZT)iT​]=i=1∑m​[ziT​zi​−2(WZT)i​zi​+(WZT)i​(WZT)iT​]​

其中
$$\begin{array}{rl}(W{Z}^T{)}_i& =W_i{Z}^T\\ & =\sum _{j=1}^m{W}_{ij}(Z^T{)}_j\\ & =\sum _{j=1}^m{w}_{ij}{z}_j\end{array}$$

代入上式可得
t r ( Z M Z T ) = ∑ i = 1 m [ z i T z i − 2 ( W Z T ) i z i + ( W Z T ) i ( W Z T ) i T ] = ∑ i = 1 m [ z i T z i − 2 ( ∑ j = 1 m w i j z j ) T z i + ( ∑ j = 1 m w i j z j ) T ( ∑ j = 1 m w i j z j ) ] = ∑ i = 1 m ( z i − ∑ j = 1 m w i j z j ) 2 \begin{aligned} tr(ZMZ^T) &=\sum_{i=1}^m\left[ z_{i}^T z_{i}- 2 (WZ^T)_{i}z_{i} + (WZ^T)_{i}(WZ^T)_{i}^T \right] \\ &=\sum_{i=1}^m\left[ z_{i}^T z_{i}- 2 \left(\sum_{j=1}^m w_{ij}z_{j} \right)^Tz_{i} + \left(\sum_{j=1}^m w_{ij}z_{j} \right)^T\left(\sum_{j=1}^m w_{ij}z_{j} \right) \right] \\ &=\sum_{i=1}^m \left( z_i- \sum_{j=1}^m w_{ij}z_{j} \right)^2 \end{aligned} tr(ZMZT)​=i=1∑m​[ziT​zi​−2(WZT)i​zi​+(WZT)i​(WZT)iT​]=i=1∑m​ ​ziT​zi​−2(j=1∑m​wij​zj​)Tzi​+(j=1∑m​wij​zj​)T(j=1∑m​wij​zj​) ​=i=1∑m​(zi​−j=1∑m​wij​zj​)2​

所以
min ⁡ Z t r ( Z M Z T ) = min ⁡ z 1 , z 2 , . . . , z m ∑ i = 1 m ( z i − ∑ j = 1 m w i j z j ) 2 = min ⁡ z 1 , z 2 , . . . , z m ∑ i = 1 m ∥ z i − ∑ j ∈ Q i w i j z j ∥ 2 2 \begin{aligned} \min_{Z} tr(ZMZ^T) &= \min_{z_1,z_2,...,z_m}\sum_{i=1}^m \left( z_i- \sum_{j=1}^m w_{ij}z_{j} \right)^2 \\ &=\min_{z_1,z_2,...,z_m}\sum_{i=1}^m \left \lVert z_i-\sum_{j\in Q_i} w_{ij} z_j\right \rVert_2^2 \end{aligned} Zmin​tr(ZMZT)​=z1​,z2​,...,zm​min​i=1∑m​(zi​−j=1∑m​wij​zj​)2=z1​,z2​,...,zm​min​i=1∑m​ ​zi​−j∈Qi​∑​wij​zj​ ​22​​
Q.E.D.