【机器学习总结】向量、矩阵求导公式

关于向量求导用到的公式实在是太多了…经常公式推着推着就被卡住，这里一次性做个总结吧。

0.引言

  正文中，元素使用字母a，b，c等表示，向量使用小写的$x，y，z$等表示，并且默认是列向量，矩阵使用大写的A,B,C进行表示。

1.向量对元素求导

• 行向量对元素求导
  $$\frac{\partial x^T}{\partial a}=\left[\begin{array}{c}\frac{\partial x_1}{\partial a},\frac{\partial x_2}{\partial a},...,\frac{\partial x_n}{\partial a}\end{array}\right]$$
• 列向量对元素求导
  $$\frac{\partial x}{\partial a}=\left[\begin{array}{c}\frac{\partial x_1}{\partial a}\\ \frac{\partial x_2}{\partial a}\\ ...\\ \frac{\partial x_n}{\partial a}\end{array}\right]$$

2.向量对向量求导

• 行向量对列向量求导
  $$\frac{\partial y^T}{\partial x}=\left[\begin{array}{c}\frac{\partial y_1}{\partial x_1},\frac{\partial y_2}{\partial x_1},...,\frac{\partial y_n}{\partial x_1}\\ \frac{\partial y_1}{\partial x_2},\frac{\partial y_2}{\partial x_2},...,\frac{\partial y_n}{\partial x_2}\\ ...\\ \frac{\partial y_1}{\partial x_n},\frac{\partial y_2}{\partial x_n},...,\frac{\partial y_n}{\partial x_n}\end{array}\right]$$
• 列向量对行向量求导
  $$\frac{\partial y}{\partial x^T}=\left[\begin{array}{c}\frac{\partial y_1}{\partial x_1},\frac{\partial y_1}{\partial x_2},...,\frac{\partial y_1}{\partial x_n}\\ \frac{\partial y_2}{\partial x_1},\frac{\partial y_2}{\partial x_2},...,\frac{\partial y_2}{\partial x_n}\\ ...\\ \frac{\partial y_n}{\partial x_1},\frac{\partial y_n}{\partial x_2},...,\frac{\partial y_n}{\partial x_n}\end{array}\right]$$
• 行向量对行向量求导
  $$\frac{\partial y^T}{\partial x^T}=\left[\begin{array}{c}\frac{\partial y_T}{\partial x_1},\frac{\partial y_T}{\partial x_2},...,\frac{\partial y_T}{\partial x_n}\end{array}\right]$$
• 列向量对列向量求导
  $$\frac{\partial y}{\partial x}=\left[\begin{array}{c}\frac{\partial y_1}{\partial x}\\ \frac{\partial y_2}{\partial x}\\ ...\\ \frac{\partial y_n}{\partial x}\end{array}\right]$$

3.矩阵对向量求导

• 矩阵对行向量求导
  $$\frac{\partial A}{\partial x^T}=\left[\begin{array}{c}\frac{\partial A}{\partial x_1},\frac{\partial A}{\partial x_2},...,\frac{\partial A}{\partial x_n}\end{array}\right]$$
• 矩阵对列向量求导
  $$\frac{\partial A}{\partial x}=\left[\begin{array}{c}\frac{\partial A_{11}}{\partial x},\frac{\partial A_{12}}{\partial x},...,\frac{\partial A_{1n}}{\partial x}\\ ...\\ \frac{\partial A_{n1}}{\partial x},\frac{\partial A_{n2}}{\partial x},...,\frac{\partial A_{nn}}{\partial x}\end{array}\right]$$

4.矩阵复合向量的求导

• $\frac{d}{dx}{x}^{T}A=A$
• $\frac{d}{d{x}^T}Ax=A$
• $\frac{d}{dx}xA=A^T$
• $\frac{d}{dx}Ax=A^T$
• $\frac{d}{dx}{x}^T=I$
• $\frac{d}{d{x}^T}x=I$
• $\frac{d}{dx}{x}^{T}y=\frac{d}{dx}{y}^{T}x=y$
• $\frac{d}{dx}{x}^{T}Ay=x{y}^T$
• $\frac{d}{dA}{x}^{T}Ax=x{x}^T$
• $\frac{d}{dA}{x}^T{A}^{T}y=y{x}^T$
• $\frac{d}{dx}{x}^{T}Ax=(A+A^T)x=2Ax$（当A为对称矩阵时第二个等式成立）
• $\frac{d}{dx}{x}^{T}x=2x$