矩阵向量求导

矩阵向量求导

  矩阵向量求导不过是一种方便的计算形式，用来保存一系列偏导结果的符号，具体形式如下(A(t)是矩阵，t是标量)：
$$A(t)=\left[\begin{array}{ccc}{a}_{11}(t)& a_{12}(t)& a_{13}(t)\\ a_{21}(t)& a_{22}(t)& a_{23}(t)\\ a_{31}(t)& a_{32}(t)& a_{33}(t)\end{array}\right]$$
$$\frac{\partial A(t)}{\partial t}=\left[\begin{array}{ccc}{a}_{11}^{\prime }(t)& a_{12}^{\prime }(t)& a_{13}^{\prime }(t)\\ a_{21}^{\prime }(t)& a_{22}^{\prime }(t)& a_{23}^{\prime }(t)\\ a_{31}^{\prime }(t)& a_{32}^{\prime }(t)& a_{33}^{\prime }(t)\end{array}\right]\text{\hspace{1em}(1)}$$
  这种矩阵向量求导直观来看就是把分散求导聚集起来，用一个符号一次性表示，没有什么复杂的地方，所以我在分类时把它分成了高等数学领域，这篇blog会介绍这种求导方式。$(1)$矩阵求导和下面的$(2)$微分方程并不一样(A是矩阵，u是向量)：
$$\frac{\partial u}{\partial t}=Au\text{\hspace{1em}(2)}$$
  $(2)$在矩阵论中用来求解微分方程，结果大体长这样：$u=e^{At}u(0)$。我们也比较容易区分这两种情况，$(1)$求导计算，$(2)$是微分方程，$(1)$中是对A求导，$(2)$中A是一个系数矩阵。关于$(2)$，我会写一个矩阵的专题单独介绍。

---

分类

  我们通过自变量和因变量来分类矩阵向量求导，自变量和因变量均可以是标量、向量或者矩阵，一共9种情况，见下图，但是向量与矩阵的两种情况，以及矩阵与矩阵的一种情况比较复杂，结果需要用张量(高维矩阵)表示，所以暂未讨论。

自变量\因变量	标量	向量	矩阵
标量	简单	同(1)	同(1)
向量	同(1)	重点讨论	复杂
矩阵	同(1)	复杂	复杂

向量对标量求导

$$u(t)=[u_1(t),u_2(t),u_3(t)]$$

$$\frac{\partial u(t)}{\partial t}=[u_1^{\prime }(t),u_2^{\prime }(t),u_3^{\prime }(t)]$$
  或者
$$u(t)=[u_1(t),u_2(t),u_3(t){]}^T$$

$$\frac{\partial u(t)}{\partial t}=[u_1^{\prime }(t),u_2^{\prime }(t),u_3^{\prime }(t){]}^T$$

矩阵对标量求导

$$A(t)=\left[\begin{array}{ccc}{a}_{11}(t)& a_{12}(t)& a_{13}(t)\\ a_{21}(t)& a_{22}(t)& a_{23}(t)\\ a_{31}(t)& a_{32}(t)& a_{33}(t)\end{array}\right]$$
$$\frac{\partial A(t)}{\partial t}=\left[\begin{array}{ccc}{a}_{11}^{\prime }(t)& a_{12}^{\prime }(t)& a_{13}^{\prime }(t)\\ a_{21}^{\prime }(t)& a_{22}^{\prime }(t)& a_{23}^{\prime }(t)\\ a_{31}^{\prime }(t)& a_{32}^{\prime }(t)& a_{33}^{\prime }(t)\end{array}\right]$$

标量对向量求导

$$u(t)=[u_1(t),u_2(t),u_3(t)]$$

$$\frac{\partial t}{\partial u(t)}=[\frac{\partial t}{\partial u_1(t)},\frac{\partial t}{\partial u_2(t)},\frac{\partial t}{\partial u_3(t)}]$$
  或者
$$u(t)=[u_1(t),u_2(t),u_3(t){]}^T$$

$$\frac{\partial t}{\partial u(t)}=[\frac{\partial t}{\partial u_1(t)},\frac{\partial t}{\partial u_2(t)},\frac{\partial t}{\partial u_3(t)}{]}^T$$

标量对矩阵求导

$$A(t)=\left[\begin{array}{ccc}{a}_{11}(t)& a_{12}(t)& a_{13}(t)\\ a_{21}(t)& a_{22}(t)& a_{23}(t)\\ a_{31}(t)& a_{32}(t)& a_{33}(t)\end{array}\right]$$
$$\frac{\partial t}{\partial A(t)}=\left[\begin{array}{ccc}\frac{\partial t}{\partial a_{11}(t)}& \frac{\partial t}{\partial a_{12}(t)}& \frac{\partial t}{\partial a_{13}(t)}\\ \frac{\partial t}{\partial a_{21}(t)}& \frac{\partial t}{\partial a_{22}(t)}& \frac{\partial t}{\partial a_{23}(t)}\\ \frac{\partial t}{\partial a_{31}(t)}& \frac{\partial t}{\partial a_{32}(t)}& \frac{\partial t}{\partial a_{33}(t)}\end{array}\right]$$

向量对向量求导

  我们先介绍分子布局以及分母布局。
分子布局：$\frac{\partial y}{\partial x}=\frac{列向量}{行向量}$
$$y=[y_1,y_2,y_3{]}^T$$
$$x=[x_1,x_2,x_3]$$
$$\frac{\partial y}{\partial x}=\left[\begin{array}{ccc}\frac{\partial y_1}{\partial x_1}& \frac{\partial y_1}{\partial x_2}& \frac{\partial y_1}{\partial x_3}\\ \frac{\partial y_2}{\partial x_1}& \frac{\partial y_2}{\partial x_2}& \frac{\partial y_2}{\partial x_3}\\ \frac{\partial y_3}{\partial x_1}& \frac{\partial y_3}{\partial x_2}& \frac{\partial y_3}{\partial x_3}\end{array}\right]$$
分母布局：$\frac{\partial y}{\partial x}=\frac{行向量}{列向量}$
$$y=[y_1,y_2,y_3]$$
$$x=[x_1,x_2,x_3{]}^T$$
$$\frac{\partial y}{\partial x}=\left[\begin{array}{ccc}\frac{\partial y_1}{\partial x_1}& \frac{\partial y_2}{\partial x_1}& \frac{\partial y_3}{\partial x_1}\\ \frac{\partial y_1}{\partial x_2}& \frac{\partial y_2}{\partial x_2}& \frac{\partial y_3}{\partial x_2}\\ \frac{\partial y_1}{\partial x_3}& \frac{\partial y_2}{\partial x_3}& \frac{\partial y_3}{\partial x_3}\end{array}\right]$$

  有时就算在同一篇论文或书中，这两种布局也会混着用，所以要结合上下文来看。在周志华《机器学习方法》书中，多数都是分母布局。
  有些说法觉得矩阵与标量以及向量与标量的导数运算也要考虑分子分母布局的问题，但我觉得一个行向量对标量求导，考虑分子布局，结果是个列向量，总觉得很怪，所以我只在向量与向量求导中使用布局，如果哪天碰到问题回来改。

---

可用公式总结

a,x是向量，a与x无关，A,B是矩阵，使用分母布局
$$\frac{\partial x^{T}x}{\partial x}=2x$$
$$\frac{\partial x^{T}a}{\partial x}=\frac{\partial a^{T}x}{\partial x}=a$$
$$\frac{\partial AB}{\partial x}=\frac{\partial A}{\partial x}B+A\frac{\partial B}{\partial x}$$

---

参考文献：
《机器学习》周志华
https://blog.csdn.net/uncle_gy/article/details/78879131
https://en.wikipedia.org/wiki/Matrix_calculus