2 Main Layout Conventions of Matrix Calculus

考虑 \(x\), \(y\) 分别是 \(n\), \(m\) 维列向量, \(A\) 是 \(m\times n\) 矩阵, \(z\) 是标量.

Numerator Layout

想象分子不变, 分母转置.

Vector by vector 符合直观. Jacobian.

\[ \frac{\partial y}{\partial x} = \begin{pmatrix} \frac{\partial y_1}{\partial x_1} & \dots & \frac{\partial y_1}{\partial x_n}\\ \vdots & \ddots &\vdots\\ \frac{\partial y_m}{\partial x_1} & \dots & \frac{\partial y_m}{\partial x_n} \end{pmatrix} \]

Scalar by matrix 要做一次转置, 不舒服.

\[ \frac{\partial z}{\partial A}= \begin{pmatrix} \frac{\partial z}{\partial a_{11}} & \dots & \frac{\partial z}{\partial a_{m1}}\\ \vdots & \ddots &\vdots\\ \frac{\partial z}{\partial a_{1n}} & \dots & \frac{\partial z}{\partial a_{mn}} \end{pmatrix} \]

Chain rule 符合直观.

\[ \frac{\partial f\circ g}{\partial x} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial x} \]

Denominator Layout

想象分母不变, 分子转置.

Vector by vector 不舒服. Hessian.

\[ \frac{\partial y}{\partial x} = \begin{pmatrix} \frac{\partial y_1}{\partial x_1} & \dots & \frac{\partial y_m}{\partial x_1}\\ \vdots & \ddots &\vdots\\ \frac{\partial y_1}{\partial x_n} & \dots & \frac{\partial y_m}{\partial x_n} \end{pmatrix} \]

Scalar by matrix 舒服.

\[ \frac{\partial z}{\partial A}= \begin{pmatrix} \frac{\partial z}{\partial a_{11}} & \dots & \frac{\partial z}{\partial a_{1n}}\\ \vdots & \ddots &\vdots\\ \frac{\partial z}{\partial a_{m1}} & \dots & \frac{\partial z}{\partial a_{mn}} \end{pmatrix} \]

Chain rule "倒过来" 了, 不舒服.

\[ \frac{\partial f\circ g}{\partial x} = \frac{\partial g}{\partial x}\frac{\partial f}{\partial g} \]

混用

混用现象很常见. 比如 CS224n, 主体是采用 numerator layout, 但是 scalar by matrix 时是不转置的.

转载于:https://www.cnblogs.com/shiina922/p/11435371.html