考虑 \(x\), \(y\) 分别是 \(n\), \(m\) 维列向量, \(A\) 是 \(m\times n\) 矩阵, \(z\) 是标量.
Numerator Layout
想象分子不变, 分母转置.
Vector by vector 符合直观. Jacobian.
\[ \frac{\partial y}{\partial x} = \begin{pmatrix} \frac{\partial y_1}{\partial x_1} & \dots & \frac{\partial y_1}{\partial x_n}\\ \vdots & \ddots &\vdots\\ \frac{\partial y_m}{\partial x_1} & \dots & \frac{\partial y_m}{\partial x_n} \end{pmatrix} \]
Scalar by matrix 要做一次转置, 不舒服.
\[ \frac{\partial z}{\partial A}= \begin{pmatrix} \frac{\partial z}{\partial a_{11}} & \dots & \frac{\partial z}{\partial a_{m1}}\\ \vdots & \ddots &\vdots\\ \frac{\partial z}{\partial a_{1n}} & \dots & \frac{\partial z}{\partial a_{mn}} \end{pmatrix} \]
Chain rule 符合直观.
\[ \frac{\partial f\circ g}{\partial x} = \frac{\partial f}{\partial g}\frac{\partial g}{\partial x} \]
Denominator Layout
想象分母不变, 分子转置.
Vector by vector 不舒服. Hessian.
\[ \frac{\partial y}{\partial x} = \begin{pmatrix} \frac{\partial y_1}{\partial x_1} & \dots & \frac{\partial y_m}{\partial x_1}\\ \vdots & \ddots &\vdots\\ \frac{\partial y_1}{\partial x_n} & \dots & \frac{\partial y_m}{\partial x_n} \end{pmatrix} \]
Scalar by matrix 舒服.
\[ \frac{\partial z}{\partial A}= \begin{pmatrix} \frac{\partial z}{\partial a_{11}} & \dots & \frac{\partial z}{\partial a_{1n}}\\ \vdots & \ddots &\vdots\\ \frac{\partial z}{\partial a_{m1}} & \dots & \frac{\partial z}{\partial a_{mn}} \end{pmatrix} \]
Chain rule "倒过来" 了, 不舒服.
\[ \frac{\partial f\circ g}{\partial x} = \frac{\partial g}{\partial x}\frac{\partial f}{\partial g} \]
混用
混用现象很常见. 比如 CS224n, 主体是采用 numerator layout, 但是 scalar by matrix 时是不转置的.