Elman神经网络

标题：Elman神经网络

by：Z.H.Gao

一.网络结构

N—L—M拓展N—L—L—M，样本数为n。

$$Fig.1Elman网络样本逐个输入模式$$

$$Fig.2Elman网络单个样本网络结构$$

二.正向传播

$$h^i=f\left(temp{h}^i\right)=f\left(v{x}^i+b_{in}+u{h}^{i-1}\right)$$
$$y^i=f\left(temp{y}^i\right)=f\left(w{h}^i\right)=f\left(w\cdot f\left(v{x}^i+b_{in}+u{h}^{i-1}\right)\right)$$

1. 所有样本逐个计算

随着输入数据的不断增加，自循环的结构把上一次的状态传递给当前输入，一起作为新的输入数据进行当前轮次的训练和学习，一直到输入或者训练结束，最终得到的输出即为最终的预测结果。

2. 如果输入为[$x_{1\times N}^i$]，那么输入权值[$v_{N\times L}$]

隐藏层：[$h_{1\times L}^i$]
承接层：[$h_{1\times L}^{i-1}$]
承接层与隐藏层之间链接权值：[$u_{L\times L}$]
输出链接权值：[$w_{L\times M}$]
输出层：[$y_{1\times M}^i$]
此时i表示样本，每次输入一个样本。因为存在承接层所以通常逐个输入样本，当然也可以逐批输入。

三.反向计算 (Back Propagation Through Time, BPTT)

1. 已知：

$h^i=f\left(temp{h}^i\right)=f\left(v{x}^i+b_{in}+u{h}^{i-1}\right)$
$y^i=f\left(temp{y}^i\right)=f\left(w{h}^i\right)=f\left(w\cdot f\left(v{x}^i+b_{in}+u{h}^{i-1}\right)\right)$
其中，i为样本序号。

2. 参数w的梯度计算

设计损失函数为$J\left(Y,Target\right)$，那么对于单个样本i。
注：统一使用表示元素乘法，×表示矩阵乘法*
$\frac{\partial J^i}{\partial temp{y}^i}=\frac{\partial J^i}{\partial y^i}\ast \frac{\partial y^i}{\partial temp{y}^i}=\frac{\partial J^i}{\partial y^i}\ast f^{\prime }\left(temp{y}^i\right)$
$\frac{\partial J^i}{\partial w}=\frac{\partial J^i}{\partial y^i}\ast \frac{\partial y^i}{\partial temp{y}^i}\times \frac{\partial temp{y}^i}{\partial w}={\left(h^i\right)}^T\times \frac{\partial J^i}{\partial y^i}\ast f^{\prime }\left(temp{y}^i\right)$
$\frac{\partial J^i}{\partial h^i}=\frac{\partial J^i}{\partial y^i}\ast \frac{\partial y^i}{\partial temp{y}^i}\times \frac{\partial temp{y}^i}{\partial h^i}=\frac{\partial J^i}{\partial y^i}\ast f^{\prime }\left(temp{y}^i\right)\times w^T$
那么对于所有的样本，w的梯度计算等同于普通前馈神经网络
$\frac{\partial J}{\partial w}=\sum _{i=1}^n{\left(h^i\right)}^T\times \frac{\partial J^i}{\partial y^i}\ast f^{\prime }\left(temp{y}^i\right)=H^T\times \frac{\partial J}{\partial Y}\ast f^{\prime }\left(tempY\right)$
同理，$\frac{\partial J}{\partial H}=\frac{\partial J}{\partial Y}\ast f^{\prime }\left(tempY\right)\times w^T$

3. 隐藏层hi与hi-1承接层的梯度计算

参数u的计算关系到当前样本与之前样本的链接，需要用“循环”计算梯度。
$\frac{\partial J}{\partial tempH}=\frac{\partial J}{\partial H}\ast f^{\prime }\left(tempH\right)=\left[\frac{\partial J}{\partial Y}\ast f^{\prime }\left(tempY\right)\times w^T\right]\ast f^{\prime }\left(tempH\right)$
则，$\frac{\partial J^i}{\partial temp{h}^i}=\frac{\partial J}{\partial tempH}\left(i,:\right)$，循环的重点，每次计算单个样本：
$$\begin{array}{l}\frac{\partial J^i}{\partial h^{i-1}}=\frac{\partial J^i}{\partial temp{h}^i}\ast \frac{\partial temp{h}^i}{\partial h^{i-1}}=\frac{\partial J^i}{\partial temp{h}^i}\times u^T\\ \frac{\partial J^i}{\partial temp{h}^{i-1}}=\frac{\partial J^i}{\partial temp{h}^i}\ast \frac{\partial temp{h}^i}{\partial h^{i-1}}\ast \frac{\partial h^{i-1}}{\partial temp{h}^{i-1}}=\left(\frac{\partial J^i}{\partial temp{h}^i}\times u^T\right)\ast f^{\prime }\left(temp{h}^{i-1}\right)\end{array}$$
$$\begin{array}{l}\frac{\partial J^i}{\partial h^{i-2}}=\frac{\partial J^i}{\partial temp{h}^{i-1}}\ast \frac{\partial temp{h}^{i-1}}{\partial h^{i-2}}=\frac{\partial J^i}{\partial temp{h}^{i-1}}\times u^T\\ \frac{\partial J^i}{\partial temp{h}^{i-2}}=\frac{\partial J^i}{\partial temp{h}^{i-1}}\ast \frac{\partial temp{h}^{i-1}}{\partial h^{i-2}}\ast \frac{\partial h^{i-2}}{\partial temp{h}^{i-2}}=\left(\frac{\partial J^i}{\partial temp{h}^{i-1}}\times u^T\right)\ast f^{\prime }\left(temp{h}^{i-2}\right)\end{array}$$
循环是为了计算当前样本误差Ji受前k次样本的影响。在计算上是利用当前样本误差Ji去计算前k次网络与当前网络之间的链接权值u。

4. 参数u与v的梯度计算

对于单个样本i而言，其对当前网络的影响可以计算相应的梯度：
$$\begin{array}{l}\frac{\partial J^i}{\partial u}=\frac{\partial J^i}{\partial temp{h}^i}\ast \frac{\partial temp{h}^i}{\partial u}={\left(h^{i-1}\right)}^T\times \frac{\partial J^i}{\partial temp{h}^i}\\ \frac{\partial J^i}{\partial v}=\frac{\partial J^i}{\partial temp{h}^i}\ast \frac{\partial temp{h}^i}{\partial x^i}={\left(x^i\right)}^T\times \frac{\partial J^i}{\partial temp{h}^i}\end{array}$$
那么前k个样本对于单个样本i的影响，都需要通过参数u和v，有
$$\begin{array}{l}\frac{\partial J^i}{\partial u}=\frac{\partial J^i}{\partial temp{h}^i}\ast \frac{\partial temp{h}^i}{\partial u}=\sum _{k=1}^i{\left(h^{k-1}\right)}^T\times \frac{\partial J^i}{\partial temp{h}^k}\\ \frac{\partial J^i}{\partial v}=\frac{\partial J^i}{\partial temp{h}^i}\ast \frac{\partial temp{h}^i}{\partial x^i}=\sum _{k=1}^i{\left(x^k\right)}^T\times \frac{\partial J^i}{\partial temp{h}^k}\end{array}$$

四．通式（针对第i个样本）

假设k=3，显然有
$\begin{array}{l}\frac{\partial J^i}{\partial h^{i-1}}=\frac{\partial J^i}{\partial temp{h}^i}\ast \frac{\partial temp{h}^i}{\partial h^{i-1}}=\frac{\partial J^i}{\partial temp{h}^i}\times u^T\\ \frac{\partial J^i}{\partial temp{h}^{i-1}}=\frac{\partial J^i}{\partial temp{h}^i}\ast \frac{\partial temp{h}^i}{\partial h^{i-1}}\ast \frac{\partial h^{i-1}}{\partial temp{h}^{i-1}}=\left(\frac{\partial J^i}{\partial temp{h}^i}\times u^T\right)\ast f^{\prime }\left(temp{h}^{i-1}\right)\end{array}$
$\begin{array}{l}\frac{\partial J^i}{\partial h^{i-2}}=\frac{\partial J^i}{\partial temp{h}^{i-1}}\ast \frac{\partial temp{h}^{i-1}}{\partial h^{i-2}}=\frac{\partial J^i}{\partial temp{h}^{i-1}}\times u^T\\ \frac{\partial J^i}{\partial temp{h}^{i-2}}=\frac{\partial J^i}{\partial temp{h}^{i-1}}\ast \frac{\partial temp{h}^{i-1}}{\partial h^{i-2}}\ast \frac{\partial h^{i-2}}{\partial temp{h}^{i-2}}=\left(\frac{\partial J^i}{\partial temp{h}^{i-1}}\times u^T\right)\ast f^{\prime }\left(temp{h}^{i-2}\right)\end{array}$
$\begin{array}{l}\frac{\partial J^i}{\partial h^{i-3}}=\frac{\partial J^i}{\partial temp{h}^{i-2}}\ast \frac{\partial temp{h}^{i-2}}{\partial h^{i-3}}=\frac{\partial J^i}{\partial temp{h}^{i-2}}\times u^T\\ \frac{\partial J^i}{\partial temp{h}^{i-3}}=\frac{\partial J^i}{\partial temp{h}^{i-2}}\ast \frac{\partial temp{h}^{i-2}}{\partial h^{i-3}}\ast \frac{\partial h^{i-3}}{\partial temp{h}^{i-3}}=\left(\frac{\partial J^i}{\partial temp{h}^{i-2}}\times u^T\right)\ast f^{\prime }\left(temp{h}^{i-3}\right)\end{array}$
可以归纳其通式：
$\begin{array}{l}\frac{\partial J^i}{\partial h^{i-k}}=\frac{\partial J^i}{\partial temp{h}^{i-k+1}}\times u^T\\ \frac{\partial J^i}{\partial temp{h}^{i-3}}=\left(\frac{\partial J^i}{\partial temp{h}^{i-k+1}}\times u^T\right)\ast f^{\prime }\left(temp{h}^{i-k}\right)\end{array}$
相应的对于参数u和v有：
$\begin{array}{l}\frac{\partial J^i}{\partial u}=\frac{\partial J^i}{\partial temp{h}^i}\times \frac{\partial temp{h}^i}{\partial u}={\left(h^{i-1}\right)}^T\times \frac{\partial J^i}{\partial temp{h}^i}\\ \frac{\partial J^i}{\partial v}=\frac{\partial J^i}{\partial temp{h}^i}\times \frac{\partial temp{h}^i}{\partial x^i}={\left(x^i\right)}^T\times \frac{\partial J^i}{\partial temp{h}^i}\end{array}$
$\begin{array}{l}\frac{\partial J^i}{\partial u}=\frac{\partial J^i}{\partial temp{h}^i}\ast \frac{\partial temp{h}^i}{\partial h^{i-1}}\ast \frac{\partial h^{i-1}}{\partial temp{h}^{i-1}}\times \frac{\partial temp{h}^{i-1}}{\partial u}={\left(h^{i-2}\right)}^T\times \frac{\partial J^i}{\partial temp{h}^{i-1}}\\ \frac{\partial J^i}{\partial v}=\frac{\partial J^i}{\partial temp{h}^i}\ast \frac{\partial temp{h}^i}{\partial h^{i-1}}\ast \frac{\partial h^{i-1}}{\partial temp{h}^{i-1}}\times \frac{\partial temp{h}^{i-1}}{\partial v}={\left(x^{i-1}\right)}^T\times \frac{\partial J^i}{\partial temp{h}^{i-1}}\end{array}$
$\begin{array}{l}\frac{\partial J^i}{\partial u}=\frac{\partial J^i}{\partial temp{h}^i}\ast \frac{\partial temp{h}^i}{\partial h^{i-1}}\ast \frac{\partial h^{i-1}}{\partial temp{h}^{i-1}}\ast \frac{\partial temp{h}^{i-1}}{\partial h^{i-2}}\ast \frac{\partial h^{i-1}}{\partial temp{h}^{i-2}}\times \frac{\partial temp{h}^{i-2}}{\partial u}={\left(h^{i-3}\right)}^T\times \frac{\partial J^i}{\partial temp{h}^{i-2}}\\ \frac{\partial J^i}{\partial v}=\frac{\partial J^i}{\partial temp{h}^i}\ast \frac{\partial temp{h}^i}{\partial h^{i-1}}\ast \frac{\partial h^{i-1}}{\partial temp{h}^{i-1}}\ast \frac{\partial temp{h}^{i-1}}{\partial h^{i-2}}\ast \frac{\partial h^{i-2}}{\partial temp{h}^{i-2}}\times \frac{\partial temp{h}^{i-2}}{\partial v}={\left(x^{i-2}\right)}^T\times \frac{\partial J^i}{\partial temp{h}^{i-2}}\end{array}$
通过将反向传播到前k层的链接权值u和v求和，得到最终的梯度结果：
$\begin{array}{l}\frac{\partial J^i}{\partial u}=\frac{\partial J^i}{\partial temp{h}^i}\ast \frac{\partial temp{h}^i}{\partial u}=\sum _{k=1}^i{\left(h^{k-1}\right)}^T\times \frac{\partial J^i}{\partial temp{h}^k}\\ \frac{\partial J^i}{\partial v}=\frac{\partial J^i}{\partial temp{h}^i}\ast \frac{\partial temp{h}^i}{\partial x^i}=\sum _{k=1}^i{\left(x^k\right)}^T\times \frac{\partial J^i}{\partial temp{h}^k}\end{array}$

Matlab实现代码

https://blog.csdn.net/vendetta_gg/article/details/106444683

参考文献

[1] https://zhuanlan.zhihu.com/p/26891871
[2] https://zhuanlan.zhihu.com/p/26892413
[3] https://zybuluo.com/hanbingtao/note/541458