神经网络的代价函数到底怎么在算

下面是就是神经网络中代价函数$J(\Theta )$的表达式，看起来还是稍微有点复杂。这个表达式到底在计算什么？下面我们先用一个简单的例子来分开一步步计算一下。

$$J(\Theta )=-\frac{1}{m}\sum _{i=1}^m\sum _{k=1}^K\left[y_k^{(i)}\log ((h_{\Theta }(x^{(i)}){)}_k)+(1-y_k^{(i)})\log (1-(h_{\Theta }(x^{(i)}){)}_k)\right]+\frac{\lambda }{2m}\sum _{l=1}^{L-1}\sum _{i=1}^{s_l}\sum _{j=1}^{s_{l+1}}({\Theta }_{j,i}^{(l)}{)}^2$$

有如下神经网络：

其中：
$$\begin{array}{rl}L& =\text{神经网络总共包含的层数}\\ s_l& =\text{第l层的神经元数目}\\ K& =\text{输出层的神经元数，亦即分类的数目}\end{array}$$
假设$s_1=3,s_2=2,s_3=3$，则${\Theta }^1$的维度为$2\times 4$，${\Theta }^2$的维度为$3\times 3$。

则有：

$$X^T=\left[\begin{array}{c}1\\ x_1\\ x_2\\ x_3\end{array}\right],{\Theta }^1={\left[\begin{array}{cccc}{\theta }_{10}^1& {\theta }_{11}^1& {\theta }_{12}^1& {\theta }_{13}^1\\ {\theta }_{20}^1& {\theta }_{21}^1& {\theta }_{22}^1& {\theta }_{23}^1\end{array}\right]}_{2\times 4},{\Theta }^2={\left[\begin{array}{ccc}{\theta }_{10}^2& {\theta }_{11}^2& {\theta }_{12}^2\\ {\theta }_{20}^2& {\theta }_{21}^2& {\theta }_{22}^2\\ {\theta }_{30}^2& {\theta }_{31}^2& {\theta }_{32}^2\end{array}\right]}_{3\times 3}$$

先回忆一下正向传播的计算公式：
$$\begin{gathered} \begin{array}{r}{z}^{(j)}={\Theta }^{(j-1)}{a}^{(j-1)}\dots \dots (1) \\ {a}^{(j)}=g(z^{(j)}),setting{a}_0^{(j)}=1\dots \dots (2) \\ {h}_{\Theta }(x)=a^{(j)}=g(z^{(j)})\dots \dots (3)\end{array} \end{gathered}$$

详解戳此处

此时我们先忽略 regularized term

①当m=1时；
$$J(\Theta )=-\frac{1}{m}\sum _{k=1}^K\left[y_k^{(i)}\log ((h_{\Theta }(x^{(i)}){)}_k)+(1-y_k^{(i)})\log (1-(h_{\Theta }(x^{(i)}){)}_k)\right]$$

$$\begin{array}{r}1.令a^1=X^T;⟹z^2={\Theta }^1\ast a^1={\left[\begin{array}{cccc}{\theta }_{10}^1& {\theta }_{11}^1& {\theta }_{12}^1& {\theta }_{13}^1\\ {\theta }_{20}^1& {\theta }_{21}^1& {\theta }_{22}^1& {\theta }_{23}^1\end{array}\right]}_{2\times 4}\times \left[\begin{array}{c}1\\ x_1\\ x_2\\ x_3\end{array}\right]={\left[\begin{array}{c}{\theta }_{10}^1+{\theta }_{11}^1\cdot x_1+{\theta }_{12}^1\cdot x_2+{\theta }_{13}^1\cdot x_3\\ {\theta }_{20}^1+{\theta }_{21}^1\cdot x_1+{\theta }_{22}^1\cdot x_2+{\theta }_{23}^1\cdot x_3\end{array}\right]}_{2\times 1}\end{array}$$

$$=\left[\begin{array}{c}{z}_1^2\\ z_2^2\end{array}\right],⟹a^2=g(z^2);$$

$$\begin{array}{r}\text{ 2.给}{a}^2\text{添加偏置项，并计算}{a}^3\text{即}{h}_{\theta }(x);a^2=\left[\begin{array}{c}1\\ a_1^2\\ a_2^2\end{array}\right];⟹z^3={\Theta }^2\ast a^2={\left[\begin{array}{ccc}{\theta }_{10}^2& {\theta }_{11}^2& {\theta }_{12}^2\\ {\theta }_{20}^2& {\theta }_{21}^2& {\theta }_{22}^2\\ {\theta }_{30}^2& {\theta }_{31}^2& {\theta }_{32}^2\end{array}\right]}_{3\times 3}\times \left[\begin{array}{c}1\\ a_1^2\\ a_2^2\end{array}\right]=\left[\begin{array}{c}{z}_1^3\\ z_2^3\\ z_3^3\end{array}\right];\end{array}$$

$$⟹h_{\theta }(x)=a^3=g(z^3)=\left[\begin{array}{c}g(z_1^3)\\ g(z_2^3)\\ g(z_3^3)\end{array}\right]=\left[\begin{array}{c}h(x{)}_1\\ h(x{)}_2\\ h(x{)}_3)\end{array}\right]$$

此时我们知道，对于每一个example，最终都会输出3个结果，那么这时代价函数所做的就是将这3个输出取对数然后乘以对应的预期期望值y之后，再累加起来。具体如下：

假设 $$input:X^T=\left[\begin{array}{c}1\\ x_1\\ x_2\\ x_3\end{array}\right];output:y=\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right]=\left[\begin{array}{c}{y}_1\\ y_2\\ y_3\end{array}\right]$$

则有：
$$\begin{array}{rl}J(\Theta )\ast m& =[-y_1\times log(h(x{)}_1)-(1-y_1)\times log(1-h(x{)}_1)]\\ & +[-y_2\times log(h(x{)}_2)-(1-y_2)\times log(1-h(x{)}_2)]\\ & +[-y_3\times log(h(x{)}_3)-(1-y_3)\times log(1-h(x{)}_3)]\\ & =[-1\times log(h(x{)}_1)-(1-1)\times log(1-h(x{)}_1)]\\ & +[-0\times log(h(x{)}_2)-(1-0)\times log(1-h(x{)}_2)]\\ & +[-0\times log(h(x{)}_3)-(1-0)\times log(1-h(x{)}_3)]\\ & =-log(h(x{)}_1)-log(1-h(x{)}_2)-log(1-h(x{)}_3)\end{array}$$

在matlab中，矢量化之后的代价函数为：
$$J(\Theta )=(1/m)\ast (sum(-labe{l}_Y.\ast log(H_{\theta })-(1-labe{l}_Y).\ast log(1-H_{\theta })));$$

---

---

②当m>1时；

$$J(\Theta )=-\frac{1}{m}\sum _{i=1}^m\sum _{k=1}^K\left[y_k^{(i)}\log ((h_{\Theta }(x^{(i)}){)}_k)+(1-y_k^{(i)})\log (1-(h_{\Theta }(x^{(i)}){)}_k)\right]$$
此时，对于每一个example都会产生一个上面的代价，所以只需要把所有的对于每一个example产生的代价累加起来即可。

再来分解一下：

$$假设，X=\left[\begin{array}{cccc}1& x_1^1& x_2^1& x_3^1\\ 1& x_1^2& x_2^2& x_3^2\\ 1& x_1^3& x_2^3& x_3^3\end{array}\right],$$

$$1.令a^1=X^T;⟹z^2={\Theta }^1\ast a^1={\left[\begin{array}{cccc}{\theta }_{10}^1& {\theta }_{11}^1& {\theta }_{12}^1& {\theta }_{13}^1\\ {\theta }_{20}^1& {\theta }_{21}^1& {\theta }_{22}^1& {\theta }_{23}^1\end{array}\right]}_{2\times 4}\times {\left[\begin{array}{ccc}1& 1& 1\\ x_1^1& x_1^2& x_1^3\\ x_2^1& x_2^2& x_2^3\\ x_3^1& x_3^2& x_3^3\end{array}\right]}_{4\times 3}=$$

$${\left[\begin{array}{ccc}{\theta }_{10}^1+{\theta }_{11}^1\cdot x_1^1+{\theta }_{12}^1\cdot x_2^1+{\theta }_{13}^1\cdot x_3^1& {\theta }_{10}^1+{\theta }_{11}^1\cdot x_1^2+{\theta }_{12}^1\cdot x_2^2+{\theta }_{13}^1\cdot x_3^2& {\theta }_{10}^1+{\theta }_{11}^1\cdot x_1^3+{\theta }_{12}^1\cdot x_2^3+{\theta }_{13}^1\cdot x_3^3\\ {\theta }_{20}^1+{\theta }_{21}^1\cdot x_1^1+{\theta }_{22}^1\cdot x_2^1+{\theta }_{23}^1\cdot x_3^1& {\theta }_{20}^1+{\theta }_{21}^1\cdot x_1^2+{\theta }_{22}^1\cdot x_2^2+{\theta }_{23}^1\cdot x_3^2& {\theta }_{20}^1+{\theta }_{21}^1\cdot x_1^3+{\theta }_{22}^1\cdot x_2^3+{\theta }_{23}^1\cdot x_3^3\end{array}\right]}_{2\times 3}$$

$$={\left[\begin{array}{ccc}{z}_{11}^2& z_{12}^2& z_{13}^2\\ z_{21}^2& z_{22}^2& z_{23}^2\end{array}\right]}_{2\times 3},⟹a^2=g(z^2);$$

$\text{ 2.给}{a}^2\text{添加偏置项，并计算}{a}^3\text{即}{h}_{\theta }(x)$;

$$a^2={\left[\begin{array}{ccc}1& 1& 1\\ a_{11}^2& a_{12}^2& a_{13}^2\\ a_{21}^2& a_{22}^2& a_{23}^2\end{array}\right]}_{3\times 3};⟹z^3={\Theta }^2\ast a^2={\left[\begin{array}{ccc}{\theta }_{10}^2& {\theta }_{11}^2& {\theta }_{12}^2\\ {\theta }_{20}^2& {\theta }_{21}^2& {\theta }_{22}^2\\ {\theta }_{30}^2& {\theta }_{31}^2& {\theta }_{32}^2\end{array}\right]}_{3\times 3}\times {\left[\begin{array}{ccc}1& 1& 1\\ a_{11}^2& a_{12}^2& a_{13}^2\\ a_{21}^2& a_{22}^2& a_{23}^2\end{array}\right]}_{3\times 3}$$

$$⟹h_{\theta }(x)=a^3=g(z^3)=\left[\begin{array}{ccc}g(z_{11}^3)& g(z_{12}^3& g(z_{13}^3))\\ g(z_{21}^3)& g(z_{22}^3& g(z_{23}^3))\\ g(z_{31}^3)& g(z_{32}^3)& g(z_{33}^3)\end{array}\right]$$

$$=\left[\begin{array}{cccc}第一个样本点对应的所有输出；& 第二个样本点& 第三个时；& \\ h(x_1{)}_1& h(x_2{)}_1& h(x_3{)}_1& \\ h(x_1{)}_2& h(x_2{)}_2& h(x_3{)}_2& \\ h(x_1{)}_3& h(x_2{)}_3& h(x_3{)}_3& \end{array}\right]$$

假设 $$input:X=\left[\begin{array}{cccc}1& x_1^1& x_2^1& x_3^1\\ 1& x_1^2& x_2^2& x_3^2\\ 1& x_1^3& x_2^3& x_3^3\end{array}\right]；output:y=\left[\begin{array}{c}1\\ 2\\ 2\end{array}\right]=\left[\begin{array}{c}{y}_1\\ y_2\\ y_3\end{array}\right]$$

该例子的背景为用神经网络识别手写体，即$y_1=1\text{表示期望输出为1,}{y}_2=y_3=2\text{,表示其期望输出为2。在计算代价函数的时候要将其每一个对应的输出转换为只含有0，1的向量}$

则有:
$$y_1=\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right];y_2=\left[\begin{array}{c}0\\ 1\\ 0\end{array}\right];y_3=\left[\begin{array}{c}0\\ 1\\ 0\end{array}\right]⟹labe{l}_Y=\left[\begin{array}{ccc}m=1& m=2& m=3\\ 1& 0& 0\\ 0& 1& 1\\ 0& 0& 0\end{array}\right]$$

对于如何将普通的输出值转换成只含有0,1的向量，戳此处

则有（Malab中的矢量化形式）：
$$J(\Theta )=(1/m)\ast (sum(sum[-labe{l}_Y.\ast log(H_{\theta })-(1-labe{l}_Y).\ast log(1-H_{\theta })]));$$

**加上 regularized term**

$$regular=\frac{\lambda }{2m}\sum _{l=1}^{L-1}\sum _{i=1}^{s_l}\sum _{j=1}^{s_{l+1}}({\Theta }_{j,i}^{(l)}{)}^2;$$

其实regularized term 就是所有每一层的参数$({\Theta }_{j,i}^l,j̸=0\text{,即除了每一层的第一列偏置项所对应的参数})$的平方和相加即可。

具体到本文的例子就是：

$${\Theta }^1={\left[\begin{array}{cccc}{\theta }_{10}^1& {\theta }_{11}^1& {\theta }_{12}^1& {\theta }_{13}^1\\ {\theta }_{20}^1& {\theta }_{21}^1& {\theta }_{22}^1& {\theta }_{23}^1\end{array}\right]}_{2\times 4},{\Theta }^2={\left[\begin{array}{ccc}{\theta }_{10}^2& {\theta }_{11}^2& {\theta }_{12}^2\\ {\theta }_{20}^2& {\theta }_{21}^2& {\theta }_{22}^2\\ {\theta }_{30}^2& {\theta }_{31}^2& {\theta }_{32}^2\end{array}\right]}_{3\times 3}$$

$$\begin{array}{rl}regular& =({\theta }_{11}^1{)}^2+({\theta }_{12}^1{)}^2+({\theta }_{13}^1{)}^2+({\theta }_{21}^1{)}^2+({\theta }_{22}^1{)}^2+({\theta }_{23}^1{)}^2\\ & +({\theta }_{11}^2{)}^2+({\theta }_{12}^2{)}^2+({\theta }_{21}^2{)}^2+({\theta }_{22}^2{)}^2+({\theta }_{31}^2{)}^2+({\theta }_{32}^2{)}^2\end{array}$$

Matlab中矢量化为：

  s_Theta1 = sum(Theta1 .^ 2);%先求所有元素的平方，然后再每一列相加
  r_Theta1 = sum(s_Theta1)-s_Theta1(1,1);%减去第一列的和
  
  s_Theta2 = sum(Theta2 .^ 2);
  r_Theta2 = sum(s_Theta2)-s_Theta2(1,1);
  
  
  regular = (lambda/(2*m))*(r_Theta1+r_Theta2);