手写BP神经网络

---

一、梯度下降的数学公式

神经网络是由一个一个神经元组成，其基本结构如下：

无数以链式结构组成在一起的神经元构成了神经网络，设置合理的$w$和$b$可以拟合任何数学模型，训练神经网络的本质就是利用梯度下降求解最佳$w$和$b$。由于神经网络颇为复杂，无法直接求解某个参数的梯度，所以主流的方法是采用反向传播($BP$)的方法一步步前推，具体详见深度学习笔记(1)——神经网络详解及改进。

1、$sigmoid$

$$a=\sigma (x)=\frac{1}{1+e^{-x}}$$
$sigmoid$作为最简单的激活函数，在$x$趋近于负无穷时，$y$趋近于0，$x$趋近于正无穷时，$y$趋近于1。

• 对$sigmoid$求导
  $$a^{\prime }=\sigma (x)(1-\sigma (x))$$
• 损失函数$l$对$z$求导：$\partial l/\partial z$
  $$\frac{\partial l}{\partial z_i}={\sigma }^{\prime }(z)\sum w_i\frac{\partial l}{\partial z_{i+1}}$$

def sigmoid(z, driv=False):    # driv 微分
    if(driv == True):   # 微分（输入z，返回dz）
        return sigmoid(z)*(1-sigmoid(z))
    return 1/(1+np.exp(-z))

2、$softmax$

$$a=\frac{e^{z_i}}{{\sum }_j^n{e}^{z_j}}$$
$softmax$是指数除以指数之和，一般作为分类任务的输出层，输出结果可以认为是几个类别选择的概率。
例如三分类问题，$z_1=3,z_2=1,z_3=-3$，经过$softmax$后$a_1=0.88,a_2=0.12,a_3=0$。所以通俗得讲$softmax$的作用就是对输出的最大值进行强化，并将其归一化至$[0,1]$之间。

• 对$softmax$求导
  如果$i=j$：
  $$\frac{\partial a_i}{\partial z_i}=a_i(1-a_i)$$
  如果$i\ne j$：
  $$\frac{\partial a_j}{\partial z_i}=-a_i{a}_j$$

• 损失函数$l$对$z$求导：$\partial l/\partial z$
  $$\frac{\partial l}{\partial z_i}=\frac{\partial l}{\partial a_j}\frac{\partial a_j}{\partial z_i}=a_i-y_i$$

def softmax(self, z=None, a=None, y=None, driv=False):  # softmax层
	if(driv==True): # 损失函数对z的微分
	    return a-y
	size = z.shape[0]
	log_out = np.exp(z)    # 对输出结果取ln
	log_sum = np.sum(log_out,axis=1).reshape(size,1)   # sum(e^output)
    return np.exp(z)/log_sum

二、代码编写

1、数据形式

• 训练集$x$:
  $x$是一个$size\ast dimension$的二维数组，$size$代表训练集$x$的数量，$dimension$代表训练集的维度。
  $x={\left[\begin{array}{cccc}{x}_{11}& x_{12}& ...& x_{1d}\\ x_{21}& x_{22}& ...& x_{2d}\\ ...& ...& & ...\\ x_{s1}& x_{s2}& ...& x_{sd}\end{array}\right]}_{size\ast dimension}$ ，其中$x_{ij}$代表第$i$组数据的第$j$个输入值。

• 神经元值$a$:
  $a$是一个$layers\ast size\ast units$的三维数组，$layers$代表神经网络层数，$size$代表训练集数量，$units$代表该层网络有几个神经元。
  $a[i]={\left[\begin{array}{cccc}{a}_{11}& a_{12}& ...& a_{1l}\\ a_{21}& a_{22}& ...& a_{2l}\\ ...& ...& & ...\\ a_{s1}& a_{s2}& ...& a_{sl}\end{array}\right]}_{size\ast layers}$，其中$a[i][j][k]$代表第$i$层网络，第$j$组数据，第$k$个神经元的值。例：$a[1]=x$，第1层网络就是输入层。

• 未经过激活函数的神经元$z$
  数据形式和$a$相同。

• 损失函数$l$对$z$的微分$dz$
  数据形式和$a$相同。
  $dz[i]={\left[\begin{array}{cccc}d{z}_{11}& d{z}_{12}& ...& d{z}_{1l}\\ d{z}_{21}& d{z}_{22}& ...& d{z}_{2l}\\ ...& ...& & ...\\ d{z}_{s1}& d{z}_{s2}& ...& d{z}_{sl}\end{array}\right]}_{size\ast layers}$

• 偏差$b$
  $b$是一个$layers\ast units$的二维数组，$b[i][j]$代表第$i$层网络第$j$个神经元上的偏差。

• 权重$w$
  $w$是一个$(layers-1)\ast units1\ast units2$的三维数组，$w[i]$表示第$i$层和第$i+1$层之间的权值，$units1$是第$i$层的神经元个数，$units2$是第$i+1$层的神经元个数。
  $w[i]={\left[\begin{array}{cccc}{w}_{11}& w_{12}& ...& w_{1u_2}\\ w_{21}& w_{22}& ...& w_{2u_2}\\ ...& ...& & ...\\ w_{u_{1}1}& w_{u_{1}2}& ...& w_{u_1{u}_2}\end{array}\right]}_{units1\ast units2}$，$w[i][j][k]$代表第$i$层第$j$个神经元和第$i+1$层第$k$个神经元之间的权重。

1、前向传播

比较简单,每一层的值等于前一层的值矩阵乘权重加上$bias$
$$z_{i+1}=a_i\ast w_i+b_i$$

def ForwardPropagation(x_train, num_layers): # 神经网络的结果
	'''
	@param num_layers:网络层数
	'''
    z = {}      # 未经过激活函数的神经元 [第x个层][第y个数据, 第z个神经元]
    a = {}      # 经过激活函数的神经元 [第x个层][第y个数据, 第z个神经元]
    a[0] = x_train
    for i in range(self.num_layers-2):
        z[i+1] = a[i]@self.w[i]+self.b[i]
        a[i+1] = sigmoid(z[i+1])
    # softmax层
    z[self.num_layers-1] = a[self.num_layers-2]@self.w[self.num_layers-2]+self.b[self.num_layers-2]
    a[self.num_layers-1] = softmax(z[self.num_layers-1])
    return z,a

2、反向传播

def BackPropagation(z, a, y_train):   # bp计算Loss对z的偏微分
    dz = {}     # dl/dz [第x个层][第y个数据, 第z个神经元]
    # softmax层
    dz[self.num_layers-1] = self.softmax(a[self.num_layers-1],y_train,driv=True)/y_train.shape[0]      
    # 隐藏层
    for i in range(self.num_layers-2,0,-1):
        dz[i] = dz[i+1]@self.w[i].T*self.sigmoid(z[i],driv=True)  
    return dz  

3、计算$dw$和$db$

def cal_driv(dz,a):   # 计算dw和db
	dw =0
	db = np.mean(dz,axis=0)
	for i in range(dz.shape[0]):
	    dw = dw+a[i].reshape(len(a[i]),1)@dz[i].reshape(1,len(dz[i]))
	dw = dw/a.shape[0]
	return dw,db

4、梯度下降

def gradient_decent(x_train, y_train):
    size = x_train.shape[0]
    z,a = ForwardPropagation(x_train)     # 进行一次前馈运算
    dz = BackPropagation(z,a,y_train)
    # 对 w,b进行梯度下降
    for i in range(self.num_layers-1):
        dw,db = cal_driv(dz[i+1],a[i])
        self.w[i] = self.w[i] - dw*learning_rate
        self.b[i] = self.b[i] - db*learning_rate

5、代码总览

import numpy as np

class BP_network():
    def __init__(self,input_dim, hidden_layer, output_dim):
        self.input_dim = input_dim
        self.output_dim = output_dim
        self.w = {}                     # 利用字典存储 w    
        self.b = {}                     # 利用字典存储 bias
        self.num_layers = len(hidden_layer)+2   # 网络层数

        hidden_layer.insert(0,input_dim)
        hidden_layer.append(output_dim)
        for i in range(len(hidden_layer)-1):
            self.w[i] = np.random.rand(hidden_layer[i],hidden_layer[i+1])
            self.b[i] = np.zeros((1,hidden_layer[i+1]))

    def sigmoid(self, z, driv=False):    # driv 微分
        if(driv == True):   # 微分（输入z，返回dz）
            return self.sigmoid(z)*(1-self.sigmoid(z))
        return 1/(1+np.exp(-z))
    
    def softmax(self, z=None, a=None, y=None, driv=False):  # softmax层
        if(driv==True): # loss函数对z微分
            return a-y
        size = z.shape[0]
        log_out = np.exp(z)    # 对输出结果取ln
        log_sum = np.sum(log_out,axis=1).reshape(size,1)   # sum(e^output)
        return np.exp(z)/log_sum

    def fit(self, x_train, y_train, batch_size, learning_rate=0.01, epochs=20):
        def Loss(output, y_train): # 交叉熵
            return -np.sum(y_train*np.log(output))/output.shape[0]

        def ForwardPropagation(x_train): # 神经网络的结果
            z = {}      # 未经过激活函数的神经元 [第x个层][第y个数据, 第z个神经元]
            a = {}      # 经过激活函数的神经元 [第x个层][第y个数据, 第z个神经元]
            a[0] = x_train
            for i in range(self.num_layers-2):
                z[i+1] = a[i]@self.w[i]+self.b[i]
                a[i+1] = self.sigmoid(z[i+1])
            # softmax层
            z[self.num_layers-1] = a[self.num_layers-2]@self.w[self.num_layers-2]+self.b[self.num_layers-2]
            a[self.num_layers-1] = self.softmax(z[self.num_layers-1])
            return z,a

        def BackPropagation(z, a, y_train):   # bp计算Loss对z的偏微分
            dz = {}     # dl/dz [第x个层][第y个数据, 第z个神经元]
            # softmax层
            dz[self.num_layers-1] = self.softmax(a=a[self.num_layers-1],y=y_train,driv=True)/y_train.shape[0]      
            # 隐藏层
            for i in range(self.num_layers-2,0,-1):
                dz[i] = dz[i+1]@self.w[i].T*self.sigmoid(z[i],driv=True)  
            return dz  

        def cal_driv(dz,a):   # 计算dw和db
            dw = 0
            db = np.mean(dz,axis=0)
            for i in range(dz.shape[0]):
                dw = dw+a[i].reshape(len(a[i]),1)@dz[i].reshape(1,len(dz[i]))
            dw = dw/a.shape[0]
            return dw,db

        def gradient_decent(x_train, y_train):
            size = x_train.shape[0]
            z,a = ForwardPropagation(x_train)     # 进行一次前馈运算
            dz = BackPropagation(z,a,y_train)
            # 对 w,b进行梯度下降
            for i in range(self.num_layers-1):
                dw,db = cal_driv(dz[i+1],a[i])
                self.w[i] = self.w[i] - dw*learning_rate
                self.b[i] = self.b[i] - db*learning_rate

        size = x_train.shape[0]  # 数据集数量
        for i in range(epochs):
			for j in range(0,size,batch_size):
  			    gradient_decent(x_train[j:j+batch_size],y_train[j:j+batch_size])

            z,a = ForwardPropagation(x_train)
            loss = Loss(a[self.num_layers-1], y_train)   # 计算loss
            accuracy = self.accuracy(self.predict(x_train),y_train)
            print('第%d个epochs: Loss=%f，accuracy:%f' %(i+1,loss,accuracy))

    def predict(self, x_train):
        z = {}      # 未经过激活函数的神经元 [第x个层][第y个数据, 第z个神经元]
        a = {}      # 经过激活函数的神经元 [第x个层][第y个数据, 第z个神经元]
        a[0] = x_train
        for i in range(self.num_layers-2):
            z[i+1] = a[i]@self.w[i]+self.b[i]
            a[i+1] = self.sigmoid(z[i+1])
        # softmax层
        z[self.num_layers-1] = a[self.num_layers-2]@self.w[self.num_layers-2]+self.b[self.num_layers-2]
        a[self.num_layers-1] = self.softmax(z[self.num_layers-1])
        y = np.zeros(a[self.num_layers-1].shape)
        y[a[self.num_layers-1]-a[self.num_layers-1].max(axis=1).reshape(a[self.num_layers-1].shape[0],1)>=0]=1
        return y
        
    def accuracy(self,y,y_true):
        return np.sum(np.where(y==1)[1]==np.where(y_true==1)[1])/y.shape[0]

6、代码测试

准备数据集(简单的三分类数据集)：

x_train = np.random.randint(0,100,size=(100,2))
temp = (2*x_train[:,0]+x_train[:,1])
y_train = np.zeros((100,3))
y_train[temp>150,0] = 1
y_train[temp<50,1] = 1
y_train[((temp<=150) & (temp>=50)),2] = 1
x_train = (x_train-x_train.min(axis=0))/(x_train.max(axis=0)-x_train.min(axis=0))

训练网络

bp = BP_network(input_dim=2, hidden_layer=[5,5], output_dim=3)
bp.fit(x_train,y_train,batch_size=10,learning_rate=0.1,epochs=2000)

结果

第1个epochs: Loss=1.053458，accuracy:0.390000
第2个epochs: Loss=1.005622，accuracy:0.390000
第3个epochs: Loss=0.968565，accuracy:0.390000
第4个epochs: Loss=0.939970，accuracy:0.390000
第5个epochs: Loss=0.917880，accuracy:0.500000
第6个epochs: Loss=0.900729，accuracy:0.570000
第7个epochs: Loss=0.887299，accuracy:0.570000
第8个epochs: Loss=0.876668，accuracy:0.570000
第9个epochs: Loss=0.868147，accuracy:0.570000
第10个epochs: Loss=0.861226，accuracy:0.570000
...
第1997个epochs: Loss=0.561577，accuracy:0.830000
第1998个epochs: Loss=0.561012，accuracy:0.830000
第1999个epochs: Loss=0.560447，accuracy:0.830000
第2000个epochs: Loss=0.559881，accuracy:0.840000

代码正常运行