CS231n课程作业(一) Two-layer Neural Network

神经网络的过程主要就是forward propagation和backward propagation。
forward propagation to evaluate score function & loss function, then back propagation 对每一层计算loss对W和b的梯度，利用梯度完成W和b的更新。
总体过程可以理解为：forward–>backward–>update–>forward–>backward–>update……

一、理论知识

1. 直观理解

这是一个两层的full-connected神经网络，分别是输入层、隐藏层和输出层(输入层不算一层)。输入层4个节点表示样本是4维的，输出层3个节点表示有3类，输出的结果是每个类的score。

FC层(全连接层)：每个神经元都连接上一层的所有神经元的layer。

在神经网络中，bigger = better，即网络模型越大越好。more neurons = more capacity，因为机器学习中经常出现模型表达能力不足的情况。

对于神经网络的更形象理解：

2. score function

这里activation function采用的是ReLU函数。
Why we set activation function？
answer：如果不设置activation function，则每层的output都是input的线性函数，则无论隐藏层有多少层，都与没有隐藏层效果一样。因此，设置activation function，目的是为了对input进行非线性转化，然后output，使神经网络更有意义。

3. loss function及求梯度

本次作业中使用的是softmax loss function。可参考softmax classifier。
完成作业的关键仍是loss function对W求梯度。下面求gradient(注意：我在角标表示上有错误，但不影响理解和代码化)：

这样，score、loss和gradient都表达出来了，2层神经网络也就可以完成了。

二、Two-layer Neural Network

1. Forward Propagation & Backward Propagation

Forward: 计算score，再根据score计算loss
Backward：分别对W2、b2、W1、b1求梯度

  def loss(self, X, y=None, reg=0.0):
    """
    Compute the loss and gradients for a two layer fully connected neural
    network.

    Inputs:
    - X: Input data of shape (N, D). Each X[i] is a training sample.
    - y: Vector of training labels. y[i] is the label for X[i], and each y[i] is
      an integer in the range 0 <= y[i] < C. This parameter is optional; if it
      is not passed then we only return scores, and if it is passed then we
      instead return the loss and gradients.
    - reg: Regularization strength.

    Returns:
    If y is None, return a matrix scores of shape (N, C) where scores[i, c] is
    the score for class c on input X[i].

    If y is not None, instead return a tuple of:
    - loss: Loss (data loss and regularization loss) for this batch of training
      samples.
    - grads: Dictionary mapping parameter names to gradients of those parameters
      with respect to the loss function; has the same keys as self.params.
    """
    # Unpack variables from the params dictionary
    W1, b1 = self.params['W1'], self.params['b1']
    W2, b2 = self.params['W2'], self.params['b2']
    N, D = X.shape

    # Compute the forward pass
    scores = None
    #############################################################################
    # TODO: Perform the forward pass, computing the class scores for the input. #
    # Store the result in the scores variable, which should be an array of      #
    # shape (N, C).                                                             #
    #############################################################################
    h1 = np.maximum(0, np.dot(X,W1) + b1) #(5,10)
    #print (h1.shape)
    scores = np.dot(h1,W2) + b2 # (5,3)
    #print (scores.shape)
    #############################################################################
    #                              END OF YOUR CODE                             #
    #############################################################################

    # If the targets are not given then jump out, we're done
    if y is None:
      return scores

    # Compute the loss
    loss = None
    #############################################################################
    # TODO: Finish the forward pass, and compute the loss. This should include  #
    # both the data loss and L2 regularization for W1 and W2. Store the result  #
    # in the variable loss, which should be a scalar. Use the Softmax           #
    # classifier loss.                                                          #
    #############################################################################
    exp_S = np.exp(scores) #(5,3)
    sum_exp_S = np.sum(exp_S,axis = 1) 
    sum_exp_S = sum_exp_S.reshape(-1,1) #(5,1)
    #print (sum_exp_S.shape)
    loss = np.sum(-scores[range(N),list(y)]) + sum(np.log(sum_exp_S))
    loss = loss / N + 0.5 * reg * np.sum(W1 * W1) +  0.5 * reg * np.sum(W2 * W2)
    #print (loss)
    #############################################################################
    #                              END OF YOUR CODE                             #
    #############################################################################

    # Backward pass: compute gradients
    grads = {}
    #############################################################################
    # TODO: Compute the backward pass, computing the derivatives of the weights #
    # and biases. Store the results in the grads dictionary. For example,       #
    # grads['W1'] should store the gradient on W1, and be a matrix of same size #
    #############################################################################
    '''
    网络为4-10-3
    对于第2层：input为h1(5,10)
              W2:(10,3)
              output为score:(5,3)
    对于第1层：input为X(5,4)
                W1(5,10)
                output为h1(5,10)
    '''
    #---------------------------------#
    dscores = np.zeros(scores.shape)
    dscores[range(N),list(y)] = -1
    dscores += (exp_S/sum_exp_S) #(5,3) 
    dscores /= N
    grads['W2'] = np.dot(h1.T, dscores)
    grads['W2'] += reg * W2
    grads['b2'] = np.sum(dscores, axis = 0)
    #---------------------------------#
    dh1 = np.dot(dscores, W2.T)  #(5,10)
    dh1_ReLU = (h1>0) * dh1
    grads['W1'] = X.T.dot(dh1_ReLU) + reg * W1
    grads['b1'] = np.sum(dh1_ReLU, axis = 0)
    #---------------------------------#
    '''
    本人之前的写法：
    缺点：无法很少的利用中间变量;
          逻辑上可读性差
    buf = np.zeros(scores.shape)
    buf[range(N),list(y)] = -1
    #print (buf+(exp_S/sum_exp_S))
    grads['W2'] = np.dot(h1.T,(buf + (exp_S/sum_exp_S))) #(10,3)
    grads['W2'] = grads['W2']/N + reg * W2
    grads['b2'] = np.sum((exp_S/sum_exp_S)+buf, axis = 0) #(3,)

    '''
    #############################################################################
    #                              END OF YOUR CODE                             #
    #############################################################################

    return loss, grads

2. Gradient Check

首先计算numeric gradient：

def eval_numerical_gradient(f, x, verbose=True, h=0.00001):
  """ 
  a naive implementation of numerical gradient of f at x 
  - f should be a function that takes a single argument
  - x is the point (numpy array) to evaluate the gradient at
  """ 

  fx = f(x) # evaluate function value at original point
  # x: 权重W1,W2,b2,b1
  grad = np.zeros_like(x)
  # iterate over all indexes in x
  it = np.nditer(x, flags=['multi_index'], op_flags=['readwrite']) #构建一个迭代器
  while not it.finished:

    # evaluate function at x+h
    ix = it.multi_index #输出结果为元素的索引
    oldval = x[ix] #权重中的元素值
    x[ix] = oldval + h # increment by h
    fxph = f(x) # evalute f(x + h)
    x[ix] = oldval - h
    fxmh = f(x) # evaluate f(x - h)
    x[ix] = oldval # restore

    # compute the partial derivative with centered formula
    grad[ix] = (fxph - fxmh) / (2 * h) # the slope
    if verbose:
      print(ix, grad[ix]) #输出数值梯度
    it.iternext() # step to next dimension，如果不加这句话，则始终保持在(0,0)

  return grad #返回的是数值梯度

然后check：

from cs231n.gradient_check import eval_numerical_gradient

# Use numeric gradient checking to check your implementation of the backward pass.
# If your implementation is correct, the difference between the numeric and
# analytic gradients should be less than 1e-8 for each of W1, W2, b1, and b2.

loss, grads = net.loss(X, y, reg=0.1)
#print (loss)
#print (grads)
# these should all be less than 1e-8 or so
for param_name in grads:
    f = lambda W: net.loss(X, y, reg=0.1)[0] #input：W; output：loss
    param_grad_num = eval_numerical_gradient(f, net.params[param_name], verbose=False) #返回数值梯度
    #print (np.abs(param_grad_num - grads[param_name]))
    print('%s max relative error: %e' % (param_name, rel_error(param_grad_num, grads[param_name])))

3. Train a Network

既然forward和backward都已经完成了，且gradient检查通过，接下来就是训练出一个较好的模型。
基本迭代过程是这样的：forward–>backward–>update–>forward–>backward–>update……

  def train(self, X, y, X_val, y_val,
            learning_rate=1e-3, learning_rate_decay=0.95,
            reg=5e-6, num_iters=100,
            batch_size=200, verbose=False):
    """
    Train this neural network using stochastic gradient descent.

    Inputs:
    - X: A numpy array of shape (N, D) giving training data.
    - y: A numpy array f shape (N,) giving training labels; y[i] = c means that
      X[i] has label c, where 0 <= c < C.
    - X_val: A numpy array of shape (N_val, D) giving validation data.
    - y_val: A numpy array of shape (N_val,) giving validation labels.
    - learning_rate: Scalar giving learning rate for optimization.
    - learning_rate_decay: Scalar giving factor used to decay the learning rate
      after each epoch.
    - reg: Scalar giving regularization strength.
    - num_iters: Number of steps to take when optimizing.
    - batch_size: Number of training examples to use per step.
    - verbose: boolean; if true print progress during optimization.
    """
    num_train = X.shape[0]
    iterations_per_epoch = max(num_train / batch_size, 1)

    # Use SGD to optimize the parameters in self.model
    loss_history = []
    train_acc_history = []
    val_acc_history = []

    for it in xrange(num_iters):
      X_batch = None
      y_batch = None

      #########################################################################
      # TODO: Create a random minibatch of training data and labels, storing  #
      # them in X_batch and y_batch respectively.                             #
      #########################################################################
      mask = np.random.choice(num_train,batch_size,replace = True)
      X_batch = X[mask]
      y_batch = y[mask]
      #########################################################################
      #                             END OF YOUR CODE                          #
      #########################################################################

      # Compute loss and gradients using the current minibatch
      loss, grads = self.loss(X_batch, y=y_batch, reg=reg)
      loss_history.append(loss)

      #########################################################################
      # TODO: Use the gradients in the grads dictionary to update the         #
      # parameters of the network (stored in the dictionary self.params)      #
      # using stochastic gradient descent. You'll need to use the gradients   #
      # stored in the grads dictionary defined above.                         #
      #########################################################################
      self.params['W1'] += -learning_rate * grads['W1']
      self.params['b1'] += -learning_rate * grads['b1']
      self.params['W2'] += -learning_rate * grads['W2']
      self.params['b2'] += -learning_rate * grads['b2']
      #########################################################################
      #                             END OF YOUR CODE                          #
      #########################################################################

      if verbose and it % 100 == 0:
        print('iteration %d / %d: loss %f' % (it, num_iters, loss))

      # Every epoch, check train and val accuracy and decay learning rate.
      if it % iterations_per_epoch == 0:
        # Check accuracy
        #print ('第%d个epoch' %it)
        train_acc = (self.predict(X_batch) == y_batch).mean()
        val_acc = (self.predict(X_val) == y_val).mean()
        train_acc_history.append(train_acc)
        val_acc_history.append(val_acc)

        # Decay learning rate
        learning_rate *= learning_rate_decay #减小学习率

    return {
      'loss_history': loss_history,
      'train_acc_history': train_acc_history,
      'val_acc_history': val_acc_history,
    }

训练完模型，即可以对未知样本做出prediction。

三、涉及到的numpy重要函数

1. np.random.seed(a)
function: 选定固定的随机数，a为任意数字(如0)。即后面每次产生的随机数都固定。

2. np.nditer(array, flags=[‘multi_index’], op_flags=[‘readwrite’])
function: 构建一个迭代器。
参数1：数组; 参数2：多重索引; 参数3：可读写
本作业中，用于输出数组的index，从而进行数值梯度的计算。