这次作业其实总的来说花费了很长时间,主要是自己不能集中去写代码,第二是基础知识很多不扎实,很多需要查,但是我查也不是深究,我就简单记录一下用法,主要还是需要多用。每次都小结一下。
s["dW" + str(l+1)] = beta2 * s["dW" + str(l+1)] + (1-beta2)* np.square(grads["dW" + str(l+1)])
#s["db" + str(l+1)] = beta2 * s["db" + str(l+1)] + (1-beta2)* math.pow(grads["db" + str(l+1)],2) 错啦
v_corrected["dW" + str(l+1)] = v["dW" + str(l+1)] / (1 - np.power(beta1,t))
#v_corrected["db" + str(l+1)] = v["db" + str(l+1)] / (1 - math.pow(beta1,l)) 错啦
s["dW"+str(l+1)]=np.zeros((parameters["W"+str(l+1)].shape[0],parameters["W"+str(l+1)].shape[1]))
s["db" + str(l+1)] = np.zeros_like(parameters['b'+ str(l+1)])
numpy中的ravel()、flatten()、squeeze()的用法与区别 https://blog.csdn.net/tymatlab/article/details/79009618
参见官方文档:
Python Numpy模块函数np.c_和np.r_学习使用 https://blog.csdn.net/Together_CZ/article/details/79548217
Lets use the following "moons" dataset to test the different optimization methods. (The dataset is named "moons" because the data from each of the two classes looks a bit like a crescent-shaped moon.)
train_X, train_Y = load_dataset()
print(train_X.shape[0],train_X.shape[1])
print(train_X[:,0])
print(train_X[0,:])
print(train_Y.shape[0],train_Y.shape[1])
print(train_Y)
print(train_Y.flatten())
We have already implemented a 3-layer neural network. You will train it with:
def model(X, Y, layers_dims, optimizer, learning_rate = 0.0007, mini_batch_size = 64, beta = 0.9,
beta1 = 0.9, beta2 = 0.999, epsilon = 1e-8, num_epochs = 10000, print_cost = True):
"""
3-layer neural network model which can be run in different optimizer modes.
Arguments:
X -- input data, of shape (2, number of examples)
Y -- true "label" vector (1 for blue dot / 0 for red dot), of shape (1, number of examples)
layers_dims -- python list, containing the size of each layer
learning_rate -- the learning rate, scalar.
mini_batch_size -- the size of a mini batch
beta -- Momentum hyperparameter
beta1 -- Exponential decay hyperparameter for the past gradients estimates
beta2 -- Exponential decay hyperparameter for the past squared gradients estimates
epsilon -- hyperparameter preventing division by zero in Adam updates
num_epochs -- number of epochs
print_cost -- True to print the cost every 1000 epochs
Returns:
parameters -- python dictionary containing your updated parameters
"""
L = len(layers_dims) # number of layers in the neural networks
costs = [] # to keep track of the cost
t = 0 # initializing the counter required for Adam update 初始化Adam更新所需的计数器
seed = 10 # For grading purposes, so that your "random" minibatches are the same as ours
# Initialize parameters
parameters = initialize_parameters(layers_dims)
# Initialize the optimizer
if optimizer == "gd":
pass # no initialization required for gradient descent
elif optimizer == "momentum":
v = initialize_velocity(parameters)
elif optimizer == "adam":
v, s = initialize_adam(parameters)
# Optimization loop
for i in range(num_epochs):
# Define the random minibatches. We increment the seed to reshuffle differently the dataset after each epoch
#定义随机的小批。我们增加种子以在每个历元之后以不同的方式重新洗牌数据集
seed = seed + 1
minibatches = random_mini_batches(X, Y, mini_batch_size, seed)
for minibatch in minibatches:
# Select a minibatch
(minibatch_X, minibatch_Y) = minibatch
# Forward propagation
a3, caches = forward_propagation(minibatch_X, parameters)
# Compute cost
cost = compute_cost(a3, minibatch_Y)
# Backward propagation
grads = backward_propagation(minibatch_X, minibatch_Y, caches)
# Update parameters
if optimizer == "gd":
parameters = update_parameters_with_gd(parameters, grads, learning_rate) #梯度下降算法(小批量)
elif optimizer == "momentum":
parameters, v = update_parameters_with_momentum(parameters, grads, v, beta, learning_rate)
elif optimizer == "adam":
t = t + 1 # Adam counter
parameters, v, s = update_parameters_with_adam(parameters, grads, v, s,
t, learning_rate, beta1, beta2, epsilon)
# Print the cost every 1000 epoch
if print_cost and i % 1000 == 0:
print ("Cost after epoch %i: %f" %(i, cost))
if print_cost and i % 100 == 0:
costs.append(cost)
# plot the cost
plt.plot(costs)
plt.ylabel('cost')
plt.xlabel('epochs (per 100)')
plt.title("Learning rate = " + str(learning_rate))
plt.show()
return parameters
You will now run this 3 layer neural network with each of the 3 optimization methods.
Run the following code to see how the model does with mini-batch gradient descent.
# train 3-layer model
layers_dims = [train_X.shape[0], 5, 2, 1]
parameters = model(train_X, train_Y, layers_dims, optimizer = "gd")
# Predict
predictions = predict(train_X, train_Y, parameters)
print(predictions)
# Plot decision boundary
plt.title("Model with Gradient Descent optimization")
axes = plt.gca() #gca=get current axis
axes.set_xlim([-1.5,2.5])
axes.set_ylim([-1,1.5])
# print(x.T)
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y.flatten())
Run the following code to see how the model does with momentum. Because this example is relatively simple, the gains from using momemtum are small; but for more complex problems you might see bigger gains.运行以下代码,查看模型如何处理动量。因为这个例子相对简单,使用momemtum的收益很小;但对于更复杂的问题,你可能会看到更大的收益。
# train 3-layer model
layers_dims = [train_X.shape[0], 5, 2, 1]
parameters = model(train_X, train_Y, layers_dims, beta = 0.9, optimizer = "momentum")
# Predict
predictions = predict(train_X, train_Y, parameters)
# Plot decision boundary
plt.title("Model with Momentum optimization")
axes = plt.gca()
axes.set_xlim([-1.5,2.5])
axes.set_ylim([-1,1.5])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y.flatten())
Run the following code to see how the model does with Adam.
# train 3-layer model
layers_dims = [train_X.shape[0], 5, 2, 1]
parameters = model(train_X, train_Y, layers_dims, optimizer = "adam")
# Predict
predictions = predict(train_X, train_Y, parameters)
# Plot decision boundary
plt.title("Model with Adam optimization")
axes = plt.gca()
axes.set_xlim([-1.5,2.5])
axes.set_ylim([-1,1.5])
plot_decision_boundary(lambda x: predict_dec(parameters, x.T), train_X, train_Y.flatten())
Momentum usually helps, but given the small learning rate and the simplistic dataset, its impact is almost negligeable. Also, the huge oscillations you see in the cost come from the fact that some minibatches are more difficult thans others for the optimization algorithm.
Adam on the other hand, clearly outperforms mini-batch gradient descent and Momentum. If you run the model for more epochs on this simple dataset, all three methods will lead to very good results. However, you've seen that Adam converges a lot faster. 动量通常是有帮助的,但是考虑到较小的学习速度和简单的数据集,它的影响几乎是可以忽略的。此外,您在成本中看到的巨大振荡来自于这样一个事实,即一些小批量比其他优化算法更困难。另一方面,Adam明显优于小批量梯度下降和动量。如果您在这个简单的数据集中运行模型更长的时间,那么这三种方法都将带来非常好的结果。不过,你已经看到Adam收敛得更快了。
Some advantages of Adam include:
References: