Project 3Classification and inference with machine learningThis notebook is arranged in cells. Texts are usually written in the markdown cells, and here you can use htmltags (make it bold, italic, colored, etc). You can double click on this cell to see the formatting.The ellipsis (...) are provided where you are expected to write your solution but feel free to change the template(not over much) in case this style is not to your taste.Hit Shift-Enter on a code cell to evaluate it. Double click a Markdown cell to edit.Link OkpyIn [1]:Imports=====================================================================Assignment: Project 3OK, version v1.12.5=====================================================================Open the following URL:https://okpy.org/client/login/ (https://okpy.org/client/login/)After logging in, copy the code from the web page and paste it into the box.Then press the Enter key on your keyboard.Paste your code here: KozofyhR7YkKXUwCK3ycaIJ6Nubck9Successfully logged in as [email protected] client.api.notebook import Notebookok = Notebook(Project3_U.ok)_ = ok.auth(inline = True)In [2]:Problem 1 - Using Keras - MNISTThe goal of this notebook is to introduce deep neural networks (DNNs) and convolutional neural networks(CNNs) using the high-level Keras package and to become familiar with how to choose its architecture, costfunction, and optimizer in Keras. We will also learn how to train neural networks.We will once again work with the MNIST dataset of hand written digits introduced in HW8. The goal is to find astatistical model which recognizes and distinguishes between the ten handwritten digits (0-9).The MNIST dataset comprises handwritten digits, each of which comes in a square image, divided into a pixel grid. Every pixel can take on nuances of the gray color, interpolating between white andblack, and hence each data point assumes any value in the set . Since there are categoriesin the problem, corresponding to the ten digits, this problem represents a generic classification task.In this Notebook, we show how to use the Keras python package to tackle the MNIST problem with the help ofdeep neural networks.28 × 28 256{0, 1, … , 255} 10import numpy as npfrom scipy.integrate import quad#For plottingimport matplotlib.pyplot as plt%matplotlib inlineimport warningswarnings.filterwarnings(ignore)Creating DNNs with KerasConstructing a Deep Neural Network to solve ML problems is a multiple-stage process. Quite generally, onecan identify the key steps as follows:step 1: Load and process the datastep 2: Define the model and its architecturestep 3: Choose the optimizer and the cost functionstep 4: Train the modelstep 5: Evaluate the model performance on the unseen test datastep 6: Modify the hyperparameters to optimize performance for the specific data setWe would like to emphasize that, while it is always possible to view steps 1-5 as independent of the particulartask we are trying to solve, it is only when they are put together in step 6 that the real gain of using DeepLearning is revealed, compared to less sophisticated methods such as the regression models. With this remarkin mind, we shall focus predominantly on steps 1-5 below. We show how one can use grid search methods tofind optimal hyperparameters in step 6.Step 1: Load and Process the DataKeras knows to download automatically the MNIST data from the web. All we need to do is import the mnistmodule and use the load_data() class, and it will create the training and test data sets or us.The MNIST set has pre-defined test and training sets, in order to facilitate the comparison of the performanceof different models on the data.Once we have loaded the data, we need to format it in the correct shape ( ).The size of each sample, i.e. the number of bare features used is N_features (whis is 784 because we have a pixel grid), while the number of potential classification categories is num_classes (which is 10,number of digits).Each pixel contains a greyscale value quantified by an integer between 0 and 255. To standardize the dataset,we normalize the input data in the interval [0, 1].(N , ) samples Nfeatures28 × 28In [3]:1. Make a plot of one MNIST digit (2D plot using X data - make sure to reshape it into a matrix) andlabel it (which digit does it correspond to?).28 × 28Using TensorFlow backend.from __future__ import print_functionimport keras,sklearn# suppress tensorflow compilation warningsimport osimport tensorflow as tfos.environ[TF_CPP_MIN_LOG_LEVEL] = 2seed=0np.random.seed(seed) # fix random seedtf.set_random_seed(seed)from keras.datasets import mnist# input image dimensionsnum_classes = 10 # 10 digitsimg_rows, img_cols = 28, 28 # number of pixels# the data, shuffled and split between train and test sets(X_train, Y_train), (X_test, Y_test) = mnist.load_data()X_train = X_train[:40000]Y_train = Y_train[:40000]# reshape data, depending on Keras backendX_train = X_train.reshape(X_train.shape[0], img_rows*img_cols)X_test = X_test.reshape(X_test.shape[0], img_rows*img_cols)# cast floats to single precesionX_train = X_train.astype(float32)X_test = X_test.astype(float32)# rescale data in interval [0,1]X_train /= 255X_test /= 255In [4]:Last, we cast the label vectors y to binary class matrices (a.k.a. one-hot format).plt.imshow(X_train[0].reshape((28,28)), cmap = plt.cm.gray)plt.title(Label = %d %Y_train[0])plt.show()In [23]:Here in this template, we use 40000 training samples and 10000 test samples. Remember that wepreprocessed data into the shape (N , ). samples NfeaturesIn [24]:before conversion -y vector : [5 0 4 1 9 2 1 3 1 4]after conversion -y vector : [[0. 0. 0. 0. 0. 1. 0. 0. 0. 0.] [1. 0. 0. 0. 0. 0. 0. 0. 0. 0.] [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.] [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.] [0. 0. 0. 0. 0. 0. 0. 0. 0. 1.] [0. 0. 1. 0. 0. 0. 0. 0. 0. 0.] [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.] [0. 0. 0. 1. 0. 0. 0. 0. 0. 0.] [0. 1. 0. 0. 0. 0. 0. 0. 0. 0.] [0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]]X_train shape: (40000, 784)Y_train shape: (40000, 10)40000 train samples10000 test samples# convert class vectors to binary class matricesprint(before conversion - )print(y vector : , Y_train[0:10])Y_train = keras.utils.to_categorical(Y_train, num_classes)Y_test = keras.utils.to_categorical(Y_test, num_classes)print(after conversion - )print(y vector : , Y_train[0:10])print(X_train shape:, X_train.shape)print(Y_train shape:, Y_train.shape)print()print(X_train.shape[0], train samples)print(X_test.shape[0], test samples)Step 2: Define the Neural Net and its ArchitectureWe can now move on to construct our deep neural net. We shall use Kerass Sequential() class toinstantiate a model, and will add different deep layers one by one.Let us create an instance of Keras Sequential() class, called model . As the name suggests, this classallows us to build DNNs layer by layer. (https://keras.io/getting-started/sequential-model-guide/(https://keras.io/getting-started/sequential-model-guide/))In [4]:We use the add() method to attach layers to our model. For the purposes of our introductory example, itsuffices to focus on Dense layers for simplicity. (https://keras.io/layers/core/ (https://keras.io/layers/core/))Every Dense() layer accepts as its first required argument an integer which specifies the number of neurons.The type of activation function for the layer is defined using the activation optional argument, the input ofwhich is the name of the activation function in string format. Examples include relu , tanh , elu ,sigmoid , softmax .In order for our DNN to work properly, we have to make sure that the numbers of input and output neurons foreach layer match. Therefore, we specify the shape of the input in the first layer of the model explicitly using theoptional argument input_shape=(N_features,) . The sequential construction of the model then allowsKeras to infer the correct input/output dimensions of all hidden layers automatically. Hence, we only need tospecify the size of the softmax output layer to match the number of categories.First, add a Dense layer with 400 output neurons and relu activation function.In [26]:Add another layer with 100 output neurons. Then, we will apply dropout, a regularization scheme that hasbeen widely adopted in the neural networks literature: during the training procedure neurons are randomly“dropped out” of the neural network with some probability giving rise to a thinned network. It preventsoverfitting by reducing spurious correlations between neurons within the network by introducing arandomization procedure.pfrom keras.models import Sequentialfrom keras.layers import Dense, Dropout, Flattenfrom keras.layers import Conv2D, MaxPooling2D# instantiate modelmodel = Sequential()model.add(Dense(400,input_shape=(img_rows*img_cols,), activation=relu))In [27]:Lastly, we need to add a soft-max layer since we have a multi-class output.In [28]:Step 3: Choose the Optimizer and the Cost FunctionNext, we choose the loss function according to which to train the DNN. For classification problems, this is thecross entropy, and since the output data was cast in categorical form, we choose thecategorical_crossentropy defined in Keras losses module. Depending on the problem of interestone can pick any other suitable loss function. To optimize the weights of the net, we choose SGD. Thisalgorithm is already available to use under Keras optimizers module (https://keras.io/optimizers/)(https://keras.io/optimizers/)), but we could use Adam() or any other built-in one as well. The parameters forthe optimizer, such as lr (learning rate) or momentum are passed using the corresponding optionalarguments of the SGD() function.While the loss function and the optimizer are essential for the training procedure, to test the performance of themodel one may want to look at a particular metric of performance. For instance, in categorical tasks onetypically looks at their accuracy , which is defined as the percentage of correctly classified data points.To complete the definition of our model, we use the compile() method, with optional arguments for theoptimizer , loss , and the validation metric as follows:In [29]:model.add(Dense(100, activation=relu))# apply dropout with rate 0.5model.add(Dropout(0.5))model.add(Dense(num_classes, activation=softmax))# compile the modelmodel.compile(loss=keras.losses.categorical_crossentropy, optimizer=SGD, metrics=[Step 4: Train the modelWe train our DNN in minibatches. Shuffling the training data during training improves stability of the model.Thus, we train over a number of training epochs.(The number of epochs is the number of complete passes through the training dataset, and the batch size is anumber of samples propagated through the network before the model is updated.)Training the DNN is a one-liner using the fit() method of the Sequential class. The first two requiredarguments are the training input and output data. As optional arguments, we specify the mini- batch_size ,the number of training epochs , and the test or validation data. To monitor the training procedure for everyepoch, we set verbose=True .Let us set batch_size = 64 and epochs = 10.In [30]:Step 5: Evaluate the Model Performance on the Unseen Test DataNext, we evaluate the model and read of the loss on the test data, and its accuracy using the evaluate()method.Train on 40000 samples, validate on 10000 samplesEpoch 1/1040000/40000 [==============================] - 4s - loss: 1.2012 - acc: 0.6446 - val_loss: 0.5087 - val_acc: 0.8839Epoch 2/1040000/40000 [==============================] - 4s - loss: 0.5895 - acc: 0.8318 - val_loss: 0.3646 - val_acc: 0.9065Epoch 3/1040000/40000 [==============================] - 4s - loss: 0.4755 - acc: 0.8646 - val_loss: 0.3081 - val_acc: 0.9193Epoch 4/1040000/40000 [==============================] - 3s - loss: 0.4100 - acc: 0.8814 - val_loss: 0.2755 - val_acc: 0.9243Epoch 5/1040000/40000 [==============================] - 4s - loss: 0.3716 - acc: 0.8975 - val_loss: 0.2527 - val_acc: 0.9288Epoch 6/1040000/40000 [==============================] - 4s - loss: 0.3445 - acc: 0.9030 - val_loss: 0.2338 - val_acc: 0.9342Epoch 7/1040000/40000 [==============================] - 4s - loss: 0.3185 - acc: 0.9105 - val_loss: 0.2203 - val_acc: 0.9383Epoch 8/1040000/40000 [==============================] - 4s - loss: 0.2991 - acc: 0.9171 - val_loss: 0.2060 - val_acc: 0.9413Epoch 9/1040000/40000 [==============================] - 3s - loss: 0.2815 - acc: 0.9213 - val_loss: 0.1972 - val_acc: 0.9437Epoch 10/1040000/40000 [==============================] - 4s - loss: 0.2656 - acc: 0.9265 - val_loss: 0.1874 - val_acc: 0.9457# training parametersbatch_size = 64epochs = 10# train DNN and store training info in historyhistory=model.fit(X_train, Y_train, batch_size=batch_size, epochs=epochs, verbose=1, validation_data=(X_test, Y_test))In [32]: 9632/10000 [===========================>..] - ETA: 0sTest loss: 0.18736902387440205Test accuracy: 0.9457# evaluate modelscore = model.evaluate(X_test, Y_test, verbose=1)# print performanceprint(Test loss:, score[0])print(Test accuracy:, score[1])# look into training history# summarize history for accuracyplt.plot(history.history[acc])plt.plot(history.history[val_acc])plt.ylabel(model accuracy)plt.xlabel(epoch)plt.legend([train, test], loc=best)plt.show()# summarize history for lossplt.plot(history.history[loss])plt.plot(history.history[val_loss])plt.ylabel(model loss)plt.xlabel(epoch)plt.legend([train, test], loc=best)plt.show()Step 6: Modify the Hyperparameters to Optimize Performance of the ModelLast, we show how to use the grid search option of scikit-learn (https://scikitlearn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html(https://scikitlearn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html))to optimize thehyperparameters of our model.First, define a function for crating a DNN:In [9]:With epochs = 1 and batch_size = 64, do grid search over the following optimization schemes: [SGD,RMSprop, Adagrad, Adadelta, Adam, Adamax, Nadam].In [34]:def create_DNN(optimizer=keras.optimizers.Adam()): model = Sequential() model.add(Dense(400,input_shape=(img_rows*img_cols,), activation=relu)) model.add(Dense(100, activation=relu)) model.add(Dropout(0.5)) model.add(Dense(num_classes, activation=softmax)) model.compile(loss=keras.losses.categorical_crossentropy, optimizer=optimizer, metrics=[accuracy]) return modelfrom sklearn.model_selection import GridSearchCVfrom keras.wrappers.scikit_learn import KerasClassifierbatch_size = 64epochs = 1model_gridsearch = KerasClassifier(build_fn=create_DNN, epochs=epochs, batch_size=batch_size, verbose=1)# list of allowed optional arguments for the optimizer, see `compile_model()`optimizer = [SGD, RMSprop, Adagrad, Adadelta, Adam, Adamax, Nadam]Epoch 1/130000/30000 [==============================] - 3s - loss: 1.3620 - acc: 0.583928928/30000 [===========================>..] - ETA: 0sEpoch 1/130000/30000 [==============================] - 2s - loss: 1.4106 - acc: 0.576328800/30000 [===========================>..machine learning作业代写、代做Markdown留学生作业、代写python, C/C++编程语言作业 代] - ETA: 0sEpoch 1/130000/30000 [==============================] - 2s - loss: 1.3413 - acc: 0.601729056/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 2s - loss: 1.3520 - acc: 0.585629632/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 0.4102 - acc: 0.878428736/30000 [===========================>..] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 0.4258 - acc: 0.872328608/30000 [===========================>..] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 0.4126 - acc: 0.877629312/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 0.4097 - acc: 0.877929824/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 0.3794 - acc: 0.888729824/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.4031 - acc: 0.881629184/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 0.3895 - acc: 0.886629824/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.3752 - acc: 0.891129632/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.5778 - acc: 0.832529184/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.5900 - acc: 0.827828864/30000 [===========================>..] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.5922 - acc: 0.8268optimizer = [SGD, RMSprop, Adagrad, Adadelta, Adam, Adamax, Nadam]# define parameter dictionaryparam_grid = dict(optimizer=optimizer)# call scikit grid search modulegrid = GridSearchCV(estimator=model_gridsearch, param_grid=param_grid, n_jobs=1, cv=grid_result = grid.fit(X_train,Y_train)Show the mean test score of all optimization schemes and determine which scheme gives the best accuracy.29184/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.5868 - acc: 0.826629952/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.4324 - acc: 0.872128864/30000 [===========================>..] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.4337 - acc: 0.871429440/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.4356 - acc: 0.871729568/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.4332 - acc: 0.872728608/30000 [===========================>..] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.4958 - acc: 0.850029312/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.4888 - acc: 0.859528928/30000 [===========================>..] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.4658 - acc: 0.862128736/30000 [===========================>..] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.4754 - acc: 0.860130000/30000 [==============================] - 1sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.3584 - acc: 0.892928992/30000 [===========================>..] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.3585 - acc: 0.894429248/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.3631 - acc: 0.891729824/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 4s - loss: 0.3624 - acc: 0.893128800/30000 [===========================>..] - ETA: 0sEpoch 1/140000/40000 [==============================] - 6s - loss: 0.3168 - acc: 0.9073 In [35]:2. Create a DNN with one Dense layer having 200 output neurons. Do the grid search over any 5 differentactivation functions from https://keras.io/activations/ (https://keras.io/activations/). Let epochs = 1, batches =64, p_dropout=0.5, and optimizer=keras.optimizers.Adam(). Make sure to print the mean test score of eachcase and determine which activation functions gives the best accuracy.Doing the grid search requires quite a bit of memory. Please restart the kernel (Kernel-Restart) and re-loadthe data before doing a new grid search.In [10]:Best: 0.951650 using {optimizer: Nadam}0.850700 (0.013746) with: {optimizer: SGD}0.947125 (0.001248) with: {optimizer: RMSprop}0.946550 (0.003741) with: {optimizer: Adagrad}0.925900 (0.002684) with: {optimizer: Adadelta}0.947200 (0.001200) with: {optimizer: Adam}0.934825 (0.002807) with: {optimizer: Adamax}0.951650 (0.000865) with: {optimizer: Nadam}# summarize resultsprint(Best: %f using %s % (grid_result.best_score_, grid_result.best_params_))means = grid_result.cv_results_[mean_test_score]stds = grid_result.cv_results_[std_test_score]params = grid_result.cv_results_[params]for mean, stdev, param in zip(means, stds, params): print(%f (%f) with: %r % (mean, stdev, param))model = Sequential()def create_DNN(activation): model = Sequential() model.add(Dense(200,input_shape=(img_rows*img_cols,), activation=activation)) model.add(Dropout(0.5)) model.add(Dense(num_classes, activation=softmax)) model.compile(loss=keras.losses.categorical_crossentropy, optimizer=Adam, metrics=[accuracy]) return modelfrom sklearn.model_selection import GridSearchCVfrom keras.wrappers.scikit_learn import KerasClassifierbatch_size = 64epochs = 1model_gridsearch = KerasClassifier(build_fn=create_DNN, epochs=epochs, batch_size=batch_size, verbose=1)Epoch 1/130000/30000 [==============================] - 3s - loss: 0.5012 - acc: 0.850629120/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 0.4967 - acc: 0.849229184/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 2s - loss: 0.4963 - acc: 0.853029376/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 2s - loss: 0.4939 - acc: 0.855929440/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 2s - loss: 0.4717 - acc: 0.861129120/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 2s - loss: 0.4722 - acc: 0.860430000/30000 [==============================] - 0sEpoch 1/130000/30000 [==============================] - 2s - loss: 0.4762 - acc: 0.858729760/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 2s - loss: 0.4679 - acc: 0.863729376/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 2s - loss: 0.4831 - acc: 0.856729952/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 2s - loss: 0.4629 - acc: 0.861228864/30000 [===========================>..] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 0.4722 - acc: 0.857928992/30000 [===========================>..] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 0.4746 - acc: 0.858029248/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 0.7890 - acc: 0.765628480/30000 [===========================>..] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 0.7785 - acc: 0.772028992/30000 [===========================>..] - ETA: 0sEpoch 1/1# list of allowed optional arguments for the optimizer, see `compile_model()`activation = [relu, tanh, elu, sigmoid, softmax]# define parameter dictionaryparam_grid = dict(activation=activation)# call scikit grid search modulegrid = GridSearchCV(estimator=model_gridsearch, param_grid=param_grid, n_jobs=1, cv=grid_result = grid.fit(X_train,Y_train)In [11]:30000/30000 [==============================] - 3s - loss: 0.7647 - acc: 0.772429312/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 0.7746 - acc: 0.772729184/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 1.9335 - acc: 0.502829760/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 1.9408 - acc: 0.504828928/30000 [===========================>..] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 1.9342 - acc: 0.476029824/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 1.9398 - acc: 0.481010000/10000 [==============================] - 0s28928/30000 [===========================>..] - ETA: 0sEpoch 1/140000/40000 [==============================] - 4s - loss: 0.4484 - acc: 0.8681Best: 0.932725 using {activation: relu}0.932725 (0.002930) with: {activation: relu}0.912850 (0.003827) with: {activation: tanh}0.913725 (0.005063) with: {activation: elu}0.895375 (0.005523) with: {activation: sigmoid}0.839475 (0.016595) with: {activation: softmax}# summarize resultsprint(Best: %f using %s % (grid_result.best_score_, grid_result.best_params_))means = grid_result.cv_results_[mean_test_score]stds = grid_result.cv_results_[std_test_score]params = grid_result.cv_results_[params]for mean, stdev, param in zip(means, stds, params): print(%f (%f) with: %r % (mean, stdev, param))3. Now, do the grid search over different combination of batch sizes (10, 30, 50, 100) and number of epochs (1,2, 5). Make sure to print the mean test score of each case and determine which activation functions gives thebest accuracy. Here, you have a freedom to create your own DNN (assume an arbitrary number of Dense layers,optimization scheme, etc).Doing the grid search requires quite a bit of memory. Please restart the kernel (Kernel-Restart) and re-loadthe data before doing a new grid search.Hint: To do the grid search over both batch_size and epochs, you can do:param_grid = dict(batch_size=batch_size, epochs=epochs)In [13]:Epoch 1/130000/30000 [==============================] - 14s - loss: 0.3851 - acc: 0.883229830/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 15s - loss: 0.3897 - acc: 0.883629960/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 15s - loss: 0.3945 - acc: 0.882029750/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 17s - loss: 0.3827 - acc: 0.882129980/30000 [============================>.] - ETA: 0sEpoch 1/230000/30000 [==============================] - 16s - loss: 0.3935 - acc: 0.8827Epoch 2/230000/30000 [==============================] - 15s - loss: 0.2171 - acc: 0.934129930/30000 [============================>.] - ETA: 0sEpoch 1/2model = Sequential()def create_DNN(): model = Sequential() model.add(Dense(200,input_shape=(img_rows*img_cols,), activation=relu)) model.add(Dropout(0.5)) model.add(Dense(num_classes, activation=softmax)) model.compile(loss=keras.losses.categorical_crossentropy, optimizer=Adam, metrics=[accuracy]) return modelfrom sklearn.model_selection import GridSearchCVfrom keras.wrappers.scikit_learn import KerasClassifierbatch_size = 64epochs = 1model_gridsearch = KerasClassifier(build_fn=create_DNN, epochs=epochs, batch_size=batch_size, verbose=1)# list of allowed optional arguments for the optimizer, see `compile_model()`batch_size = [10,30,50,100]epochs = [1,2,5]# define parameter dictionaryparam_grid = dict(batch_size=batch_size, epochs=epochs)# call scikit grid search modulegrid = GridSearchCV(estimator=model_gridsearch, param_grid=param_grid, n_jobs=1, cv=grid_result = grid.fit(X_train,Y_train)In [14]:4. Do the grid search over the number of neurons in the Dense layer and make a plot of mean test score as afunction of num_neurons. Again, you have a freedom to create your own DNN.Doing the grid search requires quite a bit of memory. Please restart the kernel (Kernel-Restart) and re-loadthe data before doing a new grid search.In [8]:Best: 0.967475 using {batch_size: 10, epochs: 5}0.944350 (0.001052) with: {batch_size: 10, epochs: 1}0.956850 (0.002544) with: {batch_size: 10, epochs: 2}0.967475 (0.000476) with: {batch_size: 10, epochs: 5}0.939225 (0.003904) with: {batch_size: 30, epochs: 1}0.951700 (0.002840) with: {batch_size: 30, epochs: 2}0.967075 (0.000536) with: {batch_size: 30, epochs: 5}0.933700 (0.002487) with: {batch_size: 50, epochs: 1}0.949700 (0.001885) with: {batch_size: 50, epochs: 2}0.965475 (0.002815) with: {batch_size: 50, epochs: 5}0.927900 (0.002847) with: {batch_size: 100, epochs: 1}0.942250 (0.001410) with: {batch_size: 100, epochs: 2}0.960925 (0.002025) with: {batch_size: 100, epochs: 5}# summarize resultsprint(Best: %f using %s % (grid_result.best_score_, grid_result.best_params_))means = grid_result.cv_results_[mean_test_score]stds = grid_result.cv_results_[std_test_score]params = grid_result.cv_results_[params]for mean, stdev, param in zip(means, stds, params): print(%f (%f) with: %r % (mean, stdev, param))model = Sequential()def create_DNN(number): model = Sequential() model.add(Dense(number,input_shape=(img_rows*img_cols,), activation=relu)) model.add(Dense(100, activation=relu)) model.add(Dropout(0.5)) model.add(Dense(num_classes, activation=softmax)) model.compile(loss=keras.losses.categorical_crossentropy, optimizer=Adam, metrics=[accuracy]) return modelfrom sklearn.model_selection import GridSearchCVfrom keras.wrappers.scikit_learn import KerasClassifierbatch_size = 64epochs = 1Epoch 1/130000/30000 [==============================] - 2s - loss: 0.5748 - acc: 0.824129696/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 2s - loss: 0.5448 - acc: 0.835928672/30000 [===========================>..] - ETA: 0sEpoch 1/130000/30000 [==============================] - 2s - loss: 0.5499 - acc: 0.835828864/30000 [===========================>..] - ETA: 0sEpoch 1/130000/30000 [==============================] - 2s - loss: 0.5480 - acc: 0.834330000/30000 [==============================] - 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 0.4889 - acc: 0.856929376/30000 [============================>.] - ETA: 0sEpoch 1/130000/30000 [==============================] - 3s - loss: 0.4796 - acc: 0.857728544/30000 [========================转自：http://ass.3daixie.com/2018121440484051.html
