Homework 7 INF 552,1. Generative Models for Text(a) In this problem, we are trying to build a generative model to mimic the writingstyle of prominent British Mathematician, Philosopher, prolific writer, andpolitical activist, Bertrand Russell.(b) i. The Problems of Philosophyii. The Analysis of Mindiii. Mysticism and Logic and Other Essaysiv. Our Knowledge of the External World as a Field for Scientific Method inPhilosophyProject Gutenberg adds a standard header and footer to each book and this isnot part of the original text. Open the file in a text editor and delete the headerand footer.The header is obvious and ends with the text:*** START OF THIS PROJECT GUTENBERG EBOOK AN INQUIRY INTOMEANING AND TRUTH ***The footer is all of the text after the line of text that says:THE ENDTo have a better model, it is strongly recommended that you download the followingbooks from The Library of Congress https://archive.org and convertthem to text files:i. The History of Western Philosophyhttps://archive.org/details/westernphilosophy4ii. The Analysis of Matterhttps://archive.org/details/in.ernet.dli.2015.221533iii. An Inquiry into Meaning and Truthhttps://archive.org/details/BertrandRussell-AnInquaryIntoMeaningAndTruthTry to only use the text of the books and throw away unwanted text before andafter the text, although in a large corpus, these are considered as noise and shouldnot make big problems.1(c) LSTM: Train an LSTM to mimic Russell’s style and thoughts:i. Concatenate your text files to create a corpus of Russell’s writings.ii. Use a character-level representation for this model by using extended ASCIIthat has N = 256 characters. Each character will be encoded into a an integerusing its ASCII code. Rescale the integers to the range [0, 1], because LSTM1If this is a large corpus for your computer’s power and it makes training LSTM hard, use as many ofthe books as possible.1Homework 7 INF 552,uses a sigmoid activation function. LSTM will receive the rescaled integersas its input.2iii. Choose a window size, e.g., W = 100.iv. Inputs to the network will be the first W ?1 = 99 characters of each sequence,and the output of the network will be the Wth character of the sequence.Basically, we are training the network to predict each character using the 99characters that precede it. Slide the window in strides of S = 1 on the text.For example, if W = 5 and S = 1 and we want to train the network with thesequence ABRACADABRA, The first input to the network will be ABRAand the corresponding output will be C. The second input will be BRAC andthe second output will be A, etc.v. Note that the output has to be encoded using a one-hot encoding scheme withN = 256 (or less) elements. This means that the network reads integers, butoutputs a vector of N = 256 (or less) elements.vi. Use a single hidden layer for the LSTM with N = 256 (or less) memory units.vii. Use a Softmax output layer to yield a probability prediction for each of thecharacters between 0 and 1. This is actually a character classification problemwith N classes. Choose log loss (cross entropy) as the objective function forthe network (research what it means).3viii. We do not use a test dataset. We are using the whole training dataset tolearn the probability of each character in a sequence. We are not seeking fora very accurate model. Instead we are interested in a generalization of thedataset that can mimic the gist of the text.ix. Choose a reasonable number of epochs4for training, considering your computationalpower (e.g., 30, although the network will need more epochs to yielda better model).x. Use model checkpointing to keep the network weights to determine each timean improvement in loss is observed at the end of the epoch. Find the best setof weights in terms of loss.xi. Use the network with the best weights to generate 1000 characters, using thefollowing text as initialization of the network:There are those who take mental phenomena naively, just as theywould physical phenomena. This school of psychologists tends not toemphasize the object.2A smarter way is to parse the whole corpus to figure out how many distinct characters you have in thecorpus (the number may be less than 256, e.g., 53). One can also disregard lowercase and uppercase lettersor even remove punctuation characters such as !.3In Keras, you can use the ADAM optimization algorithm for speed.4one epoch = one forward pass and one backward pass of all the training examples.batch size = the number of training examples in one forward/backward pass. The higher the batch size,the more memory space you’ll need.number of iterations = number of passes, each pass using [batch size] number of examples. To be clear,one pass = one forward pass + one backward pass (we do not count the forward pass and backward pass astwo different passes).See https://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network2Homework 7 INF 552, Instructor: Mohammad Reza Rajatixii. Extra Practice: Use one-hot encoding for the input sequence. Use a largenumber of epochs, e.g., 150. Add dropout to the network, and use a deeperLSTM (e.g. with 3 or more layers). Generate 3000 characters using the aboveinitialization and report if you get more meaningful text.xiii. Extra Practice- HMM: Train a Hidden Markov Model with V hidden statesand V possible outputs using Baum-Welch Algorithm (or any other modernalgoINF 552作业代做、ASCII code留学生作业代写、MATLAB课程设计作业代写、MATLAB编程语言作业调试 rithm that is available) using the Russell corpus, where V is the numberof distinct words in the corpus. Note that for HMM, you NOT use characterlevel encoding, because it may yield totally meaningless results, although thetransition matrices associated with it will be way smaller (you are welcometo try it). Generate 200 words using the model and comment on its meaningfulness.Extra extra practice: can you train a higher order HMM (i.e. anHMM that assumes dependency on more than one previous state) to get abetter model?2. (Deep) CNNs for Image Colorization(a) This assignment uses a convolutional neural network for image colorization whichturns a grayscale image to a colored image.5 By converting an image to grayscale,we loose color information, so converting a grayscale image back to a coloredversion is not an easy job. We will use the CIFAR-10 dataset. Downolad thedataset from http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz.(b) From the train and test dataset, extract the class birds. We will focus on thisclass, which has 6000 members.(c) Those 6000 images have 6000 × 32 × 32 pixels. Choose at least 10% of the pixelsrandomly. It is strongly recommended that you choose a large number or all ofthe pixels. You will have between P = 614400 and P = 6144000 pixels. Eachpixel is an RGB vector with three elements.(d) Run k-means clustering on the P vectors using k = 4. The centers of the clusterswill be your main colors. Convert the colored images to k-color images by convertingeach pixel’s value to the closest main color in terms of Euclidean distance.These are the outputs of your network, whose each pixel falls in one of those kclasses.6(e) Use any tool (e.g., openCV or scikit-learn) to obtain grayscale 32 × 32 × 1 imagesfrom the original 32 × 32 × 3 images. The grayscale images are inputs of yournetwork.5MATLAB seems to have an easy to use CNN library. https://www.mathworks.com/help/nnet/examples/train-a-convolutional-neural-network-for-regression.html6Centers of clusters have been reported too close previously, so the resultant tetra-chrome imageswill be very close to grayscale. In case you would like to see colorful images, repeat the exercisewith colors you select from https://sashat.me/2017/01/11/list-of-20-simple-distinct-colors/ orhttps://www.rapidtables.com/web/color/RGB_Color.html. A suggestion would be Navy = (0,0,128),Red =( 230, 25, 75), Mint = (170, 255, 195), and White = (255, 255, 255).3Homework 7 INF 552, �(f) Set up a deep convolutional neural network with two convolution layers (or more)and two (or more) MLP layers. Use 5 × 5 filters and a softmax output layer.Determine the number of filters, strides, and whether or not to use padding yourself.Use a minimum of one max pooling layer. Use a classification scheme, whichmeans your output must determine one of the k = 4 color classes for each pixel inyour grayscale image. Your input is a grayscale version of an image (32 × 32 × 1)and the output is 32 × 32 × 4. The output assigns one of the k = 4 colors toeach of the 32 × 32 pixels; therefore, each of the pixels is classified into one of theclasses [1 0 0 0], [0 1 0 0], [0 0 1 0], [0 0 0 1]. After each pixel is classified into oneof the main colors, the RGB code of that color can be assigned to the pixel. Forexample, if the third main color 7is [255 255 255] and pixel (32,32) of an imagehas the one-hot encoded class [0 0 1 0], i.e it was classified as the third color, the(32,32) place in the output can be associated with [255 255 255]. The size of theoutput of the convolutional part, c1 × c2 depends on the size of the convolutionallayers you choose and is a feature map, which is a matrix. That matrix must beflattened or reshaped, i.e. must be turned into a vector of size c1c2 ×1, before it isfed to the MLP part. Choose the number of neurons in the first layer of the MLP(and any other hidden layers, if you are willing to have more than one hiddenlayer) yourself, but the last layer must have 32 × 32 × 4 = 4096 neurons, each ofwhich represents a pixel being in one of the k = 4 classes. Add a softmax layer8which will choose the highest value out of its k = 4 inputs for each of the 1024pixels; therefore, the output of the MLP has to be reshaped into a 32 × 32 × 4matrix, and to get the colored image, the RGB vector of each of the k = 4 classeshas to be converted to the RGB vector, so an output image will be 32 × 32 × 3.Train at least for 5 epochs (30 epochs is strongly recommended). Plot training,(validation), and test errors in each epoch. Report the train and test errors andvisually compare the artificially colored versions of the first 10 images in the testset with the original images.9(g) Extra Practice: Repeat the whole exercise with k = 16, 24, 32 colors if yourcomputer can handle the computations.7Do not use the original CIFAR-10 images as the output. You must use the tetrachrome images youcreated as your output.8Compile the network with loss = cross entropy .9If you are using matplotlib, you may get a floating point error because to print an image, matplotlib eitherexpects ints in range 0-255 or floats in range 0-1. You might be having, for example, 153.0 representation of153 in your array and this is what makes matplotlib think that you are sending floats.Wrap your array into np.uint8(). It will convert 153.0 into 153. NOTE that you cannot use np.roundor np.int etc because matplotlib’s requirement is unsigned int of 8 bit (think 0-255).4转自：http://www.7daixie.com/2019042315276235.html
