《Python 深度学习》刷书笔记 Chapter 8 Part-1 生成式深度学习

---

import keras
keras.__version__

Using TensorFlow backend.





'2.3.1'

生成式深度学习

---

生成式深度学习的方法，就是使用前面的标记作为输入，训练一个网络，来预测接下来的一个或多个标记

在本届的例子中，我们将会用到一个LSTM层，向其输入从文本中提取的N个字符组成的字符串，来预测第N+1个字符串

这个LSTM就叫做字符级的神经网络语言模型

在生成文本时，主要有以下两种采样方式：

1. 贪婪采样：得到可重复、可预测的字符串
2. 随机采样：在过程中引入随机性（可能会更有趣？）

为了在采样过程中控制随机性的大小，我们将引入一个叫做Softmax温度的参数，用于表示采样概率分布的熵（即表示所选的下一个字符是否让人出人意料）

给定一个temperature值，按照下列的方法对原始概率分布进行重新加权，计算得到一个新的概率分布

更高的温度意味着更大的采样分布，生成的结果会更加的出人意料，更加的无结构化

---

8-1 对于不同的softmax温度，对概率进行重新加权

---

import numpy as np

# original_distribution是概率值组成的一维Numpy数组，这些概率之和为1
# 用于定量描述输出的分布熵
def reweight_distribution(original_distribution, temperature = 0.5):
    distribution = np.log(original_distribution) / temperature
    distribution = np.exp(distribution)
    
    # 返回原始分布重新加权后的结果
    # distribution的求和可能不再等于1，因此要将其归一化
    return distribution / np.sum(distribution)

实现字符级的LSTM文本生成

---

本例将应用尼采的一些作品，生成文本数据

注意：我们这个模型是针对尼采的，而不是一个英语文本的通用模型

8-2 下载并解析初始文本文件

---

import keras
import numpy as np

path = keras.utils.get_file(
    'nietzsche.txt', 
    origin = 'https://s3.amazonaws.com/text-datasets/nietzsche.txt')

text = open(path, encoding='utf-8').read().lower()
print('Corpus length:', len(text))

Corpus length: 600893

8-3 将字符序列向量化

---

# 提取60个字符组成的序列
maxlen = 60

# 每三个采样一个新序列
step = 3

# 保存所提取的序列
sentences = []

# 保存的目标(下一个字符)
next_chars = []

for i in range(0, len(text) - maxlen, step):
    sentences.append(text[i: i + maxlen])
    next_chars.append(text[i + maxlen])
    
print('Number of sequences:', len(sentences))

# 语料中唯一字符组成的列表
chars = sorted(list(set(text)))
# 打印不重复字符数
print('Unique charaters', len(chars))
# 一个字典，将唯一的字符映射为它在chars中的索引
char_indices = dict((char, chars.index(char)) for char in chars)

# 将字符one-hot编码转变为二进制数组
print('Vectorization...')
x = np.zeros((len(sentences), maxlen, len(chars)), dtype = np.bool)
y = np.zeros((len(sentences), len(chars)), dtype = np.bool)
for i, sentences in enumerate(sentences):
    for t, char in enumerate(sentences):
        x[i, t, char_indices[char]] = 1
    y[i, char_indices[next_chars[i]]] = 1

Number of sequences: 200278
Unique charaters 57
Vectorization...

8-4 用于预测下一个字符的单层LSTM模型

---

from keras import layers

model = keras.models.Sequential()
model.add(layers.LSTM(128, input_shape = (maxlen, len(chars))))
model.add(layers.Dense(len(chars), activation = 'softmax'))

---

目标是经过one-hot编码的，所以训练模型需要用categorical_crossentropy作为损失

---

8-5 模型编译设置

---

optimizer = keras.optimizers.RMSprop(lr = 0.01)
model.compile(loss = 'categorical_crossentropy', optimizer = optimizer)

8-6 给定模型预测，采样下一个字符的函数

---

def sample(preds, temperature = 1.0):
    preds = np.array(preds).astype('float64')
    preds = np.log(preds) / temperature
    exp_preds = np.exp(preds)
    preds = exp_preds / np.sum(exp_preds)
    probas = np.random.multinomial(1, preds, 1)
    return np.argmax(probas)

8-7 文本生成循环

---

import random
import sys

# 将模型训练60轮
for epoch in range(1, 60):
    print('epoch', epoch)
    
    # 将模型在数据上拟合一次
    model.fit(x, y,
              batch_size = 128,
              epochs = 1)

    # 随机选一个文本种子
    start_index = random.randint(0, len(text) - maxlen - 1)
    generated_text = text[start_index: start_index + maxlen]
    print('--- Generating with seed: "' + generated_text + '"')

    #尝试一系列不同的温度
    for temperature in [0.2, 0.5, 1.0, 1.2]:
        print('------ temperature:', temperature)
        sys.stdout.write(generated_text)

        # 从种子文本开始，生成400个字符
        for i in range(400):
            # 对目前的字符进行one-hot编码
            sampled = np.zeros((1, maxlen, len(chars)))
            for t, char in enumerate(generated_text):
                sampled[0, t, char_indices[char]] = 1.
            
            # 对下一个字符进行采样
            preds = model.predict(sampled, verbose = 0)[0]
            next_index = sample(preds, temperature)
            next_char = chars[next_index]

            generated_text += next_char
            generated_text = generated_text[1:]

            sys.stdout.write(next_char)
            sys.stdout.flush()
        print()

训练结果如下:
注：因训练结果太长，这里仅仅展示第1、2、3、20、23、58以及59轮的结果
注:第1轮

epoch 1
Epoch 1/1
200278/200278 [==============================] - 255s 1ms/step - loss: 1.8413
--- Generating with seed: "d to afford shade to generation
after generation in the futu"
------ temperature: 0.2
d to afford shade to generation
after generation in the future the sporises, and in the constinction of the some the sense of the ever the formuled the formuled the sense, and the supersting the soul the sense, in the entiorary the stance of the something in the stance of the something in the sense, and the sense of the most the still the sense, and in the more the supered the sense and really in the stance, the still the profound, the supersting the sense
------ temperature: 0.5
the stance, the still the profound, the supersting the sense of action is with the fool, with the how him in the saper to himself the storating concertion of conscience, of the become mach is the
ever the world the good the composition of the good be the concerded the conscience the semusion and in the stould possible the good as he douther and interption, endured the the himself in the sempre also become the superstical the believe of concerding the force
------ temperature: 1.0
o become the superstical the believe of concerding the forcendamely--and rality above steen
that he
condeltly in the represence from such soul, look--our reguerdly and so uniming him a infreecemy the world; himself expeads to mulon in bect so can him of aterby, siment  for outher acow of thingst is ih child peesen, sich sympathy in dissoment? goods of lifizion of himself it nowly comment
but-go intermenle.


4). the more who
excesf, and have in the fuve of
------ temperature: 1.2
termenle.


4). the more who
excesf, and have in the fuve of meunsporing ofself--it; it if themelve, beguated
their the muact
bell, in then, the sumprincting the becrmipt.
-
31.
is the ctdided believe
ink, and
the rifig the oodrengerentless mide,valshed ave foectifulence
concess be -" we havists which himdenderby hislowh6y
inafttime nowlever? so_? furwnelie, worl"l's, as hichoried testity, the comor the pain autifge the ere the
sential itse, ye
certianabli

注:第2轮

epoch 2
Epoch 1/1
200278/200278 [==============================] - 248s 1ms/step - loss: 1.5906
--- Generating with seed: " condition of existence--if it would continue, it can
only d"
------ temperature: 0.2
 condition of existence--if it would continue, it can
only does the consequence of the wander and believe the really and the superstiness and distroses the proposing and the consequently and such a souls of the conseated that the sense the consequence of the sense of the the consequence of the consequence of the prosonal consciously and delusion of the consequently to the proposing the man the desirate consequence of the consequence of the content of the s
------ temperature: 0.5
irate consequence of the consequence of the content of the same thereto has the wor the platated as it would have been consequently in the send-conduction and former to the enticonce of the same the readon and consequence and profoundly of the strong the does when in the sare and contained, and scient prosondation to the form and depression to the consequence of one standanting and cannot is conceance the instance of the religion of the prosonal strong and
------ temperature: 1.0
ance the instance of the religion of the prosonal strong and the dully reloter, deplases a fundhaldal history consint and mators cannot only not gruth to complenated, scient ope, is whologh in with sympited
in not un-he
res
to be sericital only trong
or of the charcions, it may too--throess doen considerates, countrarstancer
and an than condent winden in the deep, to cay not gruely, and special artime which the intellectual,
while of
the agming is expeash 
------ temperature: 1.2
time which the intellectual,
while of
the agming is expeash values: effine, non pow calling, re theregomer. qualitionible, knowled--pains the peile foroty? firs arvelomest, ?-=the musence-lowed to
citumvelofitiusate power itself--one rily is ulreown, te!

boded hiddoultly isnororal: god of in o
; believe and modes
brymbland can wisles every for enjequelly life to any, age heow more in
this force in these ares duls,
in seristiousldbaty in itself understood 

注:第3轮

epoch 3
Epoch 1/1
200278/200278 [==============================] - 251s 1ms/step - loss: 1.5139
--- Generating with seed: "t,
fundamentally and essentially, in the general economy of "
------ temperature: 0.2
t,
fundamentally and essentially, in the general economy of the world and sense, the same of the same the way that the conscience of the same the state and respect of the have and sense of the same the enough the way the same the conscience of the world of the english of the world and a sense of the same and a man as a morality of the same and something the delight of the fact the sense of the same and a conscience of the same the conscience and all the ca
------ temperature: 0.5
e and a conscience of the same the conscience and all the cained and accord a consequences and antisences that the morality of the deplice and a conscience, as the
fact a sense of a stander as he may the varies for every called the enewarable be are and for a moralists, and with the world the master and men and being same many the relation
of the last the sufferent in which has been are deceived by the delight in the same all uninations manfulous and manif
------ temperature: 1.0
y the delight in the same all uninations manfulous and manifestapes as confudest one real enrown, or we appearine distingumy from the firmod sphriths in one become the one
that all
infliction for a precausy snection"
accordapt mung-the yasical, who beery in like every inactings of a nmmisrery and being relatings one authris with the fau itselful
condycion this bisjorantquent weon arabed something of obenical every man remainerest educatedged reliding certa
------ temperature: 1.2
of obenical every man remainerest educatedged reliding certainerly reference.=--flatted, herew that it is dangoring element--a cakefain in theiscial want hampe other them
delus the saking dinfranged, salt may incing1yangs, as
because ame them which be loqk, amplen as the to him something that all-stectest all goesi:
.

for aqual
opporous and grectoromanne for heqused, duch
feorisings any
occupressered gorpheri, it
did, of
a
very day
becomes; by yean him, "

注:第20轮

epoch 20
Epoch 1/1
200278/200278 [==============================] - 249s 1ms/step - loss: 1.3191
--- Generating with seed: "ame
time, it should be further explained that the needs whic"
------ temperature: 0.2
ame
time, it should be further explained that the needs which is a most sufficient and the most profound of the spirit of the spirit is a most science, and the fact of the reason of the property of the spirit and the supersid and more the string and faith of the spirit of the same thing and who should as a more something in the fact of such an intellect and better.

131. the senses of the spirit is all the spirit of the fact that is a man and all the relig
------ temperature: 0.5
s all the spirit of the fact that is a man and all the religious the destruction of the object and the desire, he would find and according to formerly by the desired the experience of the same to souls the spirits of the compulsion of self-loving last contempt and the religious fact, and far the idea of a profound, with the highest sense of the grand and was a most benedy and by the desires of the nature of man and rank, as the astrement of a man and the i
------ temperature: 1.0
 nature of man and rank, as the astrement of a man and the idea which
ourselves to
our powerfue to a than the "think perhaps, tenders. that who
success when they makes metaphysical that the action of diverer to discovere
our
the often
an endiared themselves with those valid, thret releopmenty, and
stails very religious this here a onver deathion
people to at the oberquity and our mysperty
of physiological expressions, high is not, but he is not, and, and a
------ temperature: 1.2
ological expressions, high is not, but he is not, and, and and forowlosis, with reas intere; the grus, exeld of ellong mound; teo by the raiters form or facul--learning
derplise be shame through, in their fruit of stood hodardjge of modicate the case as if there is one
mean which if much and thiinge
in the whole of the knew beauthw, the gred, bstong of all that is the
philosophy-cornuny" than to villome
people, he isond, by than voluallow no, in the follog

注:第21轮

epoch 21
Epoch 1/1
200278/200278 [==============================] - 245s 1ms/step - loss: 1.3147
--- Generating with seed: "e that will captivate the
foreign ear as well as the native-"
------ temperature: 0.2
e that will captivate the
foreign ear as well as the native--and what is the moral moral and the most profound and the moral and the sense of the spirit and also the profound and the most profound of the same and the struggle of the spirit and according to the struggle of the subject to the most problem of all the same and as the sense of the spirit and and such a must not the sense of the christian the subject and and the same and something and also any o
------ temperature: 0.5
an the subject and and the same and something and also any one is always a man of the freeity to its far his entertain, and in every untersared to the same and in man, the accordion of the most bad and in the way to him who whose who tan except his forms of the most profound to a promise of the feelings that
it are the states, and and more interesting of
the own in reality is on the degree of the prevention and pressing society of the imagination in the wa
------ temperature: 1.0
prevention and pressing society of the imagination in the wait, and love of weir whatever of much in the spiritded of constitne exercised in . much
as one
allow anome of civeringel, at the own
free themselves carlior of a do no longer happens what aken saint
wa
ternipal and must also possible of the world will result is mislime to meatselness of a
snecialness, what
is lubus:
should if a more relientenance at himself against the imagination to the hyist who
------ temperature: 1.2
ntenance at himself against the imagination to the hyist who depression of datation! kulds of humazence which. shoild; there he while
only
or mebitie, and except out_ it was is be obligod of placidied which was estemout to all to our first every virtue of all too concepiuage at europe, birxatide. chorded: even wo-candably
that ilook" fourours, he amoudy, then all irnts, not successs it
terring, who conscienceselfolve of much for perhaps, it were also lettu

注:第58轮

epoch 58
Epoch 1/1
200278/200278 [==============================] - 234s 1ms/step - loss: 1.2422
--- Generating with seed: "t only by masculine shallow-pates), thus proves to be a rema"
------ temperature: 0.2
t only by masculine shallow-pates), thus proves to be a remain the democratically the stronger of the same and society, the sense of the spirit in the same things of the same the morals, and the spirit in the same things of the same things and the democratical sense of the same and the states and and spirit is a more of the same things interpretations of the same things of the same that the same that the same the stronger of the same that the sense of the 
------ temperature: 0.5
hat the same the stronger of the same that the sense of the sense of the depths, which is superficial of the last discovered, but it is a states of the spirit seems to the spirit of nature, and precisely the world be states without deceive the man of the same desires and accuracy, that he is the way of a being, and the
experience of a more the sense of a honours of the stopered the problem of the intimidations, and the spirit has a man of the charm of an e
------ temperature: 1.0
intimidations, and the spirit has a man of the charm of an endurably at the fundamental conscience as iall, man
degree of
refines, and sphinxs, this more indient within the hypether,
became and threement.

e ] quisicial interphites, and libert sweet this fashion. leables to this philosopife their spirit.

connotirs," as a god side helesly tights. it
is decisent,
in the godlown indutgurich them, and more noked ambitions, we have greman man, probaut more bes
------ temperature: 1.2
d more noked ambitions, we have greman man, probaut more best happi(s it. ther
reviteatilys, in the externt soy instenve-infest as to
the voluted functions, these hard. losess. good of invented-pastus fragt. becomian
aetlancemans of this
differing tire, where know by the leact, mysticome that
must were
subject of his fortime, and what. man--whithose, pain and beings!
leven under eviden of them we noid of ethical brosderion, in the fath judgme:
seldhe.--loo

注:第59轮

epoch 59
Epoch 1/1
200278/200278 [==============================] - 235s 1ms/step - loss: 1.2392
--- Generating with seed: "at do their retrograde
by-paths concern us! the main thing a"
------ temperature: 0.2
at do their retrograde
by-paths concern us! the main thing and superficially and still and sense of the same things and spirit to a man and self-possibility of the spirit to the same and conscience of the same morality in the command, the morality and conscience of the same and spirit to the same things will more the morality to the morality of the same thinking in the most morality and self-possibility to the same and the spirit to the morality and thereb
------ temperature: 0.5
bility to the same and the spirit to the morality and thereby predicated for the same and intermitud will be read to the senses. and he were a conscience of the words, in which there are this standard all the god-not believe the emotions and intellectualical compelline of a desire of the problem of the more and of death, in the realm of such animate the same and self-conscience of the philosophy in all the idealous worldlies of the distrust of his feminine
------ temperature: 1.0
n all the idealous worldlies of the distrust of his feminine-god's naincge of were suffering-allow the god, find and lofting the morality somethed one of herd
and must be not, and artimelntest moralistic of that science also the
realm of a diw"s whose wills to feel ventude to be such most life is injure: the pirtion is harn
failure by any proper of the bandsthies are tool! be rundinglance to satisfaction, whet it seems to  nothing like that great, a nount 
------ temperature: 1.2
faction, whet it seems to  nothing like that great, a nount dealies who is etsebway and last past of values this kind of yeats germans--but quaind
over your are desires
to ypart
your experion to precisely carry that given been cy!
with the temportaioredy interested misunderstandinating of goethredly; and as well,ist! whereg, becaused my untistory
over a thirgm. one madnered, it seems theist to nofetherouncals which
by
ecclusion or sullow wanto inder
flor a

---

总结

---

我们这里使用的随机文本种子是new faculty, and the jubilation reach its climax when kant

在第59轮的时候，文本几乎完全收敛，看起来更加的连贯

可见，较小的温度值会得到极端重复和可预测的文本，但是局部结构较为真实，所有的单词都是英文单词（单词就是字符的局部模式），但是随着温度的增大，生成的文本也会更加的出人意料，也会创造出新的单词。

---

写在最后

注：本文代码来自《Python 深度学习》，做成电子笔记的方式上传，仅供学习参考，作者均已运行成功，如有遗漏请练习本文作者

各位看官，都看到这里了，麻烦动动手指头给博主来个点赞8，您的支持作者最大的创作动力哟！
＜（＾－＾）＞
才疏学浅，若有纰漏，恳请斧正
本文章仅用于各位同志作为学习交流之用，不作任何商业用途，若涉及版权问题请速与作者联系，望悉知