使用Genism进行词向量训练:教程版
使用Genism进行词向量训练:教程版
1.起源
将文本换成向量后可以方便的使用SVM\logistics\deep learnig 等方法完成文本分类、标签、情感分析等实际任务.如何获得有效的词向量成为重要基础性工作.本博展示使用gensim包训练词向量的相关基础知识.
训练方法参考论文:
训练方法主要有CBOW与Skip-gram两种.
Deep learningvia word2vec’s “skip-gram and CBOW models”, using either hierarchical softmaxor negative sampling [1] [2].
| [1] |
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR, 2013. |
| [2] |
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS, 2013. |
CBOW Mehod
Skip-gram Method
2.完成训练的重要类、方法及参数含义
(1)model主类
sg defines thetraining algorithm. By default (sg=0), CBOW is used. Otherwise (sg=1), skip-gram isemployed.
size is thedimensionalityof the feature vectors.
window is the maximum distance between the current andpredicted word within a sentence.
alpha is the initiallearning rate (will linearly drop to zero as trainingprogresses).
min_count =低于该词频的词会被drop. ignore all words with total frequency lowerthan this.
workers = use thismanyworker threads to train the model (=faster training with multicoremachines).
hs = if 1,hierarchical softmax will be used for model training. If set to0 (default), and negative is non-zero,negative sampling will be used.
negative =if > 0, negative sampling will be used, the int for negativespecifies how many “noise words” should be drawn (usually between 5-20).Default is 5. If set to 0, no negative samping is used.
cbow_mean = if 0, use the sum of the context wordvectors. If 1 (default), use the mean. Only applies when cbow is used.
iter = number of iterations (epochs) over the corpus.
示例：
model = Word2Vec(sentences,size=100, window=5, min_count=5, workers=4)
(2)模型构建后(类实例化),再调用类的train方法之前一定要构建词树.
build_vocab(sentences, keep_raw_vocab=False, trim_rule=None)
Build vocabulary from a sequence of sentences (can be aonce-only generator stream). Each sentence must be a list of unicode strings.
词树就是基于词在语料中出现的词频构建的哈夫曼树，这样那些经常出现的词汇会在训练时更快被检索到，节省训练的搜索时间.
(3)
class gensim.models.word2vec.LineSentence(source, max_sentence_length=10000, limit=None)
一行一句话,每个单词以空格分隔，所以需要@XXX换成@”XX中”字符去除
(4)模型训练好后，可以load/save model
model.save(fname)
model = Word2Vec.load(fname) # you can continue training with the loaded model!
或者保存输出词向量
save_word2vec_format(fname, fvocab=None, binary=False)
Store theinput-hidden weight matrix in the same format used by theoriginal C word2vec-tool, for compatibility.
save_word2vec_format(fname, fvocab=None, binary=False)
(5)模型结果使用
model.most_similar(positive=['woman', 'king'], negative=['man'])
[('queen', 0.50882536), ...]
model.doesnt_match("breakfast cereal dinner lunch".split())
'cereal'#返回这个list中与其他词最大搭的词
model.similarity('woman', 'man')
0.73723527
model['computer'] # raw numpy vector of a word
array([-0.00449447, -0.00310097, 0.02421786, ...], dtype=float32)
trained_model.similarity('woman', 'man')
0.73723527#两单词cosine相似度
trained_model.n_similarity(['restaurant', 'japanese'], ['japanese', 'restaurant'])#两对词向量的cos相似度
1.0000000000000004
(6)其他相关知识
modelparameters are stored as matrices (NumPy arrays)
#vocabulary times 词汇计数
#size (size parameter)of floats 维度
内存估计:
主要内存占用：
100,000 uniquewords、200维
模型参数占用内存:100,000*200*4(float占4bytes)*3=~229MB
再加一部分内存：几MB
存储:词汇哈夫曼树(节省搜索空间)
加速:
model=Word2Vec(sentences,workers=4)
The workersparameter has only effect if you have Cython installed
参考链接:
1.Making sense of word2vec
http://rare-technologies.com/making-sense-of-word2vec/
2.Genism word2vec
http://rutumulkar.com/blog/2015/word2vec/
http://rare-technologies.com/word2vec-tutorial/
3.http://www.52nlp.cn/中英文维基百科语料上的word2vec实验
http://radimrehurek.com/gensim/models/word2vec.html