Python LDA主题模型实战
-
导入相关的包
https://github.com/lda-project/lda ：lda包的文档！采用LDA库，pip install lda
import numpy as np
import lda
12
X = lda.datasets.load_reuters()
X.shape
12
(395, 4258)
1
- 这里说明X是395行4258列的数据，说明有395个训练样本
vocab = lda.datasets.load_reuters_vocab()
len(vocab)# 这里是所有的词汇
12
4258
1
- 这里说明一个有4258个不重复的词语
1
- 选取前十个训练数据看一看
title = lda.datasets.load_reuters_titles()
title[:10]
12
('0 UK: Prince Charles spearheads British royal revolution. LONDON 1996-08-20',
'1 GERMANY: Historic Dresden church rising from WW2 ashes. DRESDEN, Germany 1996-08-21',
"2 INDIA: Mother Teresa's condition said still unstable. CALCUTTA 1996-08-23",
'3 UK: Palace warns British weekly over Charles pictures. LONDON 1996-08-25',
'4 INDIA: Mother Teresa, slightly stronger, blesses nuns. CALCUTTA 1996-08-25',
"5 INDIA: Mother Teresa's condition unchanged, thousands pray. CALCUTTA 1996-08-25",
'6 INDIA: Mother Teresa shows signs of strength, blesses nuns. CALCUTTA 1996-08-26',
"7 INDIA: Mother Teresa's condition improves, many pray. CALCUTTA, India 1996-08-25",
'8 INDIA: Mother Teresa improves, nuns pray for "miracle". CALCUTTA 1996-08-26',
'9 UK: Charles under fire over prospect of Queen Camilla. LONDON 1996-08-26')
12345678910
1
- 开始训练，这顶主题数目是20，迭代次数是1500次
model = lda.LDA(n_topics = 20, n_iter = 1500, random_state = 1) #初始化模型, n_iter 迭代次数
model.fit(X)
控制台输出：
INFO:lda:n_documents: 395
INFO:lda:vocab_size: 4258
INFO:lda:n_words: 84010
INFO:lda:n_topics: 20
INFO:lda:n_iter: 1500
INFO:lda:<0> log likelihood: -1051748
INFO:lda:<10> log likelihood: -719800
INFO:lda:<20> log likelihood: -699115
INFO:lda:<30> log likelihood: -689370
INFO:lda:<40> log likelihood: -684918
...
INFO:lda:<1450> log likelihood: -654884
INFO:lda:<1460> log likelihood: -655493
INFO:lda:<1470> log likelihood: -655415
INFO:lda:<1480> log likelihood: -655192
INFO:lda:<1490> log likelihood: -655728
INFO:lda:<1499> log likelihood: -655858
1234567891011121314151617181920212223
- 查看20个主题中的词分布
topic_word = model.topic_word_
print(topic_word.shape)
topic_word
查看输出：
(20, 4258)
array([[3.62505347e-06, 3.62505347e-06, 3.62505347e-06, ...,
3.62505347e-06, 3.62505347e-06, 3.62505347e-06],
[1.87498968e-02, 1.17916463e-06, 1.17916463e-06, ...,
1.17916463e-06, 1.17916463e-06, 1.17916463e-06],
[1.52206232e-03, 5.05668544e-06, 4.05040504e-03, ...,
5.05668544e-06, 5.05668544e-06, 5.05668544e-06],
...,
[4.17266923e-02, 3.93610908e-06, 9.05698699e-03, ...,
3.93610908e-06, 3.93610908e-06, 3.93610908e-06],
[2.37609835e-06, 2.37609835e-06, 2.37609835e-06, ...,
2.37609835e-06, 2.37609835e-06, 2.37609835e-06],
[3.46310752e-06, 3.46310752e-06, 3.46310752e-06, ...,
3.46310752e-06, 3.46310752e-06, 3.46310752e-06]])
- 得到每个主题的前8个词
for i, topic_dist in enumerate(topic_word):
print(np.array(vocab)[np.argsort(topic_dist)][:-9:-1])
12
['british' 'churchill' 'sale' 'million' 'major' 'letters' 'west' 'britain']
['church' 'government' 'political' 'country' 'state' 'people' 'party'
'against']
['elvis' 'king' 'fans' 'presley' 'life' 'concert' 'young' 'death']
['yeltsin' 'russian' 'russia' 'president' 'kremlin' 'moscow' 'michael'
'operation']
['pope' 'vatican' 'paul' 'john' 'surgery' 'hospital' 'pontiff' 'rome']
['family' 'funeral' 'police' 'miami' 'versace' 'cunanan' 'city' 'service']
['simpson' 'former' 'years' 'court' 'president' 'wife' 'south' 'church']
['order' 'mother' 'successor' 'election' 'nuns' 'church' 'nirmala' 'head']
['charles' 'prince' 'diana' 'royal' 'king' 'queen' 'parker' 'bowles']
['film' 'french' 'france' 'against' 'bardot' 'paris' 'poster' 'animal']
['germany' 'german' 'war' 'nazi' 'letter' 'christian' 'book' 'jews']
['east' 'peace' 'prize' 'award' 'timor' 'quebec' 'belo' 'leader']
["n't" 'life' 'show' 'told' 'very' 'love' 'television' 'father']
['years' 'year' 'time' 'last' 'church' 'world' 'people' 'say']
['mother' 'teresa' 'heart' 'calcutta' 'charity' 'nun' 'hospital'
'missionaries']
['city' 'salonika' 'capital' 'buddhist' 'cultural' 'vietnam' 'byzantine'
'show']
['music' 'tour' 'opera' 'singer' 'israel' 'people' 'film' 'israeli']
['church' 'catholic' 'bernardin' 'cardinal' 'bishop' 'wright' 'death'
'cancer']
['harriman' 'clinton' 'u.s' 'ambassador' 'paris' 'president' 'churchill'
'france']
['city' 'museum' 'art' 'exhibition' 'century' 'million' 'churches' 'set']
1234567891011121314151617181920212223242526
- 得到每句话在每个主题的分布，并得到每句话的最大主题
1
doc_topic = model.doc_topic_
print(doc_topic.shape) # 主题分布式395行，20列的矩阵，其中每一行对应一个训练样本在20个主题上的分布
print("第一个样本的主题分布是",doc_topic[0]) # 打印一下第一个样本的主题分布
print("第一个样本的最终主题是",doc_topic[0].argmax())
1234
(395, 20)
第一个样本的主题分布是 [4.34782609e-04 3.52173913e-02 4.34782609e-04 9.13043478e-03
4.78260870e-03 4.34782609e-04 9.13043478e-03 3.08695652e-02
5.04782609e-01 4.78260870e-03 4.34782609e-04 4.34782609e-04
3.08695652e-02 2.17826087e-01 4.34782609e-04 4.34782609e-04
4.34782609e-04 3.95652174e-02 4.34782609e-04 1.09130435e-01]
第一个样本的最终主题是 8
转载至：https://blog.csdn.net/jiangzhenkang/article/details/84335646