Python数据分析：文本相似度

Python数据分析：文本相似度

文本相似度：

• 度量文本间的相似性

• 使用词频表示文本特征

  • 文本中单词出现的频率或次数
  • 将文本表示成向量

• 向量间相似度

  • 余弦相似度
    $$\mathrm{sim}(A,B)=\cos (\theta )=\frac{A\cdot B}{\Vert A\Vert B\Vert }$$

• NLTK实现词频统计

import nltk
from nltk import FreqDist

text1 = 'I like the movie so much '
text2 = 'That is a good movie '
text3 = 'This is a great one '
text4 = 'That is a really bad movie '
text5 = 'This is a terrible movie'

text = text1 + text2 + text3 + text4 + text5
words = nltk.word_tokenize(text)
freq_dist = FreqDist(words)
print('词频：',freq_dist['is'])

运行：

# 取出常用的n=5个单词
n = 5

# 构造“常用单词列表”
most_common_words = freq_dist.most_common(n)
print(most_common_words)

运行：

def lookup_pos(most_common_words):
    """
        查找常用单词的位置
    """
    result = {}
    pos = 0
    for word in most_common_words:
        result[word[0]] = pos
        pos += 1
    return result

# 记录位置
std_pos_dict = lookup_pos(most_common_words)
print(std_pos_dict)

运行：

# 新文本
new_text = 'That one is a good movie. This is so good!'

# 初始化向量
freq_vec = [0] * n

# 分词
new_words = nltk.word_tokenize(new_text)

# 在“常用单词列表”上计算词频
for new_word in new_words:
    if new_word in list(std_pos_dict.keys()):
        freq_vec[std_pos_dict[new_word]] += 1

print(freq_vec)

运行：