VADER lexicon基本结构
$: -1.5 0.80623 [-1, -1, -1, -1, -3, -1, -3, -1, -2, -1]
%) -0.4 1.0198 [-1, 0, -1, 0, 0, -2, -1, 2, -1, 0]
%-) -1.5 1.43178 [-2, 0, -2, -2, -1, 2, -2, -3, -2, -3]
&-: -0.4 1.42829 [-3, -1, 0, 0, -1, -1, -1, 2, -1, 2]
&: -0.7 0.64031 [0, -1, -1, -1, 1, -1, -1, -1, -1, -1]
( '}{' ) 1.6 0.66332 [1, 2, 2, 1, 1, 2, 2, 1, 3, 1]
worriedly -2.0 0.44721 [-2, -2, -3, -2, -2, -2, -2, -1, -2, -2]
worrier -1.8 0.6 [-2, -2, -1, -2, -1, -3, -2, -2, -1, -2]
worriers -1.7 0.45826 [-2, -1, -2, -2, -2, -2, -1, -2, -1, -2]
worries -1.8 0.6 [-2, -2, -1, -2, -1, -2, -2, -3, -1, -2]
worriment -1.5 0.67082 [-1, -2, -1, -1, -1, -2, -1, -3, -1, -2]
worriments -1.9 0.7 [-2, -1, -2, -3, -1, -2, -3, -1, -2, -2]
worrisome -1.7 0.64031 [-1, -1, -1, -2, -1, -2, -3, -2, -2, -2]
vader_lexicon.txt文件是以tab键分割的四列字段组成的
第一列：单词或词组（token）
第二列：人类情感打分的均值
第三列：它是单词的标准偏差，假设它遵循正态分布
第四列：这是在实验中10个人对单词进行评分的列表。
实际代码或情感计算不使用第3列和第4列。因此，如果你想根据你的需求更新词典，你可以将最后两列留空，或者用一个随机数和一个列表填充。
VADER 源码如何加载lexicon?
def __init__(self, lexicon_file="sentiment/vader_lexicon.zip/vader_lexicon/vader_lexicon.txt"):
self.lexicon_file = nltk.data.load(lexicon_file)
self.lexicon = self.make_lex_dict()
def make_lex_dict(self):
"""
Convert lexicon file to a dictionary
"""
lex_dict = {}
for line in self.lexicon_file.split('\n'):
(word, measure) = line.strip().split('\t')[0:2]
lex_dict[word] = float(measure)
return lex_dict
print(self.lexicon)
{'guiltier': -2.0, 'proud': 2.1, 'freeholds': 1.0, 'madness': -1.9, 'unsecured': -1.6, 'wilco': 0.9, 'doom': -1.7, 'crazy': -1.4, '|o:': -0.9, 'faultlessness': 1.1, 'triumphs': 2.0, 'excruciatingly': -2.9, 'warsaws': -0.2, 'insecurely': -1.4, 'abusing': -2.0, 'confusions': -0.9, 'relieve': 1.5, 'futile': -1.9, 'stinkpots': -0.7, ...}
VADER lexicon更新使用
- 预判某个词的正负中极性
import nltk
from nltk.tokenize import word_tokenize, RegexpTokenizer
from nltk.sentiment.vader import SentimentIntensityAnalyzer
Analyzer = SentimentIntensityAnalyzer()
sentence = 'enter your text to test'
tokenized_sentence = nltk.word_tokenize(sentence)
pos_word_list=[]
neu_word_list=[]
neg_word_list=[]
for word in tokenized_sentence:
if (Analyzer.polarity_scores(word)['compound']) >= 0.1:
pos_word_list.append(word)
elif (Analyzer.polarity_scores(word)['compound']) <= -0.1:
neg_word_list.append(word)
else:
neu_word_list.append(word)
print('Positive:',pos_word_list)
print('Neutral:',neu_word_list)
print('Negative:',neg_word_list)
score = Analyzer.polarity_scores(sentence)
print('\nScores:', score)
对vader lexicon可以对其进行增添新领域的lexicon
- 增加更新前
sentence = 'stocks were volatile on Tuesday due to the recent calamities in the Chinese market'
Positive: []
Neutral: ['stocks', 'were', 'volatile', 'on', 'Tuesday', 'due', 'to', 'the', 'recent', 'calamities', 'in', 'the', 'Chinese', 'markets']
Negative: []
Scores: {'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0}
- 增加更新后
Analyzer.lexicon.update(Financial_Lexicon)
sentence = 'stocks were volatile on Tuesday due to the recent calamities in the Chinese market'
Positive: []
Neutral: ['stocks', 'were', 'on', 'Tuesday', 'due', 'to', 'the', 'recent', 'in', 'the', 'Chinese', 'markets']
Negative: ['volatile', 'calamities']
Scores: {'neg': 0.294, 'neu': 0.706, 'pos': 0.0, 'compound': -0.6124}