以下学习笔记来源于 Coggle 30 Days of ML(22年1&2月)
链接:https://coggle.club/blog/30days-of-ml-202201
比赛链接:https://aistudio.baidu.com/aistudio/competition/detail/45/0/task-definition
import pandas as pd
import jieba
from gensim.models import Word2Vec
from gensim.models.word2vec import LineSentence
#读取tsv文件的方法
def read_tsv(input_file,columns):
with open(input_file,"r",encoding="utf-8") as file:
lines = []
count = 1
for line in file:
if len(line.strip().split("\t")) != 1:
lines.append([count]+line.strip().split("\t"))
count += 1
df = pd.DataFrame(lines)
df.columns = columns
return df
bq_train=read_tsv('./bq_corpus/train.tsv',['index','text1','text2','label'])
lcqmc_train=read_tsv('./lcqmc/train.tsv',['index','text1','text2','label'])
pawsx_train=read_tsv('./paws-x-zh/train.tsv',['index','text1','text2','label'])
bq_test=read_tsv('./bq_corpus/test.tsv',['index','text1','text2'])
lcqmc_test=read_tsv('./lcqmc/test.tsv',['index','text1','text2'])
pawsx_test=read_tsv('./paws-x-zh/test.tsv',['index','text1','text2'])
Word2Vec第一个参数sentences要求是是预处理后的训练语料库,需要输入一个可迭代的列表,如果是句子,需要进行分词,如果为文件需要将文件处理为每一行对应一个已经分词,以空格隔开的句子。然后可以使用gensim.models.word2vec.LineSentence
把txt文件转为所需要的格式。这里我们采用第二中方法,将简单处理后的语料保存到文件,若之后任务需要的话方便直接读取。
df_train=pd.concat([bq_train,lcqmc_train,pawsx_train])
df_test=pd.concat([bq_test,lcqmc_test,pawsx_test])
corpus=df_train['text1'].values.tolist()+df_train['text2'].values.tolist()
tokenized_corpus=[' '.join(jieba.lcut(text)) for text in corpus]
f = open('./word2vec_data/corpus.txt', 'w',encoding='utf-8')
for i in tokenized_corpus:
f.write(i + '\n')
def train_word2vec():
corpus=open('./word2vec_data/corpus.txt', 'r',encoding='utf-8')
model = Word2Vec(LineSentence(corpus), sg=0,vector_size=100,
window=5, min_count=5, workers=9)
model.save('./word2vec_data/text_similarity.word2vec')
train_word2vec()
示例:计算词语之间的相似度
model=Word2Vec.load('./word2vec_data/text_similarity.word2vec')
print(model.wv.similarity('贷款','借款'))
print(model.wv.similarity('交易','准备'))