【CCF大数据与计算智能大赛】互联网新闻情感分析（使用LSTM完成中文文本多分类任务）
使用LSTM完成文本多分类任务
这只是把keras的教程用于实战的练手项目，对于真正希望打出好的比赛成绩必须要用到各种（bert等）预训练模型
赛题地址：
https://www.datafountain.cn/competitions/350
代码源于keras作者的《Deep Learning with Python"》
https://nbviewer.jupyter.org/github/fchollet/deep-learning-with-python-notebooks/blob/master/3.5-classifying-movie-reviews.ipynb
原书中使用的IMDB 的数据做二分类问题，而这个赛题下需要做的是三分类问题，读入的是中文数据，需要对代码进行一些修改和添加
数据集解析
官方给出的数据集文件共三个
Train_DataSet.csv
Test_DataSet.csv
Train_DataSet_Label.csv
训练集共id,title,content三列，存储了ID，新闻标题，新闻内容三种内容
我们需要读入数据集
import pandas as pd
train_df = pd.read_csv('Train_DataSet.csv', header=0)
test_df = pd.read_csv('Test_DataSet.csv', header=0)
train_lable = pd.read_csv('Train_DataSet_Label.csv', header=0)
print(train_df.shape,test_df.shape,train_lable)
#变量转换为字符型
train_df['title'] = train_df['title'].astype(str)
train_df['content'] = train_df['content'].astype(str)
test_df['title'] = test_df['title'].astype(str)
test_df['content'] = test_df['content'].astype(str)
#拼接标题和内容
#Phrase_list = train_df['title'].astype(str)
Phrase_list = train_df['title'].str.cat(train_df['content'],sep= '-')
X_test = test_df['title'].str.cat(test_df['content'],sep= '-')
print(Phrase_list)
然后对中文进行分词
jieba分词
import jieba
import pandas as pd
train_df = pd.read_csv('Train_DataSet.csv', header=0)
test_df = pd.read_csv('Test_DataSet.csv', header=0)
train_lable = pd.read_csv('Train_DataSet_Label.csv', header=0)
print(train_df.shape,test_df.shape,train_lable)
#变量转换为字符型
train_df['title'] = train_df['title'].astype(str)
train_df['content'] = train_df['content'].astype(str)
test_df['title'] = test_df['title'].astype(str)
test_df['content'] = test_df['content'].astype(str)
#拼接标题和内容
#Phrase_list = train_df['title'].astype(str)
Phrase_list = train_df['title'].str.cat(train_df['content'],sep= '-')
X_test = test_df['title'].str.cat(test_df['content'],sep= '-')
print(Phrase_list)
Phrase_list1 = []
X_test1 = []
k = 0
for i in Phrase_list:
seg_list = jieba.cut(i, cut_all=True)
seg_list = " ".join(seg_list)
Phrase_list1.append(seg_list)
k = k+ 1
print(k,'/',len(Phrase_list),'-----------------------------------------')
k=0
for i in X_test:
seg_list = jieba.cut(i, cut_all=True)
seg_list = " ".join(seg_list)
X_test1.append(seg_list)
k = k+ 1
print(k,'/',len(X_test),'-----------------------------------------')
k=0
Phrase_list1 = pd.Series(Phrase_list1)
X_test1 = pd.Series(X_test1)
GloVe训练词向量
中文词训练可以参考这篇博文
https://blog.csdn.net/weixin_37947156/article/details/83145778
简而言之，把title和content的内容分词后，写成一个txt文件
从GitHub下载GloVe代码，https://github.com/stanfordnlp/GloVe
将语料txt放入到Glove的主文件夹下。注释掉
将语料corpus.txt放入到Glove的主文件夹下。
修改bash 打开demo.sh，修改相应的内容
因为demo默认是下载网上的语料来训练的，因此如果要训练自己的语料，需要注释掉 修改参数设置，将CORPUS设置成语料的名字 执行bash文件
得到了词表文件，vocab.txt
将vocab.txt复制到工程下
keras搭建LSTM分类
glove_dir = r'D:\CCF互联网新闻情感分析\glove.6B\vocab.txt'
#加载词表文件
embeddings_index = { }
f = open(glove_dir,encoding='UTF-8')
for line in f:
values = line.split()
word = values[0]
coefs = np.asanyarray(values[1:],dtype='float32')
embeddings_index[word] = coefs
f.close()
print('Found %s word vectors.' % len(embeddings_index))
embedding_dim = 300
embedding_matrix = np.zeros((max_words, embedding_dim))
for word, i in word_index.items():
embedding_vector = embeddings_index.get(word)
if i < max_words:
if embedding_vector is not None:
# Words not found in embedding index will be all-zeros.
embedding_matrix[i] = embedding_vector
########################################
from keras.models import Sequential
from keras.layers import Flatten, Dense, Embedding,SimpleRNN
from keras.layers import LSTM
from keras.layers import GRU
max_features = max_words
model = Sequential()
model.add(Embedding(max_features,300))
model.add(LSTM(64,return_sequences=True))
model.add(LSTM(32,activation='relu',
dropout=0.1,
))
model.add(Dense(3,activation='softmax'))#三分类问题，此处需要换成3，并使用激活函数softmax
#######################################################################
model.layers[0].set_weights([embedding_matrix])
model.layers[0].trainable = False
##########################################################################
model.compile(optimizer='rmsprop', loss = 'categorical_crossentropy',metrics=['acc'])
#损失函数改为
history = model.fit(x_train, y_train,
epochs=37,
batch_size=128,
validation_split=0.1
)
预测结果
from keras.models import load_model
model.save('LSTM_model.h5') # creates a HDF5 file 'my_model.h5'
del model # deletes the existing model
# returns a compiled model
# identical to the previous one
model = load_model('LSTM_model.h5')
y = model.predict_classes(X_test)
print(y)
#将标签和预测结果组合并写入csv文件。
ID = test_df['id']
def write_preds(ID , fname):
pd.DataFrame({"PhraseId": ID, "Sentiment": y}).to_csv(fname, index=False, header=True)
#sentiment-analysis-on-movie-reviews
write_preds(ID , "CCF-LSTMkeras.csv")
#绘制结果
import matplotlib.pyplot as plt
acc = history.history['acc']
val_acc = history.history['val_acc']
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(1, len(acc) + 1)
plt.plot(epochs, acc,'bo',label = 'Training acc')
plt.plot(epochs,val_acc,'b',label = 'Validation acc')
plt.title('Taining and validation accuracy')
plt.legend()
plt.figure()
plt.plot(epochs, loss,'bo',label = 'Training loss')
plt.plot(epochs,val_loss,'b',label = 'Validation loss')
plt.title('Taining and validation accuracy')
plt.legend()
plt.show()
将找到过拟合之前的epochs，所得结果提交CCF-LSTMkeras.csv，完成！