NLP-Task1:基于机器学习的文本分类

NLP-Task1:基于机器学习的文本分类

实现基于logistic/softmax regression的文本分类

数据集:Classify the sentiment of sentences from the Rotten Tomatoes dataset
网盘下载链接见文末

  • 需要了解的知识点:
    • 文本特征表示:Bag-of-Word,N-gram
    • 分类器:logistic/softmax regression,损失函数、(随机)梯度下降、特征选择
    • 数据集:训练集/验证集/测试集的划分
  • 实验:
    • 分析不同的特征、损失函数、学习率对最终分类性能的影响
    • shuffle 、batch、mini-batch
本文参考:NLP-Beginner 任务一:基于机器学习的文本分类(超详细!!

一、工作简介

1.1 主要内容

利用softmax regression对文本情感进行分类

1.2 数据集

训练集共156060条英文评论,情感分共0-4类
NLP-Task1:基于机器学习的文本分类_第1张图片

二、特征提取(Feature extraction)

2.1 词袋特征(Bag-of-word)

词袋模型即所有单词相互独立

在一句话中,存在于句子的单词 (不区分大小写),则对应的向量位置上的数字为1,反之为0,通过这种方式,可以把一个句子变成一个由数字表示的0-1向量。虽然转换方式非常简单,但是它没有考虑词序

  • 词袋:i, you, him, her, love, hate, but
  • 句子:I love you
  • 向量:[1,1,0,0,1,0,0]

2.2 N元特征(N-gram)

N元特征考虑词序,如当N=2时,I love you不再看作是I, love, you这三个单词,而是 I love, love you 这两个词组

通常来说,使用N元特征时,会一并使用1, 2, …, N-1元特征,数据量大的输入时,较大的N会导致结果模型训练极度缓慢

  • 特征(N=1&2):i, you, him, her, love, hate, but, i-love, i-hate, love-you, hate-you, you-but, but-love, but-hate, love-him, hate-him
  • 句子:I love you
  • 向量:[1,1,0,0,1,0,0,0,1,0,1,0,0,0,0,0,0]

三、Softmax Regression

3.1 softmax介绍

给定一个输入x,输出一个y向量,y的长度即为类别的个数,总和为1,y中的数字是各类别的概率

  • 0-4共五类,则 y = [ 0 , 0 , 0.7 , . 25 , 0.05 ] T y=[0,0,0.7,.25,0.05]^T y=[0,0,0.7,.25,0.05]T表示类别2的概率为0.7,类别3的概率为0.25,类别4的概率为0.05

公式:

p ( y = c ∣ x ) = s o f t m a x ( w c T x ) = e x p ( w c T x ) ∑ c ′ = 1 C e x p ( w c T x ) p(y=c|x)=softmax(w^T_cx)=\frac{exp(w^T_cx)}{\sum_{c'=1}^Cexp(w^T_cx)} p(y=cx)=softmax(wcTx)=c=1Cexp(wcTx)exp(wcTx)

其中 w c w_c wc是第 c c c类的权重向量,记 W = [ w 1 , w 2 , . . . , w C ] W=[w_1,w_2,...,w_C] W=[w1,w2,...,wC]为权重矩阵,则上述公式可以表示为列向量:

y ^ = f ( x ) = s o f t m a x ( W T x ) = e x p ( W T x ) 1 ⃗ c T e x p ( W T x ) \hat{y}=f(x)=softmax(W^Tx)=\frac{exp(W^Tx)}{\vec{1}^T_cexp(W^Tx)} y^=f(x)=softmax(WTx)=1 cTexp(WTx)exp(WTx)

y ^ \hat{y} y^ 中的每一个元素,代表对应类别的概率,接下来求解参数矩 W W W

3.2 损失函数

用损失函数对模型的好坏做出评价

在此选择交叉熵损失函数,对每一个样本n,其损失值为:

L ( f W ( x ( n ) ) , y ( n ) ) = − ∑ c = 1 C y c ( n ) l o g y ^ c ( n ) = − ( y ( n ) ) T l o g y ^ ( n ) L(fW(x^{(n)}),y^{(n)})=-\sum_{c=1}^Cy_c^{(n)}log\hat{y}_c^{(n)}=-(y^{(n)})^Tlog\hat{y}^{(n)} L(fW(x(n)),y(n))=c=1Cyc(n)logy^c(n)=(y(n))Tlogy^(n)

= − ( y ( n ) ) T l o g ( e x p ( W T x ) 1 ⃗ c T e x p ( W T x ) ) =-(y^{(n)})^Tlog(\frac{exp(W^Tx)}{\vec{1}^T_cexp(W^Tx)}) =(y(n))Tlog(1 cTexp(WTx)exp(WTx))

其中 y ( n ) = ( I ( c = 0 ) , I ( c = 1 ) , . . . , I ( c = C ) ) y^{(n)}=(I(c=0),I(c=1),...,I(c=C)) y(n)=(I(c=0),I(c=1),...,I(c=C)),是一个one-hot向量,只有一个元素是1,其余都是0

对于总体样本,总的损失值则是每个样本损失值的平均,即

L ( W ) = L ( f W ( x ) , y ) = 1 N ∑ n = 1 N L ( f W ( x ( n ) ) , y ( n ) ) L(W)=L(fW(x),y)=\frac{1}{N}\sum_{n=1}^NL(fW(x^{(n)}),y^{(n)}) L(W)=L(fW(x),y)=N1n=1NL(fW(x(n)),y(n))

通过找到损失函数的最小值,确定最优的参数矩阵 W W W

3.3 梯度下降

∂ L ( W ) ∂ W = − 1 N ∑ n = 1 N x ( n ) ( y ( n ) − y ^ W ( n ) ) T \frac{\partial L(W)}{\partial W}=-\frac{1}{N}\sum_{n=1}^Nx^{(n)}(y^{(n)}-\hat{y}^{(n)}_W)^T WL(W)=N1n=1Nx(n)(y(n)y^W(n))T

1、整批量梯度下降(Batch)参数更新:

W t + 1 ← W t − α ∂ L ( W t ) ∂ W t W_{t+1}\gets W_t-\alpha\frac{\partial L(W_t)}{\partial W_t} Wt+1WtαWtL(Wt)

2、随机梯度下降(Shuffle)参数更新,每次只随机选取一个样本:

Δ W = − x ( n ) ( y ( n ) − y ^ W ( n ) ) T \Delta W=-x^{(n)}(y^{(n)}-\hat{y}^{(n)}_W)^T ΔW=x(n)(y(n)y^W(n))T

W t + 1 ← W t − α ( Δ W t ) W_{t+1}\gets W_t-\alpha(\Delta W_t) Wt+1Wtα(ΔWt)

3、小批量梯度下降(Mini-Batch)参数更新,每次随机选取一些样本(如k个),设样本集合为S:

Δ W = − 1 k ∑ n ∈ S x ( n ) ( y ( n ) − y ^ W ( n ) ) T \Delta W=-\frac{1}{k}\sum\limits_{n\in S}x^{(n)}(y^{(n)}-\hat{y}^{(n)}_W)^T ΔW=k1nSx(n)(y(n)y^W(n))T

W t + 1 ← W t − α ( Δ W t ) W_{t+1}\gets W_t-\alpha(\Delta W_t) Wt+1Wtα(ΔWt)

四、代码实现

4.1 实验基本设置

  • 样本个数:1000
  • 训练集与测试集:7:3
  • 学习率: 1 0 − 3 10^{-3} 103~ 1 0 4 10^4 104
  • Mini-Batch大小:1%

(分析三种梯度下降的优劣时,需要控制每种梯度下降中计算梯度的次数一致)

4.2 代码

4.2.1 main.py

import numpy
import csv
import random
from feature import Bag,Gram
from comparison_plot import alpha_gradient_plot

# 数据读取
with open('train.tsv') as f:
    tsvreader = csv.reader(f, delimiter='\t')
    temp = list(tsvreader)

# 初始化
data = temp[1:]
max_item=1000
random.seed(2021)
numpy.random.seed(2021)

# 特征提取
bag=Bag(data,max_item)
bag.get_words()
bag.get_matrix()

gram=Gram(data, dimension=2, max_item=max_item)
gram.get_words()
gram.get_matrix()

# 画图
alpha_gradient_plot(bag,gram,10000,10)  # 计算10000次
alpha_gradient_plot(bag,gram,100000,10)  # 计算100000次

4.2.2 特征提取—feature.py

import numpy
import random


def data_split(data, test_rate=0.3, max_item=1000):
    """把数据按一定比例划分成训练集和测试集"""
    train = list()
    test = list()
    i = 0
    for datum in data:
        i += 1
        if random.random() > test_rate:
            train.append(datum)
        else:
            test.append(datum)
        if i > max_item:
            break
    return train, test


class Bag:
    """Bag of words"""
    def __init__(self, my_data, max_item=1000):
        self.data = my_data[:max_item]
        self.max_item=max_item
        self.dict_words = dict()  # 单词到单词编号的映射
        self.len = 0  # 记录有几个单词
        self.train, self.test = data_split(my_data, test_rate=0.3, max_item=max_item)
        self.train_y = [int(term[3]) for term in self.train]  # 训练集类别
        self.test_y = [int(term[3]) for term in self.test]  # 测试集类别
        self.train_matrix = None  # 训练集的0-1矩阵(每行一个句子)
        self.test_matrix = None  # 测试集的0-1矩阵(每行一个句子)

    def get_words(self):
        for term in self.data:
            s = term[2]
            s = s.upper()  # 记得要全部转化为大写!!(或者全部小写,否则一个单词例如i,I会识别成不同的两个单词)
            words = s.split()
            for word in words:  # 一个一个单词寻找
                if word not in self.dict_words:
                    self.dict_words[word] = len(self.dict_words)
        self.len = len(self.dict_words)
        self.test_matrix = numpy.zeros((len(self.test), self.len))  # 初始化0-1矩阵
        self.train_matrix = numpy.zeros((len(self.train), self.len))  # 初始化0-1矩阵

    def get_matrix(self):
        for i in range(len(self.train)):  # 训练集矩阵
            s = self.train[i][2]
            words = s.split()
            for word in words:
                word = word.upper()
                self.train_matrix[i][self.dict_words[word]] = 1
        for i in range(len(self.test)):  # 测试集矩阵
            s = self.test[i][2]
            words = s.split()
            for word in words:
                word = word.upper()
                self.test_matrix[i][self.dict_words[word]] = 1


class Gram:
    """N-gram"""
    def __init__(self, my_data, dimension=2, max_item=1000):
        self.data = my_data[:max_item]
        self.max_item = max_item
        self.dict_words = dict()  # 特征到t正编号的映射
        self.len = 0  # 记录有多少个特征
        self.dimension = dimension  # 决定使用几元特征
        self.train, self.test = data_split(my_data, test_rate=0.3, max_item=max_item)
        self.train_y = [int(term[3]) for term in self.train]  # 训练集类别
        self.test_y = [int(term[3]) for term in self.test]  # 测试集类别
        self.train_matrix = None  # 训练集0-1矩阵(每行代表一句话)
        self.test_matrix = None  # 测试集0-1矩阵(每行代表一句话)

    def get_words(self):
        for d in range(1, self.dimension + 1):  # 提取 1-gram, 2-gram,..., N-gram 特征
            for term in self.data:
                s = term[2]
                s = s.upper()  # 记得要全部转化为大写!!(或者全部小写,否则一个单词例如i,I会识别成不同的两个单词)
                words = s.split()
                for i in range(len(words) - d + 1):  # 一个一个特征找
                    temp = words[i:i + d]
                    temp = "_".join(temp)  # 形成i d-gram 特征
                    if temp not in self.dict_words:
                        self.dict_words[temp] = len(self.dict_words)
        self.len = len(self.dict_words)
        self.test_matrix = numpy.zeros((len(self.test), self.len))  # 训练集矩阵初始化
        self.train_matrix = numpy.zeros((len(self.train), self.len))  # 测试集矩阵初始化

    def get_matrix(self):
        for d in range(1, self.dimension + 1):
            for i in range(len(self.train)):  # 训练集矩阵
                s = self.train[i][2]
                s = s.upper()
                words = s.split()
                for j in range(len(words) - d + 1):
                    temp = words[j:j + d]
                    temp = "_".join(temp)
                    self.train_matrix[i][self.dict_words[temp]] = 1
            for i in range(len(self.test)):  # 测试集矩阵
                s = self.test[i][2]
                s = s.upper()
                words = s.split()
                for j in range(len(words) - d + 1):
                    temp = words[j:j + d]
                    temp = "_".join(temp)
                    self.test_matrix[i][self.dict_words[temp]] = 1

4.2.3 Softmax regression.py

import numpy
import random


class Softmax:
    """Softmax regression"""
    def __init__(self, sample, typenum, feature):
        self.sample = sample  # 训练集样本个数
        self.typenum = typenum  # (情感)种类个数
        self.feature = feature  # 0-1向量的长度
        self.W = numpy.random.randn(feature, typenum)  # 参数矩阵W初始化

    def softmax_calculation(self, x):
        """x是向量,计算softmax值"""
        exp = numpy.exp(x - numpy.max(x))  # 先减去最大值防止指数太大溢出
        return exp / exp.sum()

    def softmax_all(self, wtx):
        """wtx是矩阵,即许多向量叠在一起,按行计算softmax值"""
        wtx -= numpy.max(wtx, axis=1, keepdims=True) # 先减去行最大值防止指数太大溢出
        wtx = numpy.exp(wtx)
        wtx /= numpy.sum(wtx, axis=1, keepdims=True)
        return wtx

    def change_y(self, y):
        """把(情感)种类转换为一个one-hot向量"""
        ans = numpy.array([0] * self.typenum)
        ans[y] = 1
        return ans.reshape(-1, 1)

    def prediction(self, X):
        """给定0-1矩阵X,计算每个句子的y_hat值(概率)"""
        prob = self.softmax_all(X.dot(self.W))
        return prob.argmax(axis=1)

    def correct_rate(self, train, train_y, test, test_y):
        """计算训练集和测试集的准确率"""
        # train set
        n_train = len(train)
        pred_train = self.prediction(train)
        train_correct = sum([train_y[i] == pred_train[i] for i in range(n_train)]) / n_train
        # test set
        n_test = len(test)
        pred_test = self.prediction(test)
        test_correct = sum([test_y[i] == pred_test[i] for i in range(n_test)]) / n_test
        print(train_correct, test_correct)
        return train_correct, test_correct

    def regression(self, X, y, alpha, times, strategy="mini", mini_size=100):
        """Softmax regression"""
        if self.sample != len(X) or self.sample != len(y):
            raise Exception("Sample size does not match!")  # 样本个数不匹配
        if strategy == "mini":  # mini-batch
            for i in range(times):
                increment = numpy.zeros((self.feature, self.typenum))  # 梯度初始为0矩阵
                for j in range(mini_size):  # 随机抽K次
                    k = random.randint(0, self.sample - 1)
                    yhat = self.softmax_calculation(self.W.T.dot(X[k].reshape(-1, 1)))
                    increment += X[k].reshape(-1, 1).dot((self.change_y(y[k]) - yhat).T)  # 梯度加和
                # print(i * mini_size)
                self.W += alpha / mini_size * increment  # 参数更新
        elif strategy == "shuffle":  # 随机梯度
            for i in range(times): 
                k = random.randint(0, self.sample - 1)  # 每次抽一个
                yhat = self.softmax_calculation(self.W.T.dot(X[k].reshape(-1, 1)))
                increment = X[k].reshape(-1, 1).dot((self.change_y(y[k]) - yhat).T)  # 计算梯度
                self.W += alpha * increment  # 参数更新
                # if not (i % 10000):
                #     print(i)
        elif strategy=="batch":  # 整批量梯度
            for i in range(times):
                increment = numpy.zeros((self.feature, self.typenum))  ## 梯度初始为0矩阵
                for j in range(self.sample):  # 所有样本都要计算
                    yhat = self.softmax_calculation(self.W.T.dot(X[j].reshape(-1, 1)))
                    increment += X[j].reshape(-1, 1).dot((self.change_y(y[j]) - yhat).T)  # 梯度加和
                # print(i)
                self.W += alpha / self.sample * increment  # 参数更新
        else:
            raise Exception("Unknown strategy")

4.2.4 结果分析及可视化—comparison_plot.py

import matplotlib.pyplot
from mysoftmax_regression import Softmax


def alpha_gradient_plot(bag,gram, total_times, mini_size):
    """Plot categorization verses different parameters."""
    alphas = [0.001, 0.01, 0.1, 1, 10, 100, 1000, 10000]

    # Bag of words

    # Shuffle
    shuffle_train = list()
    shuffle_test = list()
    for alpha in alphas:
        soft = Softmax(len(bag.train), 5, bag.len)
        soft.regression(bag.train_matrix, bag.train_y, alpha, total_times, "shuffle")
        r_train, r_test = soft.correct_rate(bag.train_matrix, bag.train_y, bag.test_matrix, bag.test_y)
        shuffle_train.append(r_train)
        shuffle_test.append(r_test)

    # Batch
    batch_train = list()
    batch_test = list()
    for alpha in alphas:
        soft = Softmax(len(bag.train), 5, bag.len)
        soft.regression(bag.train_matrix, bag.train_y, alpha, int(total_times/bag.max_item), "batch")
        r_train, r_test = soft.correct_rate(bag.train_matrix, bag.train_y, bag.test_matrix, bag.test_y)
        batch_train.append(r_train)
        batch_test.append(r_test)

    # Mini-batch
    mini_train = list()
    mini_test = list()
    for alpha in alphas:
        soft = Softmax(len(bag.train), 5, bag.len)
        soft.regression(bag.train_matrix, bag.train_y, alpha, int(total_times/mini_size), "mini",mini_size)
        r_train, r_test= soft.correct_rate(bag.train_matrix, bag.train_y, bag.test_matrix, bag.test_y)
        mini_train.append(r_train)
        mini_test.append(r_test)
    matplotlib.pyplot.subplot(2,2,1)
    matplotlib.pyplot.semilogx(alphas,shuffle_train,'r--',label='shuffle')
    matplotlib.pyplot.semilogx(alphas, batch_train, 'g--', label='batch')
    matplotlib.pyplot.semilogx(alphas, mini_train, 'b--', label='mini-batch')
    matplotlib.pyplot.semilogx(alphas,shuffle_train, 'ro-', alphas, batch_train, 'g+-',alphas, mini_train, 'b^-')
    matplotlib.pyplot.legend()
    matplotlib.pyplot.title("Bag of words -- Training Set")
    matplotlib.pyplot.xlabel("Learning Rate")
    matplotlib.pyplot.ylabel("Accuracy")
    matplotlib.pyplot.ylim(0,1)
    matplotlib.pyplot.subplot(2, 2, 2)
    matplotlib.pyplot.semilogx(alphas, shuffle_test, 'r--', label='shuffle')
    matplotlib.pyplot.semilogx(alphas, batch_test, 'g--', label='batch')
    matplotlib.pyplot.semilogx(alphas, mini_test, 'b--', label='mini-batch')
    matplotlib.pyplot.semilogx(alphas, shuffle_test, 'ro-', alphas, batch_test, 'g+-', alphas, mini_test, 'b^-')
    matplotlib.pyplot.legend()
    matplotlib.pyplot.title("Bag of words -- Test Set")
    matplotlib.pyplot.xlabel("Learning Rate")
    matplotlib.pyplot.ylabel("Accuracy")
    matplotlib.pyplot.ylim(0, 1)

    # N-gram
    # Shuffle
    shuffle_train = list()
    shuffle_test = list()
    for alpha in alphas:
        soft = Softmax(len(gram.train), 5, gram.len)
        soft.regression(gram.train_matrix, gram.train_y, alpha, total_times, "shuffle")
        r_train, r_test = soft.correct_rate(gram.train_matrix, gram.train_y, gram.test_matrix, gram.test_y)
        shuffle_train.append(r_train)
        shuffle_test.append(r_test)

    # Batch
    batch_train = list()
    batch_test = list()
    for alpha in alphas:
        soft = Softmax(len(gram.train), 5, gram.len)
        soft.regression(gram.train_matrix, gram.train_y, alpha, int(total_times / gram.max_item), "batch")
        r_train, r_test = soft.correct_rate(gram.train_matrix, gram.train_y, gram.test_matrix, gram.test_y)
        batch_train.append(r_train)
        batch_test.append(r_test)

    # Mini-batch
    mini_train = list()
    mini_test = list()
    for alpha in alphas:
        soft = Softmax(len(gram.train), 5, gram.len)
        soft.regression(gram.train_matrix, gram.train_y, alpha, int(total_times / mini_size), "mini", mini_size)
        r_train, r_test = soft.correct_rate(gram.train_matrix, gram.train_y, gram.test_matrix, gram.test_y)
        mini_train.append(r_train)
        mini_test.append(r_test)
    matplotlib.pyplot.subplot(2, 2, 3)
    matplotlib.pyplot.semilogx(alphas, shuffle_train, 'r--', label='shuffle')
    matplotlib.pyplot.semilogx(alphas, batch_train, 'g--', label='batch')
    matplotlib.pyplot.semilogx(alphas, mini_train, 'b--', label='mini-batch')
    matplotlib.pyplot.semilogx(alphas, shuffle_train, 'ro-', alphas, batch_train, 'g+-', alphas, mini_train, 'b^-')
    matplotlib.pyplot.legend()
    matplotlib.pyplot.title("N-gram -- Training Set")
    matplotlib.pyplot.xlabel("Learning Rate")
    matplotlib.pyplot.ylabel("Accuracy")
    matplotlib.pyplot.ylim(0, 1)
    matplotlib.pyplot.subplot(2, 2, 4)
    matplotlib.pyplot.semilogx(alphas, shuffle_test, 'r--', label='shuffle')
    matplotlib.pyplot.semilogx(alphas, batch_test, 'g--', label='batch')
    matplotlib.pyplot.semilogx(alphas, mini_test, 'b--', label='mini-batch')
    matplotlib.pyplot.semilogx(alphas, shuffle_test, 'ro-', alphas, batch_test, 'g+-', alphas, mini_test, 'b^-')
    matplotlib.pyplot.legend()
    matplotlib.pyplot.title("N-gram -- Test Set")
    matplotlib.pyplot.xlabel("Learning Rate")
    matplotlib.pyplot.ylabel("Accuracy")
    matplotlib.pyplot.ylim(0, 1)
    matplotlib.pyplot.tight_layout()
    matplotlib.pyplot.show()

4.3 结果

4.3.1 实验一

NLP-Task1:基于机器学习的文本分类_第2张图片

4.3.2 实验二

NLP-Task1:基于机器学习的文本分类_第3张图片
训练集的准确率高达100%,意味着模型存在着过拟合现象;而测试集的最高准确率都在55%附近

数据集下载

鉴于kaggle上直接下载要VPN,附上网盘下载链接:
数据集下载 提取码:cnex

你可能感兴趣的:(NLP-beginner,自然语言处理,机器学习,神经网络)