对Pytorch Bi-LSTM + CRF 代码详解

本文主要解释torch的官网对Pytorch Bi-LSTM + CRF 代码是如何计算$log({\sum }_{\tilde{y}\in Y_X}{e}^{S(X,\tilde{y})})$

公式为：$log({\sum }_{\tilde{y}\in Y_X}{e}^{S(X,\tilde{y})})$(后面用logsumexp表示)

在此之前可以先看完再看看这位的博客
Bi-LSTM-CRF for Sequence Labeling PENG
首先是定义$S(X,\tilde{y})$是对于输入序列$X$对应的输出tag序列$y$的分数：
$$S(X,\tilde{y})=\sum _{i=0}^{n-1}{A}_{y_i,y_{i+1}}+\sum _{i=0}^n{P}_{i,y_i},n为序列长度$$
其中$A$是tag的转移概率矩阵，$P$是Bi-LISTM的输出矩阵,其中$P_{i,j}$代表词$w_i$映射到$ta{g}_j$的非归一化概率。

代码处理

在pytorch官网提供的代码中，不是先求出每一种路径的分数在求和,而是利用前向算法的思想来求解：
对于求到序列i位置的logsumexp分数，可以先求出到序列i-1的logsumexp分数，在加上转移概率和Bi-LSTM的
输出的概率。定义$f(i,s_j)=log({\sum }_{\tilde{y}\in Y_i,y_i=s_j}{e}^{S_i(X,\tilde{y},s_j)})$表示到序列长度i，最后的状态为$s_j$的所有路径logsumexp分数和。所以要构造一个函数：
$f(i,s_j)=F(f(i-1,s_1),f(i-1,s_2),...,f(i-1,s_T))=F(i-1),T表示状态个数$。
可以利用公式$log(\sum e^{log(\sum e^x)+y})=log(\sum \sum e^{x+y})$(这个公式自己动手推一下是
成立的),来计算$f(i,s_j)$和$f(i-1,s_t)$的关系。
$$\begin{array}{rl}& log(\sum _{t=1}^T{e}^{f(i-1,s_t)+A_{s_t,s_j}+P_{i,j}})\\ & =log(\sum _{t=1}^T{e}^{A_{s_t,s_j}+P_{i,j}}\ast e^{f(i-1,s_t)})\\ & =log(\sum _{t=1}^T{e}^{A_{s_t,s_j}+P_{i,j}}\ast e^{log({\sum }_{(\tilde{y}\in Y_{i-1},y_{i-1}=s_t)}{e}^{S_{i-1}(X,\tilde{y},s_t)})})\\ & =log(\sum _{t=1}^T{e}^{A_{s_t,s_j}+P_{i,j}}\ast \sum _{\tilde{y}\in Y_{i-1},y_{i-1}=s_t}{e}^{S_{i-1}(X,\tilde{y},s_t)})\\ & =log(\sum _{t=1}^T\sum _{\tilde{y}\in Y_{i-1},y_{i-1}=s_t}{e}^{S_{i-1}(X,\tilde{y},s_t)+A_{s_t,s_j}+P_{i,j}})\\ & =log(\sum _{t=1}^T\sum _{\tilde{y}\in Y_i,y_{i-1}=s_t,y_i=s_j}{e}^{S_i(X,\tilde{y},s_j)})\\ & =log(\sum _{\tilde{y}\in Y_i,y_i=s_j}{e}^{S_i(X,\tilde{y},s_j)})\\ & =f(i,s_j)\end{array}$$

对应到代码里，每一步的状态的logsumexp分数等于上一步的每个状态的logsumexp分数+转移概率+BI-LSTM的输出概率:

$$logsumex{p}_{i,j}=log(\sum _{t=1}^T{e}^{(logsumex{p}_{i-1,t}+A_{i,j}+P_{i,t})})$$

这个公式对应标注场景:

$$\begin{array}{r}位置i的词w_i的对应的label为ta{g}_j的logsumexp分数=log(\sum _{t=1}^T{e}^{(位置i-1的标签为ta{g}_t的logsumexp分数+标签ta{g}_t到ta{g}_j的转移概率+bilstm的输出矩阵中词w_{i-1}映射到ta{g}_t的概率)})\end{array}$$
最后，长度为n的序列logsumexp分数为$Final(n)={\sum }_{t=1}^{T}f(n,t)+A_{t,S_{stop}}$,其中$A_{t,S_{stop}}$表示从$ta{g}_t$到终止符号的的概率。

以下是计算$log({\sum }_{\tilde{y}\in Y_X}{e}^{S(X,\tilde{y})})$的代码

def _forward_alg(self, feats):
    # Do the forward algorithm to compute the partition function
    init_alphas = torch.Tensor(1, self.tagset_size).fill_(-10000.)
    # START_TAG has all of the score.
    init_alphas[0][self.tag_to_ix[START_TAG]] = 0.

    # Wrap in a variable so that we will get automatic backprop
    forward_var = autograd.Variable(init_alphas)

    # Iterate through the sentence
    for feat in feats:
        alphas_t = []  # The forward variables at this timestep
        for next_tag in range(self.tagset_size):
            # broadcast the emission score: it is the same regardless of
            # the previous tag
            emit_score = feat[next_tag].view(
                1, -1).expand(1, self.tagset_size)
            # the ith entry of trans_score is the score of transitioning to
            # next_tag from i
            trans_score = self.transitions[next_tag].view(1, -1)
            # The ith entry of next_tag_var is the value for the
            # edge (i -> next_tag) before we do log-sum-exp
            next_tag_var = forward_var + trans_score + emit_score
            # The forward variable for this tag is log-sum-exp of all the
            # scores.
            alphas_t.append(log_sum_exp(next_tag_var))
        forward_var = torch.cat(alphas_t).view(1, -1)
    terminal_var = forward_var + self.transitions[self.tag_to_ix[STOP_TAG]]
    alpha = log_sum_exp(terminal_var)
    return alpha

以下是计算目标句子的$S(X,y)$分数,参数feats是bi-lstm的输出，

参数tag是句子的标注序列

 def _score_sentence(self, feats, tags):
        # Gives the score of a provided tag sequence
        score = torch.zeros(1)
        tags = torch.cat([torch.tensor([self.tag_to_ix[START_TAG]], dtype=torch.long), tags])
        for i, feat in enumerate(feats):
            score = score + \
                self.transitions[tags[i + 1], tags[i]] + feat[tags[i + 1]]
        score = score + self.transitions[self.tag_to_ix[STOP_TAG], tags[-1]]
        return score

以下是计算损失函数的代码

在bi-lstm+crf中$p(y|X)=\frac{e^{S(X,y)}}{{\sum }_{\tilde{y}\in Y_X}{e}^{S(X,\tilde{y})}}$，
所以损失函数$log(p(y|X))=S(X,y)-log({\sum }_{\tilde{y}\in Y_X}{e}^{S(X,\tilde{y})})$,
对应代码中的forward_score - gold_score

    def neg_log_likelihood(self, sentence, tags):
        feats = self._get_lstm_features(sentence)
        forward_score = self._forward_alg(feats)
        gold_score = self._score_sentence(feats, tags)
        return forward_score - gold_score