BERT的源码介绍

BERT源码解析学习链接

tf.data API使用方法

Dataset API入门教程

tf.contrib.data.parallel_interleave

Pipeline技术的优点

tf.gfile学习

tf.data.Dataset.from_tensor_slices()

BERT计算句向量

BERT大量使用tf.data

实体识别使用BERT的输出结果

self.sequence_output = self.all_encoder_layers[-1]

文本分类使用BERT的输出结果

      with tf.variable_scope("pooler"):
        # We "pool" the model by simply taking the hidden state corresponding
        # to the first token. We assume that this has been pre-trained
        first_token_tensor = tf.squeeze(self.sequence_output[:, 0:1, :], axis=1)
        self.pooled_output = tf.layers.dense(
            first_token_tensor,
            config.hidden_size,
            activation=tf.tanh,
            kernel_initializer=create_initializer(config.initializer_range))

创建预训练数据的时候：

    if rng.random() < 0.8:
      masked_token = "[MASK]"
    else:
      # 10% of the time, keep original
      if rng.random() < 0.5:
        masked_token = tokens[index]
      # 10% of the time, replace with random word
      else:
        masked_token = vocab_words[rng.randint(0, len(vocab_words) - 1)]

损失值计算：两个损失值相加（mask的损失值和下一个句子的损失值）

(masked_lm_loss,
     masked_lm_example_loss, masked_lm_log_probs) = get_masked_lm_output(
         bert_config, model.get_sequence_output(), model.get_embedding_table(),
         masked_lm_positions, masked_lm_ids, masked_lm_weights)

    (next_sentence_loss, next_sentence_example_loss,
     next_sentence_log_probs) = get_next_sentence_output(
         bert_config, model.get_pooled_output(), next_sentence_labels)

    total_loss = masked_lm_loss + next_sentence_loss

BERT+BILSTM+CRF并没使用字+词的训练方式，只使用了字向量，因此需要对BILSTM+CRF层进行修改，才能加入词和词性