笔记，阅读的时候写下，记性不太好。。
Bidirectional Encoder Representations from Transformers
介绍
在预训练bert结尾加输出层，就可以实现多种任务的最先进的模型。As a result, the pre-trained BERT representations can be fine-tuned with just one additional output layer to create state-of-the- art models for a wide range of tasks.
不需要为多种任务构建不同的模型。e.g. 问答和语言推理question answering and language inference
贡献点：
- BERT使用深度双向的模型，超越了之前的单向和浅层双向。
- 第一个通过fine-tune就能实现state of art结果
- BERT为11个NLP任务提供了最先进的技术
总之，模型简单， 但是效果好。
在bert之前有：
feature-based：需要定义结构，e.g.ELMo
fine-tuning：优化参数， Generative Pre-trained Transformer (OpenAI GPT)
both using unidirectional language models to learn general language represen- tations.
bert训练: masked language model (MLM) 随机将句子中的某个词给抹掉，然后根据左右上下文预测。
next sentence prediction
现有task结果：
GLUE：80.4%
MultiNLI：86.7%
SQuAD v1.1 ： F1 to 93.2
sentence level tasks：natural language inference， paraphrasing。
token-level tasks：named entity recognition， SQuAD question answering
相关工作：
- Feature-based Approaches：
1.1 Pre-trained word embeddings
1.2 sentence embeddings
1.3 paragraph embeddings
1.4 ELMo 他们建议从语言模型中提取上下文敏感的特性。将上下文嵌入与特定于任务的架构集成。- Fine-tuning Approaches： transfer learning
基于很少有参数需要重头学，OpenAI GPT搞了- Transfer Learning from Supervised Data
架构：
multi-layer bidirectional Transformer encoder
与这个相同，可以参考：Vaswani et al. (2017) as well as excellent guides such as “The Annotated Transformer.”
BERT-BASE: L=12, H=768, A=12, Total Pa- rameters=110M
BERT-LARGE: L=24, H=1024, A=16, Total Parameters=340M
BERT-BASE was chosen to have an identical model size as OpenAI GPT for comparison purposes
输入介绍
第一个都是[CLS]对于分类任务最后是[SEP]，非分类任务不需要
- We use WordPiece embeddings (Wu et al., 2016) with a 30,000 token vocabulary. We denote split word pieces with ##.
- We use learned positional embeddings with supported sequence lengths up to 512 tokens.
架构对比
GPT is trained on the BooksCorpus (800M words); BERT is trained on the BooksCor- pus (800M words) and Wikipedia (2,500M words).
GPT used the same learning rate of 5e-5 for all fine-tuning experiments; BERT chooses a task-specific fine-tuning learning rate which performs the best on the development set.
具体训练
pre-train BERT using two novel unsupervised prediction tasks
Masked LM. mask 15% one sentence。
Next Sentence Prediction:
50% of the time B is the actual next sentence that follows A, and 50% of the time it is a random sentence from the corpus.
They are sampled such that the combined length is ≤ 512 tokens.
training loss is the sum of the mean masked LM likelihood and mean next sentence prediction like- lihood.
实验：
GLUE : The General Language Understanding Evaluation (GLUE) benchmark
is a collection of diverse natural language understand- ing tasks.
数据集介绍：
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
MNLI：
Multi-Genre Natural Language Inference is a large-scale, crowdsourced entailment classifi- cation task
Given a pair of sentences, the goal is to predict whether the second sentence is an entailment, contradiction, or neutral with respect to the first one.
Adina Williams, Nikita Nangia, and Samuel R Bow- man. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL.
QQP：
Quora Question Pairs is a binary classifi- cation task where the goal is to determine if two questions asked on Quora are semantically equivalent
Z. Chen, H. Zhang, X. Zhang, and L. Zhao. 2018. Quora question pairs.
QNLI Question Natural Language Inference is a version of the Stanford Question Answering Dataset (Rajpurkar et al., 2016) which has been converted to a binary classification task (Wang et al., 2018). The positive examples are (ques- tion, sentence) pairs which do contain the correct answer, and the negative examples are (question, sentence) from the same paragraph which do not contain the answer.
SST-2 The Stanford Sentiment Treebank is a binary single-sentence classification task consist- ing of sentences extracted from movie reviews with human annotations of their sentiment (Socher et al., 2013).
CoLA The Corpus of Linguistic Acceptability is a binary single-sentence classification task, where the goal is to predict whether an English sentence is linguistically “acceptable” or not (Warstadt et al., 2018).
STS-B The Semantic Textual Similarity Bench- mark is a collection of sentence pairs drawn from news headlines and other sources (Cer et al., 2017). They were annotated with a score from 1 to 5 denoting how similar the two sentences are in terms of semantic meaning.
MRPC Microsoft Research Paraphrase Corpus consists of sentence pairs automatically extracted from online news sources, with human annotations for whether the sentences in the pair are semanti- cally equivalent (Dolan and Brockett, 2005).
RTE Recognizing Textual Entailment is a bi- nary entailment task similar to MNLI, but with much less training data (Bentivogli et al., 2009).6
WNLI Winograd NLI is a small natural lan- guage inference dataset deriving from (Levesque et al., 2011). The GLUE webpage notes that there are issues with the construction of this dataset, 7 and every trained system that’s been submitted to GLUE has has performed worse than the 65.1 baseline accuracy of predicting the majority class. We therefore exclude this set out of fairness to OpenAI GPT. For our GLUE submission, we al- ways predicted the majority class
SQuAD v1.1：span prediction task
100k crowdsourced question/answer pairs
At inference time, since the end prediction is not conditioned on the start, we add the constraint that the end must come after the start, but no other heuristics are used.（end必须在start之后）
Named Entity Recognition： CONLL
final hidden representation Ti ∈ RH for to each token i into a classification layer over the NER label set.
A cased WordPiece model is used for NER, whereas an uncased model is used for all other tasks.
分析
Question: Does BERT really need such a large amount of pre-training (128,000 words/batch * 1,000,000 steps) to achieve high fine-tuning accuracy?
Answer: Yes, BERTBASE achieves almost 1.0% additional accuracy on MNLI when trained on 1M steps compared to 500k steps.
不好重现结果，因为开发机的内存不同，一次能够处理的batch不同~