上半部分：Flair-1
Model训练代码
一个是获取训练集（矿），一个是构造Embedding+SequenceTagger（矿机），然后两个再trainner obj的帮助下开始工作（挖矿）
ps: downSample--debug神器
下面一个部分找了半天：文本分类请使用make_label_dictionary()，还是太菜
from flair.data import Corpus
from flair.datasets import WNUT_17
from flair.embeddings import TokenEmbeddings, FlairEmbeddings, StackedEmbeddings
from typing import List
# 训练集获取
corpus = WNUT_17()
down_sample = corpus.downsample(0.1)
print(down_sample)
# 获取标签集合->用于SequenceTagger
tag_type = 'ner'
tag_dictionary = corpus.make_tag_dictionary(tag_type=tag_type)
#文本分类请使用make_label_dictionary()
print(tag_dictionary)
# 初始化embedding对象->用于SequenceTagger
flair_embedding_forward = FlairEmbeddings('news-forward')
embeddings: StackedEmbeddings = StackedEmbeddings(embeddings=
[
flair_embedding_forward,
])
# SequenceTagger
from flair.models import SequenceTagger
tagger: SequenceTagger = SequenceTagger(hidden_size=256,
embeddings=embeddings,
tag_dictionary=tag_dictionary,
tag_type=tag_type,
use_crf=True)
# training 设定超参数
from flair.trainers import ModelTrainer
trainer: ModelTrainer = ModelTrainer(tagger, corpus)
trainer.train('resources/taggers/example-ner',
learning_rate=0.1,
mini_batch_size=32,
max_epochs=150)
# 训练曲线制作
from flair.visual.training_curves import Plotter
plotter = Plotter()
plotter.plot_weights('resources/taggers/example-ner/weights.txt')
命令行结果
2020-05-11 14:32:51,537 ----------------------------------------------------------------------------------------------------
2020-05-11 14:32:51,537 Corpus: "Corpus: 3394 train + 1009 dev + 1287 test sentences"
2020-05-11 14:32:51,538 ----------------------------------------------------------------------------------------------------
2020-05-11 14:32:51,538 Parameters:
2020-05-11 14:32:51,538 - learning_rate: "0.1"
2020-05-11 14:32:51,538 - mini_batch_size: "32"
2020-05-11 14:32:51,538 - patience: "3"
2020-05-11 14:32:51,538 - anneal_factor: "0.5"
2020-05-11 14:32:51,538 - max_epochs: "150"
2020-05-11 14:32:51,538 - shuffle: "True"
2020-05-11 14:32:51,538 - train_with_dev: "False"
2020-05-11 14:32:51,539 - batch_growth_annealing: "False"
2020-05-11 14:32:51,539 ----------------------------------------------------------------------------------------------------
2020-05-11 14:32:51,539 Model training base path: "resources/taggers/example-ner"
2020-05-11 14:32:51,539 ----------------------------------------------------------------------------------------------------
2020-05-11 14:32:51,539 Device: cpu
2020-05-11 14:32:51,539 ----------------------------------------------------------------------------------------------------
2020-05-11 14:32:51,539 Embeddings storage mode: cpu
2020-05-11 14:32:51,540 ----------------------------------------------------------------------------------------------------
2020-05-11 14:33:06,177 epoch 1 - iter 10/107 - loss 27.19113178 - samples/sec: 21.87
终止上面的代码还会完成自动测试，牛掰
问题：目前不知道这么上gpu
train函数的参数：
| 超参 | 可选/默认 | 作用 |
|---|---|---|
| embeddings_storage_mode | ['cpu', 'gpu','none'] | 内存，GPU，需要时候导入 |
| learning_rate | 0.1 | 学习率 |
| mini_batch_size | 32 | batch_size |
| patience | 3 | |
| anneal_factor | 0.5 | |
| max_epochs | 150 | 最大epoch |
| shuffle | True | |
| train_with_dev | False | 最后时刻用 |
| batch_growth_annealing | False | |
| tag_type | ['ner', 'upos','pos',''] | 文本分类中不用赋值 |
制作checkpoint 与导入重新训练
trainer: ModelTrainer = ModelTrainer(tagger, corpus)
# 7. start training
trainer.train('resources/taggers/example-ner',
learning_rate=0.1,
mini_batch_size=32,
max_epochs=150,
checkpoint=True)
# 8. stop training at any point
# 9. continue trainer at later point
from pathlib import Path
checkpoint = 'resources/taggers/example-ner/checkpoint.pt'
trainer = ModelTrainer.load_checkpoint(checkpoint, corpus)
trainer.train('resources/taggers/example-ner',
learning_rate=0.1,
mini_batch_size=32,
max_epochs=150,
checkpoint=True)
Model 测试代码
# load the model you trained
model = SequenceTagger.load('resources/taggers/example-ner/final-model.pt')
# create example sentence
sentence = Sentence('I love Berlin')
# predict tags and print
model.predict(sentence)
print(sentence.to_tagged_string())
结果展示
结果文件.png
| 文件 | 作用 |
|---|---|
| best-model.pt | dev最佳模型 |
| final-model.pt | 最后终止时候的模型 |
| loss.tsv | loss曲线 |
| test.tsv | 测试结果 |
| training.log | log |
| weights.png | 所有的权重图像化 |
| weights.txt | 所有的权重 |
超参炼丹炉
from hyperopt import hp
from flair.hyperparameter.param_selection import SearchSpace, Parameter
# define your search space
search_space = SearchSpace()
search_space.add(Parameter.EMBEDDINGS, hp.choice, options=[
[ WordEmbeddings('en') ],
[ FlairEmbeddings('news-forward'), FlairEmbeddings('news-backward') ]
])
search_space.add(Parameter.HIDDEN_SIZE, hp.choice, options=[32, 64, 128])
search_space.add(Parameter.RNN_LAYERS, hp.choice, options=[1, 2])
search_space.add(Parameter.DROPOUT, hp.uniform, low=0.0, high=0.5)
search_space.add(Parameter.LEARNING_RATE, hp.choice, options=[0.05, 0.1, 0.15, 0.2])
search_space.add(Parameter.MINI_BATCH_SIZE, hp.choice, options=[8, 16, 32])
ps:Attention: You should always add your embeddings to the search space (as shown above). If you don't want to test different kind of embeddings, simply pass just one embedding option to the search space, which will then be used in every test run
Learning Rate调节
为什么要把learning rate 单独拿出来，因为learning rate是最重要，且最多变的一个超参数。The learning rate is one of the most important hyper parameter and it fundamentally depends on the topology of the loss landscape via the architecture of your model and the training data it consumes.并且推荐了Cyclical Learning Rates for Training Learning Rate从很小开始，每个Batch都指数上升，然后到一定程度后再开始变小，（主要是未来找到合适的开始LR）
1.Cyclical Learning Rates
learning_rate_tsv = trainer.find_learning_rate('resources/taggers/example-ner',
'learning_rate.tsv')
#后面还可以追加打印
plotter.plot_learning_rate(learning_rate_tsv)
2.use ADAM or else
from torch.optim.adam import Adam
trainer = ModelTrainer(tagger, corpus,
optimizer=Adam)
trainer.train(
"resources/taggers/example",
weight_decay=1e-4
)
学习目录
- Tutorial 1: Basics
- Tutorial 2: Tagging your Text
- Tutorial 3: Embedding Words
- Tutorial 4: List of All Word Embeddings
- Tutorial 5: Embedding Documents
- Tutorial 6: Loading your own Corpus
- Tutorial 7: Training your own Models
- Tutorial 8: Optimizing your Models
- Tutorial 9: Training your own Flair Embeddings
PS
训练你自己的语言模型（embedding）
暂时pass，用处不大。