BIO 格式介绍
AGAC Track 官网
Bio-NLP课程链接
data dowmload
1.json 模块使用 资料
import json
data=[{"a":1,"b":2,"c":3,"d":4,"e":5}]
json.dumps(data)
print(json.dumps({'a': 'Runoob', 'b': 7}, sort_keys=True, indent=4, separators=(',', ':')))
###
jsonData = '{"a":1,"b":2,"c":3,"d":4,"e":5}';
text=json.loads(jsonData)
1、json.dumps()和json.loads()是json格式处理函数（可以这么理解，json是字符串）
(1)json.dumps()函数是将一个Python数据类型列表进行json格式的编码（可以这么理解，json.dumps()函数是将字典转化为字符串）
(2)json.loads()函数是将json格式数据转换为字典（可以这么理解，json.loads()函数是将字符串转化为字典）
2、json.dump()和json.load()主要用来读写json文件函数
2.spacy 模块使用 资料1 资料2
pycharm 安装第三方模块
win 添加conda 到环境变量
linux 环境如下：
- pip install spacy
- python -m spacy download en
下载en
结果如下：
前10个token
列出部分的spacy用法：请参考资料1
text = "The sequel, Yes, Prime Minister, ran from 1986 to 1988. In total there were 38 episodes, of which all but one lasted half an hour. Almost all episodes ended with a variation of the title of the series spoken as the answer to a question posed by the same character, Jim Hacker. Several episodes were adapted for BBC Radio, and a stage play was produced in 2010, the latter leading to a new television series on UKTV Gold in 2013."
text
import spacy
nlp = spacy.load("en")
doc = nlp(text)
doc
###让Spacy帮我们分析这段话中出现的全部词例（token）
for token in doc:
print('"' + token.text + '"')
for token in doc[:10]:
print("{0}\t{1}\t{2}\t{3}\t{4}\t{5}\t{6}\t{7}".format(
token.text,
token.idx,
token.lemma_,
token.is_punct,
token.is_space,
token.shape_,
token.pos_,
token.tag_
))
for ent in doc.ents:
print(ent.text, ent.label_)
from spacy import displacy
displacy.render(doc, style='ent', jupyter=True)
3.glob 模块 资料 类似linux find - name 功能
glob.glob
- 返回所有匹配的文件路径列表。它只有一个参数pathname，定义了文件路径匹配规则，这里可以是绝对路径，也可以是相对路径。
glob.iglob
-
获取一个可编历对象，使用它可以逐个获取匹配的文件路径名。与glob.glob()的区别是：glob.glob同时获取所有的匹配路径，而 glob.iglob一次只获取一个匹配路径。这有点类似于.NET中操作数据库用到的DataSet与DataReader。
glob 模块
4.训练集，测试集。7:3分
import random
import os
import shutil
def random_copyfile(srcPath,dstPath,lastpath,numfiles):
name_list=list(os.path.join(srcPath,name) for name in os.listdir(srcPath))
random_name_list=list(random.sample(name_list,numfiles))
last=[ item for item in name_list if item not in random_name_list ]
if not os.path.exists(dstPath):
os.mkdir(dstPath)
for oldname in random_name_list:
shutil.copyfile(oldname,oldname.replace(srcPath, dstPath))
for file in last:
shutil.copyfile(file,file.replace(srcPath, lastpath))
srcPath='/home/kcao/test/tmp/AGAC_training'
dstPath = '/home/kcao/test/tmp/kcao_train_data'
lastpath='/home/kcao/test/tmp/kcao_test_data'
random_copyfile(srcPath,dstPath,lastpath,175)
5.将josn转换成BIO格式，供CRF使用
将训练的josn,与测试的json 分别转成BIO格式
# -*- coding: utf-8 -*-
"""
Created on Tue Apr 10 09:35:15 2019
@author: wyx
"""
import json
from glob import glob
import spacy
nlp = spacy.load('en')
def json2bio(fpath,output,splitby = 's'):
'''
输入json文件，返回bio(token pmid label)
splitby = 's' ----以句子空格
splitby = 'a' ----以摘要空格
'''
with open(fpath) as f:
pmid = fpath[-13:-5]
annotations = json.load(f)
text = annotations['text'].replace('\n',' ')
all_words = text.split(' ')
all_words2 = [token for token in nlp(text)]
all_label = ['O']*len(all_words)
for i in annotations['denotations']:
b_location = i['span']['begin']
e_location = i['span']['end']
label = i['obj']
B_wordloc = text.count(' ',0,b_location)
I_wordloc = text.count(' ',0,e_location)
all_label[B_wordloc] = 'B-'+label
if B_wordloc != I_wordloc:
for word in range(B_wordloc+1,I_wordloc+1):
all_label[word] = 'I-'+label
#得到以空格分词的词列表和对应标签列表
for w,_ in enumerate(all_words):
all_words[w] = nlp(all_words[w])
#对单个元素分词
labelchange = []
for i,_ in enumerate(all_words):
token = [token for token in all_words[i]]
if len(token)==1:
labelchange.append(all_label[i])
else:
if all_label[i] == 'O':
labelchange.extend(['O']*len(token))
if all_label[i] != 'O':
labelchange.append(all_label[i])
if str(token[-1]) == '.' or str(token[-1]) == ',':
labelchange.extend(['I-'+all_label[i][2:]]*(len(token)-2))
labelchange.append('O')
else:
labelchange.extend(['I-'+all_label[i][2:]]*(len(token)-1))
#写入文件
with open(output,'a',encoding='utf-8') as f:
#以句子空行
if splitby == 's':
for j,_ in enumerate(all_words2):
if str(all_words2[j]) == '.' and str(all_words2[j-1]) != 'p':
line =str(all_words2[j])+'\t'+pmid+'\t'+labelchange[j]+'\n'
f.write(line+'\n')
else:
line =str(all_words2[j])+'\t'+pmid+'\t'+labelchange[j]+'\n'
f.write(line)
#以摘要空行
if splitby == 'a':
for j,_ in enumerate(all_words2):
line =str(all_words2[j])+'\t'+pmid+'\t'+labelchange[j]+'\n'
f.write(line)
f.write('\n')
if __name__ == "__main__":
fpathlist = glob('/home/kcao/test/tmp/kcao_train_data/*.json')
output = "/home/kcao/test/tmp/kcao_train_data/train.tab"
for i in fpathlist:
json2bio(i,output,'s')
6.使用Wapiti进行训练
pat内容如下：
*
U:tok:1:-1:%X[-1,0]
U:tok:1:+0:%X[0,0]
U:tok:1:+1:%X[1,0]
U:tok:2:-1:%X[-1,0]/%X[0,0]
U:tok:2:+0:%X[0,0]/%X[1,0]
U:tok:3:-2:%X[-2,0]/%X[-1,0]/%X[0,0]
U:tok:3:-1:%X[-1,0]/%X[0,0]/%X[1,0]
U:tok:3:+0:%X[0,0]/%X[1,0]/%X[2,0]
U:pre:1:+0:4:%M[0,0,"^.?.?.?.?"]
U:suf:1:+0:4:%M[0,0,".?.?.?.?$"]
test-wapiti.sh 如下
#! bin/bash
traininput_dir="kcao_train_data"
testinput_dir="kcao_test_data"
output_dir="kcao_output"
pattern_file="pat/Tok321dis.pat"
training_options=' -a sgd-l1 -t 3 -i 10 '
debug=0
verbose=0
patname=$(basename $pattern_file .pat)
corpus_name=$(basename $traininput_dir)
echo "================ Training $corpus_name (this may take some time) ================" 1>&2
# training: create a MODEL based on PATTERNS and TRAINING-CORPUS
# wapiti train -p PATTERNS TRAINING-CORPUS MODEL
echo "wapiti train $training_options -p $pattern_file <(cat $1) $output_dir/$patname-train-$corpus_name-$3.mod" 1>&2
wapiti train $training_options -p $pattern_file <(cat $traininput_dir/*.tab) $output_dir/$patname-train-$corpus_name.mod
# wapiti train -a bcd -t 2 -i 5 -p t.pat train-bio.tab t-train-bio.mod
#
# Note: The default algorithm, l-bfgs, stops early and does not succeed in annotating any token (all O)
# sgd-l1 works; bcd works
wapiti dump $output_dir/$patname-train-$corpus_name.mod $output_dir/$patname-train-$corpus_name.txt
echo "================ Inference $corpus_name ================" 1>&2
# inference (labeling): apply the MODEL to label the TEST-CORPUS, put results in TEST-RESULTS
# wapiti label -m MODEL TEST-CORPUS TEST-RESULTS
# -c : check (= evaluate)
# <(COMMAND ARGUMENTS ...) : runs COMMAND on ARGUMENTS ... and provides the results as if in a file
echo "wapiti label -c -m $output_dir/$patname-train-$corpus_name-$3.mod <(cat $1) $output_dir/$patname-train-test-$corpus_name-$3.tab" 1>&2
wapiti label -c -m $output_dir/$patname-train-$corpus_name.mod <(cat $testinput_dir/*) $output_dir/$patname-train-test-$corpus_name.tab
# wapiti label -c -m t-train-bio.mod test-bio.tab t-train-test-bio.tab
#echo "================ Evaluation with conlleval.pl $corpus_name ================" 1>&2
echo "Finished!"
# evaluate the resulting entities
# $'\t' is a way to obtain a tabulation in bash
#echo "$BINDIR/conlleval.pl -d $'\t' < $output_dir/$patname-train-test-$corpus_name-$3.tab | tee $output_dir/$patname-train-test-$corpus_name-$3.eval" 1>&2
perl conlleval.pl -d $'\t' < $output_dir/$patname-train-test-$corpus_name.tab | tee -a $output_dir/$patname-train-test-$corpus_name.eval
结果统计
显示的FBI只有12.73还是太低，可能需要进行修改tab 里面特征。
1.修改pat 增加到U:tok:4:-3:%X[-3,0]/%X[-2,0]/%X[-1,0]/%X[0,0]
image.png