python spacy库使用总结【待完善】
spacy库的使用说明
- 1.安装
- 2.用法
- 2.1 word tokenize（doc: token）
- 2.2 英文断句（doc.sents: sent）
- 2.3 词干化（doc: token, token_lemma_, token_lemma）
- 2.4 词性标注（doc: token,token.pos_,token.pos）
- 2.5 命名实体识别（doc.ents:ent, ent.label_, ent.label）
- 2.6 名词短语提取（doc.noun_chunks）
- 2.7 基于词向量计算两个单词的相似度 （doc[index_i].similarity(doc[index_j])）
1.安装
见另一篇python spacy安装问题末尾总结。
2.用法
spaCy 是一个Python自然语言处理工具包，诞生于2014年年中，号称“Industrial-Strength Natural Language Processing in Python”，是具有工业级强度的Python NLP工具包。spaCy里大量使用了 Cython 来提高相关模块的性能，这个区别于学术性质更浓的Python NLTK，因此具有了业界应用的实际价值。
import spacy
nlp = spacy.load(en_core_web_em)
官方文档见spacy（https://spacy.io/usage/linguistic-features）
主要支持英语和德语。
功能包括word tokenize, 英文断句，词干化，词性标注，命名实体识别，名词短语提取，相似度计算……
2.1 word tokenize（doc: token）
将英文单词和标点符号都分离出来，如果含有中文，则中文以多个文字之间的空格分词。
In [3]: test_doc = nlp(u"it's word tokenize test for spacy")
In [4]: print(test_doc)
it's word tokenize test for spacy
In [5]: for token in test_doc:
print(token)
...:
it
's
word
tokenize
test
for
spacy
test_doc是 spacy.tokens.doc.Doc 对象。
2.2 英文断句（doc.sents: sent）
In [6]: test_doc = nlp(u'Natural language processing (NLP) deals with the application of computational models to text or speech data. Application areas within NLP include automatic (machine) translation between languages; dialogue systems, which allow a human to interact with a machine using natural language; and information extraction, where the goal is to transform unstructured text into structured (database) representations that can be searched and browsed in flexible ways. NLP technologies are having a dramatic impact on the way people interact with computers, on the way people interact with each other through the use of language, and on the way people access the vast amount of linguistic data now in electronic form. From a scientific viewpoint, NLP involves fundamental questions of how to structure formal models (for example statistical models) of natural language phenomena, and of how to design algorithms that implement these models.')
In [7]: for sent in test_doc.sents:
print(sent)
...:
Natural language processing (NLP) deals with the application of computational models to text or speech data.
Application areas within NLP include automatic (machine) translation between languages; dialogue systems, which allow a human to interact with a machine using natural language; and information extraction, where the goal is to transform unstructured text into structured (database) representations that can be searched and browsed in flexible ways.
NLP technologies are having a dramatic impact on the way people interact with computers, on the way people interact with each other through the use of language, and on the way people access the vast amount of linguistic data now in electronic form.
From a scientific viewpoint, NLP involves fundamental questions of how to structure formal models (for example statistical models) of natural language phenomena, and of how to design algorithms that implement these models.
2.3 词干化（doc: token, token_lemma_, token_lemma）
词干化是指词形还原，比如将复数还原为单数，将过去式还原为一般式，将am,is,are还原为be……等处理。
In [8]: test_doc = nlp(u"you are best. it is lemmatize test for spacy. I love these books")
In [9]: for token in test_doc:
print(token, token.lemma_, token.lemma)
...:
(you, u'you', 472)
(are, u'be', 488)
(best, u'good', 556)
(., u'.', 419)
(it, u'it', 473)
(is, u'be', 488)
(lemmatize, u'lemmatize', 1510296)
(test, u'test', 1351)
(for, u'for', 480)
(spacy, u'spacy', 173783)
(., u'.', 419)
(I, u'i', 570)
(love, u'love', 644)
(these, u'these', 642)
(books, u'book', 1011)
2.4 词性标注（doc: token,token.pos_,token.pos）
In [10]: for token in test_doc:
print(token, token.pos_, token.pos)
....:
(you, u'PRON', 92)
(are, u'VERB', 97)
(best, u'ADJ', 82)
(., u'PUNCT', 94)
(it, u'PRON', 92)
(is, u'VERB', 97)
(lemmatize, u'ADJ', 82)
(test, u'NOUN', 89)
(for, u'ADP', 83)
(spacy, u'NOUN', 89)
(., u'PUNCT', 94)
(I, u'PRON', 92)
(love, u'VERB', 97)
(these, u'DET', 87)
(books, u'NOUN', 89)
2.5 命名实体识别（doc.ents:ent, ent.label_, ent.label）
In [11]: test_doc = nlp(u"Rami Eid is studying at Stony Brook University in New York")
In [12]: for ent in test_doc.ents:
print(ent, ent.label_, ent.label)
....:
(Rami Eid, u'PERSON', 346)
(Stony Brook University, u'ORG', 349)
(New York, u'GPE', 350)
2.6 名词短语提取（doc.noun_chunks）
In [13]: test_doc = nlp(u'Natural language processing (NLP) deals with the application of computational models to text or speech data. Application areas within NLP include automatic (machine) translation between languages; dialogue systems, which allow a human to interact with a machine using natural language; and information extraction, where the goal is to transform unstructured text into structured (database) representations that can be searched and browsed in flexible ways. NLP technologies are having a dramatic impact on the way people interact with computers, on the way people interact with each other through the use of language, and on the way people access the vast amount of linguistic data now in electronic form. From a scientific viewpoint, NLP involves fundamental questions of how to structure formal models (for example statistical models) of natural language phenomena, and of how to design algorithms that implement these models.')
In [14]: for np in test_doc.noun_chunks:
print(np)
....:
Natural language processing
Natural language processing (NLP) deals
the application
computational models
text
speech
data
Application areas
NLP
automatic (machine) translation
languages
......
2.7 基于词向量计算两个单词的相似度 （doc[index_i].similarity(doc[index_j])）
In [15]: test_doc = nlp(u"Apples and oranges are similar. Boots and hippos aren't.")
In [16]: apples = test_doc[0]
In [17]: print(apples)
Apples
In [18]: oranges = test_doc[2]
In [19]: print(oranges)
oranges
In [20]: boots = test_doc[6]
In [21]: print(boots)
Boots
In [22]: hippos = test_doc[8]
In [23]: print(hippos)
hippos
In [24]: apples.similarity(oranges)
Out[24]: 0.77809414836023805
In [25]: boots.similarity(hippos)
Out[25]: 0.038474555379008429
参考自 http://www.52nlp.cn/tag/python-spacy