spacy库的安装与使用_使用spaCy让自然语言处理变得更简单

介绍

自然语言处理(NLP)是人工智能的主要领域之一。NLP在许多智能应用程序中扮演着重要的角色，例如自动聊天机器人、文章摘要、多语言翻译和数据识别。每一个利用NLP来理解非结构化文本数据的行业，不仅要求准确，而且要迅速获得结果。

自然语言处理是一个广阔的领域，NLP中的一些任务是文本分类、机器翻译、问题应答和概念识别。在我的其中一篇文章中，我讨论了在NLP实现中使用的各种工具和组件。文章大多数都在论及库-NLTK (自然语言工具包)组件。

而在这篇文章中，我将说一说强大的spaCy组件。

1.关于spaCy和安装

1.1 关于

spaCy是用Cython语言编写的，(Python的C扩展，它的目的是将C语言的性能交给Python程序)。它是一个相当快的NLP库。spaCy提供了一个简洁的API来访问它的方法和属性，它由经过训练的机器(以及深度)学习模型来管理。

1.2 安装

spaCy的数据以及它的模型可以使用python包索引和设置工具轻松地安装。使用以下命令在您的计算机中来安装spaCy:

sudo pip install spacy

对于Python3，在上面的命令中使用“pip3”替换“pip”。或者点这里下载源代码并在解压之后运行以下命令:

python setup.py install

下载所有的数据和模型，在安装之后运行以下命令:

python -m spacy.en.download all

现在准备就绪去探索和使用spaCy啦！

2.spaCy的pipeline和属性

通过创建pipeline来启动spaCy实现和对不同属性的访问。Pipeline是通过加载模型来创建的。程序包中提供了不同类型的模型，其中包含关于语言的信息—词汇表、受过训练的向量、语法和实体。

我们将加载一个英语核心web的默认模型。

import spacy

nlp = spacy.load(“en”)

对象“NLP”用于创建文档、访问语言注释和不同的NLP属性。让我们通过在pipeline中加载文本数据来创建一个文档。我使用从tripadvisor网站获得的酒店的评论。数据文件可以在 这里下载。

document = unicode(open(filename).read().decode('utf8'))

document = nlp(document)

这份文件现在已经成为了spaCy英语模型的类的一部分，并与许多属性相关联。

文档(或Token)的属性可以使用以下命令列出:

dir(document)

>> [ 'doc', 'ents', … 'mem']

这将输出大量的文档属性，例如-Token、Token的引用索引、语音标记的一部分、实体、向量、情绪、词汇表等等。让我们来探索其中的一些属性。

2.1令牌化技术(TOKENIZATION)

每个spaCy文档都被标记为句子，并进一步进入Tokens，可以通过迭代文档来访问:

# first token of the doc

document[0]

>> Nice

# last token of the doc

document[len(document)-5]

>> boston

# List of sentences of our doc

list(document.sents)

>> [ Nice place Better than some reviews give it credit for.,

Overall, the rooms were a bit small but nice.,

...

Everything was clean, the view was wonderful and it is very well located (the Prudential Center makes shopping and eating easy and the T is nearby for jaunts out and about the city).]

2.2 词性标注(POS)

词性标注是词在语法正确的句子中所使用的词的属性。这些标记可以用作信息过滤、统计模型和基于规则的解析的文本特性。

让我们检查一下我们文档的所有pos标签:

# get all tags

all_tags = {w.pos: w.pos_ for w in document}

>> {97: u'SYM', 98: u'VERB', 99: u'X', 101: u'SPACE', 82: u'ADJ', 83: u'ADP', 84: u'ADV', 87: u'CCONJ', 88: u'DET', 89: u'INTJ', 90: u'NOUN', 91: u'NUM', 92: u'PART', 93: u'PRON', 94: u'PROPN', 95: u'PUNCT'}

# all tags of first sentence of our document

for word in list(document.sents)[0]:

print word, word.tag_

>> ( Nice, u'JJ') (place, u'NN') (Better, u'NNP') (than, u'IN') (some, u'DT') (reviews, u'NNS') (give, u'VBP') (it, u'PRP') (creit, u'NN') (for, u'IN') (., u'.')

让我们来研究一下文档的一些top unigrams。我已经创建了一个基本的预处理和文本清洗功能。

#define some parameters

noisy_pos_tags = [“PROP”]

min_token_length = 2

#Function to check if the token is a noise or not

def isNoise(token):

is_noise = False

if token.pos_ in noisy_pos_tags:

is_noise = True

elif token.is_stop == True:

is_noise = True

elif len(token.string) <= min_token_length:

is_noise = True

return is_noise

def cleanup(token, lower = True):

if lower:

token = token.lower()

return token.strip()

# top unigrams used in the reviews

from collections import Counter

cleaned_list = [cleanup(word.string) for word in document if not isNoise(word)]

Counter(cleaned_list) .most_common(5)

>> [( u'hotel', 683), (u'room', 652), (u'great', 300), (u'sheraton', 285), (u'location', 271)]

2.3 实体检测(ENTITY DETECTION)

spaCy由一个快速的实体识别模型组成，它能够识别来自文档的实体短语。实体可以是不同类型的，例如:人、位置、组织、日期、数字等等。这些实体可以通过“.ents”属性来访问。

让我们在文档中找到所有命名实体的类型。

labels = set([w.label_ for w in document.ents])

for label in labels:

entities = [cleanup(e.string, lower=False) for e in document.ents if label==e.label_]

entities = list(set(entities))

print label,entities

2.4 依存句法分析(DEPENDENCY PARSING)

spaCy的一个最强大的特性是极其快速和准确的句法依存解析(syntactic dependency parser)，它能够通过轻量级API来访问。Parser也可用于句子边界检测和短语分块。这些关系可以通过属性“.children” , “.root”, “.ancestor”等等来访问。

# extract all review sentences that contains the term - hotel

hotel = [sent for sent in document.sents if 'hotel' in sent.string.lower()]

# create dependency tree

sentence = hotel[2] for word in sentence:

print word, ': ', str(list(word.children))

>> A : [] cab : [A, from]

from : [airport, to]

the : []

airport : [the]

to : [hotel]

the : [] hotel :

[the] can : []

be : [cab, can, cheaper, .]

cheaper : [than] than :

[shuttles]

the : []

shuttles : [the, depending]

depending : [time] what : []

time : [what, of] of : [day]

the : [] day :

[the, go] you :

[]

go : [you]

. : []

让我们来解析包含“hotel”一词的所有句子的依存树，并检查“hotel”使用的形容词(用“adj”)标记。我创建了一个自定义功能，来解析依存树并提取相关的pos标记。

# check all adjectives used with a word

def pos_words (sentence, token, ptag):

sentences = [sent for sent in sentence.sents if token in sent.string]

pwrds = []

for sent in sentences:

for word in sent:

if character in word.string:

pwrds.extend([child.string.strip() for child in word.children

if child.pos_ == ptag] )

return Counter(pwrds).most_common(10)

pos_words(document, 'hotel', “ADJ”)

>> [(u'other', 20), (u'great', 10), (u'good', 7), (u'better', 6), (u'nice', 6), (u'different', 5), (u'many', 5), (u'best', 4), (u'my', 4), (u'wonderful', 3)]

2.5 名词短语(NOUN PHRASES)

依存树也能用来产生名词短语:

# Generate Noun Phrases

doc = nlp(u'I love data science on analytics vidhya')

for np in doc.noun_chunks:

print np.text, np.root.dep_, np.root.head.text

>> I nsubj love

data science dobj love

analytics pobj on

3.向量集成词汇

spaCy也提供了密集的内置集成，实值向量代表了分布相似性信息。它用GloVe来生成向量。Glove是一种不受监督的学习算法，用于获取单词的向量表示。

让我们创建一些单词向量并执行一些有趣的操作。

from numpy import dot

from numpy.linalg import norm

from spacy.en import English

parser = English()

#Generate word vector of the word - apple

apple = parser.vocab[u'apple']

#Cosine similarity function

cosine = lambda v1, v2: dot(v1, v2) / (norm(v1) * norm(v2))

others = list({w for w in parser.vocab if w.has_vector and w.orth_.islower() and w.lower_ != unicode("apple")})

# sort by similarity score

others.sort(key=lambda w: cosine(w.vector, apple.vector))

others.reverse()

print "top most similar words to apple:"

for word in others[:10]:

print word.orth_

>> apples iphone f ruit juice cherry lemon banana pie mac orange

4.机器学习文本使用spaCy

在机器学习模型中，集成spaCy是非常简单和直接的。让我们使用sklearn构建一个自定义文本分类器(classifier)。我们将创建一个具有以下组件的sklearn pipeline: cleaner, tokenizer, vectorizer, classifier.。对于tokenizer和vectorizer，我们将使用spaCy来构建我们自己的自定义模块。

from sklearn.feature_extraction.stop_words import ENGLISH_STOP_WORDS as stopwords

from sklearn.feature_extraction.text import CountVectorizer

from sklearn.metrics import accuracy_score

from sklearn.base import TransformerMixin

from sklearn.pipeline import Pipeline

from sklearn.svm import LinearSVC

import string

punctuations = string.punctuation

from spacy.en import English

parser = English()

#Custom transformer using spaCy

class predictors(TransformerMixin):

def transform(self, X, **transform_params):

return [clean_text(text) for text in X]

def fit(self, X, y=None, **fit_params):

return self

def get_params(self, deep=True):

return {}

# Basic utility function to clean the text

def clean_text(text):

return text.strip().lower()

现在让我们使用spaCy解析器和一些基本的清洗创建一个自定义的tokenizer功能。这里需要注意的一点是，文本特性可以替换为单词向量(尤其是在深度学习模型中)。

#Create spacy tokenizer that parses a sentence and generates tokens

#these can also be replaced by word vectors

def spacy_tokenizer(sentence):

tokens = parser(sentence)

tokens = [tok.lemma_.lower().strip() if tok.lemma_ != "-PRON-" else tok.lower_ for tok in tokens]

tokens = [tok for tok in tokens if (tok not in stopwords and tok not in punctuations)] return tokens

#create vectorizer object to generate feature vectors, we will use custom spacy’s tokenizer

vectorizer = CountVectorizer(tokenizer = spacy_tokenizer, ngram_range=(1,1)) classifier = LinearSVC()

现在我们准备开始创建pipeline，加载数据(这里的示例)，还有运行分类(classifier)模型。

# Create the pipeline to clean, tokenize, vectorize, and classify

pipe = Pipeline([("cleaner", predictors()),

('vectorizer', vectorizer),

('classifier', classifier)])

# Load sample data

train = [('I love this sandwich.', 'pos'),

('this is an amazing place!', 'pos'),

('I feel very good about these beers.', 'pos'),

('this is my best work.', 'pos'),

("what an awesome view", 'pos'),

('I do not like this restaurant', 'neg'),

('I am tired of this stuff.', 'neg'),

("I can't deal with this", 'neg'),

('he is my sworn enemy!', 'neg'),

('my boss is horrible.', 'neg')]

test = [('the beer was good.', 'pos'),

('I do not enjoy my job', 'neg'),

("I ain't feelin dandy today.", 'neg'),

("I feel amazing!", 'pos'),

('Gary is a good friend of mine.', 'pos'),

("I can't believe I'm doing this.", 'neg')]

# Create model and measure accuracy

pipe.fit([x[0] for x in train], [x[1] for x in train])

pred_data = pipe.predict([x[0] for x in test])

for (sample, pred) in zip(test, pred_data):

print sample, pred

print "Accuracy:", accuracy_score([x[1] for x in test], pred_data)

>> ('the beer was good.', 'pos') pos

('I do not enjoy my job', 'neg') neg

("I ain't feelin dandy today.", 'neg') neg

('I feel amazing!', 'pos') pos

('Gary is a good friend of mine.', 'pos') pos

("I can't believe I'm doing this.", 'neg') neg

Accuracy: 1.0

5.与其他库作比较

spaCy对于几乎所有的NLP任务来说，都是非常强大的。如果你想知道为什么的话? 让我们将spaCy与其他著名的实现NLP的工具(NLTK和Core NLP)进行比较

有效性特征

Feature

Spacy

NLTK

Core NLP

Easy installation

Y

Y

Y

Python API

Y

Y

N

Multi Language support

N

Y

Y

Tokenization

Y

Y

Y

Part-of-speech tagging

Y

Y

Y

Sentence segmentation

Y

Y

Y

Dependency parsing

Y

N

Y

Entity Recognition

Y

Y

Y

Integrated word vectors

Y

N

N

Sentiment analysis

Y

Y

Y

Coreference resolution

N

N

Y

速率: 主要功能 – TOKENIZER, TAGGING, PARSING

Package

Tokenizer

Tagging

Parsing

spaCy

0.2ms

1ms

19ms

CoreNLP

2ms

10ms

49ms

NLTK

4ms

443ms

–

精确度: 实体抽取

Package

Precition

Recall

F-Score

spaCy

0.72

0.65

0.69

CoreNLP

0.79

0.73

0.76

NLTK

0.51

0.65

0.58

此文为编译作品，作者 SHIVAM BANSAL，原网站https://www.analyticsvidhya.com/blog/2017/04/natural-language-processing-made-easy-using-spacy-%e2%80%8bin-python/