文本NLP噪音预处理

最近总结修改了下预处理方法，记录下

 首先download需要的依赖

pip install pyenchant

pip install nltk

 pyenchant 是用来检测拼写正确的，如果你的文本里面可能包含非正确拼写的单词，那就忽略它，nltk用来做分词的。

python -m nltk.downloader punkt
python -m nltk.downloader stopwords

from nltk.corpus import stopwords
import nltk
import enchant
import re

def is_spelled_correctly(word, language='en_US'):
        spell_checker = enchant.Dict(language)
        return spell_checker.check(word)
    
def preprocess_text(text):
        text= re.sub(r'\W+', ' ',re.sub(r'[0-9]+', '', text.replace('-', '').replace('_', ' ')))
        words=nltk.word_tokenize(text)
        stop_words = set(stopwords.words('english'))
        words = [item for word in words for item in re.findall(r'[A-Z]+[a-z]*|[a-z]+', word)if is_spelled_correctly(item) and item.lower() not in stop_words]
        return ' '.join(words).lower()

if __name__ == '__main__':
    print(preprocess_text('ServiceHandlerId caedbe-85432-xssc-dsdabffdddbea An exception of some microservice TargetDownService occurred and was test #@/*-sss '))
#service handler id exception target service occurred test

 这里最后再转小写是因为防止ServiceHandlerId这种连续的单词链接成的字符串被拼写检查剔除，只有保持驼峰情况下，才能用 re.findall(r'[A-Z]+[a-z]*|[a-z]+', word) 成功把他分成单独的单词，所以最后再处理大小写。

改进： 

之后测试的时候发现数据量一大，他就很慢，后面优化了一下，速度大大提升了

from nltk.corpus import stopwords
import nltk
import enchant
import re

spell_checker = enchant.Dict(language)

def memoize(func):
        cache = {}
        def wrapper(*args):
            if args not in cache:
                cache[args] = func(*args)
            return cache[args]
        return wrapper

@memoize
def check_spelling(word):
    return spell_checker.check(word)


def preprocess_text(text):
        text= re.sub(r'\W+', ' ',re.sub(r'[0-9]+', '', text.replace('-', '').replace('_', ' ')))
        words=nltk.word_tokenize(text)
        stop_words = set(stopwords.words('english'))
        words = [item for word in words for item in re.findall(r'[A-Z]+[a-z]*|[a-z]+', word)if check_spelling(item) and item.lower() not in stop_words]
        return ' '.join(words).lower()

if __name__ == '__main__':
    print(preprocess_text('ServiceHandlerId caedbe-85432-xssc-dsdabffdddbea An exception of some microservice TargetDownService occurred and was test #@/*-sss '))
#service handler id exception target service occurred test

这里面使用了memoization 技术，它是一种将函数调用和结果存储在一个字典中的优化技术。我这里用来缓存单词的拼写检查结果。

这样之后数据量大了之后速度依然不会太慢了。