python.nlp随笔（四）简单的全文检索系统

实现一个简单的电影评论语料库的全文检索系统

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Wed Apr  4 15:28:11 2018

@author: dag
"""
#coding:utf-8  
import nltk
import re
def raw(file):

    contents = open(file).read()

    contents = re.sub(r'<.*?>', ' ', contents)

    contents = re.sub('\s+', ' ', contents)

    return contents

 

def snippet(doc, term): # buggy

    text = ' '*30 + raw(doc) + ' '*30

    pos = text.index(term)

    return text[pos-30:pos+30]

 

print ("Building Index...")

files = nltk.corpus.movie_reviews.abspaths()

idx = nltk.Index((w, f) for f in files for w in raw(f).split())

 

query = ''

while query != "quit":

    query = input("query> ")

    if query in idx:

        for doc in idx[query]:

            print (snippet(doc, query))

    else:

        print ("Not found")

output:

runfile('/Users/dag/.spyder-py3/filesearch.py', wdir='/Users/dag/.spyder-py3')
Building Index...

query> natural
drug usage as being a hip and natural part of the art scene 
twister , this is yet another natural disaster movie that do
, such as bull durham and the natural . no curve balls here 
great abandon . rejecting her natural abilities , she has sp
 due to the exhausting of our natural resources . each membe
rn kentucky . it's beauty and natural goodness is being slow
ilming in order to get a more natural expression of fear out