关于lucene的分词（三）
到此为止这个简单的但是功能强大的分词器就写完了,下面咱们可以尝试写一个功能更强大的分词器.
如何DIY一个功能更加强大Analyzer
譬如你有词典,然后你根据正向最大匹配法或者逆向最大匹配法写了一个分词方法,却想在Lucene中应用,很简单
你只要把他们包装成Lucene的TokenStream就好了.下边我以调用中科院写的ICTCLAS接口为例,进行演示.你去中科院
网站可以拿到此接口的free版本,谁叫你没钱呢,有钱,你就可以购买了.哈哈
好,由于ICTCLAS进行分词之后,在Java中,中间会以两个空格隔开!too easy,我们直接使用继承Lucene的
WhiteSpaceTokenizer就好了.
所以TjuChineseTokenizer 看起来像是这样.
public class TjuChineseTokenizer extends WhitespaceTokenizer
{
public TjuChineseTokenizer(Reader readerInput)
{
super(readerInput);
}
}
而TjuChineseAnalyzer看起来象是这样
public final class TjuChineseAnalyzer
extends Analyzer
{
private Set stopWords;
/** An array containing some common English words that are not usually useful
for searching. */
/*
public static final String[] CHINESE_ENGLISH_STOP_WORDS =
{
"a", "an", "and", "are", "as", "at", "be", "but", "by",
"for", "if", "in", "into", "is", "it",
"no", "not", "of", "on", "or", "s", "such",
"t", "that", "the", "their", "then", "there", "these",
"they", "this", "to", "was", "will", "with",
"我", "我们"
};
*/
/** Builds an analyzer which removes words in ENGLISH_STOP_WORDS. */
public TjuChineseAnalyzer()
{
stopWords = StopFilter.makeStopSet(StopWords.SMART_CHINESE_ENGLISH_STOP_WORDS);
}
/** Builds an analyzer which removes words in the provided array. */
//提供独自的stopwords
public TjuChineseAnalyzer(String[] stopWords)
{
this.stopWords = StopFilter.makeStopSet(stopWords);
}
/** Filters LowerCaseTokenizer with StopFilter. */
public TokenStream tokenStream(String fieldName, Reader reader)
{
try
{
ICTCLAS splitWord = new ICTCLAS();
String inputString = FileIO.readerToString(reader);
//分词中间加入了空格
String resultString = splitWord.paragraphProcess(inputString);
System.out.println(resultString);
TokenStream result = new TjuChineseTokenizer(new StringReader(resultString));
result = new LowerCaseFilter(result);
//使用stopWords进行过滤
result = new StopFilter(result, stopWords);
//使用p-stemming算法进行过滤
result = new PorterStemFilter(result);
return result;
}
catch (IOException e)
{
System.out.println("转换出错");
return null;
}
}
public static void main(String[] args)
{
String string = "我爱中国人民";
Analyzer analyzer = new TjuChineseAnalyzer();
TokenStream ts = analyzer.tokenStream("dummy", new StringReader(string));
Token token;
System.out.println("Tokens:");
try
{
int n=0;
while ( (token = ts.next()) != null)
{
System.out.println((n++)+"->"+token.toString());
}
}
catch (IOException ioe)
{
ioe.printStackTrace();
}
}
}
对于此程序的输出接口可以看一下
0->Token's (termText,startOffset,endOffset,type,positionIncrement) is:(爱,3,4,word,1)
1->Token's (termText,startOffset,endOffset,type,positionIncrement) is:(中国,6,8,word,1)
2->Token's (termText,startOffset,endOffset,type,positionIncrement) is:(人民,10,12,word,1)
OK,经过这样一番讲解,你已经对Lucene的Analysis包认识的比较好了,当然如果你想更加了解,还是认真读读源码才好,
呵呵,源码说明一切!