ElasticSearch使用模板中定义的分词器
ElasticSearch使用模板中定义的分词器
ElasticSearch7.2.0
1、定义ik+english分词器
2、定义ik+english+同义词分词器
3、定义english+pinyin分词器
4、使用分词器
5、测试分词效果
对人工智能感兴趣点下面链接
现在人工智能非常火爆，很多朋友都想学，但是一般的教程都是为博硕生准备的，太难看懂了。最近发现了一个非常适合小白入门的教程，不仅通俗易懂而且还很风趣幽默。所以忍不住分享一下给大家。点这里可以跳转到教程。
https://www.cbedai.net/u014646662
1、定义ik+english分词器
在这里强调一下 "index_patterns": ["*"]，指匹配所有索引，即所有索引都可用该模板的分词器，单不等于默认使用（除非你设置的级别比较高）。只有template_default这个模板的内容才会默认使用。下同
Post _template/ik_en_analyzer
{
"index_patterns": ["*"],
"order" : 0,
"version": 0,
"settings": {
"number_of_shards": 2,
"number_of_replicas":1 ,
"analysis": {
"analyzer": {
"ik_en_analyzer": {
"type": "custom",
"tokenizer": "ik_max_word",
"filter": ["en_stemmer"]
}
},
"filter": {
"en_stemmer" : {
"type" : "stemmer",
"name" : "english"
}
}
}
}
}2、定义ik+english+同义词分词器
提前准备文件
在elasticsearch/config/analysis目录下创建synonym.txt（analysis这个目录自己创建）
synonym.txt
马铃薯,土豆
番茄,西红柿
i,me,我
you,你模板定义
Post _template/template_synonym
{
"index_patterns": ["*"],
"order" : 0,
"version": 0,
"settings": {
"number_of_shards": 2,
"number_of_replicas":1 ,
"analysis": {
"analyzer": {
"ik_en_synonym_analyzer": {
"type": "custom",
"tokenizer": "ik_max_word",
"filter": ["en_stemmer","interim_synonym"]
}
},
"filter": {
"en_stemmer" : {
"type" : "stemmer",
"name" : "english"
},
"interim_synonym": {
"type": "synonym",
"synonyms_path" : "analysis/synonym.txt"
}
}
}
}
}
3、定义english+pinyin分词器
个人认为拼音分词器不建议与ik一起用（有相应的需求除外）
Post _template/template_pinyin
{
"index_patterns": ["*"],
"order" : 0,
"version": 0,
"settings": {
"number_of_shards": 2,
"number_of_replicas":1 ,
"analysis": {
"analyzer": {
"en_pinyin_analyzer": {
"type": "custom",
"tokenizer": "pinyin",
"filter": ["en_stemmer"]
}
},
"filter": {
"en_stemmer" : {
"type" : "stemmer",
"name" : "english"
}
}
}
}
}
4、使用分词器
在定义索引的时候就可以直接使用了
这里定义了三个字段，分别用以上刚刚定义好的分词器
PUT test_analyzer
{
"settings": {
"number_of_shards": "1",
"number_of_replicas": "0"
},
"mappings": {
"properties": {
"pinyin": {
"type": "text",
"analyzer": "en_pinyin_analyzer"
},
"ik": {
"type": "text",
"analyzer": "ik_en_analyzer"
},
"synonym": {
"type": "text",
"analyzer": "ik_en_synonym_analyzer"
}
}
}
}5、测试分词效果
测试pinyin+english
GET test_analyzer/_analyze
{
"field": "pinyin",
"text": "赵本山"
}
GET test_analyzer/_analyze
{
"field": "pinyin",
"text": "zhaobenshan"
}
第一种分词结果（赵本山）
{
"tokens" : [
{
"token" : "zhao",
"start_offset" : 0,
"end_offset" : 0,
"type" : "word",
"position" : 0
},
{
"token" : "zb",
"start_offset" : 0,
"end_offset" : 0,
"type" : "word",
"position" : 0
},
{
"token" : "ben",
"start_offset" : 0,
"end_offset" : 0,
"type" : "word",
"position" : 1
},
{
"token" : "shan",
"start_offset" : 0,
"end_offset" : 0,
"type" : "word",
"position" : 2
}
]
}
第二种分词结果（zhaobenshan）
{
"tokens" : [
{
"token" : "zhao",
"start_offset" : 0,
"end_offset" : 0,
"type" : "word",
"position" : 0
},
{
"token" : "zhaobenshan",
"start_offset" : 0,
"end_offset" : 0,
"type" : "word",
"position" : 0
},
{
"token" : "ben",
"start_offset" : 0,
"end_offset" : 0,
"type" : "word",
"position" : 1
},
{
"token" : "shan",
"start_offset" : 0,
"end_offset" : 0,
"type" : "word",
"position" : 2
}
]
}
测试ik+english
GET test_analyzer/_analyze
{
"field": "ik",
"text": "我有100个梨 I have 100 pears"
}
测试结果
{
"tokens" : [
{
"token" : "我",
"start_offset" : 0,
"end_offset" : 1,
"type" : "CN_CHAR",
"position" : 0
},
{
"token" : "有",
"start_offset" : 1,
"end_offset" : 2,
"type" : "CN_CHAR",
"position" : 1
},
{
"token" : "100",
"start_offset" : 2,
"end_offset" : 5,
"type" : "ARABIC",
"position" : 2
},
{
"token" : "个",
"start_offset" : 5,
"end_offset" : 6,
"type" : "COUNT",
"position" : 3
},
{
"token" : "梨",
"start_offset" : 6,
"end_offset" : 7,
"type" : "CN_CHAR",
"position" : 4
},
{
"token" : "i",
"start_offset" : 8,
"end_offset" : 9,
"type" : "ENGLISH",
"position" : 5
},
{
"token" : "have",
"start_offset" : 10,
"end_offset" : 14,
"type" : "ENGLISH",
"position" : 6
},
{
"token" : "100",
"start_offset" : 15,
"end_offset" : 18,
"type" : "ARABIC",
"position" : 7
},
{
"token" : "pear", //把复数形式的s去掉了
"start_offset" : 19,
"end_offset" : 24,
"type" : "ENGLISH",
"position" : 8
}
]
}
测试同义词
GET test_analyzer/_analyze
{
"field": "synonym",
"text": "土豆的果实是长在底下的"
}
测试结果：既有土豆也有马铃薯
{
"tokens" : [
{
"token" : "土豆",
"start_offset" : 0,
"end_offset" : 2,
"type" : "CN_WORD",
"position" : 0
},
{
"token" : "马铃薯",
"start_offset" : 0,
"end_offset" : 2,
"type" : "SYNONYM",
"position" : 0
},
{
"token" : "的",
"start_offset" : 2,
"end_offset" : 3,
"type" : "CN_CHAR",
"position" : 1
},
{
"token" : "果实",
"start_offset" : 3,
"end_offset" : 5,
"type" : "CN_WORD",
"position" : 2
},
{
"token" : "实是",
"start_offset" : 4,
"end_offset" : 6,
"type" : "CN_WORD",
"position" : 3
},
{
"token" : "长在",
"start_offset" : 6,
"end_offset" : 8,
"type" : "CN_WORD",
"position" : 4
},
{
"token" : "底下",
"start_offset" : 8,
"end_offset" : 10,
"type" : "CN_WORD",
"position" : 5
},
{
"token" : "的",
"start_offset" : 10,
"end_offset" : 11,
"type" : "CN_CHAR",
"position" : 6
}
]
}