进阶-第17__深度探秘搜索技术_在案例实战中掌握phrase matching搜索技术

 

1、什么是近似匹配

 

两个句子

 

java is my favourite programming language, and I also think spark is a very good big data system.

java spark are very related, because scala is spark's programming language and scala is also based on jvm like java.

 

match query，搜索java spark

 

{

        "match": {

                 "content": "java spark"

        }

}

 

match query，只能搜索到包含java和spark的document，但是不知道java和spark是不是离的很近

 

包含java或包含spark，或包含java和spark的doc，都会被返回回来。我们其实并不知道哪个doc，java和spark距离的比较近。如果我们就是希望搜索java spark，中间不能插入任何其他的字符，那这个时候match去做全文检索，能搞定我们的需求吗？答案是，搞不定。

 

如果我们要尽量让java和spark离的很近的document优先返回，要给它一个更高的relevance score，这就涉及到了proximity match，近似匹配

 

如果说，要实现两个需求：

 

1、java spark，就靠在一起，中间不能插入任何其他字符，就要搜索出来这种doc

2、java spark，但是要求，java和spark两个单词靠的越近，doc的分数越高，排名越靠前

 

要实现上述两个需求，用match做全文检索，是搞不定的，必须得用proximity match，近似匹配

 

phrase match，proximity match：短语匹配，近似匹配

 

这一讲，要学习的是phrase match，就是仅仅搜索出java和spark靠在一起的那些doc，比如有个doc，是java use'd spark，不行。必须是比如java spark are very good friends，是可以搜索出来的。

 

phrase match，就是要去将多个term作为一个短语，一起去搜索，只有包含这个短语的doc才会作为结果返回。不像是match，java spark，java的doc也会返回，spark的doc也会返回。

 

2、match_phrase

Match query 测试（反例）

GET /forum/article/_search {   "query": {     "match": {       "content": "java spark"     }   } } 结果： {   "took": 3,   "timed_out": false,   "_shards": {     "total": 5,     "successful": 5,     "failed": 0   },   "hits": {     "total": 2,     "max_score": 0.68640786,     "hits": [       {         "_index": "forum",         "_type": "article",         "_id": "2",         "_score": 0.68640786,         "_source": {           "articleID": "KDKE-B-9947-#kL5",           "userID": 1,           "hidden": false,           "postDate": "2017-01-02",           "tag": [             "java"           ],           "tag_cnt": 1,           "view_cnt": 50,           "title": "this is java blog",           "content": "i think java is the best programming language",           "sub_title": "learned a lot of course",           "author_first_name": "Smith",           "author_last_name": "Williams"         }       },       {         "_index": "forum",         "_type": "article",         "_id": "5",         "_score": 0.56008905,         "_source": {           "articleID": "DHJK-B-1395-#Ky5",           "userID": 3,           "hidden": false,           "postDate": "2017-03-01",           "tag": [             "elasticsearch"           ],           "tag_cnt": 1,           "view_cnt": 10,           "title": "this is spark blog",           "content": "spark is best big data solution based on scala ,an programming language similar to java",           "sub_title": "haha, hello world",           "author_first_name": "Tonny",           "author_last_name": "Peter Smith"         }       }     ]   } }

 

单单包含java的doc也返回了，不是我们想要的结果

添加测试数据

POST /forum/article/5/_update {   "doc": {     "content": "spark is best big data solution based on scala ,an programming language similar to java spark"   } }

 

 

将一个doc的content设置为恰巧包含java spark这个短语

 

match_phrase语法

GET /forum/article/_search {     "query": {         "match_phrase": {             "content": "java spark"//必须包含这个短语，而且顺序必须的是         }     } } 结果： {   "took": 1,   "timed_out": false,   "_shards": {     "total": 5,     "successful": 5,     "failed": 0   },   "hits": {     "total": 1,     "max_score": 0.5753642,     "hits": [       {         "_index": "forum",         "_type": "article",         "_id": "5",         "_score": 0.5753642,         "_source": {           "articleID": "DHJK-B-1395-#Ky5",           "userID": 3,           "hidden": false,           "postDate": "2017-03-01",           "tag": [             "elasticsearch"           ],           "tag_cnt": 1,           "view_cnt": 10,           "title": "this is spark blog",           "content": "spark is best big data solution based on scala ,an programming language similar to java spark",           "sub_title": "haha, hello world",           "author_first_name": "Tonny",           "author_last_name": "Peter Smith"         }       }     ]   } }

 

成功了，只有包含java spark这个短语的doc才返回了，只包含java的doc不会返回

 

3、term position

hello world, java spark         doc1

hi, spark java                      doc2

解析：

hello             doc1(0)         

wolrd             doc1(1)

java            doc1(2) doc2(2)

spark             doc1(3) doc2(1)

 

了解什么是分词后的position

GET _analyze {   "text": "hello world, java spark",   "analyzer": "standard" } 结果： {   "tokens": [     {       "token": "hello",       "start_offset": 0,       "end_offset": 5,       "type": "",       "position": 0     },     {       "token": "world",       "start_offset": 6,       "end_offset": 11,       "type": "",       "position": 1     },     {       "token": "java",       "start_offset": 13,       "end_offset": 17,       "type": "",       "position": 2     },     {       "token": "spark",       "start_offset": 18,       "end_offset": 23,       "type": "",       "position": 3     }   ] }

 

4、match_phrase的基本原理

 

索引中的position，match_phrase

 

hello world, java spark         doc1

hi, spark java                      doc2

 

hello             doc1(0)         

wolrd             doc1(1)

java            doc1(2) doc2(2)

spark             doc1(3) doc2(1)

 

java spark --> match phrase

 

java spark --> java和spark

 

java --> doc1(2) doc2(2)

spark --> doc1(3) doc2(1)

 

(1)要找到每个term都在的一个共有的那些doc，就是要求一个doc，必须包含每个term，才能拿出来继续计算

 

(2)doc1 --> java和spark --> spark position恰巧比java大1 --> java的position是2，spark的position是3，恰好满足条件

 

doc1符合条件

 

doc2 --> java和spark --> java position是2，spark position是1，spark position比java position小1，而不是大1 --> 光是position就不满足，那么doc2不匹配

 

必须理解这块原理！！！！

 

因为后面的proximity match就是原理跟这个一模一样！！！