ELASTIC 搜索开发实战-笔记
本文是《Elastic 搜索开发实战 》的学习笔记
《Elastic 搜索开发实战 》
需求描述：
- 智能补全提示
- 结果的高亮显示
- 结果的聚合统计和过滤
- 相关搜索结果的推荐
- 短语纠错(fuzziness)
- 数据的实时同步与更新(logstash)
- 查得到
- 查得全
- 查得准
- 支持同义词
- 支持简繁体
- 支持拼音(pinyin)
- 支持 PPT 的搜索
- 支持自定义排序，按时间，按相关度
- 支持结果过滤，按分类、按标签、按时间范围等
- 搜索结果展示待加强，UI 设计
版本：
- Elasticsearch v6.2.4
- Kibana v6.2.4
- Logstash v6.2.4
- mysql-connector-java-8.0.20.tar.gz
kibana dev tools快捷键
command + i 自动缩进
command + enter 提交请求
常用 API 介绍
1、增删改查
# 创建文档
POST twitter/doc/1
{
"name": "Jack",
"age": 30
}
# 取回文档
GET twitter/doc/1
# 完全替换
PUT twitter/doc/1
{
"name": "Jack",
"age": 35
}
# 部分更新
POST twitter/doc/1/_update
{
"doc": {
"name": "Mark"
}
}
# 删除文档
DELETE twitter/doc/1
2、搜索的使用
# 创建两个索引文档
POST twitter/doc/1
{
"name": "Jack",
"age": 30
}
POST twitter/doc/2
{
"name": "Mark",
"age": 35
}
# 通过名称搜索
GET twitter/_search?q=Jack
# 通过年龄检索
GET twitter/_search?q=35
# 限定查询年龄字段
GET twitter/_search?q=age:35
# QueryDSL查询表达式
POST twitter/_search
{
"query": {
"match": {
"age": 35
}
}
}
3、聚合的使用
# 再创建两个索引文档
POST twitter/doc/3
{
"name":"john",
"age":30
}
POST twitter/doc/4
{
"name":"mark",
"age":40
}
# 统计年龄的分布情况
POST twitter/_search
{
"size": 0,
"aggs": {
"age_stats": {
"terms": {
"field": "age",
"size": 10
}
}
}
}
聚合查询表达式说明
- 最外层的 size 为 0 不返回搜索命中的文档，只返回聚合的统计结果，
- aggs 就是我们描述聚合查询语句根节点，
- 使用 terms 聚合类型来对 age 字段的值进行统计，并且只返回前 10 个统计值，
- 然后这些统计结果我们命名为 age_stats
4、索引的管理
# 查看索引列表
GET _cat/indices?v
# 删除索引
DELETE twitter
# 查看集群的健康状况
GET _cluster/health
| 字段名 | 说明 |
|---|---|
| health | 索引健康状态，green 表示健康；yellow 表示数据完整，但是缺少副本；red 则表示有数据损坏。 |
| status | 索引工作状态，open 表示索引打开中，可以被使用；close 表示索引被关闭，不能使用。 |
| index | 索引名称。 |
| uuid | 索引的唯一 ID 标识。 |
| pri | 索引的主分片个数。 |
| rep | 索引的副本分片个数。 |
| docs.count | 索引内的文档个数。 |
| docs.deleted | 索引内已经删除的文档个数。 |
搜索示例
数据准备
创建表
CREATE TABLE `blog` (
`id` int(11) NOT NULL AUTO_INCREMENT COMMENT '主键id',
`title` varchar(60) DEFAULT NULL COMMENT '标题',
`author` varchar(60) DEFAULT NULL COMMENT '作者',
`content` text COMMENT '内容',
`create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',
`update_time` datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT '更新时间',
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4
获取测试数据
# -*- coding: utf-8 -*-
from pprint import pprint
import requests
from parsel import Selector
from puremysql import PureMysql
def get_data(url):
"""
获取古诗文网数据
eg: https://www.gushiwen.cn/
:return: list
"""
response = requests.get(url)
sel = Selector(text=response.text)
rows = sel.css(".main3 .left .sons")
lst = []
for row in rows:
title = row.css("b::text").extract_first()
author = row.css(".source").xpath("string(.)").extract_first()
content = row.css(".contson").xpath("string(.)").extract_first()
if not title:
continue
item = {
"title": title.strip(),
"author": author.strip(),
"content": content.replace('\n', ''),
}
pprint(item)
lst.append(item)
return lst
def insert_data(lst):
"""
数据入库
"""
con = PureMysql(db_url="mysql://root:[email protected]:3306/data?charset=utf8")
table = con.table("blog")
ret = table.insert(lst)
con.close()
print("成功入库", ret)
def main():
# url = "https://www.gushiwen.cn/"
for page in range(1, 11):
url = f"https://www.gushiwen.cn/default.aspx?page={page}"
lst = get_data(url)
insert_data(lst)
if __name__ == '__main__':
main()
logstash同步数据配置
config/jdbc.conf
input {
jdbc {
jdbc_driver_library => "mysql-connector-java-8.0.16.jar"
jdbc_driver_class => "com.mysql.cj.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/data"
jdbc_user => "root"
jdbc_password => "123456"
statement => "SELECT id, title, content, author, create_time, update_time FROM blog"
jdbc_paging_enabled => "true"
jdbc_page_size => "5000"
}
}
filter {
}
output {
stdout {
codec => rubydebug
}
elasticsearch {
index => "blog",
document_id => "%{id}"
}
}
同步数据
# 检查配置文件
$ ./bin/logstash -t -f config/jdbc.conf
# 执行配置文件
$ ./bin/logstash -f config/jdbc.conf
问题及处理
处理elasticsearch跨域问题
config/elasticsearch.yml
http.cors.enabled: true
http.cors.allow-origin: "*"
搜索提示
高亮结果显示
POST /blog/_search
{
"query": {
"match": {
"author": "李白"
}
},
"highlight": {
"fields": {
"author": {
}
}
}
}
搜索模板
将查询和参数分离
POST /blog/_search/template
{
"source": {
"query": {
"match": {
"{
{key}}": "{
{value}}"
}
},
"size": "{
{size}}"
},
"params": {
"key": "author",
"value": "李白",
"size": 10
}
}
其他语句
# 调试模板渲染结果：
GET _render/template
# 取回模板定义的语法：
GET _scripts/<templatename>
# 删除模板定义的语法：
DELETE _scripts/<templatename>
创建模板
POST /_scripts/blog_template_v1
{
"script": {
"lang": "mustache",
"source": {
"query": {
"match": {
"{
{key}}": "{
{value}}"
}
},
"highlight": {
"fields": {
"{
{key}}": {
}
}
},
"size": "{
{size}}"
}
}
}
使用模板
POST /blog/_search/template
{
"id": "blog_template_v1",
"params": {
"key": "author",
"value": "李白",
"size": 10
}
}
模糊查询
GET test/_search
{
"query": {
"match": {
"doc":{
"query": "elastix",
"fuzziness": "AUTO"
}
}
}
}
优化查询
POST _scripts/blog_template_v1
{
"script": {
"lang": "mustache",
"source": {
"size": "{
{size}}",
"query": {
"bool": {
"should": [
{
"prefix": {
"{
{field}}.keyword": {
"value": "{
{query}}",
"boost": 10
}
}
},
{
"match_phrase_prefix": {
"{
{field}}": {
"query": "{
{query}}",
"boost": 2
}
}
},
{
"match": {
"{
{field}}": "{
{query}}"
}
}
]
}
},
"_source": [
"title",
"id",
"uid",
"views"
]
}
}
}
重建索引
# 新建索引
PUT blog_v1
# 查看原索引的mapping
GET blog/_mapping
# 设置索引的mapping
POST blog_v1/doc/_mapping
{
"doc": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"author": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"content": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"create_time": {
"type": "date"
},
"id": {
"type": "long"
},
"title": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"update_time": {
"type": "date"
}
}
}
}
# 索引迁移
POST _reindex
{
"source": {
"index": "blog"},
"dest": {
"index": "blog_v1"}
}
# 查询测试
POST /blog_v1/_search
索引别名
# 查看别名
GET _cat/aliases
# 添加别名
POST /_aliases
{
"actions": [
{
"add": {
"index": "blog",
"alias": "my-blog"
}
}
]
}
# 切换别名
POST /_aliases
{
"actions": [
{
"add": {
"index": "blog_v1",
"alias": "my-blog"
}
},
{
"remove": {
"index": "blog",
"alias": "my-blog"
}
}
]
}
# 通过别名搜索
POST my-blog/_search
拼音处理的插件
https://github.com/medcl/elasticsearch-analysis-pinyin/releases/tag/v6.3.2
添加拼音搜索字段
# 关闭索引
POST my-blog/_close
# 设置索引支持拼音分析器
PUT my-blog/_settings
{
"index": {
"analysis": {
"analyzer": {
"pinyin_analyzer": {
"tokenizer": "my_pinyin"
}
},
"tokenizer": {
"my_pinyin": {
"type": "pinyin",
"keep_first_letter": true,
"keep_separate_first_letter": true,
"keep_full_pinyin": true,
"keep_original": false,
"limit_first_letter_length": 16,
"lowercase": true
}
}
}
}
}
# 打开索引
POST my-blog/_open
# 获取原索引mapping
GET my-blog/_mapping
# 添加字段
PUT my-blog/doc/_mapping
{
"doc": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"author": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
},
"pinyin": {
"type": "text",
"analyzer": "pinyin_analyzer"
}
}
},
"content": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"create_time": {
"type": "date"
},
"id": {
"type": "long"
},
"title": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"update_time": {
"type": "date"
}
}
}
}
# 更新索引
POST my-blog/_update_by_query?conflicts=proceed
# 测试拼音搜索
POST my-blog/_search
{
"query": {
"match": {
"author.pinyin": "libai"
}}
}
前端显示
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Documenttitle>
<script src="https://cdn.jsdelivr.net/npm/vue/dist/vue.js">script>
<link
rel="stylesheet"
href="https://unpkg.com/element-ui/lib/theme-chalk/index.css"
/>
<script src="https://unpkg.com/element-ui/lib/index.js">script>
<script src="https://unpkg.com/axios/dist/axios.min.js">script>
<style>
/* 居中显示 */
#app {
width: 200px;
margin: 0 auto;
margin-top: 300px;
}
/* 搜索结果高亮 */
em {
color: red;
}
style>
head>
<body>
<div id="app">
<el-autocomplete
v-model="state"
:fetch-suggestions="querySearchAsync"
placeholder="请输入内容"
@select="handleSelect"
>
<template slot-scope="{ item }">
<div v-html="item.highlight.author[0]">div>
template>
el-autocomplete>
div>
<script>
new Vue({
el: "#app",
data() {
return {
list: [],
state: "",
};
},
methods: {
async querySearchAsync(queryString, cb) {
// 查询地址
const QUERY_URL = "http://localhost:9200/blog/_search";
// 查询语句
let query = {
query: {
match: {
author: queryString,
},
},
highlight: {
fields: {
author: {
} },
},
};
const res = await axios.post(QUERY_URL, query);
console.log(res.data.hits.hits);
cb(res.data.hits.hits);
},
handleSelect(item) {
console.log(item);
},
},
});
script>
body>
html>
