目录
- ElasticSearch
- 安装
- 运行
- 初步检索
- 新增文档
- 查询文档
- 更新文档
- 删除文档/索引
- 批量操作
- 进阶检索
- Search API
- Query DSL
- match
- bool
- filter
- term
- 字段.keyword以及match区分
- Aggregations
- Mapping
- 创建映射关系
- 查看映射信息
- 修改映射信息
- 分词
- 自定义词库
- 安装nginx
- 创建自定义词库
- 自定义词库
- 整合SpringBoot
- 创建索引
- 获取
- 删除
- 检索
ElasticSearch
安装
ElasticSearch
docker pull elasticsearch:7.4.2
Kibana用于可视化
docker pull kibana:
运行
准备es
mkdir -p /mydata/elasticsearch/config #存放ES配置
mkdir -p /mydata/elasticsearch/data #存放ES数据
chmod -R 777 /mydata/elasticsearch #修改权限
#进入config目录
echo "http.host: 0.0.0.0">>/mydata/elasticsearch/config/elasticsearch.yml #将可远程访问写入配置文件
运行es
# 9200是我们向es发请求的http端口，9300是es集群节点之间的通信端口
docker run --name es -p 9200:9200 -p 9300:9300 \
-e "discovery.type=single-node" \ #单节点设置
-e ES_JAVA_OPTS="-Xms64m -Xmx512m" \ #内存占用，此配置为开发测试
#### 挂载###
-v /mydata/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml \ #配置
-v /mydata/elasticsearch/data:/usr/share/elasticsearch/data \ #数据
-v /mydata/elasticsearch/plugins:/usr/share/elasticsearch/plugins \ #插件
-d elasticsearch:7.4.2
运行kibana
docker run --name kibana -e ELASTICSEARCH_HOSTS=http://123.56.16.54:9200 -p 5601:5601 -d kibana:7.4.2
初步检索
- GET请求：_cat/nodes
- GET请求：_cat/health
- GET请求：_cat/master 查看主节点
- GET请求：_cat/indices 查看索引
新增文档
-
PUT请求：index/type/id （不存在该id）新增数据，（存在该id）更新数据
-
POST请求：index/type/id （不带id，随机指定）新增数据，（带且存在该id）更新数据
查询文档
-
GET请求：index/type/id 查找
乐观锁，插叙的时候带上
?if_sq_no=1&if_primary_term=1
更新文档
-
POST请求：index/type/id/_update
方法1：若是此次更新数据与上次一样，则不会增加版本号，提示noop无操作
方法2：不带_update，同样的更新也能增加版本号，注意json数据写法不一样
- PUT请求：index/type/id
- _update只能是post请求
- 不会检查是否与上次更新操作一样
删除文档/索引
- DELETE请求：index/type/id
- DELETE请求：index
批量操作
两行一个数据，删除没有请求体，只有一行
进阶检索
Search API
#方法1 uri+检索参数
GET index01/_search?q=*&sort=_id:asc
#方法2 uri+请求体
GET index01/_search
{
"query": {
"match_all": {}
},
"sort": [
{
"_id": {
"order": "asc" #按id升序 desc降序
}
},
...
]
}
#默认只显示10个信息
Query DSL
match
# 1. match_all 查询所有
GET /bank/_search
{
"query": {
"match_all": {}
},
"sort": [
{
"balance": {
"order": "asc" #按id升序 desc降序
}
}
],
"from": 5, # 指定从哪里开始
"size": 5, # 指定一次查询多少个
"_source": ["balance","firstname"] # 指定返回哪些字段
}
# 2. match用于查找字段xxx,值为xxx的数据，
GET /bank/_search
{
"query": {
"match": {
"account_number": 20
}
}
}
# 也可以是模糊匹配，单词用空格隔开,按匹配度（匹配单词数量）降序 ”max_score“匹配最高的得分
GET /bank/_search
{
"query": {
"match": {
"address": "mill lane"
}
}
}
# 3.match_phrase 短语完全匹配
GET /bank/_search
{
"query": {
"match_phrase": {
"address": "Lane mill"
}
}
}
# 4.multi_match多字段匹配 指定字段内包含query值越多越匹配
GET /bank/_search
{
"query": {
"multi_match": {
"query": "Lane Brogan",
"fields": ["address","city"]
}
}
}
bool
- must
- must_not
- should：并非必要，只是匹配的话得分更高
# 以下条件筛选"gender"为"M"，"address"包含 "mill"，但是age不为28，且最好last有Holland的数据
GET /bank/_search
{
"query":{
"bool": {
"must": [
{"match": {
"gender": "M"
}},
{
"match": {
"address": "mill"
}
}
],
"must_not": [
{"match": {
"age": "28"
}}
],
"should": [
{"match": {
"lastname": "Holland"
}}
]
}
}
}
filter
GET /bank/_search
{
"query":{
"bool": {
"filter": { #不影响得分
"range": {
"age": {
"gte": 10,
"lte": 40
}
}
}
}
}
}
term
非text字段检索，match适合全文检索字段
GET /bank/_search
{
"query": {
"term": {
"balance": "32838"
}
}
}
字段.keyword以及match区分
必须完全匹配，空格也要，大小写也要
GET /bank/_search
{
"query": {
"match": {
"address.keyword": "789 Madison Street"
}
}
}
#下例中 match_phrase，只要存在该完整短语(完整单词A 完整单词B)即可（大小写无关，但是单词顺序不能乱）
GET /bank/_search
{
"query": {
"match_phrase": {
"address": "789 madison"
}
}
}
#下例中 match 只要存在当中任意一个单词即可，与单词顺序、拼写大小写无关，匹配越多单词得分越高
GET /bank/_search
{
"query": {
"match": {
"address": "789 madison"
}
}
}
# 注意：上述匹配均是需要匹配完整单词，每个单词长度一样才可匹配
Aggregations
GET bank/_search
{
"query": {
"match_all": {}
},
"aggs": {
"myagg1": {
"terms": {
"field": "age" #查看年龄分布
}
},
"myaggAVG":{
"avg": {
"field": "balance" #查看平均余额
}
}
},
"size": 0
}
GET bank/_search
{
"query": {
"match_all": {}
},
"aggs": {
"myagg1": {
"terms": {
"field": "age"
},
"aggs": {
"myaggAVG": { # 子聚合 即查看每个年龄的分布清空以及每个年龄的平均余额
"avg": {
"field": "balance"
}
}
}
}
},
"size": 0 #不显示查出的结果，只显示聚合结果
}
GET bank/_search
{
"query": {
"match_all": {}
},
"aggs": {
"myagg": {
"terms": {
"field": "age",
"size": 10
},
"aggs": {
"myaggGender": {
"terms": {
"field": "gender.keyword",
"size": 10
},
"aggs": {
"myaggGenderAVG": {
"avg": {
"field": "balance"
}
}
}
},
"myaggAllAVG":{
"avg": {
"field": "balance"
}
}
}
}
},
"size": 0
}
# 查看各个年龄段的男和女分别的平均余额以及这个年龄段的平均余额
Mapping
创建映射关系
PUT /my_index
{
"mappings": {
"properties": {
"age": {"type": "integer"},
"email":{"type": "keyword"},
"name": {"type": "text"，"index":true} #index默认true，默认可以被索引
}
}
}
查看映射信息
GET请求：index/_mapping
修改映射信息
添加新字段
PUT /my_index/_mapping
{
"properties":{
"address":{
"type":"keyword",
"index":false
}
}
}
注意：以及存在的映射信息不可修改
如果要修改，只能指定正确的映射关系，然后数据迁移
PUT /new_bank #创建新的映射
{
"mappings": {
"properties": {
"account_number": {
"type": "long"
},
"address": {
"type": "text"
},
"age": {
"type": "integer"
},
"balance": {
"type": "long"
},
"city": {
"type": "keyword"
},
"email": {
"type": "keyword"
},
"employer": {
"type": "keyword"
},
"firstname": {
"type": "text"
},
"gender": {
"type": "keyword"
},
"lastname": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"state": {
"type": "keyword"
}
}
}
}
POST _reindex #数据迁移，不再用type
{
"source": {
"index": "bank",
"type": "account" #原始数据拥有类型，6.0以后不用type
},
"dest": {
"index": "new_bank" #不用type
}
}
原始type是自己创建的适合定义的
不用type之后所有的type为_doc
分词
安装插件
下载elasticsearch-analysis-ik-7.4.2
解压后放到挂载的plugins目录，或者进入容器放进去
检查插件是否安装成功
docker exec -it es /bin/bash
cd bin
elasticsearch-plugin list # 列出所有插件
POST _analyze
{
"analyzer": "ik_max_word",
"text": ["我是中国人"]
}
POST _analyze
{
"analyzer": "ik_smart",
"text": ["我是中国人"]
}
自定义词库
安装nginx
# 随便启动一个nginx实例
docker run -p 80:80 --name nginx -d nginx:1.10
# 把容器内配置文件复制出来,(后面有个 空格+. )
docker container cp nginx:/etc/nginx .
# 停止、删除这个容器
# 把配置放到conf目录再转移到/mydata/nginx目录下
mv nginx conf
mkdir nginx
mv conf /mydata/nginx/
# 运行
docker run -p 80:80 --name nginx \
-v /mydata/nginx/html:/usr/share/nginx/html \
-v /mydata/nginx/logs:/var/log/nginx \
-v /mydata/nginx/conf:/etc/nginx \
-d nginx:1.10
#测试是否启动成功
cd html
vi index.html
#写入
hahahha~~~~~
#保存退出，访问80端口
创建自定义词库
-
在nginx的html目录下新建es文件夹新建txt文件，录入单词，保存访问ip:80/es/xxx.txt
-
进入plugins/elasticsearch-analysis-ik-7.4.2/config目录修改
IKAnalyzer.cfg.xml文件在扩展字典那里写上各个的访问路径
IK Analyzer 扩展配置 http://你的ip/es/participle.txt -
重启
整合SpringBoot
Rest Client文档
- 创建springboot项目，选择web不选择springboot整合的elasticsearch，因其版本没有更新到7.0之后
- 导入依赖
org.elasticsearch.client
elasticsearch-rest-high-level-client
7.4.2
- 调整版本，springboot 2.2.7内置6.8.8，指定版本即可
1.8
7.4.2
- 配置
@Configuration
public class ElasticSearchConfig {
public static final RequestOptions COMMON_OPTIONS;
static {
RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();
/*builder.addHeader("Authorization", "Bearer " + TOKEN);
builder.setHttpAsyncResponseConsumerFactory(
new HttpAsyncResponseConsumerFactory
.HeapBufferedResponseConsumerFactory(30 * 1024 * 1024 * 1024));*/
COMMON_OPTIONS = builder.build();
}
@Bean
public RestHighLevelClient esRestCilent(){
RestHighLevelClient client = new RestHighLevelClient(
RestClient.builder(
new HttpHost("ip...", 9200, "http")));
return client;
}
}
创建索引
@Test
void index() throws IOException{
//新建一个索引
IndexRequest indexRequest = new IndexRequest("users");
//指定id
indexRequest.id("1");
User user = new User();
user.setUserName("小明");
user.setGender("M");
user.setAge(18);
// 转为json字符
String u = JSON.toJSONString(user);
// 保存json
indexRequest.source(u, XContentType.JSON);
// 执行操作
IndexResponse index = client.index(indexRequest, ElasticSearchConfig.COMMON_OPTIONS);
System.out.println(index);
}
输出结果
IndexResponse[index=users,type=_doc,id=1,version=1,result=created,seqNo=0,primaryTerm=1,shards={"total":2,"successful":1,"failed":0}]
获取
@Test
public void get() throws IOException{
GetRequest getRequest = new GetRequest(
"users",
"1");
GetResponse fields = client.get(getRequest, ElasticSearchConfig.COMMON_OPTIONS);
System.out.println(fields);
}
输出
{"_index":"users","_type":"_doc","_id":"1","_version":1,"_seq_no":0,"_primary_term":1,"found":true,"_source":{"age":18,"gender":"M","userName":"小明"}}
删除
@Test
public void delete() throws IOException{
DeleteRequest request = new DeleteRequest(
"users",
"1");
DeleteResponse delete = client.delete(request, ElasticSearchConfig.COMMON_OPTIONS);
System.out.println(delete);
}
检索
json工具网站
点击此处
@Test
public void search() throws IOException{
//创建请求
SearchRequest searchRequest = new SearchRequest("new_bank");
//封装条件
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
//筛选address有mill单词的数据
searchSourceBuilder.query(QueryBuilders.matchQuery("address","mill"));
//按年龄分布聚合，
TermsAggregationBuilder ageTerm = AggregationBuilders.terms("ageTerm").field("age");
//求余额平均值
AvgAggregationBuilder balanceAvg = AggregationBuilders.avg("balanceAvg").field("balance");
//添加聚合
searchSourceBuilder.aggregation(ageTerm);
searchSourceBuilder.aggregation(balanceAvg);
System.out.println("请求"+searchSourceBuilder);
//保存
searchRequest.source(searchSourceBuilder);
//执行
SearchResponse response = client.search(searchRequest, ElasticSearchConfig.COMMON_OPTIONS);
System.out.println("检索返回信息"+response);
//分析返回信息
SearchHits hits = response.getHits();
System.out.println("检索结果有几个："+hits.getTotalHits());
SearchHit[] searchHits = hits.getHits();
for(SearchHit hit :searchHits){
String string = hit.getSourceAsString();
Account account = JSON.parseObject(string,Account.class);
System.out.println("检索结果"+account);
}
//分析聚合结果
Aggregations aggregations = response.getAggregations();
Terms term = aggregations.get("ageTerm");
for (Terms.Bucket bucket : term.getBuckets()) {
String keyAsString = bucket.getKeyAsString();
System.out.println("年龄为："+keyAsString+"有："+bucket.getDocCount()+"个");
}
Avg avg = aggregations.get("balanceAvg");
System.out.println("求得余额平均值为："+avg.getValue());
}
输出的结果
请求
{
"query": {
"match": {
"address": {
"query": "mill",
"operator": "OR",
"prefix_length": 0,
"max_expansions": 50,
"fuzzy_transpositions": true,
"lenient": false,
"zero_terms_query": "NONE",
"auto_generate_synonyms_phrase_query": true,
"boost": 1.0
}
}
},
"aggregations": {
"ageTerm": {
"terms": {
"field": "age",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [{
"_count": "desc"
}, {
"_key": "asc"
}]
}
},
"balanceAvg": {
"avg": {
"field": "balance"
}
}
}
}
检索返回信息
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 4,
"relation": "eq"
},
"max_score": 5.4032025,
"hits": [{
"_index": "new_bank",
"_type": "_doc",
"_id": "970",
"_score": 5.4032025,
"_source": {
"account_number": 970,
"balance": 19648,
"firstname": "Forbes",
"lastname": "Wallace",
"age": 28,
"gender": "M",
"address": "990 Mill Road",
"employer": "Pheast",
"email": "[email protected]",
"city": "Lopezo",
"state": "AK"
}
}, ...]
},
"aggregations": {
"avg#balanceAvg": {
"value": 25208.0
},
"lterms#ageTerm": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [{
"key": 38,
"doc_count": 2
}, {
"key": 28,
"doc_count": 1
}, {
"key": 32,
"doc_count": 1
}]
}
}
}
检索结果有几个：4 hits
检索结果Account(account_number=970, balance=19648, firstname=Forbes, lastname=Wallace, age=28, gender=M, address=990 Mill Road, employer=Pheast, [email protected], city=Lopezo, state=AK)
检索结果Account(account_number=136, balance=45801, firstname=Winnie, lastname=Holland, age=38, gender=M, address=198 Mill Lane, employer=Neteria, [email protected], city=Urie, state=IL)
检索结果Account(account_number=345, balance=9812, firstname=Parker, lastname=Hines, age=38, gender=M, address=715 Mill Avenue, employer=Baluba, [email protected], city=Blackgum, state=KY)
检索结果Account(account_number=472, balance=25571, firstname=Lee, lastname=Long, age=32, gender=F, address=288 Mill Street, employer=Comverges, [email protected], city=Movico, state=MT)
年龄为：38有：2个
年龄为：28有：1个
年龄为：32有：1个
求得余额平均值为：25208.0
用kibana测试
