ElasticSearch-Kibana常用操作
启动
ElasticSearch
1、进入目录 elasticsearch-5.6.16
2、打开Terminal，键入：./elasticsearch
3、检查是否启动成功
Kibana
1、进入目录 kibana-5.6.16-darwin-x86_64
2、打开Terminal，键入：./kibana
3、检查是否启动成功
4、打开http://localhost:5601
Kibana常用操作
1、快速检查集群的健康状况
GET /_cat/health?v
注意：后面如果跟上?v会显示表头，否则不显示表头
- green：每个索引的primary shard和replica shard都是active状态的
- yellow：每个索引的primary shard都是active状态的，但是部分replica shard不是active状态，处于不可用的状态
- red：不是所有索引的primary shard都是active状态的，部分索引有数据丢失了
Q&A 为什么现在会处于一个yellow状态？
/**
我们现在就一个笔记本电脑，就启动了一个es进程，相当于就只有一个node。现在es中有一个index，就是kibana自己内置建立的index。
由于默认的配置是给每个index分配5个primary shard和5个replica shard，而且primary shard和replica shard不能在同一台机器上（为了容错）。
现在kibana自己建立的index是1个primary shard和1个replica shard。当前就一个node，
所以只有1个primary shard被分配了和启动了，但是一个replica shard没有第二台机器去启动。
*/
2、快速查看集群中有哪些索引
GET /_cat/indices?v
3、简单的索引操作
创建索引：PUT /test_index?pretty
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open test_index XmS9DTAtSkSZSwWhhGEKkQ 5 1 0 0 650b 650b
yellow open .kibana rUm9n9wMRQCCrRDEhqneBg 1 1 1 0 3.1kb 3.1kb
删除索引：DELETE /test_index?pretty
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana rUm9n9wMRQCCrRDEhqneBg 1 1 1 0 3.1kb 3.1kb
4、商品的CRUD操作
PUT /Index/Type/Id：新增/替换
GET /Index/Type/id: 查询
POST /Index/Type/id/_update : 修改
DELETE /Index/Type/Id : 删除
4.1 新增商品：新增文档，建立索引
PUT /Index/Type/Id
{
"json数据"
}
键入如下操作
PUT /ecommerce/product/1
{
"name" : "gaolujie yagao",
"desc" : "gaoxiao meibai",
"price" : 30,
"producer" :"gaolujie producer",
"tags": [ "meibai", "fangzhu" ]
}
返回操作结果
{
"_index": "ecommerce",
"_type": "product",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"created": true
}
es会自动建立index和type，不需要提前创建，而且es默认会对document每个field都建立倒排索引，让其可以被搜索
4.2 查询商品：检索文档
GET /Index/Type/id
键入：
GET /ecommerce/product/1
返回：
{
"_index": "ecommerce",
"_type": "product",
"_id": "1",
"_version": 1,
"found": true,
"_source": {
"name": "gaolujie yagao",
"desc": "gaoxiao meibai",
"price": 30,
"producer": "gaolujie producer",
"tags": [
"meibai",
"fangzhu"
]
}
}
4.3 修改商品：替换文档
PUT /ecommerce/product/1
{
"name" : "jiaqiangban gaolujie yagao",
"desc" : "gaoxiao meibai",
"price" : 30,
"producer" : "gaolujie producer",
"tags": [ "meibai", "fangzhu" ]
}
替换方式有一个不好，即使必须带上所有的field，才能去进行信息的修改，例如
PUT /ecommerce/product/1
{
"name" : "jiaqiangban gaolujie yagao"
}
通过查询，发现id为1的商品只有那么字段了
4.4 修改商品：更新文档
POST /ecommerce/product/1/_update
{
"doc": {
"name": "jiaqiangban gaolujie yagao"
}
}
得到返回结果
{
"_index": "ecommerce",
"_type": "product",
"_id": "1",
"_version": 8,
"result": "updated",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
}
}
只会更新name字段
4.5 删除商品：删除文档
DELETE /ecommerce/product/1
返回结果:
{
"found": true,
"_index": "ecommerce",
"_type": "product",
"_id": "1",
"_version": 9,
"result": "deleted",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
}
}
如果删除的document在原来的type中不存在
{
"_index": "ecommerce",
"_type": "product",
"_id": "1",
"found": false
}
5、商品的查询操作
六种查询方式
- 1、query string search
GET /Index/Type/_search : 搜索全部 - 2、query DSL
- 3、query filter
- 4、full-text search
- 5、phrase search
- 6、highlight search
5.1、query string search（不常用)
GET /taobao/goods/_search
结果如下：
{
"took": 5,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 4,
"max_score": 1,
"hits": [
{
"_index": "taobao",
"_type": "goods",
"_id": "4",
"_score": 1,
"_source": {
"name": "heiren",
"desc": "youxiao qingjie",
"price": 70,
"producer": "yagao producer"
}
},
......
]
}
}
took：耗费了几毫秒
timed_out：是否超时，这里是没有
_shards：数据拆成了5个分片，所以对于搜索请求，会打到所有的primary shard（或者是它的某个replica shard也可以）
hits.total：查询结果的数量，3个document
hits.max_score：score的含义，就是document对于一个search的相关度的匹配分数，越相关，就越匹配，分数也高
hits.hits：包含了匹配搜索的document的详细数据
适用于临时的在命令行使用一些工具，比如curl，快速的发出请求，来检索想要的信息；但是如果查询请求很复杂，是很难去构建的。在生产环境中，几乎很少使用query string search，例如这样的需求：
搜索商品名称中包含yagao的商品，而且按照售价降序排序
GET /taobao/goods/_search?q=name:yagao&sort=price:desc
5.2、query DSL
DSL：Domain Specified Language，特定领域的语言
http request body：请求体，可以用json的格式来构建查询语法，比较方便，可以构建各种复杂的语法，比query string search肯定强大多了
5.2.1 查询所有
GET /taobao/goods/_search
{
"query": {
"match_all": {}
}
}
5.2.2 查询名称包含yagao的商品，同时按照价格降序排序
GET /taobao/goods/_search
{
"query": {
"match":{
"name":"yagao"
}
},
"sort": [
{
"price": "desc"
}
]
}
5.2.3 分页查询商品，总共3条商品，假设每页就显示1条商品，现在显示第2页，所以就查出来第2个商品
GET /taobao/goods/_search
{
"query": {
"match_all": {}
},
"from": 1,
"size": 1
}
5.2.4 指定查询返回的字段列表
GET /taobao/goods/_search
{
"query": {"match_all": {}},
"_source":["name","producer"]
}
返回结果如下所示
{
"took": 6,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 4,
"max_score": 1,
"hits": [
{
"_index": "taobao",
"_type": "goods",
"_id": "4",
"_score": 1,
"_source": {
"name": "heiren",
"producer": "yagao producer"
}
},
......
]
}
}
5.3、query filter
搜索商品名称包含yagao，而且售价大于20元的商品
GET /taobao/goods/_search
{
"query": {
"bool": {
"must":{
"match":{
"name":"yagao"
}
},
"filter": {
"range":{
"price":{
"gt":20
}
}
}
}
}
}
5.4、full-text search（全文检索）
查询producer中包含 yagao和producer的商品
即：用空格分隔match里面的查询条件
GET /taobao/goods/_search
{
"query":{
"match":{
"producer": "yagao producer"
}
}
}
查询的结果是
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 4,
"max_score": 0.78549397,
"hits": [
{
"_index": "taobao",
"_type": "goods",
"_id": "4",
"_score": 0.78549397,
"_source": {
"name": "heiren",
"desc": "youxiao qingjie",
"price": 70,
"producer": "yagao producer"
}
},
{
"_index": "taobao",
"_type": "goods",
"_id": "1",
"_score": 0.25811607,
"_source": {
"name": "gaolujie yagao",
"desc": "youxiao fangzhu",
"price": 25,
"producer": "gaolujie producer"
}
},
{
"_index": "taobao",
"_type": "goods",
"_id": "3",
"_score": 0.25811607,
"_source": {
"name": "zhonghua",
"desc": "youxiao zhonghua",
"price": 55,
"producer": "zhonghua producer"
}
},
{
"_index": "taobao",
"_type": "goods",
"_id": "2",
"_score": 0.16358379,
"_source": {
"name": "jiajieshi yagao",
"desc": "youxiao fangzhu",
"price": 30,
"producer": "jiajieshi producer"
}
}
]
}
}
一个复杂的查询
GET /taobao/goods/_search
{
"query":{
"bool":{
"must":{
"match":{
"producer":"yagao producer"
}
},
"filter": {
"range":{
"price":{
"gt":30
}
}
}
}
},
"sort":{
"price":"asc"
}
}
5.5、phrase search（短语搜索）
跟全文检索相对应，相反，全文检索会将输入的搜索串拆解开来，去倒排索引里面去一一匹配，只要能匹配上任意一个拆解后的单词，就可以作为结果返回phrase search，要求输入的搜索串，必须在指定的字段文本中，完全包含一模一样的，才可以算匹配，才能作为结果返回
GET /taobao/goods/_search
{
"query":{
"match_phrase":{
"producer":"yagao producer"
}
}
}
6、数据分析
6.1 计算每个tag下的商品数量
GET /taobao/goods/_search
{
"size":0,//表示返回hits/hits为空
"aggs": { //aggs表示聚合分析，固定写法
"group_by_tag": {//为此次分组命名
"terms": {// 分组terms，固定写法
"field": "tag"//按照tag的内容进行分组，tag对应的是一个jsonarray
}
}
}
}
首次使用tag进行分组会报错，这个时候需要将tag字段fielddata置为true，如下所示
PUT /taobao/_mapping/goods
{
"properties": {
"tag":{
"type":"text",
"fielddata": true
}
}
}
得到的结果如下所以
{
"took": 7,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 4,
"max_score": 0,
"hits": []
},
"aggregations": {
"group_by_tag": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "fangzhu",
"doc_count": 2
},
{
"key": "meibai",
"doc_count": 2
},
{
"key": "qingxin",
"doc_count": 2
}
]
}
}
}
6.2 对名称中包含yagao的商品，计算每个tag下的商品数量
对比上面的aggs，多了query的条件
GET /taobao/goods/_search
{
"size":0,
"query": {
"match": {
"name": "yagao"
}
},
"aggs":{
"group_by_tag":{
"terms": {
"field": "tag"
}
}
}
}
6.3 先分组，再算每组的平均值，计算每个tag下的商品的平均价格
GET /taobao/goods/_search
{
"size":0,
"aggs": {
"group_by_tag": {
"terms": {
"field": "tag"
},
"aggs": {
"avg_price": {
"avg": {
"field": "price"
}
}
}
}
}
}
返回的结果是：
{
"took": 9,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 4,
"max_score": 0,
"hits": []
},
"aggregations": {
"group_by_tag": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "fangzhu",
"doc_count": 2,
"avg_price": {
"value": 55.5
}
},
{
"key": "meibai",
"doc_count": 2,
"avg_price": {
"value": 35.5
}
},
{
"key": "qingxin",
"doc_count": 2,
"avg_price": {
"value": 40
}
}
]
}
}
}
6.4
算每个tag下的商品的平均价格，并且按照平均价格降序排序
GET /taobao/goods/_search
{
"size":0,
"aggs":{
"group_by_tag":{
"terms":{
"field": "tag",
"order": {
"avg_price": "desc"
}
},
"aggs": {
"avg_price": {
"avg": {
"field": "price"
}
}
}
}
}
}
返回结果是
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 4,
"max_score": 0,
"hits": []
},
"aggregations": {
"group_by_tag": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "fangzhu",
"doc_count": 2,
"avg_price": {
"value": 55.5
}
},
{
"key": "qingxin",
"doc_count": 2,
"avg_price": {
"value": 40
}
},
{
"key": "meibai",
"doc_count": 2,
"avg_price": {
"value": 35.5
}
}
]
}
}
}
6.5 按照指定的价格范围区间进行分组，然后在每组内再按照tag进行分组，最后再计算每组的平均价格
GET /taobao/goods/_search
{
"size":0,
"aggs": {
"group_by_price": {
"range": {
"field": "price",
"ranges":[
{
"from":0,
"to":20
},
{
"from":20,
"to":40
},
{
"from":40,
"to":60
}
]
},
"aggs": {
"group_by_tag":{
"terms": {
"field": "tag"
},
"aggs": {
"avg_price": {
"avg": {
"field": "price"
}
}
}
}
}
}
}
}
返回结果是：
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 4,
"max_score": 0,
"hits": []
},
"aggregations": {
"group_by_price": {
"buckets": [
{
"key": "0.0-20.0",
"from": 0,
"to": 20,
"doc_count": 0,
"group_by_tag": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": []
}
},
{
"key": "20.0-40.0",
"from": 20,
"to": 40,
"doc_count": 1,
"group_by_tag": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "meibai",
"doc_count": 1,
"avg_price": {
"value": 30
}
},
{
"key": "qingxin",
"doc_count": 1,
"avg_price": {
"value": 30
}
}
]
}
},
{
"key": "40.0-60.0",
"from": 40,
"to": 60,
"doc_count": 2,
"group_by_tag": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "fangzhu",
"doc_count": 1,
"avg_price": {
"value": 41
}
},
{
"key": "meibai",
"doc_count": 1,
"avg_price": {
"value": 41
}
},
{
"key": "qingxin",
"doc_count": 1,
"avg_price": {
"value": 50
}
}
]
}
}
]
}
}
}