应用场景: 修复与增强写入数据
Ingest Node
Pipeline & Processor
内置的 Processors
# 测试 split tags
POST _ingest/pipeline/_simulate
{"pipeline":{"description":"to split blog tags","processors": [
{"split":{"field":"tags","separator":","}}]},"docs": [
{"_index": "index","_id": "id","_source": {
"title": "Introducing big data......",
"tags": "hadoop,elasticsearch,spark",
"content": "You konw, for big data"}},
{"_index":"index","_id":"idxx","_source":{
"title":"Introducing cloud computering",
"tags":"openstack,k8s",
"content":"You konw, for cloud"}}]}
# 同时为文档增加一个字段
POST _ingest/pipeline/_simulate
{"pipeline":{"description":"to split blog tags","processors":[
{"split":{"field":"tags","separator":","}},
{"set":{"field":"views","value":0}}]},
"docs": [
{"_index":"index","_id":"id","_source":{
"title":"Introducing big data......",
"tags":"hadoop,elasticsearch,spark",
"content":"You konw, for big data"}},
{"_index":"index","_id":"idxx","_source":{
"title":"Introducing cloud computering",
"tags":"openstack,k8s",
"content":"You konw, for cloud"}}]}
# 为 ES 添加一个 Pipeline
PUT _ingest/pipeline/blog_pipeline
{"description":"a blog pipeline","processors":[
{"split":{"field":"tags","separator":","}},
{"set":{"field":"views","value":0}}]}
# 使用 pipeline 更新数据
PUT tech_blogs/_doc/2?pipeline=blog_pipeline
{"title":"Introducing cloud computering",
"tags":"openstack,k8s",
"content":"You konw, for cloud"}
#增加update_by_query的条件
POST tech_blogs/_update_by_query?pipeline=blog_pipeline
{"query":{"bool":{"must_not":{"exists":{"field":"views"}}}}}
Logstash | Ingest Node | |
---|---|---|
数据输入与输出 | 支持从不同的数据源读取,并写 入不同的数据源 |
支持从 ES REST API 获取数据, 并且写入 Elasticsearch |
数据缓冲 | 实现了简单的数据队列,支持重写 | 不支持缓冲 |
数据处理 | 支持大量的插件,也支持定制开发 | 内置的插件,可以开发Plugin进 行扩展 (Plugin更新需要重启) |
配置和使用 | 增加了一定的架构复杂度 | 无需额外部署 |
ctx.field_name
ctx._source.field_name
doc["field_name"]
# 增加一个 Script Prcessor
POST _ingest/pipeline/_simulate
{"pipeline":{"description":"to split blog tags","processors":[
{"split":{"field":"tags","separator":","}},
{"script":{"source":"""
if(ctx.containsKey("content")){
ctx.content_length = ctx.content.length();
}else{
ctx.content_length=0;
}"""}},
{"set":{"field":"views","value":0}}]},
"docs": [{"_index":"index","_id":"id","_source":{
"title":"Introducing big data......",
"tags":"hadoop,elasticsearch,spark",
"content":"You konw, for big data"}},
{"_index":"index","_id":"idxx","_source":{
"title":"Introducing cloud computering",
"tags":"openstack,k8s",
"content":"You konw, for cloud"}}]}
PUT tech_blogs/_doc/1
{"title":"Introducing big data......",
"tags":"hadoop,elasticsearch,spark",
"content":"You konw, for big data",
"views":0}
POST tech_blogs/_update/1
{"script":{"params":{"new_views":100},
"source":"ctx._source.views += params.new_views"}}
# 查看 views 计数
POST tech_blogs/_search
# 保存脚本在 Cluster State
POST _scripts/update_views
{"script":{"lang":"painless","source":"ctx._source.views += params.new_views"}}
GET tech_blogs/_search
{"script_fields":{"rnd_views":{"script":{"lang":"painless","source":"""
java.util.Random rnd = new Random();
doc['views'].value+rnd.nextInt(1000);"""}}},
"query":{"match_all":{}}}
脚本缓存:脚本编译的开销较大,Elasticsearch会将脚本编译后缓存在Cache 中
script.cache.max_size
设置最大缓存数
script.cache.expire
设置缓存超时
script.max_compilations_rate
默认5分钟最多75次编译 (75/5m)