elasticsearch ingest node
elasticsearch.yml中定义node为ingest node
node.ingest: false
可以在request或者bulk request命令提交pipeline到ingest node
PUT my-index/my-type/my-id?pipeline=my_pipeline_id
{
"foo": "bar"
}
{
"description" : "...",
"processors" : [ ... ]
}
description描述功能，processors定义处理列表
put api
可以更新和创建新的pipeline
PUT _ingest/pipeline/my-pipeline-id
{
"description" : "describe pipeline",
"processors" : [
{
"set" : {
"field": "foo",
"value": "bar"
}
}
]
}
修改可以即时更新
get api
GET _ingest/pipeline/my-pipeline-id
返回
{
"my-pipeline-id" : {
"description" : "describe pipeline",
"processors" : [
{
"set" : {
"field" : "foo",
"value" : "bar"
}
}
]
}
}
delet api
DELETE _ingest/pipeline/my-pipeline-id
模拟pepeline api
创建模拟的pipeline
POST _ingest/pipeline/_simulate
{
"pipeline" : {
// pipeline definition here
},
"docs" : [
{ /** first document **/ },
{ /** second document **/ },
// ...
]
}
根据现有的pipeline
POST _ingest/pipeline/my-pipeline-id/_simulate
{
"docs" : [
{ /** first document **/ },
{ /** second document **/ },
// ...
]
}
example
POST _ingest/pipeline/_simulate
{
"pipeline" :
{
"description": "_description",
"processors": [
{
"set" : {
"field" : "field2",
"value" : "_value"
}
}
]
},
"docs": [
{
"_index": "index",
"_type": "type",
"_id": "id",
"_source": {
"foo": "bar"
}
},
{
"_index": "index",
"_type": "type",
"_id": "id",
"_source": {
"foo": "rab"
}
}
]
}
返回值
{
"docs": [
{
"doc": {
"_id": "id",
"_ttl": null,
"_parent": null,
"_index": "index",
"_routing": null,
"_type": "type",
"_timestamp": null,
"_source": {
"field2": "_value",
"foo": "bar"
},
"_ingest": {
"timestamp": "2016-01-04T23:53:27.186+0000"
}
}
},
{
"doc": {
"_id": "id",
"_ttl": null,
"_parent": null,
"_index": "index",
"_routing": null,
"_type": "type",
"_timestamp": null,
"_source": {
"field2": "_value",
"foo": "rab"
},
"_ingest": {
"timestamp": "2016-01-04T23:53:27.186+0000"
}
}
}
]
}
访问pipeline数据
处理时可以访问或者设置数据的字段，元数据等信息
访问source filed
{
"set": {
"field": "_source.my_field"
"value": 582.1
}
}
修改元素数据字段，修改document _id
{
"set": {
"field": "_id"
"value": "1"
}
}
_index, _type, _id, _routing,_parent是可以接受访问的
访问ingest元数据
{
"set": {
"field": "received"
"value": "{{_ingest.timestamp}}"
}
}
访问field与metafield在template中
{
"set": {
"field": "field_c"
"value": "{{field_a}} {{field_b}}"
}
}
{
"set": {
"field": "_index"
"value": "{{geoip.country_iso_code}}"
}
}
pipelin错误处理
下面的例子可以将foo字段转换成bar字段,如果没有foo将保存错误在elasticsearch中分析
{
"description" : "my first pipeline with handled exceptions",
"processors" : [
{
"rename" : {
"field" : "foo",
"target_field" : "bar",
"on_failure" : [
{
"set" : {
"field" : "error",
"value" : "field \"foo\" does not exist, cannot rename to \"bar\""
}
}
]
}
}
]
}
怱略异常
{
"description" : "my first pipeline with handled exceptions",
"processors" : [
{
"rename" : {
"field" : "foo",
"target_field" : "bar",
"ignore_failure" : true
}
}
]
}
访问错误信息在pipeline
{
"description" : "my first pipeline with handled exceptions",
"processors" : [
{
"rename" : {
"field" : "foo",
"to" : "bar",
"on_failure" : [
{
"set" : {
"field" : "error",
"value" : "{{ _ingest.on_failure_message }}"
}
}
]
}
}
]
}
现有的process
https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-processors.html