Flume1.6.0 ES2.3.5 sink

1.删除flume1.6.0/lib里面的guava-.jar and jackson-core-.jar

2.拷贝es2.3.5/lib里es2.3.5.jar包，到flume1.6.0/lib

3.拷贝elasticsearch-sink2-1.0.jar，到flume1.6.0/lib

4.编辑flume1.6.0/conf/es.conf

5.启动es

6.创建测试文件，并启动flume

3.拷贝elasticsearch-sink2-1.0.jar，到flume1.6.0/lib

也可以build出来jar包
https://github.com/lucidfrontier45/ElasticsearchSink2

ElasticsearchSink2

This is Flume-NG Sink for Elasticsearch >= 2.0. I developed this because the official version does not support Elasticsearch >= 2.0 due to API changes. Hope Flume dev team will fix this soon.

Requirements

Flume-NG >= 1.6
Elasticsearch >= 2.0
Build

Build standard jar by the following command

$ ./gradlew build
Build fat jar which contains elasticsearch dependencies

$ ./gradlew assembly
Jar will be generated in build/libs

Usage

Append the built jar to Flume's classpath
remove guava-*.jar and jackson-core-*.jar in flume's default libs dir. They are outdated and newer version are included in Elasticsearch.
set sink type to com.frontier45.flume.sink.elasticsearch2.ElasticSearchSink in flume.conf
start flume agent

4.编辑flume1.6.0/conf/es.conf

agent.sources = tail
agent.channels = memoryChannel
agent.channels.memoryChannel.type = memory
agent.sources.tail.channels = memoryChannel
agent.sources.tail.type = exec
agent.sources.tail.command = tail -F /tmp/ttt

# Describe the sink ES
agent.sinks = elasticsearch
agent.sinks.elasticsearch.type = com.frontier45.flume.sink.elasticsearch2.ElasticSearchSink
agent.sinks.elasticsearch.hostNames = 127.0.0.1:9300
agent.sinks.elasticsearch.indexType = logs
agent.sinks.elasticsearch.indexName = flume
agent.sinks.elasticsearch.clusterName = elas
agent.sinks.elasticsearch.batchSize = 500
agent.sinks.elasticsearch.ttl = 5d
agent.sinks.elasticsearch.serializer = com.frontier45.flume.sink.elasticsearch2.ElasticSearchDynamicSerializer
agent.sinks.elasticsearch.indexNameBuilder = com.frontier45.flume.sink.elasticsearch2.TimeBasedIndexNameBuilder
agent.sinks.elasticsearch.channel = memoryChannel

5.启动es

bin/elasticsearch

6.创建测试文件，并启动flume

touch /tmp/ttt
echo "test" >> /tmp/ttt

bin/flume-ng agent -c . -f conf/es.conf -n agent -Dflume.root.logger=INFO,console