flume-kafka 实例 详细

阅读更多

 

flume 下载：http://download.csdn.net/download/knight_black_bob/9454497

 

spring-kafka 接受消息实例   http://knight-black-bob.iteye.com/blog/2344424

 

flume kafka 安装 详细 http://knight-black-bob.iteye.com/blog/2343192

flume 安装 

 

20.flume 安装

tar -xf flume-ng-1.6.0-cdh5.5.0.tar.gz
mv   apache-flume-1.6.0-cdh5.5.0-bin ../soft/flume-1.6.0
cd ../soft/
ln -s /home/baoy/soft/flume-1.6.0 /home/baoy/soft/flume
chown -R root:root /home/baoy/soft/flume-1.6.0 /home/baoy/soft/flume
cd /home/baoy/soft/flume
cd /home/baoy/soft/flume/config
cp  /home/baoy/soft/flume/config/flume-env.sh.template /home/baoy/soft/flume/config/flume-env.sh  
vim flume-env.sh
	export JAVA_HOME=/usr/java/jdk1.7.0_79
/home/baoy/soft/flume/bin/flume-ng version


cp /usr/local/kafka/libs/kafka_2.10-0.10.0.0.jar /home/baoy/soft/flume/lib/
cp /usr/local/kafka/libs/scala-library-2.10.6.jar /home/baoy/soft/flume/lib/ 
cp /usr/local/kafka/libs/metrics-core-2.2.0.jar /home/baoy/soft/flume/lib/



vim /home/baoy/soft/flume/config/exec_tail_kafka.conf

agent.sources = s1
agent.sinks = k1
agent.channels = c1

agent.sources.s1.type=exec
agent.sources.s1.command=tail -F /home/project/flume/logs/flume.log
agent.sources.s1.channels=c1
agent.channels.c1.type=memory
agent.channels.c1.capacity=10000
agent.channels.c1.transactionCapacity=100

#设置Kafka接收器
agent.sinks.k1.type= org.apache.flume.sink.kafka.KafkaSink
#设置Kafka的broker地址和端口号
agent.sinks.k1.brokerList=dev10.aoiplus.openpf:9092
#设置Kafka的Topic
agent.sinks.k1.topic=baoy-topic
#设置序列化方式
agent.sinks.k1.serializer.class=kafka.serializer.StringEncoder

agent.sinks.k1.channel=c1
 
/home/baoy/soft/flume/bin/flume-ng agent -n agent -c conf -f /home/baoy/soft/flume/conf/exec_tail_kafka.conf -Dflume.root.logger=INFO,console
 

 

 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

捐助开发者

在兴趣的驱动下,写一个免费的东西，有欣喜，也还有汗水，希望你喜欢我的作品，同时也能支持一下。 当然，有钱捧个钱场（右上角的爱心标志，支持支付宝和PayPal捐助），没钱捧个人场，谢谢各位。

 
 谢谢您的赞助，我会做的更好！

 

 

 

 

 

 

 

• 
• 大小: 204 KB

• 
• 大小: 69.8 KB

• 
• 大小: 5.6 KB

• 查看图片附件