使用flume中遇见的问题整理

先发一下牢骚： 这玩意真恶心遇见个问题都搜不到。。。。 要么就是英文的，比啃牛皮还难受。 所以写一个汇集， 要是大家有新问题或者别的解决方法，有什么不对的地方，尽情留言拍砖，大家共同学习进步

1 开发的jar包 是取得 flume安装目录下的lib内的所有

 

2 提示log4j WARN

   解决方法：在src下补充log4j.properties 文件

 

3 运行程序向avor的source里面写东西，提示异常  NettyAvroRpcClient { host: n1, port: 44444 }: RPC connection error

完整异常：

Exception in thread "main" org.apache.flume.FlumeException: NettyAvroRpcClient { host: n1, port: 44444 }: RPC connection error

    at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:117)

    at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:93)

    at org.apache.flume.api.NettyAvroRpcClient.configure(NettyAvroRpcClient.java:507)

    at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:169)

    at org.apache.flume.api.RpcClientFactory.getDefaultInstance(RpcClientFactory.java:128)

    at cn.vko.flume.MyRpcClientFacade.init(MyRpcClientFacade.java:22)

    at cn.vko.flume.ClientApp.main(ClientApp.java:10)

Caused by: java.io.IOException: Error connecting to /192.168.1.16:44444

    at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)

    at org.apache.avro.ipc.NettyTransceiver.(NettyTransceiver.java:203)

    at org.apache.avro.ipc.NettyTransceiver.(NettyTransceiver.java:152)

    at org.apache.flume.api.NettyAvroRpcClient.connect(NettyAvroRpcClient.java:106)

    ... 6 more

Caused by: java.net.ConnectException: Connection refused: no further information

    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)

    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:396)

    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:358)

    at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:274)

    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

    at java.lang.Thread.run(Thread.java:662)

    解决方法： 这是flume-ng 没有启动，检查是否启动着java进程 Application 。。。

 

4 运行程序时，$FLUME_HOME/logs/flume.log 中异常 Flume wasn't able to parse timestamp header in the event to resolve time based bucketing：

19 Feb 2013 14:18:59,799 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160)  - Unable to deliver event. Exception follows.

org.apache.flume.EventDeliveryException: java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using TimestampInterceptor source interceptor).

        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:464)

        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)

        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)

        at java.lang.Thread.run(Thread.java:662)

Caused by: java.lang.RuntimeException: Flume wasn't able to parse timestamp header in the event to resolve time based bucketing. Please check that you're correctly populating timestamp header (for example using TimestampInterceptor source interceptor).

        at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:160)

        at org.apache.flume.formatter.output.BucketPath.escapeString(BucketPath.java:343)

        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:397)

        ... 3 more

Caused by: java.lang.NumberFormatException: null

        at java.lang.Long.parseLong(Long.java:375)

        at java.lang.Long.valueOf(Long.java:525)

        at org.apache.flume.formatter.output.BucketPath.replaceShorthand(BucketPath.java:158)

        ... 5 more

 

解决方法: 这是HDFS sink  提示的异常，这是因为在hdfs路径中有时间参数，然后取event中的header的 timestamp 参数，取不到提示的异常； 所以解决方法是

    1 检查配置文件conf/flume-conf.properties 中是否用到了时间参数，去掉他们。

    2 或者在提交event的时候添加header的timestamp 参数，如下：

    

        Event event = EventBuilder.withBody(data, Charset.forName("UTF-8"));

        Map headers = new HashMap();

        headers.put("timestamp", String.valueOf(System.currentTimeMillis()));/**/

        event.setHeaders(headers);

        // Send the event

        try {

            client.append(event);

        } catch (EventDeliveryException e) {

            // clean up and recreate the client

            client.close();

            client = null;

            client = RpcClientFactory.getDefaultInstance(hostname, port);

        }

    3 使用多路符合那种方案部署时怎么弄，不是怎么清楚呢。。。 待查清

（未完待完善。。。。。。 欢迎补充。。。 ）