flume sink hdfs异常

1、异常消息如下:

016-08-26 14:19:17,704 (hdfs-sink1-call-runner-2) [ERROR - org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync(AbstractHDFSWriter.java:267)] Error while trying to hflushOrSync!
2016-08-26 14:19:17,704 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:455)] HDFS IO error
java.io.IOException: Callable timed out after 10000 ms on file: hdfs://cluster/data01/kafka2hdfs/REMOTE_LOGIN_VERIFICATIOON_ONLINE/ds=2016-08-26/FlumeData.1472140808764.tmp
        at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:693)
        at org.apache.flume.sink.hdfs.BucketWriter.doFlush(BucketWriter.java:459)
        at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:421)
        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:442)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException
        at java.util.concurrent.FutureTask.get(FutureTask.java:201)
        at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:686)




2、发现flume配置实时三个参数被配置0了:

tier1.sinks.sink1.hdfs.rollInterval = 0
tier1.sinks.sink1.hdfs.rollSize = 0
tier1.sinks.sink1.hdfs.rollCount = 0


3、修改后的配置:

tier1.sinks.sink1.hdfs.path = hdfs://cluster/data01/kafka2hdfs/%{topic}/ds=%Y-%m-%d
tier1.sinks.sink1.hdfs.rollInterval = 60
tier1.sinks.sink1.hdfs.rollSize = 10240
tier1.sinks.hdfs-write.hdfs.batchSize = 100
tier1.sinks.sink1.hdfs.rollCount = 100
tier1.sinks.sink1.hdfs.fileType = DataStream
tier1.sinks.sink1.channel = channel1


你可能感兴趣的:(flume,hadoop)