java.lang.RuntimeException: native-lzo library not available

flume配置lzo压缩将数据存在hdfs上,报错信息如下:

20/05/30 11:42:50 INFO hdfs.BucketWriter: Creating /origin_data/gmall/log/topic_event/2020-05-29/logevent-.1590810087777.lzo.tmp
20/05/30 11:42:50 ERROR hdfs.HDFSEventSink: process failed
java.lang.RuntimeException: native-lzo library not available
	at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
	at org.apache.flume.sink.hdfs.HDFSCompressedDataStream.open(HDFSCompressedDataStream.java:97)
	at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:249)
	at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:232)
	at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:668)
	at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
	at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:665)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
20/05/30 11:42:50 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.RuntimeException: native-lzo library not available
	at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:451)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: native-lzo library not available
	at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.java:155)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
	at org.apache.flume.sink.hdfs.HDFSCompressedDataStream.open(HDFSCompressedDataStream.java:97)
	at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:249)
	at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:232)
	at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:668)
	at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50)
	at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:665)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more

看了一眼hdfs,上面有很多大小为0的以.lzo.tmp结尾的小文件,于是停掉服务开始找错

在经过了格式化namenode、从头到尾检查及同步配置文件、重启服务后还是没有解决,于是格式化namenode、删除所有组件的日志文件、安装lzo(yum install lzo,安装完后突然想起没指定版本,想卸掉重装,吃个饭的时间忘了。。。我估计不是因为这个)、重启服务还是老样子,然后停掉flume服务,再次查看配置文件对比官网查博客,不知道哪里出问题了,重启flume,它就好了。。。折腾了半天,也不知道是怎么死的。。。

你可能感兴趣的:(#,flume)