cdh5.9运行mapreduce uber任务报java.lang.RuntimeException: native snappy library not available错误

1.问题描述

在对yarn进行了uber任务配置之后，运行了一个word count的mapreduce任务。控制台显示的已经在uber model下了：

但是运行完了之后发现任务失败。接着查看了运行日志，发现报这个错误：

2019-11-12 15:20:18,565 INFO [uber-SubtaskRunner] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1573542707169_0002_m_000000_0 is : 0.0
2019-11-12 15:20:18,565 INFO [uber-SubtaskRunner] org.apache.hadoop.mapred.Task: Runnning cleanup for the task
2019-11-12 15:20:18,571 WARN [uber-SubtaskRunner] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not delete hdfs://nameservice1/data/result/_temporary/1/_temporary/attempt_1573542707169_0002_m_000000_0
2019-11-12 15:20:18,571 INFO [uber-SubtaskRunner] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1573542707169_0002_m_000000_0: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.
	at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)
	at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
	at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
	at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
	at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1607)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1486)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:388)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:302)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:187)
	at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:230)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

大致意思就是我的Hadoop版本不支持snappy压缩。

2. 解决办法

参考了https://my.oschina.net/yulongblog/blog/758405这篇文章之后，将yarn的yarn.app.mapreduce.am.command-opts值修改成
-Djava.net.preferIPv4Stack=true（原有部分） -Djava.library.path=/opt/cloudera/parcels/CDH-5.9.3-1.cdh5.9.3.p0.4/lib/hadoop/lib/native/（新增部分）之后，再重新运行uber任务就能运行成功了。

3. uber任务的配置

cdh5.9由四个参数可以控制uber任务的开启：

uber任务就是小作业，application master判断在新的容器中分配和运行任务的开销大于并行运行它们的开销时，就会决定是否开启uber模式运行任务。
小作业就是少于10个mapper且只有1个reducer且输入大小小于一个HDFS块的作业。由以下三个参数控制。
mapreduce.job.ubertask.maxmaps：最大map个数，超过这个值就是大作业mapreduce.job.ubertask.maxreduces：最大reduce个数，超过就是大作业 mapreduce.job.ubertask.maxbytes：Ubertask 最大作业大小，超过就是大作业。