主要描述下,在Win7环境下,通过eclipse往集群提交MapReduce程序的过程。
一、环境说明:
开发环境:WIN7
Eclipse版本:eclipse-jee-indigo
Hadoop版本:Hadoop2.5.2
MR运行模式:Yarn
二、使用MapReduce的Eclipse插件:
插件名称:hadoop-eclipse-plugin-2.5.2.jar
可以网上下载,本文档附件中也有上传。
将插件复制到${ECLIPSE_HOME}/plugins/目录下。
三、运行简单的MR(如:Wordcount)程序,碰到下面问题:
1、运行测试程序时,会出现以下报错:
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
解决方法:
需要在yarn-site.xml上,配置以下内容:
<property>
<name>mapreduce.job.ubertask.enable</name>
<value>true</value>
</property>
当在WIN7下用Eclipse提交MR程序时,如果还报这样的错误,需要在客户端代码增加如下配置:
conf.set("mapreduce.job.ubertask.enable","true");
2、运行测试程序时,会出现以下报错:
ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
解决方法:
conf.set("mapreduce.app-submission.cross-platform", "true");
或:
conf.setBoolean("mapreduce.app-submission.cross-platform", true);
3、正常情况下,在Win7下通过Eclipse向集群提交运行程序时,MapReduce的Main方法需要增加以下配置项:
conf.set("fs.default.name", "hdfs://xxx.xxx.xxx.xxx:8020");
conf.set("yarn.resourcemanager.address", "xxx.xxx.xxx.xxx:8032");
conf.set("yarn.resourcemanager.scheduler.address", "xxx.xxx.xxx.xxx:8030");
conf.set("mapreduce.jobhistory.address", "xxx.xxx.xxx.xxx:10020");
conf.set("mapreduce.framework.name", "yarn");
conf.setBoolean("mapreduce.app-submission.cross-platform", true);
conf.setBoolean("mapreduce.job.ubertask.enable", true);