Win7下用Eclipse向Hadoop2.5.2集群提交MapReduce程序的注意事项

 

主要描述下,在Win7环境下,通过eclipse往集群提交MapReduce程序的过程。

一、环境说明:

开发环境:WIN7

Eclipse版本:eclipse-jee-indigo

Hadoop版本:Hadoop2.5.2

MR运行模式:Yarn

 

二、使用MapReduce的Eclipse插件:

插件名称:hadoop-eclipse-plugin-2.5.2.jar

可以网上下载,本文档附件中也有上传。

将插件复制到${ECLIPSE_HOME}/plugins/目录下。

 

三、运行简单的MR(如:Wordcount)程序,碰到下面问题:

1、运行测试程序时,会出现以下报错:

ExitCodeException exitCode=1: 

at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)

at org.apache.hadoop.util.Shell.run(Shell.java:455)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)

at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Container exited with a non-zero exit code 1

解决方法:

需要在yarn-site.xml上,配置以下内容:

  <property>

        <name>mapreduce.job.ubertask.enable</name>

        <value>true</value>

  </property>

当在WIN7下用Eclipse提交MR程序时,如果还报这样的错误,需要在客户端代码增加如下配置:

conf.set("mapreduce.job.ubertask.enable","true");

 

2、运行测试程序时,会出现以下报错:

ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control

 

at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)

at org.apache.hadoop.util.Shell.run(Shell.java:455)

at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)

at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

 

 

Container exited with a non-zero exit code 1

.Failing this attempt.. Failing the application.

解决方法:

conf.set("mapreduce.app-submission.cross-platform", "true");

或:

conf.setBoolean("mapreduce.app-submission.cross-platform", true);

              

3、正常情况下,在Win7下通过Eclipse向集群提交运行程序时,MapReduce的Main方法需要增加以下配置项:

conf.set("fs.default.name", "hdfs://xxx.xxx.xxx.xxx:8020");

conf.set("yarn.resourcemanager.address", "xxx.xxx.xxx.xxx:8032");

conf.set("yarn.resourcemanager.scheduler.address", "xxx.xxx.xxx.xxx:8030");

conf.set("mapreduce.jobhistory.address", "xxx.xxx.xxx.xxx:10020");

conf.set("mapreduce.framework.name", "yarn");

        conf.setBoolean("mapreduce.app-submission.cross-platform", true);

        conf.setBoolean("mapreduce.job.ubertask.enable", true);

你可能感兴趣的:(mapreduce,hadoop,yarn)