hadoop项目使用

该引用适合hadoop使用1.2.1，提供参考

 

maven安装好之后即可使用

 

 D:\workspace\java>mvn archetype:generate -DarchetypeGroupId=org.apache.maven.archetypes -DgroupId=org.conan.myhadoop.mr
-DartifactId=myHadoop -DpackageName=org.conan.myhadoop.mr -Dversion=1.0-SNAPSHOT -DinteractiveMode=false
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] >>> maven-archetype-plugin:2.2:generate (default-cli) @ standalone-pom >>>
[INFO]
[INFO] <<< maven-archetype-plugin:2.2:generate (default-cli) @ standalone-pom <<<
[INFO]

.......

期间会出现需要填写的groupId，即是你的项目名称，其他输入直接回车即可。最后点Y

 

成功之后：

D:\workspace\java>cd myHadoop
~ D:\workspace\java\myHadoop>mvn clean install
此时在你的路径下就用你敲的groupId命名的项目，导入到eclipse中即可

 

最后编辑pom.xml  追加

<dependency>
   <groupId>org.apache.hadoop</groupId>
   <artifactId>hadoop-core</artifactId>
   <version>1.0.3</version>
  </dependency>

最后执行命令下载依赖mvn clean install  

 

此时你可以直接运行main方法了， 针对这个注意期间出现的问题

严重: PriviledgedActionException as:Administrator cause:java.io.IOException: Failed to set permissions of path: \tmp\hadoop-Administrator\mapred\staging\Administrator1702422322\.staging to 0700
Exception in thread "main" java.io.IOException: Failed to set permissions of path: \tmp

修改下源码文件FileUtil.java重新编译即可

 

运行时可能还会出现

严重: PriviledgedActionException as:xuanxy cause:org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
Exception in thread "main" org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client version 4
 at org.apache.hadoop.ipc.Client.call(Client.java:1070)
 at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
 at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
 at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
 at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)

 

注意hadoop的版本