在Win7下配置好hadoop集成eclipse开发环境后,运行WordCount程序的时候,可能会出现的问题:
java.io.IOException: Cannot run program "chmod": CreateProcess error=2, ????????? at java.lang.ProcessBuilder.start(Unknown Source) at org.apache.hadoop.util.Shell.runCommand(Shell.java:149) at org.apache.hadoop.util.Shell.run(Shell.java:134) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:286) at org.apache.hadoop.util.Shell.execCommand(Shell.java:354) at org.apache.hadoop.util.Shell.execCommand(Shell.java:337) at org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:481) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:473) at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:280) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:372) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:484) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:465) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:372) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:208) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:142)解决方式: 在windows7的 环境变量 path中配置增加;C:\hadoop4win\bin;C:\hadoop4win\usr\sbin
注意path中的前后值要用分号(;)隔开。
java.lang.OutOfMemoryError: Java heap space at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:781) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:350) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:177) 11/08/16 10:31:39 INFO mapred.JobClient: map 0% reduce 0% 11/08/16 10:31:39 INFO mapred.JobClient: Job complete: job_local_0001 11/08/16 10:31:39 INFO mapred.JobClient: Counters: 0 Exception in thread "main" java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252) at mapred.WordCount.main(WordCount.java:110)
原因是客户端程序运行时java虚拟机内存分配不够,修改run configuration(run as >run configurations),选择Arguments选项卡,在VM arguments栏中加入-Xmx1024m。如图:
或者选择中项目右键-->Properties-->在左边点击java Build Path-->选择 Libararies 选项卡-->找到JRE,点击Edit-->Installed JRE-->选择对应的JDK,Edit,如图设置
修改上面1,2 任意一项都要重启eclipse,否则看不到效果的!
Task Id : attempt_201003251936_0001_r_000000_0, Status : FAILED
NullPointerException
ConcurrentHashMap.get(ConcurrentHashMap.java:768)
10/03/25 19:40:21 INFO mapred.JobClient: Task Id : attempt_201003251936_0001_r_000000_0, Status : FAILED Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605) 10/03/25 19:40:21 WARN mapred.JobClient: Error reading task outputhttp://localhost:50060/tasklog?plaintext=true&taskid=attempt_201003251936_0001_r_000000_0&filter=stdout 10/03/25 19:40:21 WARN mapred.JobClient: Error reading task outputhttp://localhost:50060/tasklog?plaintext=true&taskid=attempt_201003251936_0001_r_000000_0&filter=stderr 10/03/25 19:40:27 INFO mapred.JobClient: Task Id : attempt_201003251936_0001_r_000000_1, Status : FAILED Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:768) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2683) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2605) 10/03/25 19:40:27 WARN mapred.JobClient: Error reading task outputnode2.1036dhcp
原因是 windows 的计算机名称是不能包含下划线 不能以数字开头,否则会报上面的错误。
如我的计算机名称是Think_L440 ,含有下划线,故出现上面的错误。
修改计算机名称为Think440,并对应修改C:\hadoop4win\etc\networks、hosts等相关的配置文件。问题解决了!!!
修改后一定要重启电脑哦!!!