Python运行MapReducer程序时所遇异常

landen@Master:~/UntarFile/hadoop-1.0.4$ bin/hadoop jar contrib/streaming/hadoop-streaming-1.0.4.jar -mapper home/landen/UntarFile/hadoop-1.0.4/PythonMR/wordMapper.py -reducer /home/landen/UntarFile/hadoop-1.0.4/PythonMR/wordReducer.py -input /input/* -output wordCountOutput
Warning: $HADOOP_HOME is deprecated.

packageJobJar: [/home/landen/UntarFile/hadoop-1.0.4/datas/tmp/hadoop-unjar2023262079914179173/] [] /tmp/streamjob1615815049526219730.jar tmpDir=null
14/03/19 11:22:49 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/03/19 11:22:49 WARN snappy.LoadSnappy: Snappy native library not loaded
14/03/19 11:22:49 INFO mapred.FileInputFormat: Total input paths to process : 1
14/03/19 11:22:50 INFO streaming.StreamJob: getLocalDirs(): [/home/landen/UntarFile/hadoop-1.0.4/datas/tmp/mapred/local]
14/03/19 11:22:50 INFO streaming.StreamJob: Running job: job_201403182127_0006
14/03/19 11:22:50 INFO streaming.StreamJob: To kill this job, run:
14/03/19 11:22:50 INFO streaming.StreamJob: /home/landen/UntarFile/hadoop-1.0.4/libexec/../bin/hadoop job  -Dmapred.job.tracker=Master:9001 -kill job_201403182127_0006
14/03/19 11:22:50 INFO streaming.StreamJob: Tracking URL: http://Master:50030/jobdetails.jsp?jobid=job_201403182127_0006
14/03/19 11:22:51 INFO streaming.StreamJob:  map 0%  reduce 0%
14/03/19 11:23:27 INFO streaming.StreamJob:  map 100%  reduce 100%
14/03/19 11:23:27 INFO streaming.StreamJob: To kill this job, run:
14/03/19 11:23:27 INFO streaming.StreamJob: /home/landen/UntarFile/hadoop-1.0.4/libexec/../bin/hadoop job  -Dmapred.job.tracker=Master:9001 -kill job_201403182127_0006
14/03/19 11:23:27 INFO streaming.StreamJob: Tracking URL: http://Master:50030/jobdetails.jsp?jobid=job_201403182127_0006
bug出现:
14/03/19 11:23:27 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201403182127_0006_m_000000
14/03/19 11:23:27 INFO streaming.StreamJob: killJob...
Streaming Command Failed!

查看hadoop logs文件发现:
Caused by: java.io.IOException: Cannot run program "./PythonMR/wordMapper.py": error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214)
    ... 23 more

Reason 1:执行py文件时开头没指定相关的python执行文件地址路径和编码
#!/usr/bin/python
# coding=utf-8
Reason 2:未给python文件相关执行权限
chmod a+x *.py
Reason 3:在提交作业时,采用-file选项指定这些文件, 比如上面例子中,可以使用“-file Mapper -file Reducer” 或者 “-file Mapper.py -file Reducer.py”, 这样,Hadoop会将这两个文件自动分发到各个节点上(Distributed Cache).

landen@Master:~/UntarFile/hadoop-1.0.4$ bin/hadoop jar contrib/streaming/hadoop-streaming-1.0.4.jar -mapper ./PythonMR/wordMapper.py -reducer ./PythonMR/wordReducer.py -input /input/* -output wordCountOutput -file ./PythonMR/wordMapper.py -file ./PythonMR/wordReducer.py
Warning: $HADOOP_HOME is deprecated.

packageJobJar: [./PythonMR/wordMapper.py, ./PythonMR/wordReducer.py, /home/landen/UntarFile/hadoop-1.0.4/datas/tmp/hadoop-unjar3733581910057274756/] [] /tmp/streamjob8413860595071502704.jar tmpDir=null
14/03/19 11:33:51 INFO util.NativeCodeLoader: Loaded the native-hadoop library
14/03/19 11:33:51 WARN snappy.LoadSnappy: Snappy native library not loaded
14/03/19 11:33:51 INFO mapred.FileInputFormat: Total input paths to process : 1
14/03/19 11:33:51 INFO streaming.StreamJob: getLocalDirs(): [/home/landen/UntarFile/hadoop-1.0.4/datas/tmp/mapred/local]
14/03/19 11:33:51 INFO streaming.StreamJob: Running job: job_201403182127_0007
14/03/19 11:33:51 INFO streaming.StreamJob: To kill this job, run:
14/03/19 11:33:51 INFO streaming.StreamJob: /home/landen/UntarFile/hadoop-1.0.4/libexec/../bin/hadoop job  -Dmapred.job.tracker=Master:9001 -kill job_201403182127_0007
14/03/19 11:33:51 INFO streaming.StreamJob: Tracking URL: http://Master:50030/jobdetails.jsp?jobid=job_201403182127_0007
14/03/19 11:33:52 INFO streaming.StreamJob:  map 0%  reduce 0%
14/03/19 11:34:06 INFO streaming.StreamJob:  map 50%  reduce 0%
14/03/19 11:34:07 INFO streaming.StreamJob:  map 100%  reduce 0%
14/03/19 11:34:18 INFO streaming.StreamJob:  map 100%  reduce 100%
14/03/19 11:34:24 INFO streaming.StreamJob: Job complete: job_201403182127_0007
14/03/19 11:34:24 INFO streaming.StreamJob: Output: wordCountOutput

你可能感兴趣的:(mapreduce)