hadoop3 提交任务出错（找不到类）
hadoop3 提交任务出错（找不到类）
hadoop3的配置和hadoop2的配置基本相同，但还是有一定的差距
hadoop3的配置注意事项
1.首先要一堆环境变量（HADOOP_PREFIX不要配）
export JAVA_HOEM=/usr/java/default
export HADOOP_HOME=/opt/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_CONF_DIR=$HADOOP_HOME
export HADOOP_LIBEXEC_DIR=$HADOOP_HOME/libexec
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=.:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH2.所有的主机要写在workers文件中
3.mapred-site.xml要在hadoop2的基础上添加
yarn.app.mapreduce.am.env
HADOOP_MAPRED_HOME=${HADOOP_HOME}
mapreduce.map.env
HADOOP_MAPRED_HOME=${HADOP_HOME}
mapreduce.reduce.env
HADOOP_MAPRED_HOME=${HADOOP_HOME}
mapreduce.application.classpath
${HADOOP_HOME}/etc/hadoop,
${HADOOP_HOME}/share/hadoop/common/*,
${HADOOP_HOME}/share/hadoop/common/lib/*,
${HADOOP_HOME}/share/hadoop/hdfs/*,
${HADOOP_HOME}/share/hadoop/hdfs/lib/*,
${HADOOP_HOME}/share/hadoop/mapreduce/*,
${HADOOP_HOME}/share/hadoop/mapreduce/lib/*,
${HADOOP_HOME}/share/hadoop/yarn/*,
${HADOOP_HOME}/share/hadoop/yarn/lib/*
4.在没有添加以上内容前的报错，找不到各种各样的类
hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.1.jar pi 1 1
WARNING: HADOOP_PREFIX has been replaced by HADOOP_HOME. Using value of HADOOP_PREFIX.
Number of Maps = 1
Samples per Map = 1
2018-09-28 14:19:01,238 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Wrote input for Map #0
Starting Job
2018-09-28 14:19:02,297 INFO client.RMProxy: Connecting to ResourceManager at standlone/192.168.114.135:8032
2018-09-28 14:19:02,779 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/hdfs/.staging/job_1538169417840_0003
2018-09-28 14:19:02,874 INFO input.FileInputFormat: Total input files to process : 1
2018-09-28 14:19:02,918 INFO mapreduce.JobSubmitter: number of splits:1
2018-09-28 14:19:02,953 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2018-09-28 14:19:03,447 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1538169417840_0003
2018-09-28 14:19:03,452 INFO mapreduce.JobSubmitter: Executing with tokens: []
2018-09-28 14:19:03,642 INFO conf.Configuration: resource-types.xml not found
2018-09-28 14:19:03,642 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2018-09-28 14:19:03,729 INFO impl.YarnClientImpl: Submitted application application_1538169417840_0003
2018-09-28 14:19:03,770 INFO mapreduce.Job: The url to track the job: http://standlone:8088/proxy/application_1538169417840_0003/
2018-09-28 14:19:03,771 INFO mapreduce.Job: Running job: job_1538169417840_0003
2018-09-28 14:19:09,887 INFO mapreduce.Job: Job job_1538169417840_0003 running in uber mode : false
2018-09-28 14:19:09,888 INFO mapreduce.Job: map 0% reduce 0%
2018-09-28 14:19:11,943 INFO mapreduce.Job: Task Id : attempt_1538169417840_0003_m_000000_0, Status : FAILED
[2018-09-28 14:19:11.432]Exception from container-launch.
Container id: container_1538169417840_0003_01_000002
Exit code: 1
[2018-09-28 14:19:11.434]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild
[2018-09-28 14:19:11.435]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild
2018-09-28 14:19:15,005 INFO mapreduce.Job: Task Id : attempt_1538169417840_0003_m_000000_1, Status : FAILED
[2018-09-28 14:19:14.181]Exception from container-launch.
Container id: container_1538169417840_0003_01_000003
Exit code: 1
[2018-09-28 14:19:14.182]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild
[2018-09-28 14:19:14.183]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild
2018-09-28 14:19:18,045 INFO mapreduce.Job: Task Id : attempt_1538169417840_0003_m_000000_2, Status : FAILED
[2018-09-28 14:19:17.200]Exception from container-launch.
Container id: container_1538169417840_0003_01_000004
Exit code: 1
[2018-09-28 14:19:17.202]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild
[2018-09-28 14:19:17.203]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapred.YarnChild
2018-09-28 14:19:23,081 INFO mapreduce.Job: map 100% reduce 100%
2018-09-28 14:19:24,118 INFO mapreduce.Job: Job job_1538169417840_0003 failed with state FAILED due to: Task failed task_1538169417840_0003_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0
2018-09-28 14:19:24,207 INFO mapreduce.Job: Counters: 13
Job Counters
Failed map tasks=4
Killed reduce tasks=1
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3549
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=3549
Total vcore-milliseconds taken by all map tasks=3549
Total megabyte-milliseconds taken by all map tasks=3634176
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Job job_1538169417840_0003 failed!