环境准备（06）YARN环境搭建 & 提交作业到YARN

1. 配置文件目录

/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop

2. mapred-site.xml

[hadoop@hadoop001 hadoop]$ cp mapred-site.xml.template mapred-site.xml


    
        mapreduce.framework.name
        yarn
    

3. yarn-site.xml

[hadoop@hadoop001 hadoop]$ vi yarn-site.xml

    
        yarn.nodemanager.aux-services
        mapreduce_shuffle
    

4. 启动YARN

• sbin/start-yarn.sh

[hadoop@hadoop001 sbin]$ ./start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-resourcemanager-hadoop001.out
localhost: starting nodemanager, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/yarn-hadoop-nodemanager-hadoop001.out

5. 检查YARN启动结果

[hadoop@hadoop001 sbin]$ jps
154 NameNode
2732 NodeManager
272 DataNode
429 SecondaryNameNode
3047 Jps
2639 ResourceManager

注：

• 把当前的容器commit成一个新的镜像yarn，跑yarn的时候把端口8088映射出去，就可以从宿主机访问yarn的web控制台了；
• localhost:8088

docker commit hadoop001 yarn

docker stop hadoop001

docker rm hadoop001

docker run -itd --name hadoop001 -h hadoop001 -p 9010:22 -p 50070:50070 -p 8088:8088 yarn /usr/sbin/sshd -D

6. 提交MR作业到YARN上

• 创建输入文件的目录

hadoop fs -mkdir -p /input/wc/

• 拷贝文件到HDFS输入目录里

hadoop fs -put hello.txt /input/wc/

• 查看拷贝结果

[hadoop@hadoop001 data]$ hadoop fs -cat /input/wc/hello.txt
18/07/10 13:18:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hello   world   welcome
hello   welcome

• 提交作业到YARN
  • jar包位置：/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce2/

[hadoop@hadoop001 mapreduce2]$ hadoop jar hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar wordcount /input/wc/hello.txt /output/wc/
18/07/10 13:05:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/07/10 13:05:45 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
18/07/10 13:05:46 INFO input.FileInputFormat: Total input paths to process : 1
18/07/10 13:05:46 INFO mapreduce.JobSubmitter: number of splits:1
18/07/10 13:05:46 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1531222938551_0001
18/07/10 13:05:47 INFO impl.YarnClientImpl: Submitted application application_1531222938551_0001
18/07/10 13:05:47 INFO mapreduce.Job: The url to track the job: http://hadoop001:8088/proxy/application_1531222938551_0001/
18/07/10 13:05:47 INFO mapreduce.Job: Running job: job_1531222938551_0001
18/07/10 13:05:56 INFO mapreduce.Job: Job job_1531222938551_0001 running in uber mode : false
18/07/10 13:05:56 INFO mapreduce.Job:  map 0% reduce 0%
18/07/10 13:06:03 INFO mapreduce.Job:  map 100% reduce 0%
18/07/10 13:06:11 INFO mapreduce.Job:  map 100% reduce 100%
18/07/10 13:06:11 INFO mapreduce.Job: Job job_1531222938551_0001 completed successfully
18/07/10 13:06:11 INFO mapreduce.Job: Counters: 49
    File System Counters
        FILE: Number of bytes read=44
        FILE: Number of bytes written=222959
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=139
        HDFS: Number of bytes written=26
        HDFS: Number of read operations=6
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=2
    Job Counters 
        Launched map tasks=1
        Launched reduce tasks=1
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=5194
        Total time spent by all reduces in occupied slots (ms)=4864
        Total time spent by all map tasks (ms)=5194
        Total time spent by all reduce tasks (ms)=4864
        Total vcore-seconds taken by all map tasks=5194
        Total vcore-seconds taken by all reduce tasks=4864
        Total megabyte-seconds taken by all map tasks=5318656
        Total megabyte-seconds taken by all reduce tasks=4980736
    Map-Reduce Framework
        Map input records=2
        Map output records=5
        Map output bytes=54
        Map output materialized bytes=44
        Input split bytes=105
        Combine input records=5
        Combine output records=3
        Reduce input groups=3
        Reduce shuffle bytes=44
        Reduce input records=3
        Reduce output records=3
        Spilled Records=6
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=29
        CPU time spent (ms)=1460
        Physical memory (bytes) snapshot=419115008
        Virtual memory (bytes) snapshot=2821779456
        Total committed heap usage (bytes)=295698432
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters 
        Bytes Read=34
    File Output Format Counters 
        Bytes Written=26

• 查看作业运行结果

[hadoop@hadoop001 mapreduce2]$ hadoop fs -cat /output/wc/part-r-00000
18/07/10 13:07:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hello   2
welcome 2
world   1