黑猴子的家：Hadoop 日志聚集

Job运行完成以后，将日志信息上传到HDFS系统上

1、配置yarn-site.xml

    yarn.log-aggregation-enable
    true




    yarn.log-aggregation.retain-seconds
    604800




    yarn.log-aggregation.retain-check-interval-seconds         
    86400




    yarn.nodemanager.remote-app-log-dir         
    /bigdata/logs/hadoop/Yarnlogs



    yarn.log.server.url         
    http://hadoop102:19888/jobhistory/logs 



    mapred.child.java.opts         
    -Xmx1024m

2、关闭nodemanager 、resourcemanager和historyserver

[victor@node1 hadoop-2.8.2]$ sbin/mr-jobhistory-daemon.sh stop historyserver
[victor@node1 hadoop-2.8.2]$ sbin/yarn-daemon.sh stop nodemanager
[victor@node1 hadoop-2.8.2]$ sbin/yarn-daemon.sh stop resourcemanager

3、、启动nodemanager 、resourcemanager和historyserver

[victor@node1 hadoop-2.8.2]$ sbin/yarn-daemon.sh start resourcemanager
[victor@node1 hadoop-2.8.2]$ sbin/yarn-daemon.sh start nodemanager
[victor@node1 hadoop-2.8.2]$ sbin/mr-jobhistory-daemon.sh start historyserver

4、删除hdfs上已经存在的hdfs文件

[victor@node1 hadoop-2.8.2]$ bin/hdfs dfs -rm -R /user/victor/wordcount/output

5、执行wordcount程序

[victor@node1 hadoop-2.8.2]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.2.jar wordcount /user/victor/wordcount/input /user/victor/wordcount/output

6、查看日志

$ http://node1:19888/jobhistory