【wordcount】mapreduce.Job: Running job: job_1553100392548_0001

hadoop配置

查看yarn-site.xml配置

文件位置：/usr/local/hadoop/etc/hadoop/yarn-site.xml

#我的hadoop安装目录：/usr/local/hadoop/

sudo gedit /usr/local/hadoop/etc/hadoop/yarn-site.xml





   yarn.nodemanager.aux-services
   mapreduce_shuffle



   yarn.resourcemanager.resource-tracker.address
   master:8025


   yarn.resourcemanager.scheduler.address
   master:8030


   yarn.resourcemanager.address
   master:8050

查看mapred-site.xml文件

文件位置：/usr/local/hadoop/etc/hadoop/mapred-site.xml

sudo gedit /usr/local/hadoop/etc/hadoop/mapred-site.xml



    mapreduce.framework.name
    yarn

测试：HDFS上测试 hadoop

建立input文件夹

hadoop  fs -mkdir -p /user/python/wordcount/input

hadoop jar  +（hadoop自带测试文件） wordcount  +hdfs配置的input(输入文件夹)+输出文件夹（注意不需要建立output）文件夹

hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.7.jar  wordcount /user/python/wordcount/input/me.txt /user/python/wordcount/output

附上我使用wordcount的命令行（建立文件需要修改权限：例如1031行）

 1028  start-all.sh
 1029  sudo gedit me.txt
 1030  ll
 1031  chown -R python:python me.txt

 1033  sudo chown -R python:python me.txt
 1034  ll
 1035  hdfs fs -copyFromLocal me.txt /user/python/wordcount/input
 1036  hadoop fs -copyFromLocal me.txt /user/python/wordcount/input

 1040  hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.7.jar  wordcount /user/python/wordcount/input/me.txt /user/python/wordcount/output
 1041  hadoop fs -ls /user/python/wordcount/output
 1042  hadoop fs -cat /user/python/wordcount/output/part-r-00000 | more