想学习运行一个mapreduce程序试试，记录如下
本地运行word count
新建maven项目，添加hadoop-client，版本比如3.1.2
官方的wordcount直接拿来用就可以
需要把winutils.exe和hadoop.dll放到环境变量HADOOP_HOME中，这两个文件夹需要在bin子文件夹中，下载链接
添加两个运行参数，一个输入文件名，一个输出文件名，直接就可以运行了
单服务器配置（伪分布式
服务器使用centos安装，主机名c1，静态ip地址
使用rpm安装了oracle的jdk8，解压hadoop-3.1.2到/hadoop/文件夹下
ssh-keygen
ssh-copy-id localhost
core-site.xml如下
fs.default.name
hdfs://c1:8020
hdfs-site.xml
dfs.replication
1
hadoop.tmp.dir
/hadoop/tmp
yarn-site.xml
yarn.resourcemanager.hostname
c1
yarn.nodemanager.aux-services
mapreduce_shuffle
这时候运行还是会出错，找不到MRAppMaster类，类似如下：
Container exited with a non-zero exit code 1. Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster
Please check whether your etc/hadoop/mapred-site.xml contains the below configuration:
yarn.app.mapreduce.am.env
HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}
mapreduce.map.env
HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}
mapreduce.reduce.e nv
HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}
修改如下
mapred-site.xml
yarn.app.mapreduce.am.env
HADOOP_MAPRED_HOME=/hadoop/hadoop-3.1.2
mapreduce.map.env
HADOOP_MAPRED_HOME=/hadoop/hadoop-3.1.2
mapreduce.reduce.env
HADOOP_MAPRED_HOME=/hadoop/hadoop-3.1.2
修改workers或slaves文件，不同版本的hadoop文件名不一样
把localhost改成主机名，比如c1
要在/etc/hosts加入c1的静态地址，否则，就可能连不上nodemanager，比如
192.168.1.111 c1
启动dfs/yarn服务
格式化只需要第一次运行一下。
bin/hadoop namenode -format
bin/start-dfs.sh
bin/start-yarn.sh
stop之后想再start可能需要等待文件列表同步，正常要等30秒，可以在网页startup信息看。
开发机可以看到c1的50070（hadoop2）或9870（hadoop3）端口就说明hdfs起来了
可以看到8088端口说明yarn起来了
修改程序，加入YARN配置
wordcount程序需要服务端的xml配置，可以放到conf文件夹，把这个文件夹在IDE里点右键设置为Resources，不然程序写相对路径也访问不到
修改wordcount程序，加入xml配置
Configuration conf = new Configuration();
conf.addResource("core-site.xml");
conf.addResource("yarn-site.xml");
conf.addResource("hdfs-site.xml");
conf.addResource("mapred-site.xml");
conf.set("mapreduce.app-submission.cross-platform","true");
conf.set("mapreduce.framework.name","yarn");
conf.set("mapreduce.job.jar","target\\mr1-1.0-SNAPSHOT.jar");
在maven工具中调用package，生成的jar文件名和路径，写到上面程序最后一行
在hdfs中放一个input.txt文件作为输入，如果有权限问题可以参考命令
hdfs dfs -chown user:group input.txt
hdfs dfs -chmod 777 input.txt
提交到yarn运行程序
结果如下
2019-03-26 16:49:33,942 INFO [main] client.RMProxy (RMProxy.java:newProxyInstance(133)) - Connecting to ResourceManager at c1/192.168.1.111:8032
2019-03-26 16:49:34,363 WARN [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadResourcesInternal(147)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2019-03-26 16:49:34,372 INFO [main] mapreduce.JobResourceUploader (JobResourceUploader.java:disableErasureCodingForPath(883)) - Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/cdarling/.staging/job_1553590163847_0001
2019-03-26 16:49:34,500 INFO [main] input.FileInputFormat (FileInputFormat.java:listStatus(292)) - Total input files to process : 1
2019-03-26 16:49:35,349 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(202)) - number of splits:1
2019-03-26 16:49:35,824 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(298)) - Submitting tokens for job: job_1553590163847_0001
2019-03-26 16:49:35,826 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(299)) - Executing with tokens: []
2019-03-26 16:49:35,932 INFO [main] conf.Configuration (Configuration.java:getConfResourceAsInputStream(2752)) - resource-types.xml not found
2019-03-26 16:49:35,932 INFO [main] resource.ResourceUtils (ResourceUtils.java:addResourcesFileToConf(418)) - Unable to find 'resource-types.xml'.
2019-03-26 16:49:36,278 INFO [main] impl.YarnClientImpl (YarnClientImpl.java:submitApplication(324)) - Submitted application application_1553590163847_0001
2019-03-26 16:49:36,302 INFO [main] mapreduce.Job (Job.java:submit(1574)) - The url to track the job: http://c1:8088/proxy/application_1553590163847_0001/
2019-03-26 16:49:36,302 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1619)) - Running job: job_1553590163847_0001
2019-03-26 16:49:41,367 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1640)) - Job job_1553590163847_0001 running in uber mode : false
2019-03-26 16:49:41,367 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1647)) - map 0% reduce 0%
2019-03-26 16:49:45,410 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1647)) - map 100% reduce 0%
2019-03-26 16:49:49,437 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1647)) - map 100% reduce 100%
2019-03-26 16:49:49,447 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1658)) - Job job_1553590163847_0001 completed successfully
2019-03-26 16:49:49,510 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1665)) - Counters: 53
File System Counters
FILE: Number of bytes read=99
FILE: Number of bytes written=433951
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=165
HDFS: Number of bytes written=61
HDFS: Number of read operations=8
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=1608
Total time spent by all reduces in occupied slots (ms)=1685
Total time spent by all map tasks (ms)=1608
Total time spent by all reduce tasks (ms)=1685
Total vcore-milliseconds taken by all map tasks=1608
Total vcore-milliseconds taken by all reduce tasks=1685
Total megabyte-milliseconds taken by all map tasks=1646592
Total megabyte-milliseconds taken by all reduce tasks=1725440
Map-Reduce Framework
Map input records=3
Map output records=13
Map output bytes=123
Map output materialized bytes=99
Input split bytes=94
Combine input records=13
Combine output records=8
Reduce input groups=8
Reduce shuffle bytes=99
Reduce input records=8
Reduce output records=8
Spilled Records=16
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=75
CPU time spent (ms)=930
Physical memory (bytes) snapshot=509763584
Virtual memory (bytes) snapshot=5578809344
Total committed heap usage (bytes)=410517504
Peak Map Physical memory (bytes)=294195200
Peak Map Virtual memory (bytes)=2786607104
Peak Reduce Physical memory (bytes)=215568384
Peak Reduce Virtual memory (bytes)=2792202240
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=71
File Output Format Counters
Bytes Written=61
Process finished with exit code 0