Windows 10 平台使用 Eclipse 运行官方 WordCount 示例

1. Hadoop 下载并配置环境变量

HADOOP_HOME=D:\ProgramFiles\hadoop-3.2.0
PATH 中追加：%HADOOP_HOME%\bin

2. winutils 下载

地址：https://github.com/steveloughran/winutils

2.1 解压下载的文件，拷贝到$HADOOP_HOME\bin 下

2.2 拷贝 hadoop.dll 到C:\Windows\System32 下

3. 以管理员身份启动 Eclipse，构建 Maven 项目

pom.xml

    org.apache.hadoop
    hadoop-common
    3.2.0




    org.apache.hadoop
    hadoop-hdfs
    3.2.0




    org.apache.hadoop
    hadoop-client
    3.2.0

WordCount.java

package hadoopDemo;

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

    public static class TokenizerMapper extends Mapper {

        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();

        public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
            StringTokenizer itr = new StringTokenizer(value.toString());
            while (itr.hasMoreTokens()) {
                word.set(itr.nextToken());
                context.write(word, one);
            }
        }
    }

    public static class IntSumReducer extends Reducer {
        private IntWritable result = new IntWritable();

        public void reduce(Text key, Iterable values, Context context)
                throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable val : values) {
                sum += val.get();
            }
            result.set(sum);
            context.write(key, result);
        }
    }

    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf, "word count");
        job.setJarByClass(WordCount.class);
        job.setMapperClass(TokenizerMapper.class);
        job.setCombinerClass(IntSumReducer.class);
        job.setReducerClass(IntSumReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }

}

4. 在 hdfs 上创建目录及统计的文本文件

1. 在 hdfs 上创建 input 文件夹

hdfs dfs -mkdir –p  /input

2. 把要统计的文本上传到 hdfs 的 /input 目录

hdfs dfs -put 要上传的文件路径 /input

或

# 通过搭建的HDFS网址，Browse the file system 上传
http://192.168.52.129:9870

3. 查看

hdfs dfs -cat /input/words.txt

5. 配置 Java Application Arguments 并运行

hdfs://192.168.52.129:9000/input/words.txt
hdfs://192.168.52.129:9000/output

注：

1. 如果不指定 words.txt，则为 input/ 下所有文件；
2. output 文件夹不能存在，程序执行后会自动生成。

6. 查看运行结果

hdfs dfs -cat /output/part-r-00000

或

# 通过以下地址下载生成的输出文件
hdfs://192.168.52.129:9000/output

参考链接：

官方示例程序：
https://hadoop.apache.org/docs/r3.2.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html

其它：
https://wiki.apache.org/hadoop/WindowsProblems
https://blog.csdn.net/congcong68/article/details/42043093
https://blog.csdn.net/congcong68/article/details/42098391
https://blog.csdn.net/weixin_38763887/article/details/79157652
https://www.jianshu.com/p/7dae895bea4d