hadoop (四)开发环境及Word Count

hadoop (四)开发环境及Word Count

开发环境搭建

开发工具:IDEA
构件工具:MAVEN

引入依赖:

    
      org.apache.hadoop
      hadoop-core
      0.20.2
  



  
      commons-logging
      commons-logging
      1.0.3
  

  
      junit
      junit
      4.11
      test
  
  
      org.hamcrest
      hamcrest-library
      1.3
      test
  

核心文件引入【与服务器端核心文件一致】:

image

实例代码:

package org.cnliu.myhadoop.ex;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.*;

import java.io.IOException;
import java.util.Iterator;
import java.util.StringTokenizer;

public class WordCount {
    public static class WordCountMapper extends MapReduceBase implements Mapper{

        private final static IntWritable one = new IntWritable(1);
        private Text word =new Text();

        public void map(Object o, Text text, OutputCollector outputCollector, Reporter reporter) throws IOException {

            StringTokenizer itr = new StringTokenizer(text.toString());
            while (itr.hasMoreTokens()){
                word.set(itr.nextToken());
                outputCollector.collect(word,one);
            }
        }
    }

    public static class WordCountReducer extends MapReduceBase implements Reducer{

        private IntWritable result = new IntWritable();

        public void reduce(Text text, Iterator iterator, OutputCollector outputCollector, Reporter reporter) throws IOException {
            int sum=0;
            while (iterator.hasNext()){
                sum+=iterator.next().get();
            }

            result.set(sum);
            outputCollector.collect(text,result);
        }
    }

    public static void main(String[] args) throws Exception{

        //配置job在fs中的输入和输出目录----备注:目录不符合则不会有执行结果
        String input = "/user/liuzd/in";
        String output = "/user/liuzd/o_t_account/result";

        JobConf conf=new JobConf(WordCount.class);

        conf.setJobName("WordCount");
        conf.addResource("classpath:/core-site.xml");
        conf.addResource("classpath:/hdfs-site.xml");
        conf.addResource("classpath:/mapred-site.xml");

        conf.setOutputKeyClass(Text.class);
        conf.setOutputValueClass(IntWritable.class);


        //设置自定义map过程的类 及 输出的key 和 value 的数据类型
        conf.setMapperClass(WordCountMapper.class);
        conf.setCombinerClass(WordCountReducer.class);
        conf.setReducerClass(WordCountReducer.class);

        conf.setInputFormat(TextInputFormat.class);
        conf.setOutputFormat(TextOutputFormat.class);

        FileInputFormat.setInputPaths(conf,new Path(input));
        FileOutputFormat.setOutputPath(conf,new Path(output));

        JobClient.runJob(conf);
        System.exit(0);
    }
}
 
 

执行结果:

image

原始数据

image

数据计算查询结果:

image

附录(遇到的问题及解决办法)

hadoop集群启动成功但live node为0

我的问题是网络问题:
hosts 配置了 
127.0.0.1 master 
192.168.56.101 master

导致hadoop解析别名master时ip为127.0.0.1 因是分布式集群部署,故主节点在其他服务器主机上,删除此项配置即可。  

启动Hadoop时,DataNode启动后一会儿自动消失的解决方法

原因:在第一次格式化dfs后,启动并使用了hadoop,后来又重新执行了格式化命令(hdfs namenode -format),这时namenode的clusterID会重新生成,而datanode的clusterID 保持不变。
解决方案:重新拷贝集群hadoop环境
>https://www.cnblogs.com/sasan/p/5740367.html

IDE远程提交mapreduce任务至linux,遇到ClassNotFoundException: Mapper

尝试

(1)手动打包成jar,上传到linux(运行成功)

(2)手动打包成jar,Windows下调用“java -jar ..”(运行成功)

(3)IDEA提交任务

解决方案

core-site.xml中配置路径:



    mapred.jar

    E:\java\myHadoop\out\artifacts\myHadoop_jar\myHadoop.jar


https://blog.csdn.net/qq_19648191/article/details/56684268

INFO mapred.JobClient: Task Id : attempt_201210161256_0009_r_000000_0, Status : FAILEDShuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out. WARN mapred.JobClient: Error reading task outputNo route to host

解决方案:主机hostname需要与hosts文件中的别名一致。

你可能感兴趣的:(hadoop (四)开发环境及Word Count)