大数据Mapreduce统计

大数据Mapreduce统计

一、准备工作

1.创建hadoop用户
代码如下

$ sudo useradd -m hadoop -s /bin/bash

2.设置密码，添加管理员权限

$ sudo passwd hadoop
$ sudo adduser hadoop sudo

3.更新apt

$ sudo apt-get update

4.安装配置文件VIM

$ sudo apt-get install vim

5.安装SSH、配置SSH无密码登陆

$ sudo apt-get install openssh-server

6.安装JAVA环境
下载文件jdk-8u162-linux-x64.tar.gz 使用Filezilla传输到“/home/linziyu/Downloads/”目录下

$ cd /usr/lib
$ sudo mkdir jvm 
$ cd ~ #进入hadoop用户的主目录
$ cd Downloads 
$ sudo tar -zxvf ./jdk-8u162-linux-x64.tar.gz -C /usr/lib/jvm 

7.设置环境变量

$ cd ~
$ vim ~/.bashrc

8.继续执行如下命令让.bashrc文件的配置立即生效

$ source ~/.bashrc

二、安装eclipse

在ubuntu软件中心中下载安装

在 Eclipse 中创建 MapReduce 项目
1.在Eclipse中创建项目


在弹出来的 General 选项面板中，General 的设置要与 Hadoop 的配置一致

在 Eclipse 中操作 HDFS 中的文件
配置好后，点击左侧 Project Explorer 中的 MapReduce Location 就能直接查看 HDFS 中的文件列表

源代码

public static void main(String[] args) throws Exception {
     
    Configuration conf = new Configuration();
    //*String[] otherArgs = (new GenericOptionsParser(conf, args)).getRemainingArgs();
    String[] otherArgs=new String[]{
     "input","outpu2"};
    if(otherArgs.length < 2) {
     
        System.err.println("Usage: wordcount  [...] ");
        System.exit(2);
    }

    Job job = Job.getInstance(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(WordCount.TokenizerMapper.class);
    job.setCombinerClass(WordCount.IntSumReducer.class);
    job.setReducerClass(WordCount.IntSumReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);

    for(int i = 0; i < otherArgs.length - 1; ++i) {
     
        FileInputFormat.addInputPath(job, new Path(otherArgs[i]));
    }

    FileOutputFormat.setOutputPath(job, new Path(otherArgs[otherArgs.length - 1]));
    System.exit(job.waitForCompletion(true)?0:1);
}

public static class IntSumReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
     
    private IntWritable result = new IntWritable();

    public IntSumReducer() {
     
    }

    public void reduce(Text key, Iterable<IntWritable> values, Reducer<Text, IntWritable, Text, IntWritable>.Context context) throws IOException, InterruptedException {
     
        int sum = 0;

        IntWritable val;
        for(Iterator i$ = values.iterator(); i$.hasNext(); sum += val.get()) {
     
            val = (IntWritable)i$.next();
        }

        this.result.set(sum);
        context.write(key, this.result);
    }
}

public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> {
     
    private static final IntWritable one = new IntWritable(1);
    private Text word = new Text();

    public TokenizerMapper() {
     
    }

    public void map(Object key, Text value, Mapper<Object, Text, Text, IntWritable>.Context context) throws IOException, InterruptedException {
     
        StringTokenizer itr = new StringTokenizer(value.toString());

        while(itr.hasMoreTokens()) {
     
            this.word.set(itr.nextToken());
            context.write(this.word, one);
        }

    }
}

**

部分统计文本

**

(For children and primary school pupils) (一) Developing oral communication skills Oral communication skills here refer to listening and speaking skills. They are very important skills for beginners. Children learning their native language begin by understanding through listening. They comprehend the language long before they can speak. With primary school pupils, it is better that we let them listen and understand the language first. Let them speak gradually when they feel ready and comfortable with it. In fact, listening and speaking are always used inseparably in real life situations. However, listening is an important basis for speaking. The following are some listening and speaking activities you can use in the classroom. Read them and discuss which ones are adaptable in your teaching and which ones are not practical for your situations and explain why. Listening Activities Listen and act/follow instructions This type of activity is also known as TPR (Total Physical Response). TPR is an extremely useful and adaptable teaching technique in the primary classroom for language learning. With TPR, children listen to their teacher telling them what to do and then do it. This is also the way how children acquired their first language. They are able to do things according to what they hear long before they are able to speak the language. By doing TPR, you are also giving chil- dren a sense of security. It does not matter if they cannot pronounce the word or say the sentence. They can simply listen and watch others to understand the meaning. For TPR activities, the easiest is to start with classroom commands, ‘touch’ activities and verbs in action, etc. （二）Speaking activities Speaking activities do not need to be always reading aloud, reciting dialogues or repeating what the teacher said. When we speak, we speak with a purpose and we speak with interest as well. Therefore, we need to create interesting topics and genuine purposes for children to speak the language. Of course some imitations and repetitions are necessary to prepare children to speak, but even with imi- tations and repetitions, we can make them more interesting and meaningful. One important point to note is that speaking only develops gradually with plenty of practice and with a lot of encouragement. We should have realistic expectations with children beginning with English. The following are some speaking activities. Some of them are basic speaking activities and some of them are more advanced speaking activities. They can only be applied with different levels of learners and in different contexts.

在终端中输入下列命令可查看结果