学习篇-Hadoop-MapReduce-流量统计

文章目录

          • 一、Hadoop-MapReduce-流量统计-需求分析
          • 二、Hadoop-MapReduce-流量统计-代码实现
          • 三、Hadoop-MapReduce-流量统计-Partitioner

一、Hadoop-MapReduce-流量统计-需求分析

现有一份access.log日志文件

1363157985066 	13726230503	00-FD-07-A4-72-B8:CMCC	120.196.100.82	i02.c.aliimg.com		24	27	2481	24681	200
1363157995052 	13826544101	5C-0E-8B-C7-F1-E0:CMCC	120.197.40.4			4	0	264	0	200
1363157991076 	13926435656	20-10-7A-28-CC-0A:CMCC	120.196.100.99			2	4	132	1512	200
1363154400022 	13926251106	5C-0E-8B-8B-B1-50:CMCC	120.197.40.4			4	0	240	0	200
1363157993044 	18211575961	94-71-AC-CD-E6-18:CMCC-EASY	120.196.100.99	iface.qiyi.com	视频网站	15	12	1527	2106	200
1363157995074 	84138413	5C-0E-8B-8C-E8-20:7DaysInn	120.197.40.4	122.72.52.12		20	16	4116	1432	200
1363157993055 	13560439658	C4-17-FE-BA-DE-D9:CMCC	120.196.100.99			18	15	1116	954	200
1363157995033 	15920133257	5C-0E-8B-C7-BA-20:CMCC	120.197.40.4	sug.so.360.cn	信息安全	20	20	3156	2936	200
1363157983019 	13719199419	68-A1-B7-03-07-B1:CMCC-EASY	120.196.100.82			4	0	240	0	200
1363157984041 	13660577991	5C-0E-8B-92-5C-20:CMCC-EASY	120.197.40.4	s19.cnzz.com	站点统计	24	9	6960	690	200
1363157973098 	15013685858	5C-0E-8B-C7-F7-90:CMCC	120.197.40.4	rank.ie.sogou.com	搜索引擎	28	27	3659	3538	200
1363157986029 	15989002119	E8-99-C4-4E-93-E0:CMCC-EASY	120.196.100.99	www.umeng.com	站点统计	3	3	1938	180	200
1363157992093 	13560439658	C4-17-FE-BA-DE-D9:CMCC	120.196.100.99			15	9	918	4938	200
1363157986041 	13480253104	5C-0E-8B-C7-FC-80:CMCC-EASY	120.197.40.4			3	3	180	180	200
1363157984040 	13602846565	5C-0E-8B-8B-B6-00:CMCC	120.197.40.4	2052.flash2-http.qq.com	综合门户	15	12	1938	2910	200
1363157995093 	13922314466	00-FD-07-A2-EC-BA:CMCC	120.196.100.82	img.qfc.cn		12	12	3008	3720	200
1363157982040 	13502468823	5C-0A-5B-6A-0B-D4:CMCC-EASY	120.196.100.99	y0.ifengimg.com	综合门户	57	102	7335	110349	200
1363157986072 	18320173382	84-25-DB-4F-10-1A:CMCC-EASY	120.196.100.99	input.shouji.sogou.com	搜索引擎	21	18	9531	2412	200
1363157990043 	13925057413	00-1F-64-E1-E6-9A:CMCC	120.196.100.55	t3.baidu.com	搜索引擎	69	63	11058	48243	200
1363157988072 	13760778710	00-FD-07-A4-7B-08:CMCC	120.196.100.82			2	2	120	120	200
1363157985066 	13726238888	00-FD-07-A4-72-B8:CMCC	120.196.100.82	i02.c.aliimg.com		24	27	2481	24681	200
1363157993055 	13560436666	C4-17-FE-BA-DE-D9:CMCC	120.196.100.99			18	15	1116	954	200
1363157985066 	13726238888	00-FD-07-A4-72-B8:CMCC	120.196.100.82	i02.c.aliimg.com	

需求:读取access.log文件

关注:第二个字段:手机号 倒数第三字段:上行流量 倒数第二个字段:下行流量

需求:统计每个手机号的上行流量和、下行流量和、总流量和

分析:

  • 自定义复杂返回类型Access.java

    • 手机号、上行流量、下行流量、总流量
  • 根据需求需要将手机号进行分组,然后将上行流量和下行流量相加

  • Mapper:把手机号、上行流量、下行流量拆开

    • 把手机号作为key,把Access作为value作为输出
  • Reducer:输出到reducer的结果:(13390905421,)

二、Hadoop-MapReduce-流量统计-代码实现
  • 为了简化代码添加lombok

    • pom.xml中的dependencies节点下添加依赖

      
      <dependency>
        <groupId>org.projectlombokgroupId>
        <artifactId>lombokartifactId>
        <version>${lombok.version}version>
      dependency>
      
    • pom.xml中的properties节点下添加版本号

      
      <lombok.version>1.18.12lombok.version>
      
    • pom.xml中的repositories节点下添加仓库

      
      <repository>
        <id>centralid>
        <name>Central Repositoryname>
        <url>https://repo.maven.apache.org/maven2url>
      repository>
      
  • 代码实现

    • 自定义复杂返回类型-Access.java

      /**
       * @ClassName Access
       * @Description 定义输出的复杂类型,需要实现Writable接口
       * @Author eastern
       * @Date 2020/4/30 下午4:36
       * @Version 1.0
       **/
      @Data
      @NoArgsConstructor
      public class Access implements Writable {
      
      	/**
      	 * 手机号码
      	 */
      	private String phone;
      
      	/**
      	 * 上行流量
      	 */
      	private Long up;
      
      	/**
      	 * 下行流量
      	 */
      	private Long down;
      
      	/**
      	 * 总流量
      	 */
      	private Long sum;
      
      	/**
      	 * 自定义构造,方便后续构造Access实例
      	 * @param phone
      	 * @param up
      	 * @param down
      	 */
      	public Access(String phone, Long up, Long down){
      		this.phone = phone;
      		this.up = up;
      		this.down = down;
      		this.sum = up + down;
      	}
      
      	/**
      	 * 重写toString方法,为了输出更加友好
      	 * @return
      	 */
      	@Override
      	public String toString() {
      		return  phone + "," + up + "," + down + "," + sum;
      	}
      
      	/**
      	 * 写入值
      	 * @param out
      	 * @throws IOException
      	 */
      	@Override
      	public void write(DataOutput out) throws IOException {
      		out.writeUTF(phone);
      		out.writeLong(up);
      		out.writeLong(down);
      		out.writeLong(sum);
      	}
      
      	/**
      	 * 读取值,跟写入的顺序需要一一对应
      	 * @param in
      	 * @throws IOException
      	 */
      	@Override
      	public void readFields(DataInput in) throws IOException {
      		this.phone = in.readUTF();
      		this.up = in.readLong();
      		this.down = in.readLong();
      		this.sum = in.readLong();
      	}
      }
      
    • 自定义mapper-AccessMapper.java

      /**
       * @ClassName AccessMapper
       * @Description 自定义Mapper处理类
       * @Author eastern
       * @Date 2020/4/30 下午10:20
       * @Version 1.0
       **/
      public class AccessMapper extends Mapper<LongWritable, Text, Text, Access> {
      
      	@Override
      	protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
      		String[] lines = value.toString().split("\t");
      		// 取出手机号
      		String phone = lines[1];
      		// 取出上行流量
      		long up = Long.parseLong(lines[lines.length-3]);
      		// 取出下行流量
      		long down = Long.parseLong(lines[lines.length-2]);
      
      		context.write(new Text(phone), new Access(phone, up, down));
      	}
      }
      
    • 自定义Reducer-AccessReducer.java

      /**
       * @ClassName AccessReducer
       * @Description 自定义Reducer
       * @Author eastern
       * @Date 2020/4/30 下午10:30
       * @Version 1.0
       **/
      public class AccessReducer extends Reducer<Text, Access, NullWritable, Access> {
      
      	/**
      	 *
      	 * @param key	手机号
      	 * @param values	
      	 * @param context
      	 * @throws IOException
      	 * @throws InterruptedException
      	 */
      	@Override
      	protected void reduce(Text key, Iterable<Access> values, Context context) throws IOException,
      			InterruptedException {
      		long ups = 0;
      		long downs = 0;
      		for (Access access : values) {
      			ups += access.getUp();
      			downs += access.getDown();
      		}
      		context.write(NullWritable.get(), new Access(key.toString(), ups, downs));
      	}
      }
      
    • 自定义driver-AccessLocalApp.java

      /**
       * @ClassName AccessLocalApp
       * @Description Access的Driver
       * @Author eastern
       * @Date 2020/4/30 下午10:36
       * @Version 1.0
       **/
      public class AccessLocalApp {
      	public static void main(String[] args) throws Exception {
      		Configuration configuration = new Configuration();
      		Job job = Job.getInstance(configuration);
      		job.setJarByClass(AccessLocalApp.class);
      
      		job.setMapperClass(AccessMapper.class);
      		job.setReducerClass(AccessReducer.class);
      
      		job.setMapOutputKeyClass(Text.class);
      		job.setOutputValueClass(Access.class);
      
      		job.setOutputKeyClass(NullWritable.class);
      		job.setOutputValueClass(Access.class);
      
      		FileInputFormat.setInputPaths(job, new Path("/Users/xxx/IdeaProjects/bigdata/hadoop-mapreduce/src/main/resources/access.log"));
      		FileOutputFormat.setOutputPath(job, new Path("/Users/xxx/IdeaProjects/bigdata/hadoop-mapreduce/src/main/resources/access"));
      
      		job.waitForCompletion(true);
      	}
      }
      
三、Hadoop-MapReduce-流量统计-Partitioner

学习篇-Hadoop-MapReduce-流量统计_第1张图片

  • 源码分析:

    • 进入Partitioner源码

      @InterfaceAudience.Public
      @InterfaceStability.Stable
      public abstract class Partitioner<KEY, VALUE> {
        
        /** 
         * Get the partition number for a given key (hence record) given the total 
         * number of partitions i.e. number of reduce-tasks for the job.
         *   
         * 

      Typically a hash function on a all or a subset of the key.

      * * @param key the key to be partioned. * @param value the entry value. * @param numPartitions the total number of partitions. * @return the partition number for the key. */
      public abstract int getPartition(KEY key, VALUE value, int numPartitions); }
      • 抽象类
      • Partitioner决定mapTask输出的数据交由哪个reducetask处理
    • 寻找sufflem默认分组的实现。

      • 我们去Job源码中查看搜索partitioner,发现只提供了setPartitionerClass,不是我们要的答案。

      • 在Job类上还集成了JobContextImpl,进入JobContextImpl搜索partitioner,发现有个getPartitionerClass,这边设置的默认实现是HashPartitioner.class。

          public Class<? extends Partitioner<?,?>> getPartitionerClass() 
             throws ClassNotFoundException {
            return (Class<? extends Partitioner<?,?>>) 
              conf.getClass(PARTITIONER_CLASS_ATTR, HashPartitioner.class);
          }
        
      • 在drive程序中打印试试,看看是不是这个HashPartitioner

        System.out.println("PartitionerClass--->" + job.getPartitionerClass().toString());
        
        • 控制台显示:
          学习篇-Hadoop-MapReduce-流量统计_第2张图片
        • 可以看到确实是HashPartitioner
      • 进入HashPartitioner源码

        @InterfaceAudience.Public
        @InterfaceStability.Stable
        public class HashPartitioner<K, V> extends Partitioner<K, V> {
        
          /** Use {@link Object#hashCode()} to partition. */
          public int getPartition(K key, V value,
                                  int numReduceTasks) {
            return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks;
          }
        
        }
        
        • numReduceTasks:你的作业所指定的reducer的个数,决定了reducer作业输出文件的个数。

          • 默认numReduceTasks为1

            • 源码:Job–>JobContextImpl–>JobConf中

              public int getNumReduceTasks() { return getInt(JobContext.NUM_REDUCES, 1); }
              
        • key.hashCode() & Integer.MAX_VALU:保证结果谓非负数

    • 案例需求:

      • 将统计结果按照手机的前缀进行区分,并输出到不同的输出文件中

        13* ==> …

        15* ==> …

        other ==> …

      • 代码实现

        • 自定义Partitioner-AccessPartitioner

          /**
           * @ClassName AccessPartitioner
           * @Description MapReducer自定义分区规则
           * @Author eastern
           * @Date 2020/5/2 上午11:54
           * @Version 1.0
           **/
          public class AccessPartitioner extends Partitioner<Text, Access> {
          
          	/**
          	 * @param phone 手机号
          	 * @param access
          	 * @param numPartitions
          	 * @return
          	 */
          	@Override
          	public int getPartition(Text phone, Access access, int numPartitions) {
          		if (phone.toString().startsWith("13")) {
          			return 0;
          		} else if (phone.toString().startsWith("15")) {
          			return 1;
          		} else {
          			return 2;
          		}
          	}
          }
          
        • Driver设置Partitioner和reduceTasks

          // 设置自定义分区规则
          job.setPartitionerClass(AccessPartitioner.class);
          // 设置分区数 注意:需要与分区规则对应
          job.setNumReduceTasks(3);
          
        • 输出效果
          学习篇-Hadoop-MapReduce-流量统计_第3张图片

你可能感兴趣的:(hadoop,hadoop,mapreduce)