1、 数据清洗：按照进行数据清洗，并将清洗后的数据导入hive数据库中。 两阶段数据清洗：
（1）第一阶段：把需要的信息从原始日志中提取出来
ip: 199.30.25.88
time: 10/Nov/2016:00:01:03 +0800
traffic: 62
文章： article/11325
视频： video/3235
（2）第二阶段：根据提取出来的信息做精细化操作
ip--->城市 city（IP）
date--> time:2016-11-10 00:01:03
day: 10
traffic:62
type:article/video
id:11325
（3）hive数据库表结构:
create table data( ip string, time string , day string, traffic bigint,
type string, id string )
今天着重进行第一阶段的文件数据处理
package com.test.dao;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
public class test1{
public static List ips=new ArrayList();
public static List times=new ArrayList();
public static List traffic=new ArrayList();
public static List wen=new ArrayList();
public static List shi=new ArrayList();
public static class Map extends Mapper{
private static Text Name =new Text();
private static Text num=new Text();
public void map(Object key,Text value,Context context) throws IOException, InterruptedException{
String line =value.toString();
String arr[] =line.split(" , " );
Name. set (arr[0 ]);
num. set (arr[0 ]);
context.write(Name,num);
}
}
public static class Reduce extends Reducer< Text, Text,Text, Text>{
private static Text result= new Text();
int i=0 ;
public void reduce(Text key,Iterable values,Context context) throws IOException, InterruptedException{
for (Text val:values){
context.write(key, val);
ips.add(val.toString());
}
}
}
public static int run()throws IOException, ClassNotFoundException, InterruptedException
{
Configuration conf =new Configuration();
conf. set (" fs.defaultFS " , " hdfs://localhost:9000 " );
FileSystem fs =FileSystem.get (conf);
Job job =new Job(conf," OneSort " );
job.setJarByClass(test1. class );
job.setMapperClass(Map. class );
job.setReducerClass(Reduce. class );
job.setOutputKeyClass(Text. class );
job.setOutputValueClass(Text. class );
job.setInputFormatClass(TextInputFormat. class );
job.setOutputFormatClass(TextOutputFormat. class );
Path in =new Path(" hdfs://localhost:9000/test2/in/result.txt " );
Path out =new Path(" hdfs://localhost:9000/test2/out/ip/1 " );
FileInputFormat.addInputPath(job, in );
fs.delete( out ,true );
FileOutputFormat.setOutputPath(job, out );
return (job.waitForCompletion(true ) ? 0 : 1 );
}
public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException{
run();
}
}
}
2、数据处理： ·统计最受欢迎的视频/文章的Top10访问次数 （video/article）
·按照地市统计最受欢迎的Top10课程 （ip）
·按照流量统计最受欢迎的Top10课程 （traffic）
3、数据可视化：将统计结果倒入MySql数据库中，通过图形化展示的方式展现出来。
