大数据基础（九）Maven构建Hadoop日志清洗项目（一）
Maven Hadoop日志清洗项目（一）
hadoop 2.7.2
参考：
Maven Hadoop：
http://www.cnblogs.com/Leo_wl/p/4862820.html
http://blog.csdn.net/kongxx/article/details/42339581
日志清洗：
http://www.cnblogs.com/edisonchou/p/4458219.html
1、新建Maven工程
Eclipse-》新建Maven工程
http://mvnrepository.com/search?q=hadoop-mapreduce-client
groupid:com
artifactid:first
依赖包
hadoop-common
hadoop-hdfs
hadoop-mapreduce-client-core
hadoop-mapreduce-client-jobclient
hadoop-mapreduce-client-common
我加了个hadoop-yarn-common，这个可以不要
pom.xml【注意：版本改成你自己的】
点击保存，开始构建。
构建完成后可以在Maven Dependencies下看到依赖包。
2、新建LogCleanJob类
代码见附录【详细代码解释参考原文http://www.cnblogs.com/edisonchou/p/4458219.html】
注意：pom.xml要添加assembly插件，直接用jar导出一直报错，没找到原因。
还有原文的@Override在run方法编译不通过，注释掉了。
E:\fm-workspace\workspace_2\first>mvn assembly:assembly
cd first\target下
first-0.0.1-SNAPSHOT-jar-with-dependencies.jar
E:\fm-workspace\workspace_2\first\target>dir
2016/08/13 18:21
2016/08/13 18:21
2016/08/13 18:19
2016/08/13 17:34
2016/08/13 18:21 42,996,951 first-0.0.1-SNAPSHOT-jar-with-dependencies.jar
2016/08/13 18:21 9,266 first-0.0.1-SNAPSHOT.jar
2016/08/13 18:19
2016/08/13 17:31
2016/08/13 18:19
2016/08/13 17:34
2 个文件 43,006,217 字节
8 个目录 113,821,888,512 可用字节
重命名first-0.0.1-SNAPSHOT-jar-with-dependencies.jar 为first.jar并拷贝到linux下
root@py-server:/projects/data# ll
总用量 42008
drwxr-xr-x 4 root root 4096 8月 13 18:52 ./
drwxr-xr-x 7 root root 4096 8月 11 16:29 ../
-rw-r--r-- 1 root root 42996951 8月 13 18:21 first.jar
drwxr-xr-x 2 root root 4096 8月 13 15:36 hadoop-logs/
drwxr-xr-x 2 root root 4096 8月 3 21:04 test/
5、上传数据到HDFS
数据文件在原文找吧：http://www.cnblogs.com/edisonchou/p/4458219.html，大概200MB左右。
也可以用你自己的日志文件，不过格式要一致。
root@py-server:/projects/data/hadoop-logs# ll
总用量 213056
drwxr-xr-x 2 root root 4096 8月 13 15:36 ./
drwxr-xr-x 4 root root 4096 8月 13 18:25 ../
-rw-r--r-- 1 root root 61084192 4月 26 2015 access_2013_05_30.log
-rw-r--r-- 1 root root 157069653 4月 26 2015 access_2013_05_31.log
HDFS默认路径是 /user/root/
root@py-server:/projects/data# hadoop fs -put hadoop-logs/ .
root@py-server:/projects/data# hadoop fs -ls
Found 14 items
drwxr-xr-x - root supergroup 0 2016-08-09 23:59 .sparkStaging
drwxr-xr-x - root supergroup 0 2016-08-13 15:38 hadoop-logs
-rw-r--r-- 2 root supergroup 85285 2016-08-06 07:59 imdb_labelled.txt
-rw-r--r-- 2 root supergroup 72 2016-08-04 09:29 kmeans_data.txt
drwxr-xr-x - root supergroup 0 2016-08-09 23:59 kmeans_result
drwxr-xr-x - root supergroup 0 2016-08-05 16:16 kmeans_result.txt
-rw-r--r-- 2 root supergroup 43914 2016-08-04 12:33 ks_aio.py
drwxr-xr-x - root supergroup 0 2016-08-09 10:51 mymlresult
drwxr-xr-x - root supergroup 0 2016-08-09 10:28 naive_bayes_result
-rw-r--r-- 2 root supergroup 66288 2016-08-09 23:57 price_data.txt
-rw-r--r-- 2 root supergroup 1619 2016-08-08 17:54 price_data2.txt
-rw-r--r-- 2 root supergroup 1619 2016-08-09 09:13 price_train_data.txt
-rw-r--r-- 2 root supergroup 120 2016-08-04 09:24 sample_kmeans_data.txt
-rw-r--r-- 2 root supergroup 104736 2016-08-08 17:14 sample_libsvm_data.txt
6、Hadoop测试
root@py-server:/projects/data# hadoop jar first.jar /user/root/hadoop-logs/ /user/root/logcleanjob_output
结果：【速度超快，不到瞬间啊！36s】
在hadoop UI看（本人的是：py-server:8088）下看：
| User: | root |
|---|---|
| Name: | LogCleanJob |
| Application Type: | MAPREDUCE |
| Application Tags: | |
| YarnApplicationState: | FINISHED |
| FinalStatus Reported by AM: | SUCCEEDED |
| Started: | 星期六 八月 13 18:46:18 +0800 2016 |
| Elapsed: | 36sec |
| Tracking URL: | History |
| Diagnostics: |
Clean process success!
root@py-server:/projects/data# hadoop fs -ls /user/root/
Found 15 items
drwxr-xr-x - root supergroup 0 2016-08-09 23:59 /user/root/.sparkStaging
drwxr-xr-x - root supergroup 0 2016-08-13 18:45 /user/root/hadoop-logs
-rw-r--r-- 2 root supergroup 85285 2016-08-06 07:59 /user/root/imdb_labelled.txt
-rw-r--r-- 2 root supergroup 72 2016-08-04 09:29 /user/root/kmeans_data.txt
drwxr-xr-x - root supergroup 0 2016-08-09 23:59 /user/root/kmeans_result
drwxr-xr-x - root supergroup 0 2016-08-05 16:16 /user/root/kmeans_result.txt
-rw-r--r-- 2 root supergroup 43914 2016-08-04 12:33 /user/root/ks_aio.py
drwxr-xr-x - root supergroup 0 2016-08-13 18:46 /user/root/logcleanjob_output
drwxr-xr-x - root supergroup 0 2016-08-09 10:51 /user/root/mymlresult
drwxr-xr-x - root supergroup 0 2016-08-09 10:28 /user/root/naive_bayes_result
-rw-r--r-- 2 root supergroup 66288 2016-08-09 23:57 /user/root/price_data.txt
-rw-r--r-- 2 root supergroup 1619 2016-08-08 17:54 /user/root/price_data2.txt
-rw-r--r-- 2 root supergroup 1619 2016-08-09 09:13 /user/root/price_train_data.txt
-rw-r--r-- 2 root supergroup 120 2016-08-04 09:24 /user/root/sample_kmeans_data.txt
-rw-r--r-- 2 root supergroup 104736 2016-08-08 17:14 /user/root/sample_libsvm_data.txt
root@py-server:/projects/data# hadoop fs -ls /user/root/logcleanjob_output
Found 2 items
-rw-r--r-- 2 root supergroup 0 2016-08-13 18:46 /user/root/logcleanjob_output/_SUCCESS
-rw-r--r-- 2 root supergroup 50810594 2016-08-13 18:46 /user/root/logcleanjob_output/part-r-00000
root@py-server:/projects/data# hadoop fs -cat /user/root/logcleanjob_output/part-r-00000
118.112.191.88 20130530204006 source/plugin/wsh_wx/img/wsh_zk.css
113.107.237.31 20130530204005 thread-10500-1-1.html
110.251.129.203 20130531081904 forum.php?mod=ajax&action=forumchecknew&fid=111&time=1369959258&inajax=yes
118.112.191.88 20130530204006 data/cache/style_1_common.css?y7a
220.231.55.69 20130530204005 home.php?mod=spacecp&ac=pm&op=checknewpm&rand=1369917603
110.75.174.58 20130531081903 thread-21066-1-1.html
118.112.191.88 20130530204006 data/cache/style_1_forum_viewthread.css?y7a
110.75.174.55 20130531081904 home.php?do=thread&from=space&mod=space&uid=71469&view=me
14.17.29.89 20130530204006 home.php?mod=misc&ac=sendmail&rand=1369917604
121.25.131.148 20130531081906 data/attachment/common/c2/common_12_usergroup_icon.jpg
59.174.191.135 20130530204003 forum.php?mod=forumdisplay&fid=111&page=1&filter=author&orderby=dateline
118.112.191.88 20130530204007 data/attachment/common/65/common_11_usergroup_icon.jpg
121.25.131.148 20130531081905 home.php?mod=misc&ac=sendmail&rand=1369959541
101.229.199.98 20130530204007 data/cache/style_1_widthauto.css?y7a
59.174.191.135 20130530204005 home.php?mod=space&uid=71081&do=profile&from=space
#######################################
问题解决：
1. mave 中断怎么办
http://www.cnblogs.com/tangyanbo/p/4329303.html
右键项目：maven->update project并勾选force选项，如果勾选force，那么不用删除未下载成功的残余文件，在大量jar包未下载成功的时候可以选择勾选force
重新build一下。
2. hadoop jar 没有指定主类名，直接将主类名放在first.jar后会提示找不到input那个文件夹
hadoop jar first.jar /user/root/hadoop-logs/ /user/root/logcleanjob_output
#######################################
附录：LogCleanJob.java
package com.first;
//package techbbs;
import java.net.URI;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Locale;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
public class LogCleanJob extends Configured implements Tool {
public static void main(String[] args) {
Configuration conf = new Configuration();
try {
int res = ToolRunner.run(conf, new LogCleanJob(), args);
System.exit(res);
} catch (Exception e) {
e.printStackTrace();
}
}
//@Override
public int run(String[] args) throws Exception {
final Job job = new Job(new Configuration(),
LogCleanJob.class.getSimpleName());
// 设置为可以打包运行
job.setJarByClass(LogCleanJob.class);
FileInputFormat.setInputPaths(job, args[0]);
job.setMapperClass(MyMapper.class);
job.setMapOutputKeyClass(LongWritable.class);
job.setMapOutputValueClass(Text.class);
job.setReducerClass(MyReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(NullWritable.class);
FileOutputFormat.setOutputPath(job, new Path(args[1]));
// 清理已存在的输出文件
FileSystem fs = FileSystem.get(new URI(args[0]), getConf());
Path outPath = new Path(args[1]);
if (fs.exists(outPath)) {
fs.delete(outPath, true);
}
boolean success = job.waitForCompletion(true);
if(success){
System.out.println("Clean process success!");
}
else{
System.out.println("Clean process failed!");
}
return 0;
}
static class MyMapper extends
Mapper
LogParser logParser = new LogParser();
Text outputValue = new Text();
protected void map(
LongWritable key,
Text value,
org.apache.hadoop.mapreduce.Mapper
throws java.io.IOException, InterruptedException {
final String[] parsed = logParser.parse(value.toString());
// step1.过滤掉静态资源访问请求
if (parsed[2].startsWith("GET /static/")
|| parsed[2].startsWith("GET /uc_server")) {
return;
}
// step2.过滤掉开头的指定字符串
if (parsed[2].startsWith("GET /")) {
parsed[2] = parsed[2].substring("GET /".length());
} else if (parsed[2].startsWith("POST /")) {
parsed[2] = parsed[2].substring("POST /".length());
}
// step3.过滤掉结尾的特定字符串
if (parsed[2].endsWith(" HTTP/1.1")) {
parsed[2] = parsed[2].substring(0, parsed[2].length()
- " HTTP/1.1".length());
}
// step4.只写入前三个记录类型项
outputValue.set(parsed[0] + "\t" + parsed[1] + "\t" + parsed[2]);
context.write(key, outputValue);
}
}
static class MyReducer extends
Reducer
protected void reduce(
LongWritable k2,
java.lang.Iterable
org.apache.hadoop.mapreduce.Reducer
throws java.io.IOException, InterruptedException {
for (Text v2 : v2s) {
context.write(v2, NullWritable.get());
}
};
}
/*
* 日志解析类
*/
static class LogParser {
public static final SimpleDateFormat FORMAT = new SimpleDateFormat(
"d/MMM/yyyy:HH:mm:ss", Locale.ENGLISH);
public static final SimpleDateFormat dateformat1 = new SimpleDateFormat(
"yyyyMMddHHmmss");
public static void main(String[] args) throws ParseException {
final String S1 = "27.19.74.143 - - [30/May/2013:17:38:20 +0800] \"GET /static/image/common/faq.gif HTTP/1.1\" 200 1127";
LogParser parser = new LogParser();
final String[] array = parser.parse(S1);
System.out.println("样例数据： " + S1);
System.out.format(
"解析结果： ip=%s, time=%s, url=%s, status=%s, traffic=%s",
array[0], array[1], array[2], array[3], array[4]);
}
/**
* 解析英文时间字符串
*
* @param string
* @return
* @throws ParseException
*/
private Date parseDateFormat(String string) {
Date parse = null;
try {
parse = FORMAT.parse(string);
} catch (ParseException e) {
e.printStackTrace();
}
return parse;
}
/**
* 解析日志的行记录
*
* @param line
* @return 数组含有5个元素，分别是ip、时间、url、状态、流量
*/
public String[] parse(String line) {
String ip = parseIP(line);
String time = parseTime(line);
String url = parseURL(line);
String status = parseStatus(line);
String traffic = parseTraffic(line);
return new String[] { ip, time, url, status, traffic };
}
private String parseTraffic(String line) {
final String trim = line.substring(line.lastIndexOf("\"") + 1)
.trim();
String traffic = trim.split(" ")[1];
return traffic;
}
private String parseStatus(String line) {
final String trim = line.substring(line.lastIndexOf("\"") + 1)
.trim();
String status = trim.split(" ")[0];
return status;
}
private String parseURL(String line) {
final int first = line.indexOf("\"");
final int last = line.lastIndexOf("\"");
String url = line.substring(first + 1, last);
return url;
}
private String parseTime(String line) {
final int first = line.indexOf("[");
final int last = line.indexOf("+0800]");
String time = line.substring(first + 1, last).trim();
Date date = parseDateFormat(time);
return dateformat1.format(date);
}
private String parseIP(String line) {
String ip = line.split("- -")[0].trim();
return ip;
}
}
}
######################################################
完整的pom.xml