Hive之——RegexSerDe来处理标准格式Apache Web日志

转载请注明出处:https://blog.csdn.net/l1028386804/article/details/88617592

这里,我使用的Hive版本是2.3.4。

SerDe是序列化/反序列化的简写。在内部Hive引擎使用定义的InputFormat来读取一行数据记录。这行记录之后会被传递给SerDe.deserialize()方法进行处理。
下面这个例子使用一个RegexSerde来处理标准格式的ApacheWeb日志。这个RegexSerDe作为Hive分支的标准功能使用:

首先我们在Hive中创建表serde_regex

CREATE TABLE serde_regex( 
host STRING,    
identity STRING ,
username STRING,
time STRING,
request STRING,
status STRING,
size STRING,
referer STRING,
agent STRING )
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' 
WITH SERDEPROPERTIES (
"input.regex" = "([\\d|.]+)\\s+([^ ]+)\\s+([^ ]+)\\s+\\[(.+)\\]\\s+\"([^ ]+)\\s(.+)\\s([^ ]+)\"\\s+([^ ]+)\\s+([^ ]+)\\s+\"(.+)\"\\s+\"(.+)\"?",
"output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s"
)
STORED AS TEXTFILE ;

然后载入数据查询:

hive> load data local inpath '/usr/local/src/apache_log.txt' into table serde_regex;
hive> add jar /usr/local/hive-2.3.4/lib/hive-contrib-2.3.4.jar;
hive> select * from serde_regex order by time limit 10;
Query ID = root_20190317125919_86f6f8d9-ddb5-44e2-95fb-595efc165a68
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=
In order to set a constant number of reducers:
  set mapreduce.job.reduces=
Starting Job = job_1552794249837_0003, Tracking URL = http://liuyazhuang11:8088/proxy/application_1552794249837_0003/
Kill Command = /usr/local/hadoop-2.9.2/bin/hadoop job  -kill job_1552794249837_0003
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2019-03-17 12:59:28,518 Stage-1 map = 0%,  reduce = 0%
2019-03-17 12:59:36,797 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 5.71 sec
2019-03-17 12:59:42,971 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 7.6 sec
MapReduce Total cumulative CPU time: 7 seconds 600 msec
Ended Job = job_1552794249837_0003
MapReduce Jobs Launched: 
Stage-Stage-1: Map: 1  Reduce: 1   Cumulative CPU: 7.6 sec   HDFS Read: 15991118 HDFS Write: 1281 SUCCESS
Total MapReduce CPU Time Spent: 7 seconds 600 msec
OK
61.160.224.138  -       -       11/Jul/2014:01:01:13 +0800      GET     /home.php?mod=space&do=notice&view=system       HTTP/1.0        200     7519
61.160.224.138  -       -       11/Jul/2014:01:01:15 +0800      GET     /home.php?mod=spacecp&ac=follow&op=checkfeed&rand=1405011673    HTTP/1.0        200     815
61.160.224.138  -       -       11/Jul/2014:01:01:15 +0800      GET     /home.php?mod=spacecp&ac=pm&op=checknewpm&rand=1405011673       HTTP/1.0        200     709
113.17.174.44   -       -       11/Jul/2014:01:01:17 +0800      POST    /api/manyou/my.php      HTTP/1.0        200     653
61.160.224.145  -       -       11/Jul/2014:01:01:18 +0800      GET     /static/image/feed/favorite_b.png       HTTP/1.0        200     2581
61.160.224.143  -       -       11/Jul/2014:01:01:18 +0800      GET     /static/image/feed/thread_b.png HTTP/1.0        200     2947
61.160.224.144  -       -       11/Jul/2014:01:01:18 +0800      GET     /static/image/feed/friend_b.png HTTP/1.0        200     2887
61.160.224.138  -       -       11/Jul/2014:01:01:18 +0800      GET     /forum.php?mod=ajax&action=forumjump&jfid=0&inajax=1&ajaxtarget=fjump_menu      HTTP/1.0        200     1670
61.160.224.145  -       -       11/Jul/2014:01:01:21 +0800      GET     /static/image/feed/magic_b.png  HTTP/1.0        200     2909
61.160.224.138  -       -       11/Jul/2014:01:01:22 +0800      GET     /forum.php      HTTP/1.0        200     16169
Time taken: 24.908 seconds, Fetched: 10 row(s)

附加:

Apache标准日志格式如下:

61.160.224.145 - - [11/Jul/2014:15:41:50 +0800] "GET /data/attachment/forum/201405/08/204852bmroqgmzigwqkaqm.png HTTP/1.0" 200 25136 "http://www.aboutyun.com/thread-7648-1-1.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 SE 2.X MetaSr 1.0"
61.160.224.144 - - [11/Jul/2014:15:41:50 +0800] "GET /data/attachment/forum/201405/08/204710dfaifzt954lk4atu.png HTTP/1.0" 200 26939 "http://www.aboutyun.com/thread-7648-1-1.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 SE 2.X MetaSr 1.0"
61.160.224.143 - - [11/Jul/2014:15:41:50 +0800] "GET /data/attachment/forum/201405/08/204901ry3om753yt330sq7.png HTTP/1.0" 200 16326 "http://www.aboutyun.com/thread-7648-1-1.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 SE 2.X MetaSr 1.0"
61.160.224.145 - - [11/Jul/2014:15:41:50 +0800] "GET /data/attachment/forum/201405/08/204851xfgog3tbiofypoog.png HTTP/1.0" 200 34306 "http://www.aboutyun.com/thread-7648-1-1.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 SE 2.X MetaSr 1.0"
61.160.224.144 - - [11/Jul/2014:15:41:50 +0800] "GET /data/attachment/forum/201405/08/205513bf020h08bo033jb4.png HTTP/1.0" 200 64326 "http://www.aboutyun.com/thread-7648-1-1.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 SE 2.X MetaSr 1.0"
61.160.224.145 - - [11/Jul/2014:15:41:50 +0800] "GET /data/attachment/forum/201405/08/205810ma0aa7cfbwlbeb0b.png HTTP/1.0" 200 63042 "http://www.aboutyun.com/thread-7648-1-1.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 SE 2.X MetaSr 1.0"
61.160.224.137 - - [11/Jul/2014:15:41:50 +0800] "GET /uc_server/avatar.php?uid=4579&size=middle HTTP/1.0" 301 426 "http://www.aboutyun.com/thread-7648-1-1.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 SE 2.X MetaSr 1.0"
61.160.224.143 - - [11/Jul/2014:15:41:50 +0800] "GET /data/attachment/forum/201405/08/204859ubv2zsz25lz0elrf.png HTTP/1.0" 200 42997 "http://www.aboutyun.com/thread-7648-1-1.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 SE 2.X MetaSr 1.0"
61.160.224.145 - - [11/Jul/2014:15:41:50 +0800] "GET /data/attachment/forum/201405/08/204110xrvv9s74r9gr4427.png HTTP/1.0" 200 136137 "http://www.aboutyun.com/thread-7648-1-1.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 SE 2.X MetaSr 1.0"
61.160.224.137 - - [11/Jul/2014:15:41:50 +0800] "GET /uc_server/avatar.php?uid=5216&size=middle HTTP/1.0" 301 426 "http://www.aboutyun.com/thread-7648-1-1.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 SE 2.X MetaSr 1.0"
61.160.224.137 - - [11/Jul/2014:15:41:50 +0800] "GET /uc_server/avatar.php?uid=6165&size=middle HTTP/1.0" 301 426 "http://www.aboutyun.com/thread-7648-1-1.html" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 SE 2.X MetaSr 1.0"

温馨提示:

大家可以到链接https://download.csdn.net/download/l1028386804/11029052 下载Apache日志文件,进行测试

 

你可能感兴趣的:(Hadoop,Hive,Hadoop生态)