利用 Python 分析网站访问日志文件

场景介绍

有一个前台的 Web 应用，框架会记录访问日志，并定期归档，存储在特定的目录，目录格式如下：
/onlinelogs/应用名/环境名/年/月/日/小时/，例如 /onlinelogs//Prod/2018/08/07/01。
在该目录下：

• 访问日志文件可能有多个，文件名以 access-log 开头
• 已压缩为 .gz 文件，并且只读，例如：
  
  

  访问日志文件
• 访问日志文件中，有部分行是记录 HTTP 请求的，格式如下所示：
  • 从中可以看出，请求的目标资源，响应码，客户端信息

222.67.225.134 - - [04/Aug/2018:01:16:44 +0000] "GET /?ref=as_cn_ags_resource_tb&ck-tparam-anchor=123067 HTTP/1.1" 200 7798 "https://gs.amazon.cn/resources.html/ref=as_cn_ags_hnav1_re_class" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.2 Safari/605.1.15"

222.67.225.134 - - [04/Aug/2018:01:16:54 +0000] "GET /tndetails?tnid=3be3f34dee8a4bf08baa072a478fc882 HTTP/1.1" 200 9152 "https://gs.amazon.cn/sba/?ref=as_cn_ags_resource_tb&ck-tparam-anchor=123067&tnm=Offline" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.2 Safari/605.1.15"

222.67.225.134 - - [04/Aug/2018:01:17:37 +0000] "GET /paymentinfo?oid=10763561&rtxref=164f9f8e2b5e4d7790d02d1220eae435 HTTP/1.1" 200 7138 "https://gs.amazon.cn/sba/paymentinfo?oid=10763561&rtxref=164f9f8e2b5e4d7790d02d1220eae435" "Mozilla/5.0 (Linux; Android 8.1.0; DE106 Build/OPM1.171019.026; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/62.0.3202.84 Mobile Safari/537.36 AliApp(DingTalk/4.5.3) com.alibaba.android.rimet/0 Channel/10006872 language/zh-CN"

222.67.225.134 - - [04/Aug/2018:01:17:39 +0000] "GET /paymentinfo?oid=10763561&rtxref=164f9f8e2b5e4d7790d02d1220eae435 HTTP/1.1" 200 7138 "https://gs.amazon.cn/sba/paymentinfo?oid=10763561&rtxref=164f9f8e2b5e4d7790d02d1220eae435" "Mozilla/5.0 (Linux; Android 8.1.0; DE106 Build/OPM1.171019.026; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/62.0.3202.84 Mobile Safari/537.36 AliApp(DingTalk/4.5.3) com.alibaba.android.rimet/0 Channel/10006872 language/zh-CN"

222.67.225.134 - - [04/Aug/2018:01:17:40 +0000] "GET /paymentinfo?oid=10763561&rtxref=164f9f8e2b5e4d7790d02d1220eae435 HTTP/1.1" 200 7138 "https://gs.amazon.cn/sba/paymentinfo?oid=10763561&rtxref=164f9f8e2b5e4d7790d02d1220eae435" "Mozilla/5.0 (Linux; Android 8.1.0; DE106 Build/OPM1.171019.026; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/62.0.3202.84 Mobile Safari/537.36 AliApp(DingTalk/4.5.3) com.alibaba.android.rimet/0 Channel/10006872 language/zh-CN"

140.243.121.197 - - [04/Aug/2018:01:17:41 +0000] "GET /?ref=as_cn_ags_resource_tb HTTP/1.1" 302 - "https://gs.amazon.cn/resources.html/ref=as_cn_ags_hnav1_re_class" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"

分析目标

• 记录不同时间段的访问量
• 统计 PC 和 Mobile 端访问量
• 统计不同页面的访问量

基本思想：

• 由于原日志文件已压缩，并且只读，所以需要创建一个临时目录 /tmp/logs/unziped_logs 来解压缩日志文件
• 利用正则表达式 ^access\-log.*gz+$ 过滤日志文件
• 利用正则表达式 ^.*GET (.*) HTTP.*$ 过滤日志文件中的 HTTP 请求
• 通过日志行是否有 Mobile 来判断客户端

代码如下：（部分内容屏蔽）

#!/usr/bin/python3

import os
import os.path
import re
import shutil
import gzip
from collections import defaultdict

# define the ziped and unziped log file directorys
source_logs_dir = '/onlinelogs//Prod'
unziped_logs_dir = '/tmp/logs/unziped_logs'

# clear the unziped log file directory if exists
if os.path.exists(unziped_logs_dir):
    shutil.rmtree(unziped_logs_dir)

# create the unziped log file directory
os.mkdir(unziped_logs_dir)

# regex used to match target log file name
log_file_name_regex = re.compile(r'^access\-log.*gz+$')

# regex used to match HTTP request
http_request_regex = re.compile(r'^.*GET.*gs.amazon.cn')

# regex used to match request page
request_page_regex = re.compile(r'^.*GET (.*) HTTP.*$')
# request_page_regex = re.compile(r'^.*GET (.*)\?.*$')

# a dictionary to store the HTTP request count of each day
day_count = defaultdict(int)

# a dictionary to store the count of each device (PC or Mobile)
device_count = defaultdict(int)
device_count['PC'] = 0
device_count['Mobile'] = 0

# a dictionary to store the count of each request page
request_page_count = defaultdict(int)

for root, dirs, files in os.walk(source_logs_dir):
    for name in files:
        # find the target log files
        if log_file_name_regex.search(name):
            # parst the day
            day = root[-13:-3]

            # copy the target log files
            shutil.copyfile(os.path.join(root, name), os.path.join(unziped_logs_dir, name))

            # unzip the log files
            unziped_log_file = gzip.open(os.path.join(unziped_logs_dir, name), 'rb')

            http_request_count = 0
            pc_count = 0
            mobile_count = 0
            for line in unziped_log_file:
                if(http_request_regex.search(line)):
                    # parse the request page
                    regex_obj = request_page_regex.search(line)
                    request_page = regex_obj.group(1)
                    # remove params of the request page
                    if('?' in request_page):
                        request_page = request_page[:request_page.find('?')]

                    http_request_count = http_request_count + 1

                    if('Mobile' in line):
                        mobile_count = mobile_count + 1
                    else:
                        pc_count = pc_count + 1

                    # update the count of each request page
                    if(request_page in request_page_count):
                                        request_page_count[request_page] = request_page_count[request_page] + 1
                                else:
                                        request_page_count[request_page] = 1

            # update the HTTP request count of each day
            if(day in day_count):
                day_count[day] = day_count[day] + http_request_count
            else:
                day_count[day] = http_request_count

            # update the count of each device (PC or Mobile)
            device_count['PC'] = device_count['PC'] + pc_count
            device_count['Mobile'] = device_count['Mobile'] + mobile_count

            # remvoe the original zip log files
            os.remove(os.path.join(unziped_logs_dir, name))

# print the HTTP request count of each day
total = 0
print 'HTTP request count of each day'
for day, count in sorted(day_count.items()):
    print day, ':', count
    total = total + count
print 'Total = ', total

print '###############################'

total = 0
print 'count of each device (PC or Mobile)'
# print the count of each device (PC or Mobile)
for device, count in sorted(device_count.items()):
        print device, ':', count
    total = total + count
print 'Total = ', total

print '###############################'

total = 0
print 'count of each request page'
# print the count of each request page
for request_page, count in sorted(request_page_count.items()):
        print request_page, ':', count
    total = total + count
print 'Total = ', total