快速实现Python多进程logging日志按天切割
前几天填了一个前人留下的“日志”坑，程序日志只有程序重启才可以生成一个新日志文件，这样就会导致程序长时间运行之后，会生成一个巨大的日志文件，不方便后期问题查询或磁盘清理，所以日志按天切割势在必行。
python 原生logging日志模块可以满足大部分需求，但是唯独不适合多进程下日志切割，具体的原因可以自行百度一下，网上有很多讲解的，我就说一下，如何用最小的改动logging源码的方法来实现pyhon多进程下日志按天(分钟、小时)切割。
直接上代码：
1、修改logging模块中的TimedRotatingFileHandler(BaseRotatingHandler)类方法doRollover(self):
class TimedRotatingFileHandler(BaseRotatingHandler):
def doRollover(self):
"""
do a rollover; in this case, a date/time stamp is appended to the filename
when the rollover happens. However, you want the file to be named for the
start of the interval, not the current time. If there is a backup count,
then we have to get a list of matching filenames, sort them and remove
the one with the oldest suffix.
"""
if self.stream:
self.stream.close()
self.stream = None
# get the time that this sequence started at and make it a TimeTuple
currentTime = int(time.time())
dstNow = time.localtime(currentTime)[-1]
t = self.rolloverAt - self.interval
if self.utc:
timeTuple = time.gmtime(t)
else:
timeTuple = time.localtime(t)
dstThen = timeTuple[-1]
if dstNow != dstThen:
if dstNow:
addend = 3600
else:
addend = -3600
timeTuple = time.localtime(t + addend)
dfn = self.baseFilename + "." + time.strftime(self.suffix, timeTuple)
self.baseFilename = self.baseFilename.split('.log')[0] + '.log-' + time.strftime('%Y%m%d',time.localtime(time.time()))
（新增一条语句）
''' 源码注释部分，其他的不需要改动
if os.path.exists(dfn):
os.remove(dfn)
os.rename(self.baseFilename, dfn)
'''
os.remove(dfn)
os.rename(self.baseFilename, dfn)
'''
if self.backupCount > 0:
# find the oldest log file and delete it
#s = glob.glob(self.baseFilename + ".20*")
#if len(s) > self.backupCount:
# s.sort()
# os.remove(s[0])
for s in self.getFilesToDelete():
os.remove(s)
#print "%s -> %s" % (self.baseFilename, dfn)
self.stream = self._open()
newRolloverAt = self.computeRollover(currentTime)
while newRolloverAt <= currentTime:
newRolloverAt = newRolloverAt + self.interval
#If DST changes and midnight or weekly rollover, adjust for this.
if (self.when == 'MIDNIGHT' or self.when.startswith('W')) and not self.utc:
dstAtRollover = time.localtime(newRolloverAt)[-1]
if dstNow != dstAtRollover:
if not dstNow: # DST kicks in before next rollover, so we need to deduct an hour
addend = -3600
else: # DST bows out before next rollover, so we need to add an hour
addend = 3600
newRolloverAt += addend
self.rolloverAt = newRolloverAt
2、写了一个通用模块，方便以后去调用 log.py
# -*- coding: UTF8 -*-
#!/usr/bin/python
import sys
reload(sys)
sys.setdefaultencoding('utf8')
import logging, logging.handlers
import time
import os
filePath = os.path.split(os.path.realpath(__file__))[0].replace("common", "logs")
def set_log(shellname):
filename = filePath + '/' + shellname.replace('.log','') + '.log-' + time.strftime('%Y%m%d',time.localtime(time.time()))
print filename
fmt_str = '%(asctime)s %(filename)s[line:%(lineno)d] %(levelname)s %(message)s'
fileshandle = logging.handlers.TimedRotatingFileHandler(filename, when='M', interval=1, backupCount=0)
fileshandle.suffix = "%Y%m%d_%H%M%S.log"
fileshandle.setLevel(logging.DEBUG)
formatter = logging.Formatter(fmt_str)
fileshandle.setFormatter(formatter)
logger = logging.getLogger('')
logger.setLevel(logging.INFO)
logger.addHandler(fileshandle)
3、调用模块方法：
import log
log.set_log("ReceiveCustomerDetailForHashtable_")
log.logging.error(".....")
4、自此就实现的python多进程按天切割日志。
实现原理：利用logging模块自带的按时间自定义切割日志模块，稍加修改一下，就可以实现。
但是就是有一个坑：日志生成格式是固定的，要么是按天或按小时或按分钟生成
不过日志切割一般都不会发生变化，所以这个坑还可以接受，或者分享一下，你自己的好方法。