python logging模块注册流程(以logging.config.dictConfig流程为例)
最近在查看python logging相关的模块，用到了dictConfig这一函数，尝试着跟踪了一下，捋一捋在调用dictConfig之后，这些都发生了什么,我想，用fileConfig的流程也差不多，先记载dictConfig的加载流程，后面跟踪源码了之后再做更新。
logging.config.dictconfig源码：:
dictConfigClass = DictConfigurator
def dictConfig(config):
"""Configure logging using a dictionary."""
dictConfigClass(config).configure()
1. 代码中可以看到执行config操作的实际是DictConfigurator这一实例的configure方法.那么就涉及到DictConfigurator实例的初始化：
class DictConfigurator(BaseConfigurator):
...
1.1 DictConfigurator继承自BaseConfigurator,查看其__init__方法:
class BaseConfigurator(object):
....
def __init__(self, config):
self.config = ConvertingDict(config)
self.config.configurator = self
代码中有两点: ConvertingDict的实例self.config, 以及self.config.configurator引用了当前的Baseconfigurator实例,
那么， 代码中的ConvertingDict(config)是个什么？继续跟踪:
1.2 ConvertingDict解析
class ConvertingDict(dict, ConvertingMixin):
"""A converting dictionary wrapper."""
def __getitem__(self, key):
value = dict.__getitem__(self, key)
return self.convert_with_key(key, value)
def get(self, key, default=None):
value = dict.get(self, key, default)
return self.convert_with_key(key, value)
def pop(self, key, default=None):
value = dict.pop(self, key, default)
return self.convert_with_key(key, value, replace=False)
可以看到，Convertingdict继承自dict, 重写了dict中的__getitem__, get, pop三个方法，先调用dict原有的方法，然后对获取的值使用self.convert_with_key进行转换,其代码在ConvertingMixin类中。
1.3 ConvertingMixin类
class ConvertingMixin(object):
"""For ConvertingXXX's, this mixin class provides common functions"""
def convert_with_key(self, key, value, replace=True):
result = self.configurator.convert(value)
#If the converted value is different, save for next time
if value is not result:
if replace:
self[key] = result
if type(result) in (ConvertingDict, ConvertingList,
ConvertingTuple):
result.parent = self
result.key = key
return result
def convert(self, value):
result = self.configurator.convert(value)
if value is not result:
if type(result) in (ConvertingDict, ConvertingList,
ConvertingTuple):
result.parent = self
return result
看到convert_with_key方法中，每次都会调用它所引用的self.configurator的convert方法, 在这个流程中，self.configurator就是当前的Baseconfigurator实例.
1.4 Baseconfigurator的convert方法：
def convert(self, value):
"""
Convert values to an appropriate type. dicts, lists and tuples are
replaced by their converting alternatives. Strings are checked to
see if they have a conversion format and are converted if they do.
"""
if not isinstance(value, ConvertingDict) and isinstance(value, dict):
value = ConvertingDict(value)
value.configurator = self
elif not isinstance(value, ConvertingList) and isinstance(value, list):
value = ConvertingList(value)
value.configurator = self
elif not isinstance(value, ConvertingTuple) and\
isinstance(value, tuple):
value = ConvertingTuple(value)
value.configurator = self
elif isinstance(value, str): # str for py3k
m = self.CONVERT_PATTERN.match(value)
if m:
d = m.groupdict()
prefix = d['prefix']
converter = self.value_converters.get(prefix, None)
if converter:
suffix = d['suffix']
converter = getattr(self, converter)
value = converter(suffix)
return value
前面都是在递归，只有当value类型为str类型时，才会去进行转换，其他都是递归转换类型
1.4.1 str数据的转化
CONVERT_PATTERN = re.compile(r'^(?P[a-z]+)://(?P.*)$')
CONVERT_PATTERN是一个正则，匹配类似于一个协议，如：prefix://suffix.
接着value_converters：
value_converters = {
'ext' : 'ext_convert',
'cfg' : 'cfg_convert',
}
可以看到，这对应了两种协议，ext和cfg. 看ext://协议对应的ext_convert方法：
def ext_convert(self, value):
"""Default converter for the ext:// protocol."""
return self.resolve(value)
它其实调用了self.resolve(value), self是当前的的Baseconfigurator, 查看resolve方法：
1.4.1 Baseconfigurator的resolve方法:
def resolve(self, s):
"""
Resolve strings to objects using standard import and attribute
syntax.
"""
name = s.split('.')
used = name.pop(0)
try:
found = self.importer(used)
for frag in name:
used += '.' + frag
try:
found = getattr(found, frag)
except AttributeError:
self.importer(used)
found = getattr(found, frag)
return found
except ImportError:
e, tb = sys.exc_info()[1:]
v = ValueError('Cannot resolve %r: %s' % (s, e))
v.__cause__, v.__traceback__ = e, tb
raise v
根据官方文档，ext协议主要是用来引用其他的模块，例如自定义模块ext://a.b.c, 分析源码：
先通过.号分割出各个小段，self.importer其实就是__import__函数, resolve方法会一直去循环加载模块,最后把模块返回。
1.5 总结一下：
-
Baseconfigurator初始化过程是：初始化一个ConvertingDict实例, 并将该实例的configurator指向Baseconfigurator自身, -
ConvertingDict的作用: 以ext为例，在获取对应的key:value值时，转化为协议对应的模块，以供使用。
2. DictConfigurator的configure方法：
源码如下：
"""Do the configuration."""
config = self.config
if 'version' not in config:
raise ValueError("dictionary doesn't specify a version")
if config['version'] != 1:
raise ValueError("Unsupported version: %s" % config['version'])
incremental = config.pop('incremental', False)
EMPTY_DICT = {}
logging._acquireLock()
try:
if incremental:
handlers = config.get('handlers', EMPTY_DICT)
for name in handlers:
if name not in logging._handlers:
raise ValueError('No handler found with '
'name %r' % name)
else:
try:
handler = logging._handlers[name]
handler_config = handlers[name]
level = handler_config.get('level', None)
if level:
handler.setLevel(logging._checkLevel(level))
except Exception as e:
raise ValueError('Unable to configure handler '
'%r: %s' % (name, e))
loggers = config.get('loggers', EMPTY_DICT)
for name in loggers:
try:
self.configure_logger(name, loggers[name], True)
except Exception as e:
raise ValueError('Unable to configure logger '
'%r: %s' % (name, e))
root = config.get('root', None)
if root:
try:
self.configure_root(root, True)
except Exception as e:
raise ValueError('Unable to configure root '
'logger: %s' % e)
else:
disable_existing = config.pop('disable_existing_loggers', True)
logging._handlers.clear()
del logging._handlerList[:]
# Do formatters first - they don't refer to anything else
formatters = config.get('formatters', EMPTY_DICT)
for name in formatters:
try:
formatters[name] = self.configure_formatter(
formatters[name])
except Exception as e:
raise ValueError('Unable to configure '
'formatter %r: %s' % (name, e))
# Next, do filters - they don't refer to anything else, either
filters = config.get('filters', EMPTY_DICT)
for name in filters:
try:
filters[name] = self.configure_filter(filters[name])
except Exception as e:
raise ValueError('Unable to configure '
'filter %r: %s' % (name, e))
# Next, do handlers - they refer to formatters and filters
# As handlers can refer to other handlers, sort the keys
# to allow a deterministic order of configuration
handlers = config.get('handlers', EMPTY_DICT)
deferred = []
for name in sorted(handlers):
try:
handler = self.configure_handler(handlers[name])
handler.name = name
handlers[name] = handler
except Exception as e:
if 'target not configured yet' in str(e):
deferred.append(name)
else:
raise ValueError('Unable to configure handler '
'%r: %s' % (name, e))
# Now do any that were deferred
for name in deferred:
try:
handler = self.configure_handler(handlers[name])
handler.name = name
handlers[name] = handler
except Exception as e:
raise ValueError('Unable to configure handler '
'%r: %s' % (name, e))
# Next, do loggers - they refer to handlers and filters
#we don't want to lose the existing loggers,
#since other threads may have pointers to them.
#existing is set to contain all existing loggers,
#and as we go through the new configuration we
#remove any which are configured. At the end,
#what's left in existing is the set of loggers
#which were in the previous configuration but
#which are not in the new configuration.
root = logging.root
existing = list(root.manager.loggerDict.keys())
#The list needs to be sorted so that we can
#avoid disabling child loggers of explicitly
#named loggers. With a sorted list it is easier
#to find the child loggers.
existing.sort()
#We'll keep the list of existing loggers
#which are children of named loggers here...
child_loggers = []
#now set up the new ones...
loggers = config.get('loggers', EMPTY_DICT)
for name in loggers:
if name in existing:
i = existing.index(name) + 1 # look after name
prefixed = name + "."
pflen = len(prefixed)
num_existing = len(existing)
while i < num_existing:
if existing[i][:pflen] == prefixed:
child_loggers.append(existing[i])
i += 1
existing.remove(name)
try:
self.configure_logger(name, loggers[name])
except Exception as e:
raise ValueError('Unable to configure logger '
'%r: %s' % (name, e))
#Disable any old loggers. There's no point deleting
#them as other threads may continue to hold references
#and by disabling them, you stop them doing any logging.
#However, don't disable children of named loggers, as that's
#probably not what was intended by the user.
#for log in existing:
# logger = root.manager.loggerDict[log]
# if log in child_loggers:
# logger.level = logging.NOTSET
# logger.handlers = []
# logger.propagate = True
# elif disable_existing:
# logger.disabled = True
_handle_existing_loggers(existing, child_loggers,
disable_existing)
# And finally, do the root logger
root = config.get('root', None)
if root:
try:
self.configure_root(root)
except Exception as e:
raise ValueError('Unable to configure root '
'logger: %s' % e)
finally:
logging._releaseLock()
2.1 incremental
从代码中可一个看到有个incremental参数:
incremental = config.pop('incremental', False)
这个参数的作用是什么?
2.1.1 假设incremental参数为True其核心在这里:
for name in loggers:
try:
self.configure_logger(name, loggers[name], True)
except Exception as e:
raise ValueError('Unable to configure logger '
这个例子中， configure_logger在DictConfigurator中， 代码如下:
def common_logger_config(self, logger, config, incremental=False):
"""
Perform configuration which is common to root and non-root loggers.
"""
level = config.get('level', None)
if level is not None:
logger.setLevel(logging._checkLevel(level))
if not incremental:
#Remove any existing handlers
for h in logger.handlers[:]:
logger.removeHandler(h)
handlers = config.get('handlers', None)
if handlers:
self.add_handlers(logger, handlers)
filters = config.get('filters', None)
if filters:
self.add_filters(logger, filters)
def configure_logger(self, name, config, incremental=False):
"""Configure a non-root logger from a dictionary."""
logger = logging.getLogger(name)
self.common_logger_config(logger, config, incremental)
propagate = config.get('propagate', None)
if propagate is not None:
logger.propagate = propagate
从以上代码中可以看到， 当incremental为True时，只会将日志的等级level修改.
2.2 incremental参数为False时：
2.2.1 清除logging中的_handler和_handlerList信息
logging._handlers.clear()
del logging._handlerList[:]
2.2.2 初始化formmaters, 源码如下
formatters = config.get('formatters', EMPTY_DICT)
for name in formatters:
try:
formatters[name] = self.configure_formatter(
formatters[name])
except Exception as e:
raise ValueError('Unable to configure '
'formatter %r: %s' % (name, e))
先会获取所有的formatter,然后通过configure_formatter方法来初始化.
def configure_formatter(self, config):
"""Configure a formatter from a dictionary."""
if '()' in config:
factory = config['()'] # for use in exception handler
try:
result = self.configure_custom(config)
except TypeError as te:
if "'format'" not in str(te):
raise
#Name of parameter changed from fmt to format.
#Retry with old name.
#This is so that code can be used with older Python versions
#(e.g. by Django)
config['fmt'] = config.pop('format')
config['()'] = factory
result = self.configure_custom(config)
else:
fmt = config.get('format', None)
dfmt = config.get('datefmt', None)
style = config.get('style', '%')
cname = config.get('class', None)
if not cname:
c = logging.Formatter
else:
c = _resolve(cname)
result = c(fmt, dfmt, style)
return result
可以看到，参数当formatter配置中，有两种情况：
- 有名为
()的key
此时，调用configure_custom方法:
def configure_custom(self, config):
"""Configure an object with a user-supplied factory."""
c = config.pop('()')
if not callable(c):
c = self.resolve(c)
props = config.pop('.', None)
# Check for valid identifiers
kwargs = dict([(k, config[k]) for k in config if valid_ident(k)])
result = c(**kwargs)
if props:
for name, value in props.items():
setattr(result, name, value)
return result
从代码中可以看到，config.pop('()')的调用，在上文中有提到：
def pop(self, key, default=None):
value = dict.pop(self, key, default)
return self.convert_with_key(key, value, replace=False)
这就是来获取需要调用的模块, 最后调用对应的模块，生成formatter,
注意一点： props = config.pop('.', None), 配置.属性可以设置对应实例的自身属性，如:
"console": {
"class": "logging.StreamHandler",
"level": "DEBUG",
"formatter": "simple",
".": {
"hei": 1
}
}
这个配置加上了.号属性的设置，初始化之后，对应的Streamhandler就会有属性hei
- 正常的key
直接调用对应的class模块，生成formatter
2.2.3 生成filter、Handler，与formatter类似, Handler最后会加上对应的formmaters以及filters
2.2.4 生成logger
这些就是logging.config.dictConfig的初始化过程，一步步解析，通过自带协议：ext://和cfg://协议来引用外部模块或是已有的配置,初始化你自己需求的logger，也可自定义相关模块，完成特殊化需求,粗糙之问，有点乱。有错误，麻烦之处，好更正。