python 64式: 第28式、分布式锁与群组管理__3、tooz应用之分布式锁
python中分布式锁与群组管理系列
最近有接触到分布式锁的相关问题。
基于openstack相关组件源码, tooz官网文档和自己对组件使用的一点点心得，
想整理一下这部分的内容。
主要想分为四个部分介绍:
分布式锁与群组管理 1、 tooz介绍
分布式锁与群组管理 2、 tooz应用之负载均衡
分布式锁与群组管理 3、 tooz应用之分布式锁
分布式锁与群组管理 4、 tooz源码分析
下面是第3部分的内容
1 监控数据处理总入口
gnocchi/cli.py的metricd方法，对应内容如下:
def metricd():
conf = cfg.ConfigOpts()
conf.register_cli_opts([
cfg.IntOpt("stop-after-processing-metrics",
default=0,
min=0,
help="Number of metrics to process without workers, "
"for testing purpose"),
])
conf = service.prepare_service(conf=conf)
if conf.stop_after_processing_metrics:
metricd_tester(conf)
else:
MetricdServiceManager(conf).run()
分析:
1.1) 为了调试，这里将
conf.stop_after_processing_metrics的值设置为1，这样就进入了
metricd_tester(conf)的处理逻辑。
后续可以尝试采用ForkedPdb方式进行调试,具体请参考:
在需要调试的代码中添加如下内容:
import sys
import pdb
class ForkedPdb(pdb.Pdb):
def interaction(self, *args, **kwargs):
_stdin = sys.stdin
try:
sys.stdin = open('/dev/stdin')
pdb.Pdb.interaction(self, *args, **kwargs)
finally:
sys.stdin = _stdin
在需要添加断点的地方，使用如下方法添加断点：
ForkedPdb().set_trace()
1.2) 打印参数
'''
(Pdb) p conf
(Pdb) p type(conf)
(Pdb) p conf.__dict__
{'_groups': {'statsd':
'''
2 分析metricd_tester(conf)方法
对应代码内容如下:
def metricd_tester(conf):
# NOTE(sileht): This method is designed to be profiled, we
# want to avoid issues with profiler and os.fork(), that
# why we don't use the MetricdServiceManager.
index = indexer.get_driver(conf)
index.connect()
s = storage.get_driver(conf)
metrics = s.incoming.list_metric_with_measures_to_process(
conf.stop_after_processing_metrics, 0)
s.process_new_measures(index, metrics, True)
分析:
2.1) 里面先获取了index的driver，具体调用:
index = indexer.get_driver(conf)
查看indexer.get_driver(conf)的具体内容如下:
def get_driver(conf):
"""Return the configured driver."""
split = netutils.urlsplit(conf.indexer.url)
d = driver.DriverManager('gnocchi.indexer',
split.scheme).driver
return d(conf)
解释:
这里conf.index.url样例如下:
'indexer': {'url': ['mysql+pymysql://gnocchi:[email protected]/gnocchi?charset=utf8']}
所以: split.scheme的内容应该为 'mysql+pymysql'
查看gnocchi的setup.cfg中有如下内容:
gnocchi.indexer =
mysql = gnocchi.indexer.sqlalchemy:SQLAlchemyIndexer
mysql+pymysql = gnocchi.indexer.sqlalchemy:SQLAlchemyIndexer
postgresql = gnocchi.indexer.sqlalchemy:SQLAlchemyIndexer
2.2) 分析 s = storage.get_driver(conf)
对应代码在:
gnocchi/storage/__init__.py文件中，内容如下:
def get_driver(conf):
"""Return the configured driver."""
incoming = get_driver_class('gnocchi.incoming', conf.incoming)(
conf.incoming)
return get_driver_class('gnocchi.storage', conf.storage)(
conf.storage, incoming)
分析:
2.2.1) 打印调试参数
(Pdb) p incoming
(Pdb) p type(incoming)
2.2.2) 分析
get_driver_class('gnocchi.storage', conf.storage)(
conf.storage, incoming)
对应内容如下:
def get_driver_class(namespace, conf):
"""Return the storage driver class.
:param conf: The conf to use to determine the driver.
"""
return driver.DriverManager(namespace,
conf.driver).driver
分析:
2.2.2.1) 这里实际上是初始化storage的对象
查看gnocchi的setup.cfg中有
gnocchi.storage =
swift = gnocchi.storage.swift:SwiftStorage
ceph = gnocchi.storage.ceph:CephStorage
file = gnocchi.storage.file:FileStorage
s3 = gnocchi.storage.s3:S3Storage
最终进入了gnocchi/storage/ceph.py中的CephStorage类，
具体代码如下
class CephStorage(_carbonara.CarbonaraBasedStorage):
WRITE_FULL = False
def __init__(self, conf, incoming):
super(CephStorage, self).__init__(conf, incoming)
self.rados, self.ioctx = ceph.create_rados_connection(conf)
2.2.2.2) 打印参数
(Pdb) p incoming
(Pdb) p incoming.__dict__
{'rados':
> /usr/lib/python2.7/site-packages/gnocchi/storage/_carbonara.py(63)__init__()
2.3) 打印调试参数
对于这一行代码:
s = storage.get_driver(conf)
打印结果如下:
(Pdb) p s
(Pdb) p s.incoming
(Pdb) p s.__dict__
{'incoming':
2.4）分析
metrics = s.incoming.list_metric_with_measures_to_process(
conf.stop_after_processing_metrics, 0)
具体代码位于gnocchi/storage/incoming/ceph.py的CephStorage类
内容如下:
class CephStorage(_carbonara.CarbonaraBasedStorage):
def __init__(self, conf):
super(CephStorage, self).__init__(conf)
self.rados, self.ioctx = ceph.create_rados_connection(conf)
# NOTE(sileht): constants can't be class attributes because
# they rely on presence of rados module
# NOTE(sileht): We allow to read the measure object on
# outdated replicats, that safe for us, we will
# get the new stuffs on next metricd pass.
self.OMAP_READ_FLAGS = (rados.LIBRADOS_OPERATION_BALANCE_READS |
rados.LIBRADOS_OPERATION_SKIPRWLOCKS)
# NOTE(sileht): That should be safe to manipulate the omap keys
# with any OSDs at the same times, each osd should replicate the
# new key to others and same thing for deletion.
# I wonder how ceph handle rm_omap and set_omap run at same time
# on the same key. I assume the operation are timestamped so that will
# be same. If not, they are still one acceptable race here, a rm_omap
# can finish before all replicats of set_omap are done, but we don't
# care, if that occurs next metricd run, will just remove it again, no
# object with the measure have already been delected by previous, so
# we are safe and good.
self.OMAP_WRITE_FLAGS = rados.LIBRADOS_OPERATION_SKIPRWLOCKS
def list_metric_with_measures_to_process(self, size, part, full=False):
names = self._list_object_names_to_process(limit=-1 if full else
size * (part + 1))
if full:
objs_it = names
else:
objs_it = itertools.islice(names, size * part, size * (part + 1))
return set([name.split("_")[1] for name in objs_it])
分析:
2.4.1) 打印入参
(Pdb) p size
1
(Pdb) p part
0
(Pdb) p full
False
2.4.2）调用了_list_object_names_to_process方法
该方法具体内容如下:
def _list_object_names_to_process(self, prefix="", marker="", limit=-1):
with rados.ReadOpCtx() as op:
omaps, ret = self.ioctx.get_omap_vals(op, marker, prefix, limit)
try:
self.ioctx.operate_read_op(
op, self.MEASURE_PREFIX, flag=self.OMAP_READ_FLAGS)
except rados.ObjectNotFound:
# API have still written nothing
return ()
# NOTE(sileht): after reading the libradospy, I'm
# not sure that ret will have the correct value
# get_omap_vals transforms the C int to python int
# before operate_read_op is called, I dunno if the int
# content is copied during this transformation or if
# this is a pointer to the C int, I think it's copied...
if ret == errno.ENOENT:
return ()
return (k for k, v in omaps)
分析:
2.4.2.1) 打印入参
(Pdb) p prefix
'measure_01f0658b-f147-482b-bca9-f474a79320dc'
(Pdb) p marker
''
(Pdb) p limit
2.4.2.2) 打印其他变量
(Pdb) p op
(Pdb) p type(op)
(Pdb) p op.__dict__
{}
(Pdb) p self.ioctx
2.4.2.3) 调用了get_omap_vals方法
处理结果:
(Pdb) p omaps
(Pdb) p type(omaps)
(Pdb) p ret
0
其中:
MEASURE_PREFIX = "measure"
self.OMAP_READ_FLAGS = (rados.LIBRADOS_OPERATION_BALANCE_READS |
rados.LIBRADOS_OPERATION_SKIPRWLOCKS)
2.4.2.3) 分析return (k for k, v in omaps)
(Pdb) p ret
0
(Pdb) p errno.ENOENT
2
(Pdb) p k
u'measure_01f0658b-f147-482b-bca9-f474a79320dc_0173ad9a-2553-4429-a475-11cc9f41e79e_20180419_01:31:58'
(Pdb) p v
None
这里果然omap存放的实际就是一个监控项
2.4.2.4) 总结
_list_object_names_to_process(self, prefix="", marker="", limit=-1)：
作用是: 从ceph中读取limit个数个对象，遍历omaps获取该对象的measure字符串名称组成的生成器并返回
2.5) 继续回到list_metric_with_measures_to_process方法
该方法内容如下:
def list_metric_with_measures_to_process(self, size, part, full=False):
names = self._list_object_names_to_process(limit=-1 if full else
size * (part + 1))
if full:
objs_it = names
else:
objs_it = itertools.islice(names, size * part, size * (part + 1))
return set([name.split("_")[1] for name in objs_it])
分析:
2.5.1) 根据上面的分析_list_object_names_to_process方法
作用是: 从ceph中读取limit个数个对象，遍历omaps获取该对象的measure的字符串名称组成的生成器并返回
处理结果如下:
(Pdb) p names
(Pdb) p type(names)
2.5.2) 其他处理结果
(Pdb) p full
False
(Pdb) p size * part
0
(Pdb) p size
1
(Pdb) p part
0
(Pdb) p size * (part + 1)
1
(Pdb) p objs_it
所以itertools是一个迭代器
参考: https://jingyan.baidu.com/article/6b97984db9b7931ca2b0bfe6.html
itertools: 用于实现各种生成器的工具包
切片itertools.islice：有时候我们并不想迭代所有的元素，
比如我只想处理d-g这些字母，看下面代码，我们可以使用islice(astr,3,len(astr)-1)
所以这里得到的是迭代器
islice(iterable, [start,] stop [, step]) --> islice object
这里的关键就是names，作为可迭代的对象
(Pdb) p name
u'measure_01f0658b-f147-482b-bca9-f474a79320dc_0173ad9a-2553-4429-a475-11cc9f41e79e_20180419_01:31:58'
最终返回结果是一个集合，样例如下:
set([u'01f0658b-f147-482b-bca9-f474a79320dc'])
2.5.3) 总结
list_metric_with_measures_to_process(self, size, part, full=False):
作用：
返回的是一个包含metric id的集合，这里本质上的处理流程是: 获取ceph中以measure开头的对象
从该对象名中截取出监控项id(即metric id)，塞入待处理的监控项id集合中返回
2.6) 继续返回到gnocchi/cli.py中的metricd_tester方法
该方法内容如下:
def metricd_tester(conf):
# NOTE(sileht): This method is designed to be profiled, we
# want to avoid issues with profiler and os.fork(), that
# why we don't use the MetricdServiceManager.
index = indexer.get_driver(conf)
index.connect()
s = storage.get_driver(conf)
metrics = s.incoming.list_metric_with_measures_to_process(
conf.stop_after_processing_metrics, 0)
s.process_new_measures(index, metrics, True)
分析:
2.6.1) 根据之前的分析list_metric_with_measures_to_process方法主要是
返回的是一个包含metric id的集合，这里本质上的处理流程是: 获取ceph中以measure开头的对象
从该对象名中截取出监控项id(即metric id)，塞入待处理的监控项id集合中返回
处理结果样例如下:
(Pdb) print metrics
set([u'01f0658b-f147-482b-bca9-f474a79320dc'])
2.6.2) 调用了process_new_measures方法，
具体参见3的分析。
3 分析process_new_measures方法
3.1）这里调用了
gnocchi/storage/_carbonara.py文件的CarbonaraBasedStorage类的process_new_measures()
内容如下:
class CarbonaraBasedStorage(storage.StorageDriver):
UPGRADE_BATCH_SIZE = 1000
def __init__(self, conf, incoming):
super(CarbonaraBasedStorage, self).__init__(conf, incoming)
self.aggregation_workers_number = conf.aggregation_workers_number
if self.aggregation_workers_number == 1:
# NOTE(jd) Avoid using futures at all if we don't want any threads.
self._map_in_thread = self._map_no_thread
else:
self._map_in_thread = self._map_in_futures_threads
self.coord, my_id = utils.get_coordinator_and_start(
conf.coordination_url)
def process_new_measures(self, indexer, metrics_to_process,
sync=False):
metrics = indexer.list_metrics(ids=metrics_to_process)
# This build the list of deleted metrics, i.e. the metrics we have
# measures to process for but that are not in the indexer anymore.
deleted_metrics_id = (set(map(uuid.UUID, metrics_to_process))
- set(m.id for m in metrics))
for metric_id in deleted_metrics_id:
# NOTE(jd): We need to lock the metric otherwise we might delete
# measures that another worker might be processing. Deleting
# measurement files under its feet is not nice!
try:
with self._lock(metric_id)(blocking=sync):
self.incoming.delete_unprocessed_measures_for_metric_id(
metric_id)
except coordination.LockAcquireFailed:
LOG.debug("Cannot acquire lock for metric %s, postponing "
"unprocessed measures deletion", metric_id)
for metric in metrics:
lock = self._lock(metric.id)
# Do not block if we cannot acquire the lock, that means some other
# worker is doing the job. We'll just ignore this metric and may
# get back later to it if needed.
if not lock.acquire(blocking=sync):
continue
try:
locksw = timeutils.StopWatch().start()
LOG.debug("Processing measures for %s", metric)
# process_measure_for_metric(self, metric):返回待处理监控项对应的监控数据列表，
# 每个元素是时间戳和对应的值，样例：[(Timestamp('2018-04-19 02:29:04.925214'), 4.785732057729687)]
with self.incoming.process_measure_for_metric(metric) \
as measures:
self._compute_and_store_timeseries(metric, measures)
LOG.debug("Metric %s locked during %.2f seconds",
metric.id, locksw.elapsed())
except Exception:
LOG.debug("Metric %s locked during %.2f seconds",
metric.id, locksw.elapsed())
if sync:
raise
LOG.error("Error processing new measures", exc_info=True)
finally:
lock.release()
分析:
3.1.1) 输入参数打印
(Pdb) p indexer
(Pdb) p indexer.__dict__
{'facade':
'conf':
(Pdb) p type(indexer)
(Pdb) p metrics_to_process
set([u'01f0658b-f147-482b-bca9-f474a79320dc'])
(Pdb) p sync
True
3.1.2) 处理过程分析
metrics = indexer.list_metrics(ids=metrics_to_process)
这里调用了gnocchi/indexer/sqlalchemy.py文件中SQLAlchemyIndexer的类的
list_metrics方法，具体内容如下:
class SQLAlchemyIndexer(indexer.IndexerDriver):
_RESOURCE_TYPE_MANAGER = ResourceClassMapper()
def list_metrics(self, names=None, ids=None, details=False,
status='active', limit=None, marker=None, sorts=None,
**kwargs):
sorts = sorts or []
if ids is not None and not ids:
return []
if names is not None and not names:
return []
with self.facade.independent_reader() as session:
q = session.query(Metric).filter(
Metric.status == status)
if names is not None:
q = q.filter(Metric.name.in_(names))
if ids is not None:
q = q.filter(Metric.id.in_(ids))
for attr in kwargs:
q = q.filter(getattr(Metric, attr) == kwargs[attr])
if details:
q = q.options(sqlalchemy.orm.joinedload('resource'))
sort_keys, sort_dirs = self._build_sort_keys(sorts)
if marker:
metric_marker = self.list_metrics(ids=[marker])
if metric_marker:
metric_marker = metric_marker[0]
else:
raise indexer.InvalidPagination(
"Invalid marker: `%s'" % marker)
else:
metric_marker = None
try:
q = oslo_db_utils.paginate_query(q, Metric, limit=limit,
sort_keys=sort_keys,
marker=metric_marker,
sort_dirs=sort_dirs)
except ValueError as e:
raise indexer.InvalidPagination(e)
except exception.InvalidSortKey as e:
raise indexer.InvalidPagination(e)
return list(q.all())
分析:
3.1.2.1) 入参如下
(Pdb) p names
None
(Pdb) p ids
set([u'01f0658b-f147-482b-bca9-f474a79320dc'])
这里实际上就是从indexer数据库中获取到输入的监控项id列表对应
在数据库中的Metric对象列表。
处理结果是一个Metric对象的列表
[
3.1.3) 继续回到process_new_measures方法
(Pdb) p deleted_metrics_id
set([])
分析下面的代码:
for metric in metrics:
lock = self._lock(metric.id)
# Do not block if we cannot acquire the lock, that means some other
# worker is doing the job. We'll just ignore this metric and may
# get back later to it if needed.
if not lock.acquire(blocking=sync):
continue
try:
locksw = timeutils.StopWatch().start()
LOG.debug("Processing measures for %s", metric)
# process_measure_for_metric(self, metric):返回待处理监控项对应的监控数据列表，
# 每个元素是时间戳和对应的值，样例：[(Timestamp('2018-04-19 02:29:04.925214'), 4.785732057729687)]
with self.incoming.process_measure_for_metric(metric) \
as measures:
self._compute_and_store_timeseries(metric, measures)
LOG.debug("Metric %s locked during %.2f seconds",
metric.id, locksw.elapsed())
3.1.3.1) 分析 lock = self._lock(metric.id)
这里对应代码如下:
def _lock(self, metric_id):
lock_name = b"gnocchi-" + str(metric_id).encode('ascii')
return self.coord.get_lock(lock_name)
分析:
3.1.3.1.1) 打印入参
(Pdb) p metric_id
UUID('01f0658b-f147-482b-bca9-f474a79320dc')
(Pdb) str(metric_id)
'01f0658b-f147-482b-bca9-f474a79320dc'
3.1.3.1.2) 处理分析
(Pdb) p self.coord
(Pdb) p self.coord.__dict__
{'heart':
> /usr/lib/python2.7/site-packages/tooz/drivers/mysql.py(130)get_lock()
130 -> def get_lock(self, name):
131 return MySQLLock(name, self._parsed_url, self._options)
本质分析:
这里实际上就是用tooz.coordination来获取mariadb作为tooz后端的分布式锁。
这里就是获取所有待处理的监控项列表，对每个监控项，先获取当前监控项的分布式锁，
如果获取不到锁，说明其他进程也在处理当前监控项(因为gnocchi采用的是cotyledon库，
这是一个多进程框架，多进程之间防止同时对一个数据进行处理则需要使用分布式锁)，
则不对该监控项处理；
否则，获取该监控的所有监控数据，进行后续处理。
3.1.3.1.3) 处理结果
(Pdb) p lock
(Pdb) p lock.__dict__
{'acquired': False, '_conn':
'_name': 'gnocchi-01f0658b-f147-482b-bca9-f474a79320dc'}
(Pdb) p locksw
(Pdb) p locksw.__dict__
{'_started_at': 69810.789720241, '_splits': [], '_state': 'STARTED', '_duration': None, '_stopped_at': None}
3.1.4) 分析process_measure_for_metric方法
在process_new_measures方法中获取了当前监控项，
然后获取该监控项的监控数据，代码如下:
with self.incoming.process_measure_for_metric(metric) \
as measures:
self._compute_and_store_timeseries(metric, measures)
具体的细节分析，参见4
3.1.5) 这里先写上对process_new_measures最后的总结，
process_new_measures(self, indexer, metrics_to_process,sync=False)
一. 根据待处理的监控项集合，判断如果有已经删除的监控项，则删除对应incoming storage中的监控数据
二. 遍历待处理的监控项列表，获取每个监控项在incoming storage中的监控数据列表，然后根据监控项及其待处理监控数据来计算并存储时间序列
具体步骤二需要具体细分为如下步骤
1. 对待处理监控数据按照时间从旧到新排序，获取监控项中聚合方法，采样间隔等信息
2. 获取未聚合的时间序列数据进行反序列化，来重新构建为新的时间序列，具体步骤如下
2.0. 先根据监控项id构建需要获取的对象名称，形如:gnocchi_01f0658b-f147-482b-bca9-f474a79320dc_none_v3
从ceph中读取该对象存储的值(是一个字符串)
2.1. 先解压从步骤0中读取的数据（实际是一个字符串），前面一半为时间，后面一半为时间对应的值
2.2. 解压的时间由于采用差值，所以累加计算每个时间；
2.3. 将时间列表，值列表来构建时间序列，然后根据block_size(实际是最大采样间隔)对序列计算出这个时间序列中最后一个数据，
在一天之前的起始时间，以该时间为基础，对此时间序列进行切片，得到最终需要处理的时间序列
2.4. 用步骤2.3的时间序列，block_size等实例化并返回最终需要处理的BoundTimeSerie
3 计算聚合后的时间序列，具体过程如下
3.1. 对给定的已经合并了待处理数据生成的时间序列和未聚合的时间序列的合并时间序列boundTimeSerie进行如下操作
3.2. 遍历归档策略，根据采样间隔，聚合方法：
3.2.1计算每个boundTimeSerie聚合后的时间序列；具体计算聚合的时间序列算法如下
步骤1:对时间序列ts的索引ts.index用采样间隔进行分组得到indexes
步骤2:对分组后的索引indexes通过numpy.unique重计算得到uniqeIndexes
步骤3:调用ndimage.mean方法，进行如下操作
ndimage.mean(ts.value , labels=indexes, index=uniqueIndexes)
即可得到聚合结果aggregatedValues
步骤4: 将uniqueIndexes还原为datetime64[ns]类型的numpy数组
timestamps
步骤5: 重新用步骤3得到的aggregatedValues和步骤4得到的timestamps
构建新的时间序列，该时间序列即为最终聚合的时间序列newTimeSerie
步骤6：根据需要保存的点的个数n,进行切片处理，获取newTimeSerie[-n:]
作为最终保存的时间序列的结果
3.2.2并对该聚合的时间序列分割，计算分割序列的偏移量和对应序列化的值；
3.2.2根据偏移量，将序列化的值写入到对应的ceph对象
总结：步骤3实现了: 计算聚合后的时间序列，将聚合后的时间序列写入到ceph对象中
4. 更新未聚合的时间序列(即里面还保留了原始监控数据，用于下一次计算)，具体过程如下
4.1. 对时间序列的索引进行numpy.diff的求差值操作，并
在所求的索引差值列表的最前面加上该时间序列的第一个值，
得到差值索引列表
4.2. 对差值索引列表的类型转换为uint64类型
4.3. 对时间序列的值列表类型转换为浮点型
4.4 对差值索引列表转换为字节 + 对时间序列的值列表转换为字节，
得到字符串
4.5. 对该字符串调用lz4.dumps进行压缩，返回该压缩后的字符串
4.6 构建类似gnocchi_01f0658b-f147-482b-bca9-f474a79320dc_none_v3的对象名称，
向该对象中写入未聚合的时间序列的压缩后的字符串
该字符串序列化前的前半部分为：时间序列索引，后半部分为时间序列值
总结整个时间序列处理过程【概览】：
遍历待处理的监控项列表
|
对每个待处理监控项，从incoming storage中获取该监控项的待处理监控数据
|
获取storage中该监控项对应对象的数据经过反序列化得到未聚合的时间序列
|
将该监控项待处理的监控数据转换未待处理时间序列并和其未聚合的时间序列进行合并
|
遍历该监控项的聚合方法，采样间隔
|
对每个的聚合方法，采样间隔下的合并后的时间序列进行聚合处理
|
对聚合后的时间序列分割，每个分割后的时间序列序列化后写入storage中的对应聚合方法和采样间隔的对象
|
对合并后的时间序列进行序列化,写入到storage中对应的对象中
4 分析process_measure_for_metric方法
由于incoming storage配置的后端为ceph，所以对应
具体代码位置在:
gnocchi/storage/incoming/ceph.py文件的CephStorage类中
具体代码如下:
@contextlib.contextmanager
def process_measure_for_metric(self, metric):
object_prefix = self.MEASURE_PREFIX + "_" + str(metric.id)
object_names = list(self._list_object_names_to_process(object_prefix))
measures = []
ops = []
bufsize = 8192 # Same sa rados_read one
tmp_measures = {}
def add_to_measures(name, comp, data):
if name in tmp_measures:
tmp_measures[name] += data
else:
tmp_measures[name] = data
if len(data) < bufsize:
measures.extend(self._unserialize_measures(name,
tmp_measures[name]))
del tmp_measures[name]
else:
ops.append(self.ioctx.aio_read(
name, bufsize, len(tmp_measures[name]),
functools.partial(add_to_measures, name)
))
for name in object_names:
ops.append(self.ioctx.aio_read(
name, bufsize, 0,
functools.partial(add_to_measures, name)
))
while ops:
op = ops.pop()
op.wait_for_complete_and_cb()
yield measures
# Now clean objects and omap
with rados.WriteOpCtx() as op:
# NOTE(sileht): come on Ceph, no return code
# for this operation ?!!
self.ioctx.remove_omap_keys(op, tuple(object_names))
self.ioctx.operate_write_op(op, self.MEASURE_PREFIX,
flags=self.OMAP_WRITE_FLAGS)
for n in object_names:
self.ioctx.aio_remove(n)
分析:
4.1) 打印入参
(Pdb) p metric
(Pdb) p metric.__dict__
{'status': u'active', '_sa_instance_state':
'name': u'cpu_util', 'creator': u'6a18a77646104fcb93e92cb3daf10c91:55e9bc42c004471b9111ffbb516a9bbe',
'resource_id': UUID('d872305c-94b3-4f35-a2d5-602af219945d'),
'archive_policy':
'archive_policy_name': u'frequency_300s', 'id': UUID('01f0658b-f147-482b-bca9-f474a79320dc'),
'unit': None}
4.2) 处理过程分析
(Pdb) p self.MEASURE_PREFIX
'measure'
(Pdb) p metric.id
UUID('01f0658b-f147-482b-bca9-f474a79320dc')
(Pdb) p str(metric.id)
'01f0658b-f147-482b-bca9-f474a79320dc'
(Pdb) p object_prefix
'measure_01f0658b-f147-482b-bca9-f474a79320dc'
分析
object_names = list(self._list_object_names_to_process(object_prefix))
4.2.1) 调用_list_object_names_to_process方法
该方法具体内容如下:
def _list_object_names_to_process(self, prefix="", marker="", limit=-1):
with rados.ReadOpCtx() as op:
omaps, ret = self.ioctx.get_omap_vals(op, marker, prefix, limit)
try:
self.ioctx.operate_read_op(
op, self.MEASURE_PREFIX, flag=self.OMAP_READ_FLAGS)
except rados.ObjectNotFound:
# API have still written nothing
return ()
# NOTE(sileht): after reading the libradospy, I'm
# not sure that ret will have the correct value
# get_omap_vals transforms the C int to python int
# before operate_read_op is called, I dunno if the int
# content is copied during this transformation or if
# this is a pointer to the C int, I think it's copied...
if ret == errno.ENOENT:
return ()
return (k for k, v in omaps)
分析:
4.2.1.1) 打印入参
(Pdb) p prefix
'measure_01f0658b-f147-482b-bca9-f474a79320dc'
(Pdb) p marker
''
(Pdb) p limit
-1
4.2.1.2) 处理过程分析
(Pdb) p op
(Pdb) p type(op)
(Pdb) p op.__dict__
{}
(Pdb) p self.ioctx
(Pdb) p omaps
(Pdb) p type(omaps)
(Pdb) p ret
0
(Pdb) p self.MEASURE_PREFIX
'measure'
(Pdb) p self.OMAP_READ_FLAGS
17
4.2.1.3) 总结
_list_object_names_to_process(self, prefix="", marker="", limit=-1)：
从ceph中读取limit个数个对象，遍历omaps获取该对象的measure名称组成的生成器
生成器中遍历的元素样例如下: u'measure_01f0658b-f147-482b-bca9-f474a79320dc_014eacd6-fc6e-4c95-a8ff-f5f035166116_20180419_02:29:05'
4.3）继续回到process_measure_for_metric分析
该方法具体内容如下:
@contextlib.contextmanager
def process_measure_for_metric(self, metric):
object_prefix = self.MEASURE_PREFIX + "_" + str(metric.id)
object_names = list(self._list_object_names_to_process(object_prefix))
measures = []
ops = []
bufsize = 8192 # Same sa rados_read one
tmp_measures = {}
def add_to_measures(name, comp, data):
if name in tmp_measures:
tmp_measures[name] += data
else:
tmp_measures[name] = data
if len(data) < bufsize:
measures.extend(self._unserialize_measures(name,
tmp_measures[name]))
del tmp_measures[name]
else:
ops.append(self.ioctx.aio_read(
name, bufsize, len(tmp_measures[name]),
functools.partial(add_to_measures, name)
))
for name in object_names:
ops.append(self.ioctx.aio_read(
name, bufsize, 0,
functools.partial(add_to_measures, name)
))
while ops:
op = ops.pop()
op.wait_for_complete_and_cb()
yield measures
# Now clean objects and omap
'''
对已经从incoming storage中提取的数据，清空ceph中对应的对象
'''
with rados.WriteOpCtx() as op:
# NOTE(sileht): come on Ceph, no return code
# for this operation ?!!
self.ioctx.remove_omap_keys(op, tuple(object_names))
self.ioctx.operate_write_op(op, self.MEASURE_PREFIX,
flags=self.OMAP_WRITE_FLAGS)
for n in object_names:
self.ioctx.aio_remove(n)
分析:
4.3.1) 根据上面的分析
_list_object_names_to_process方法实际就是：
从ceph中读取limit个数个对象，遍历omaps获取该对象的measure名称组成的生成器
生成器中遍历的元素样例如下: u'measure_01f0658b-f147-482b-bca9-f474a79320dc_014eacd6-fc6e-4c95-a8ff-f5f035166116_20180419_02:29:05'
object_names = list(self._list_object_names_to_process(object_prefix))
的处理结果样例如下:
(Pdb) p object_names
[u'measure_01f0658b-f147-482b-bca9-f474a79320dc_014eacd6-fc6e-4c95-a8ff-f5f035166116_20180419_02:29:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_0173ad9a-2553-4429-a475-11cc9f41e79e_20180419_01:31:58', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_02321ee7-2361-4ece-b34b-e86a05b4acfc_20180418_12:20:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_027cb3b1-39d7-4b72-8a36-7793bd79008b_20180419_03:06:07', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_03bf12fb-3f87-4f2f-9958-fa598bf71d9b_20180418_12:42:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_04a6f4c0-c774-494e-9de4-2003ca1ea78b_20180419_02:09:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_0900b2fe-6b6c-4be4-bf1b-76ada25512db_20180419_02:00:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_0a95f66e-1fb7-41d4-ae54-144c3e6ce2bb_20180418_11:31:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_0fec40a8-943c-4c73-9f8a-76aca3c522d2_20180419_02:59:07', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_1021f511-36be-4025-b317-7c1333d6e840_20180419_01:42:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_1107c9b0-f564-46e7-ab2e-f06aeeadf7d6_20180419_02:30:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_155c59ab-17a2-4cac-b0bb-be0ab15fd84f_20180419_02:52:16', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_178e2e3c-6f80-4a1f-a9b7-d299d130d078_20180419_02:05:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_182df465-e25f-4c72-8f40-3dd972efd0e2_20180418_11:43:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_18d5e87c-889b-4636-9130-3df6d978bd93_20180419_01:57:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_1a191b26-06ae-4799-9a01-0099fb97b559_20180419_02:36:09', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_1c0d1d3b-9b01-47b5-ab1f-2014e5282cd9_20180419_02:39:07', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_1c6cd652-98cc-41ca-8061-b4cbf83c7307_20180419_01:33:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_1cbb8a06-7f64-4968-8f5d-c82763d177e0_20180418_11:58:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_1d34200e-7a1f-4612-9a26-0c54d3975d8c_20180418_12:33:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_1f7e2cc0-670d-4c8b-aaea-4f9ca9fa869b_20180419_02:51:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_203595b4-bd0b-4fee-8c61-97c517910554_20180419_03:13:10', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_243d0a84-6dc8-4040-904d-86415482d18f_20180419_02:06:04', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_25aafde0-9b0b-4495-bb05-8d306291b63c_20180419_02:17:09', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_27526033-99c4-4032-b1aa-3e7ae229e267_20180418_11:40:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_27edc799-9111-4cb4-88cc-d27310980e6d_20180418_11:25:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_291d0b6f-e6be-41b6-8926-c4c1ba3d448b_20180419_01:51:04', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_2cd0b86f-01d3-4c4a-a933-9a76ed516369_20180418_12:41:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_2d7af9f3-d933-4a89-aa58-37a5c6750ed1_20180419_01:34:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_2d8f9ca3-a218-4754-95a3-59cddbe4bbde_20180418_11:33:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_2dc310cf-af5b-4516-b713-2aca6698504b_20180419_03:01:08', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_30294747-6a6b-4629-8e98-8dc6f52dc2aa_20180418_12:34:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_304144a8-9add-421a-bd26-5694521d3c91_20180418_12:59:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_3180f999-962d-496f-b04f-e192772cf99a_20180418_11:23:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_31ed994e-0f4e-4168-b042-594f7704f06d_20180419_02:11:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_32da565c-4159-402e-b983-02ff1788bcab_20180418_11:49:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_3553d290-538a-4f7b-890b-2a5f8306739b_20180419_01:43:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_366ac18f-4470-493d-8678-5987fa79e695_20180418_12:29:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_36c36ef0-2634-4de7-8a8a-83f16859080a_20180418_12:23:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_37825104-51c0-4af1-b4e8-670762d90c0b_20180418_11:44:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_38b4178f-42e7-4d57-9860-811ef9d375ef_20180419_03:05:09', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_3bcce9b6-f6c3-4308-8ff9-e3afd9a8e2b3_20180418_12:17:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_3d22638d-eea2-45f7-a3c8-c52cd023170d_20180418_12:04:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_3d2ec553-4442-4c02-a1f4-56e27098511c_20180419_02:41:09', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_3de8a274-1f83-4691-a358-e6cf7e24ef87_20180419_02:15:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_3f3bc7b2-ad51-4ea5-8ea1-f34b58327ca2_20180419_02:04:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_3f464fcb-e822-4e96-927a-f13693f322b4_20180419_02:13:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_4260a1fc-2c9f-4eb0-ae4e-73dabd039733_20180418_12:37:32', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_43018947-5e49-474c-b3eb-81889ca04150_20180419_02:53:10', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_43938551-9079-4d7c-a8ae-bedb55b138f3_20180418_11:32:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_469f5d19-d3c9-437e-847e-4df527cb52f9_20180418_11:57:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_48597ab5-dacd-45de-832c-0a9ce6737123_20180418_12:30:32', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_4a7c2c66-1853-46ca-9add-eb1bd9654205_20180419_02:12:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_4b57e9b3-cfe9-4e4c-942a-d4022e51a4f8_20180418_11:35:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_4c9e7420-e3f7-4316-ae02-1f3ff6eb9f04_20180418_12:16:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_4caa4177-e743-4f93-91ff-ba78c8b767ac_20180418_12:31:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_4de02f96-9d67-4246-a4c7-81d9045b8f00_20180418_12:26:33', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_4e1c202e-5dec-49e5-a610-2227865b97b6_20180419_02:37:08', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_4f593617-c7f9-4d0d-94d0-693997ce0985_20180418_11:29:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_5104e234-70a3-4ef6-8c39-7909005e35b5_20180418_12:46:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_525b6198-e8d2-48a1-8878-c75fe26012d0_20180418_12:24:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_53cc87db-a047-452f-8f0c-4cd53ec47623_20180419_01:32:08', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_53ec8445-81b1-40fc-822b-4787a16e5e0c_20180419_02:16:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_5758c30f-6c86-4344-b5b7-40457fc06ff6_20180418_12:09:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_58a2bde1-855d-4b39-b676-5a1a9282ebe9_20180418_11:38:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_58e8828d-5982-4569-bfaf-d7e1e9322e55_20180418_12:03:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_591ce9f4-10b9-46a9-a29b-f315e944a0ff_20180418_12:14:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_5a6d318c-f20f-47f6-bd1c-718917d67877_20180418_12:27:32', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_5b00b2c8-a2e0-43ed-afd9-5d021a41cd64_20180418_11:34:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_5bececed-4c53-4165-9552-6703a0d3ee54_20180419_02:19:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_5dc9af63-cfa4-4e46-96dc-bf28b4a428e3_20180419_02:14:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_5ded4fc6-2fc6-479e-b847-2f08f015612d_20180418_12:53:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_5e6393d4-0538-45a9-b196-b59ee070f04b_20180418_12:47:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_5f658aca-26b3-46d4-af08-08077fbcc58e_20180419_02:18:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_60048e9e-e437-4598-89b9-a8cb1f83f7c3_20180419_01:35:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_602433d4-258d-4641-8d19-51a23382d2ba_20180419_02:03:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_60884c6a-d494-4118-ace6-d2ae7f55e2a0_20180418_11:46:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_62924930-383a-4d8b-8e81-13c0e5b79c71_20180418_12:44:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_654e6938-064a-4089-b69d-dfe66a125f8d_20180419_02:26:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_666a3ee7-1108-4ac6-ad25-f2460d705d8d_20180419_01:46:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_666c1012-5721-40b8-9b20-ca489cb1eca2_20180418_12:40:32', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_66bccf94-0a9f-4783-ae71-459cdd035c3a_20180419_01:50:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_67ae5639-de79-4d0c-bfea-0c8ab71f0431_20180419_02:44:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_6828f66f-a365-4240-b4ad-86565cc82458_20180418_12:48:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_6bf75621-8e64-4915-9ede-817a40fa1ba5_20180418_11:47:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_6c2a1ef2-3073-4fac-b1ea-c8562172a414_20180418_12:19:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_6d9fdee3-b9e6-4c23-81e2-a86db864cac9_20180419_02:57:07', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_6f3dc45f-a4b5-4859-8ea7-8ddf4d6327dd_20180418_12:05:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_6f7a0575-434e-41b4-8dd0-47703858f74e_20180418_12:13:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_70f57e91-4847-4470-9a45-bf826156a5db_20180418_12:55:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_70f76174-e804-48e9-bf45-9700afa89e67_20180419_02:08:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_72435fb4-f374-469f-ba23-1117438e9195_20180419_01:41:04', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_738f13db-91c1-4a23-9f46-2f3e4675f554_20180419_02:49:08', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_758ebfbb-3ed1-484b-bba4-20034e1bce37_20180419_02:43:07', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_764dda5f-5fdb-4e23-9366-71cd0e05b201_20180418_11:36:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_7737b4f8-e085-47d2-82d7-2094a56b2438_20180419_01:40:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_78f97724-53de-455c-825a-875047e9f022_20180419_03:11:07', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_7960e47d-d575-4d5e-b7a0-55df9ac170eb_20180419_02:28:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_79f19886-a7a3-4c2c-af4b-58203b13247c_20180419_01:56:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_7ac97665-33bc-4b3a-93b6-a0e2039fa07b_20180419_02:31:08', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_7b493534-a82c-4136-b1a2-ac4901a60380_20180418_12:21:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_7eb79db5-9d00-4831-8e3a-3b9df21741a4_20180419_02:38:09', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_7ef0ad70-dd24-43a2-8477-8bdfffcf6563_20180419_02:35:07', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_80d21476-d7b9-4f86-a3e9-fc4f9acf117e_20180418_11:59:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_8268676b-2a97-4a47-9ee2-f8adb9846d21_20180418_12:35:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_83b00bdf-a566-4455-8682-4b972d764de2_20180418_11:21:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_83b316dd-9ec5-4409-b72f-105e6bcf67ab_20180419_01:36:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_85bf3b25-a120-41f0-8b32-caec57b13c1a_20180419_01:38:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_88411966-82e3-487c-9ae4-a88d334cf491_20180418_12:38:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_890d453c-76f2-4c2e-a88d-46ddd1d0e1da_20180419_02:48:09', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_89f2458d-4e84-4f50-bae0-448164b5fbc9_20180419_02:42:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_8cae1056-4ecc-439d-bdea-dff945724254_20180418_11:26:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_8e9367c0-61d4-4e32-bbbe-80ec8c18a423_20180419_02:46:08', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_8fa80205-e929-4379-b1d4-52795ab93eb0_20180418_12:07:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_9007a36b-94de-474f-b608-5412347b8bd9_20180418_11:41:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_90f3c58f-1126-4f62-86a2-11cad0bf19b9_20180419_01:55:07', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_91f643c1-6d8f-45bb-a1e5-550adf56f07f_20180419_02:40:14', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_93dff8e5-5864-473f-8ab4-28fcaff7f725_20180418_12:12:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_942c1046-46ce-4858-94cc-4503ef02d2f8_20180419_02:58:08', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_942fc1f9-4736-45f6-aa54-77fe92a37564_20180419_01:52:09', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_948ce246-04bd-420e-b8a9-1c58ce29aeb7_20180419_01:49:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_950768e0-818b-4319-b01c-399f6b350def_20180419_02:21:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_9aa88315-9219-437f-a9db-d281a38ccca0_20180418_12:39:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_9b5418d4-4c75-4ac9-b456-e2dfc2bfa208_20180418_11:50:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_9c0cf376-c896-4b97-bb68-a667e6721b52_20180418_12:58:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_9d3a407f-5ba7-4aae-9ccd-9efe33e24da8_20180419_02:34:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_9dc8eb59-1bd5-49e5-b9fb-c592983be522_20180419_02:20:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_9dee7644-d8d7-4863-b72c-e98881255417_20180418_12:57:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_9e9c028e-38f1-43e4-9068-4d45a70aaa4e_20180418_11:53:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_9fd12c86-2ddd-4e25-8887-32c326d91eb0_20180419_01:48:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_a3441a38-31d7-4955-a13b-299b1e955198_20180418_12:00:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_a40abc5b-13a7-49a2-a1fe-e632dab691a7_20180418_12:36:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_a65e1c6f-a322-4457-a439-ae0cfc6df81e_20180418_12:54:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_a6eb64dc-73bf-43ad-8b48-2cf2e9dd0ba2_20180418_12:50:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_a7365be9-4f45-47af-beac-c14b0bfdde1b_20180418_11:54:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_a93d9966-e667-40bc-8d19-0ec1b27a7712_20180418_12:22:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_aaed6b0a-8280-4266-bdb9-9ebc23e8690e_20180419_01:58:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_ab0b6783-6d23-4277-9f24-cd8a9510eadf_20180419_02:47:09', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_ac4ff98d-609e-429e-98ae-cdf817ccf89c_20180418_12:49:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_ae26878c-36bb-4bb5-b9ea-bbae05240a3c_20180418_12:25:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_ae4295dd-52c7-44b8-a468-e012bda8d86d_20180418_12:18:32', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_ae6bf2af-c965-49a8-b5a1-9c6f53a04967_20180418_11:30:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_ae80058a-1c8e-4a54-8af6-6de695609274_20180419_03:08:08', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_af35dbab-dd5f-4e34-8436-0639ec794573_20180418_11:52:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_af9aecb5-d29f-465b-a809-c274f6a93a70_20180419_02:10:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_b082c054-468f-4f16-aec0-f5b68894bac3_20180418_11:45:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_b131e8a3-0e8b-46e9-bc01-c7b23bda89a0_20180419_02:56:09', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_b2801738-b888-43de-ae68-a60f2ed9fd3b_20180418_12:15:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_b2ff238f-ddd4-4b18-a6bb-75e7235e333b_20180418_11:51:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_b306204f-faa9-4c68-abb4-7471fbaafe42_20180419_01:53:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_b406d57d-a557-42d4-b4f5-dd113ac858ee_20180418_11:37:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_b4acbf3d-c585-4e2d-81d7-428ee3678542_20180419_02:07:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_b55da9aa-646a-4db9-a676-0295746b5771_20180419_03:09:07', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_b812a38c-2520-4abf-9fa5-eacd910dcb5b_20180418_12:28:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_b9ee721f-838d-481d-bcf5-18c0e05dedd6_20180419_02:24:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_bb5049a1-f16f-4dfa-90d8-5c9f9ec11314_20180418_11:56:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_bd2c3be3-20d1-4a40-8be5-180b39e6742a_20180418_12:08:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_bdad4b97-62e8-4de4-835e-7cf99fa74224_20180419_02:01:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_be1e3afb-2ebd-437d-9355-1c0879596573_20180419_02:22:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_bf24d9b0-6c15-4ae7-8fef-246531fb8a13_20180418_12:10:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_c1006225-b273-4e28-9f14-c3efdd1802dd_20180418_12:51:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_c2da77ab-537a-4068-aff6-5d7630e618db_20180418_12:32:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_c38bf88f-67fa-4f64-ac53-58650eceeb35_20180419_02:23:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_c5847d62-cc38-4f31-b407-6005f2a4090d_20180418_11:24:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_c9a39be3-9554-4f32-81c5-1aea37889589_20180418_12:52:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_cb8c7ee0-e735-4fcb-a34b-eb646ceddb94_20180419_02:45:07', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_cbe149e5-8a5f-4a0c-83ef-a0f6837b8300_20180419_02:27:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_cdb6ed85-3cee-4523-8d18-baf523eaa564_20180419_01:39:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_d144f67a-c907-42e3-90a9-b99b02da3a1d_20180419_02:55:08', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_d1808b0a-7599-4c7e-8c15-4b6a5dbe38b0_20180419_01:45:08', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_d19cb4b8-f371-42a3-9b6e-6f65926c5dd4_20180418_11:42:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_d31f8925-bb74-486a-8cc2-e9d9ef6c0117_20180418_11:55:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_d3537530-8553-4970-a4e2-0d684d87db5a_20180419_01:54:07', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_d3b0e86b-ffe5-4805-af8e-8c8113c81841_20180418_12:43:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_d4a85eca-970a-4252-80a5-5d87eca8c9c6_20180419_03:02:10', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_d9234995-1726-45e0-b3f6-f4bd5811c819_20180419_02:02:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_de87be2e-6173-4ad8-8d96-cf1507e87dfc_20180419_03:12:08', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_dffe55fd-a372-4964-8a4e-1670f3aca40b_20180419_02:32:06', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_e1707a79-dcfc-44b5-b8bc-74a48d1dbe70_20180419_02:54:07', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_e1b3defa-6bbb-4969-a93a-cc9510b99f44_20180419_03:14:08', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_e2a875f8-59b1-430d-a96f-15587698d15d_20180419_01:37:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_e2bdde85-1526-47a1-9ac1-cae7ba92090f_20180419_03:07:10', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_e2dc5cea-e449-42ab-89dc-8d5aec9e8df0_20180418_12:56:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_e3b9456e-1e34-42d1-a881-ef7722c805db_20180418_12:02:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_e4318736-06bb-4641-af92-dfe876b86dcc_20180419_02:50:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_e461e37a-550c-4d80-9e35-dc7d20809909_20180418_12:45:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_e5a381da-a936-4ca9-be5a-66697b1ad055_20180419_02:33:07', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_e6f1b21b-f04f-4533-a079-0d701bc5de57_20180419_03:03:10', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_e8b0871f-0ded-4237-90c7-1d3adff6ca93_20180418_11:48:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_ea667c6f-2a04-4738-9100-ad5e987bafe2_20180419_03:00:09', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_eae1c606-52a6-4ee0-8214-aa2604a099d3_20180418_12:06:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_ecaa9e22-9061-479d-b78f-f7c48070bd52_20180418_12:01:30', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_ef4898b9-acd3-4331-9571-56446203c332_20180419_02:25:07', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_f07ddbac-9c0e-479f-8385-3fd577f61a0e_20180419_03:10:11', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_f1a76c07-4a80-4858-b6d3-76df10c2ccb6_20180418_11:28:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_f1fdc82b-882d-4e71-9620-2abe691439cb_20180418_12:11:32', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_f6832105-a989-4556-bdf7-836b683a3963_20180418_11:39:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_f6b641c9-d16c-4044-a493-d98e00813210_20180419_01:59:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_f726f34d-f585-41e3-8eba-5f289eacac88_20180419_01:47:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_f92f5553-4120-470b-b8e4-609da09b7896_20180418_11:22:32', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_f9eb079c-a175-4083-8faa-246989940396_20180419_01:44:05', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_fb1dabaa-2493-4fb3-82b1-218e7e0658aa_20180418_11:27:31', u'measure_01f0658b-f147-482b-bca9-f474a79320dc_fc31a147-3acb-4a45-929e-5a1d0f55e534_20180419_03:04:09']
(Pdb) p len(object_names)
203
4.3.2) 分析
for name in object_names:
ops.append(self.ioctx.aio_read(
name, bufsize, 0,
functools.partial(add_to_measures, name)
))
分析:
4.3.2.1)
(Pdb) p name
u'measure_01f0658b-f147-482b-bca9-f474a79320dc_014eacd6-fc6e-4c95-a8ff-f5f035166116_20180419_02:29:05'
其中self.ioctx.aio_read方法返回的结果样例如下:
(Pdb) p ops
[
(Pdb) p len(ops)
203
4.3.3) 分析
while ops:
op = ops.pop()
op.wait_for_complete_and_cb()
yield measures
分析:
4.3.3.1)
(Pdb) p op
(Pdb) p measures
[(Timestamp('2018-04-19 02:29:04.925214'), 4.785732057729687), (Timestamp('2018-04-19 01:31:57.004015'), 0.038519154635059005), (Timestamp('2018-04-18 12:20:30.465645'), 4.44247276885732), (Timestamp('2018-04-18 11:31:30.479519'), 4.630697293574757), (Timestamp('2018-04-19 02:30:04.802941'), 4.72629831122347), (Timestamp('2018-04-19 02:05:04.802314'), 4.899465312822332), (Timestamp('2018-04-19 03:06:07.265474'), 4.606219026908934), (Timestamp('2018-04-19 02:00:05.068829'), 4.985014177514147), (Timestamp('2018-04-19 01:42:04.879282'), 4.742942389783928), (Timestamp('2018-04-19 02:52:15.188803'), 4.164035307458439), (Timestamp('2018-04-19 02:09:04.734396'), 4.864783021566218), (Timestamp('2018-04-19 02:59:07.423876'), 4.762563821321215), (Timestamp('2018-04-18 11:43:30.481515'), 4.408241206936367), (Timestamp('2018-04-19 01:33:05.714625'), 4.775880629319837), (Timestamp('2018-04-19 01:57:05.898123'), 4.716622350276037), (Timestamp('2018-04-19 02:36:06.950848'), 5.005093670657329), (Timestamp('2018-04-18 11:58:30.633590'), 4.516955155071388), (Timestamp('2018-04-18 12:42:31.045815'), 4.521539770784765), (Timestamp('2018-04-18 12:33:30.688886'), 4.333678428076205), (Timestamp('2018-04-19 02:51:05.544821'), 4.754449419185193), (Timestamp('2018-04-19 02:39:07.148106'), 4.782045485846876), (Timestamp('2018-04-19 03:13:08.375850'), 4.648345083580437), (Timestamp('2018-04-18 12:41:30.668153'), 4.54115258806749), (Timestamp('2018-04-18 11:33:30.688185'), 4.414019789743311), (Timestamp('2018-04-19 01:43:05.305159'), 4.898563574013167), (Timestamp('2018-04-18 12:23:30.734720'), 4.253013016932383), (Timestamp('2018-04-18 12:17:30.498588'), 4.40150171902817), (Timestamp('2018-04-19 02:41:07.876715'), 4.592598710636145), (Timestamp('2018-04-18 12:30:32.318810'), 4.526942931338195), (Timestamp('2018-04-18 12:24:30.575716'), 4.478535083206168), (Timestamp('2018-04-18 12:14:30.652537'), 4.4852612819189055), (Timestamp('2018-04-19 02:18:05.632908'), 4.579984694779116), (Timestamp('2018-04-19 02:03:06.244939'), 4.987482665380561), (Timestamp('2018-04-18 12:44:30.664472'), 4.416037402325606), (Timestamp('2018-04-19 01:50:05.875464'), 4.603611199959174), (Timestamp('2018-04-18 12:48:30.966850'), 4.53283927488386), (Timestamp('2018-04-19 02:57:07.155598'), 4.392161876301822), (Timestamp('2018-04-19 01:41:04.579161'), 4.675403858911123), (Timestamp('2018-04-19 02:28:04.746333'), 4.869157240595231), (Timestamp('2018-04-18 12:21:30.613058'), 4.48897112166736), (Timestamp('2018-04-18 11:26:30.460244'), 4.370145812583828), (Timestamp('2018-04-19 02:46:06.781752'), 4.9701315664592425), (Timestamp('2018-04-18 11:41:31.389126'), 4.495910772251435), (Timestamp('2018-04-18 12:12:30.770047'), 4.388332822547206), (Timestamp('2018-04-19 02:34:05.902956'), 4.809059683880285), (Timestamp('2018-04-18 12:54:30.837672'), 4.3849504845245715), (Timestamp('2018-04-18 11:25:30.508011'), 4.421065221579064), (Timestamp('2018-04-19 02:06:04.642609'), 4.779388203216579), (Timestamp('2018-04-18 12:29:30.908714'), 4.5616065525447045), (Timestamp('2018-04-18 11:44:30.474084'), 4.467219931855227), (Timestamp('2018-04-19 03:05:08.649069'), 4.650786864968689), (Timestamp('2018-04-19 02:13:04.644721'), 4.610771573892409), (Timestamp('2018-04-19 02:53:09.873988'), 3.968167978219329), (Timestamp('2018-04-19 02:16:05.802788'), 4.859242678492049), (Timestamp('2018-04-19 02:17:08.645818'), 4.869275081102868), (Timestamp('2018-04-19 01:51:04.688511'), 4.692836268115815), (Timestamp('2018-04-19 01:34:04.961521'), 4.844135632016908), (Timestamp('2018-04-19 03:01:07.438882'), 4.702765182102024), (Timestamp('2018-04-18 12:34:31.000673'), 4.410414833173489), (Timestamp('2018-04-19 02:12:05.001775'), 4.639312620683888), (Timestamp('2018-04-18 12:31:30.480969'), 4.418680537632723), (Timestamp('2018-04-19 02:19:04.826207'), 4.797840377168367), (Timestamp('2018-04-18 11:29:30.787777'), 4.610401757926145), (Timestamp('2018-04-18 11:46:30.680061'), 4.45697877689301), (Timestamp('2018-04-19 01:32:07.086694'), 3.768839610980375), (Timestamp('2018-04-18 12:40:31.872507'), 4.384769774402943), (Timestamp('2018-04-18 11:38:30.352019'), 4.448383103047564), (Timestamp('2018-04-18 11:47:30.872743'), 4.4523684789456635), (Timestamp('2018-04-19 02:14:05.091567'), 4.681799278658807), (Timestamp('2018-04-18 11:36:30.646821'), 4.465378937700627), (Timestamp('2018-04-18 12:47:30.739698'), 4.336252993945942), (Timestamp('2018-04-19 02:48:08.437136'), 4.793475961021752), (Timestamp('2018-04-18 12:19:30.814182'), 4.369432000326476), (Timestamp('2018-04-19 01:55:05.626460'), 4.908340323457582), (Timestamp('2018-04-19 02:21:04.630142'), 4.878854940765438), (Timestamp('2018-04-18 12:49:30.463490'), 4.554879065439661), (Timestamp('2018-04-18 12:58:30.682071'), 4.461795572346424), (Timestamp('2018-04-19 03:08:07.287444'), 4.646870619194993), (Timestamp('2018-04-18 12:50:30.516833'), 4.279528618415131), (Timestamp('2018-04-19 02:11:05.294654'), 4.918873109181008), (Timestamp('2018-04-18 11:54:30.798116'), 4.354427676795497), (Timestamp('2018-04-18 11:40:30.444856'), 4.682041465410015), (Timestamp('2018-04-18 12:22:31.012354'), 4.4205813259810185), (Timestamp('2018-04-18 12:59:30.634203'), 4.436873070669113), (Timestamp('2018-04-19 01:58:05.742995'), 4.71218319257162), (Timestamp('2018-04-18 11:23:30.518884'), 4.687476499387155), (Timestamp('2018-04-18 12:18:31.538743'), 4.456083048937212), (Timestamp('2018-04-18 12:04:30.649676'), 4.536891483820938), (Timestamp('2018-04-18 12:15:30.979473'), 4.541918057963361), (Timestamp('2018-04-19 02:15:04.887946'), 5.150813563476811), (Timestamp('2018-04-18 12:16:30.519059'), 4.450820333214946), (Timestamp('2018-04-19 02:37:07.907961'), 4.7410381787602045), (Timestamp('2018-04-18 12:46:30.549483'), 4.321215294569709), (Timestamp('2018-04-18 12:09:30.931871'), 4.5007778568612995), (Timestamp('2018-04-18 11:49:30.571003'), 4.424979285252632), (Timestamp('2018-04-18 12:03:30.476317'), 4.479640168656953), (Timestamp('2018-04-18 11:34:30.589675'), 4.624258929118457), (Timestamp('2018-04-18 12:53:30.631747'), 4.355617843476749), (Timestamp('2018-04-19 02:08:04.916723'), 4.827346003653761), (Timestamp('2018-04-18 12:13:30.455375'), 4.406443070900105), (Timestamp('2018-04-19 03:11:07.255345'), 4.644506443984071), (Timestamp('2018-04-19 01:56:05.473527'), 4.6785918514603235), (Timestamp('2018-04-19 02:35:06.212724'), 4.841670092314067), (Timestamp('2018-04-18 12:35:30.810640'), 4.430699652450904), (Timestamp('2018-04-19 01:36:04.565389'), 4.696738395311353), (Timestamp('2018-04-18 12:38:30.818006'), 4.601354800659197), (Timestamp('2018-04-18 12:07:30.711686'), 4.4456014478341555), (Timestamp('2018-04-19 02:40:13.223562'), 4.721874337121487), (Timestamp('2018-04-19 02:43:06.045525'), 4.687168705675826), (Timestamp('2018-04-19 01:52:05.673186'), 4.7224979062362795), (Timestamp('2018-04-19 01:40:05.974598'), 4.865087451661152), (Timestamp('2018-04-18 12:39:30.523790'), 4.32118938426468), (Timestamp('2018-04-19 02:31:07.593714'), 4.55480935073056), (Timestamp('2018-04-18 12:25:30.682680'), 4.425443946894406), (Timestamp('2018-04-18 11:59:30.672687'), 4.663627769085202), (Timestamp('2018-04-18 11:50:30.527954'), 4.686695959572727), (Timestamp('2018-04-19 01:48:04.862497'), 4.740639922515654), (Timestamp('2018-04-18 11:45:31.222744'), 4.329313601320588), (Timestamp('2018-04-18 11:51:30.614182'), 4.476899431929726), (Timestamp('2018-04-19 03:09:07.147123'), 4.610783161733961), (Timestamp('2018-04-18 11:56:30.568675'), 4.369053518812446), (Timestamp('2018-04-18 12:08:30.497873'), 4.332104337077058), (Timestamp('2018-04-19 02:01:05.311450'), 4.9466639242007755), (Timestamp('2018-04-18 11:52:30.591304'), 4.468370456321661), (Timestamp('2018-04-19 02:23:05.061099'), 4.980562563369955), (Timestamp('2018-04-18 12:37:31.487654'), 4.482132078975561), (Timestamp('2018-04-18 11:32:30.425655'), 4.520725072254866), (Timestamp('2018-04-18 11:57:30.416034'), 4.361094697595594), (Timestamp('2018-04-18 11:35:30.405573'), 4.56400403785629), (Timestamp('2018-04-18 12:26:31.356136'), 4.400606420046354), (Timestamp('2018-04-18 12:27:31.703994'), 4.358066859638995), (Timestamp('2018-04-19 01:35:04.949550'), 4.900977826759402), (Timestamp('2018-04-19 02:26:05.124636'), 4.627460507516731), (Timestamp('2018-04-19 01:46:04.910136'), 4.83613552050249), (Timestamp('2018-04-19 02:44:05.303248'), 4.893876870699199), (Timestamp('2018-04-19 02:04:04.999870'), 4.816617008706895), (Timestamp('2018-04-18 12:51:30.705844'), 4.502483019699393), (Timestamp('2018-04-18 12:55:30.475755'), 4.326094787453178), (Timestamp('2018-04-18 11:37:30.555011'), 4.640433970714188), (Timestamp('2018-04-18 12:52:30.479546'), 4.533766371037216), (Timestamp('2018-04-19 02:55:07.756245'), 4.817978266100042), (Timestamp('2018-04-18 11:42:30.593685'), 4.425334880038545), (Timestamp('2018-04-19 02:02:06.094354'), 4.6559144327819535), (Timestamp('2018-04-19 03:12:07.278837'), 4.7981213755440955), (Timestamp('2018-04-19 03:07:09.183825'), 4.570535155240164), (Timestamp('2018-04-18 12:06:30.652321'), 4.319691314525096), (Timestamp('2018-04-18 11:22:31.211920'), 4.61634070520078), (Timestamp('2018-04-19 02:32:06.147551'), 4.4745146248912775), (Timestamp('2018-04-19 01:54:06.950822'), 4.915800608894196), (Timestamp('2018-04-19 03:10:09.552766'), 4.454725352321103), (Timestamp('2018-04-19 03:03:07.560772'), 4.715476374442231), (Timestamp('2018-04-18 11:28:30.923146'), 4.530601493071297), (Timestamp('2018-04-19 02:27:04.777023'), 4.626805629756275), (Timestamp('2018-04-19 03:00:08.112084'), 4.514880386647765), (Timestamp('2018-04-19 02:07:05.670918'), 4.801050607513965), (Timestamp('2018-04-18 12:01:30.490037'), 4.38055539601681), (Timestamp('2018-04-19 02:24:05.322718'), 5.028075996431493), (Timestamp('2018-04-19 01:59:05.289661'), 5.004478336368991), (Timestamp('2018-04-19 02:22:05.027720'), 4.801517040964788), (Timestamp('2018-04-19 01:44:04.922816'), 4.847557159114791), (Timestamp('2018-04-18 12:32:30.462913'), 4.384652821522423), (Timestamp('2018-04-18 11:27:30.445525'), 4.51777495215868), (Timestamp('2018-04-19 03:04:08.444203'), 4.598952381642223), (Timestamp('2018-04-18 11:24:31.246275'), 4.446099125187183), (Timestamp('2018-04-19 02:45:06.018773'), 4.743432589934781), (Timestamp('2018-04-19 01:39:04.721845'), 4.92302276209809), (Timestamp('2018-04-18 12:28:30.842157'), 4.413393767405322), (Timestamp('2018-04-18 12:43:31.108817'), 4.511928990828664), (Timestamp('2018-04-19 02:54:07.149911'), 4.417213843939276), (Timestamp('2018-04-19 03:14:07.646178'), 4.690373908509499), (Timestamp('2018-04-18 12:45:30.844057'), 4.536422110587834), (Timestamp('2018-04-19 02:33:07.055693'), 4.7941045386017525), (Timestamp('2018-04-18 11:48:30.683717'), 4.647976473347516), (Timestamp('2018-04-18 12:11:31.749911'), 4.398298319635849), (Timestamp('2018-04-18 11:39:31.282629'), 4.628215604603334), (Timestamp('2018-04-18 12:05:30.694333'), 4.396727588934349), (Timestamp('2018-04-19 02:38:07.341063'), 4.71117930206638), (Timestamp('2018-04-18 11:21:30.557818'), 4.488482255300747), (Timestamp('2018-04-19 01:38:05.002438'), 4.655968492662127), (Timestamp('2018-04-19 02:42:05.454576'), 3.8382808281120413), (Timestamp('2018-04-19 02:58:08.422042'), 4.586523742099345), (Timestamp('2018-04-19 01:49:04.619205'), 4.769339033870473), (Timestamp('2018-04-18 11:55:30.372571'), 4.414643826787841), (Timestamp('2018-04-19 02:49:07.672686'), 4.878826988185303), (Timestamp('2018-04-19 02:20:04.780033'), 4.937132786154464), (Timestamp('2018-04-18 12:57:31.288942'), 4.346607003997397), (Timestamp('2018-04-18 12:00:30.908551'), 4.432575251182584), (Timestamp('2018-04-18 12:36:30.579141'), 4.467236011155776), (Timestamp('2018-04-19 02:47:07.521007'), 4.758043212746024), (Timestamp('2018-04-18 11:30:30.661314'), 4.5429084972882094), (Timestamp('2018-04-19 02:10:05.321568'), 4.720471191492483), (Timestamp('2018-04-19 02:56:07.959237'), 4.617710694511661), (Timestamp('2018-04-19 01:53:04.702569'), 4.8450447127322995), (Timestamp('2018-04-18 12:10:30.589894'), 4.458746479077927), (Timestamp('2018-04-19 01:45:07.839775'), 4.49799234575212), (Timestamp('2018-04-19 03:02:07.333559'), 4.791744682085855), (Timestamp('2018-04-19 01:37:04.864574'), 4.610344235995893), (Timestamp('2018-04-18 12:56:30.551912'), 4.477649926908607), (Timestamp('2018-04-18 12:02:30.426851'), 4.504743945849373), (Timestamp('2018-04-19 02:50:05.180313'), 4.729807404503059), (Timestamp('2018-04-19 02:25:06.345096'), 4.8506795326789796), (Timestamp('2018-04-19 01:47:04.744031'), 4.7297606147819735), (Timestamp('2018-04-18 11:53:30.629474'), 4.363890504990408)]
(Pdb) p len(measures)
203
4.3.4) 分析
def add_to_measures(name, comp, data):
if name in tmp_measures:
tmp_measures[name] += data
else:
tmp_measures[name] = data
if len(data) < bufsize:
measures.extend(self._unserialize_measures(name,
tmp_measures[name]))
del tmp_measures[name]
else:
ops.append(self.ioctx.aio_read(
name, bufsize, len(tmp_measures[name]),
functools.partial(add_to_measures, name)
))
4.3.4.1）分析
(Pdb) add_to_measures ############ name:measure_01f0658b-f147-482b-bca9-f474a79320dc_014eacd6-fc6e-4c95-a8ff-f5f035166116_20180419_02:29:05, data: type:
add_to_measures ############ data: content:
0�R_�&�x���$@
(Pdb) add_to_measures ############ name:
measure_01f0658b-f147-482b-bca9-f474a79320dc_0173ad9a-2553-4429-a475-11cc9f41e79e_20180419_01:31:58,
data: type:
add_to_measures ############ data: content:
���2A�&Ȳ�Zȸ�?
这个实际上就是建立:
对象名称到对象的监控数据的映射
4.4）继续回到process_measure_for_metric方法
总结:
process_measure_for_metric(self, metric):
1. 实际上从incoming storage根据监控项id,找到measures-meatricId开头的所有该监控项的监控数据
2. 返回待处理监控项对应的监控数据列表，每个元素是时间戳和对应的值
样例：[(Timestamp('2018-04-19 02:29:04.925214'), 4.785732057729687)]
4.5) 继续回到gnocchi/storage/_carbonara.py中CarbonaraBasedStorage类的
process_new_measures方法
具体代码如下:
def process_new_measures(self, indexer, metrics_to_process,
sync=False):
metrics = indexer.list_metrics(ids=metrics_to_process)
# This build the list of deleted metrics, i.e. the metrics we have
# measures to process for but that are not in the indexer anymore.
deleted_metrics_id = (set(map(uuid.UUID, metrics_to_process))
- set(m.id for m in metrics))
for metric_id in deleted_metrics_id:
# NOTE(jd): We need to lock the metric otherwise we might delete
# measures that another worker might be processing. Deleting
# measurement files under its feet is not nice!
try:
with self._lock(metric_id)(blocking=sync):
self.incoming.delete_unprocessed_measures_for_metric_id(
metric_id)
except coordination.LockAcquireFailed:
LOG.debug("Cannot acquire lock for metric %s, postponing "
"unprocessed measures deletion", metric_id)
for metric in metrics:
lock = self._lock(metric.id)
# Do not block if we cannot acquire the lock, that means some other
# worker is doing the job. We'll just ignore this metric and may
# get back later to it if needed.
if not lock.acquire(blocking=sync):
continue
try:
locksw = timeutils.StopWatch().start()
LOG.debug("Processing measures for %s", metric)
# process_measure_for_metric(self, metric):返回待处理监控项对应的监控数据列表，
# 每个元素是时间戳和对应的值，样例：[(Timestamp('2018-04-19 02:29:04.925214'), 4.785732057729687)]
with self.incoming.process_measure_for_metric(metric) \
as measures:
self._compute_and_store_timeseries(metric, measures)
LOG.debug("Metric %s locked during %.2f seconds",
metric.id, locksw.elapsed())
except Exception:
LOG.debug("Metric %s locked during %.2f seconds",
metric.id, locksw.elapsed())
if sync:
raise
LOG.error("Error processing new measures", exc_info=True)
finally:
lock.release()
分析:
4.5.1) 刚才已经分析了process_measure_for_metric方法
该方法的作用是:
返回待处理监控项对应的监控数据列表，
每个元素是时间戳和对应的值，样例：[(Timestamp('2018-04-19 02:29:04.925214'), 4.785732057729687)]
4.5.2) 接下来调用了_compute_and_store_timeseries方法
具体分析，参见5
5 分析_compute_and_store_timeseries方法
待补充