一种在 python 中用 asyncio 和协程实现的IO并发
An IO-concurrency implement by async && coroutine in python
1 此篇文字围绕的主题
基于IO驱动程序1，本机从IO读取或发送数据——读/写IO的有效方法。
+--------------------------+ +----+
| | +==>| |
| +--+ +---+ | | +----+ 1
| +-----+ | | | I | | | .
| | CPU |<==>|MM|<==>| |<==+ .
| +-----+ | | | O | | | .
| +--+ +---+ | | +----+
| | +==>| |
+--------------------------+ +----+ n
local-computer remote-computers
MM: memory。
<==>: 数据流向。
2 IO多路复用 ≈ IO并发
基于IO驱动程序对IO的读/写，可看作是对IO缓冲区的读/写。
+--------------------------------------+ +----+
| | +===>| |
| +--+ +---+ | |1路 +----+ 1
| +==>|b1|<==>| | | |
| +-----+ concurrency| |--| | I | | | .
| | CPU |<===========+ |MM| | |<===+ .
| +-----+ | |--| | O | | | .
| +==>|bn|<==>| | | |
| +--+ +---+ | | +----+
| | +===>| |
+--------------------------------------+ n路 +----+ n
local-computer remote-computers
b: IO buffer。
本机IO能实时接收远程计算机传输来的数据，（对于诸如网卡类IO，）当数据目的地为本机时 其会被上传到IO缓冲区。
IO多路复用指本机与远程计算机建立多路数据传输的软连接（每一路数据对应一个IO缓冲区），CPU通过调度CPU并发单位2 并发读/写IO缓冲区。
由CPU调度CPU并发单位读/写IO缓冲区实现了IO硬件并发传输各路数据。相比于CPU层面的并发，IO并发粒度更大（一定程度上削弱了IO的并发现象）。此篇文字将IO多路复用称为大粒度并发——这是本小节标题“IO多路复用 ≈ IO并发”的由来。
另外，此篇文字认为构成IO并发粒度比CPU并发粒度大的原因主要有三个。
[1] 数据粒度——让IO一次性（不切换）发送某路数据的量较大（如一整个缓冲区）；
[2] IO速度比CPU低几个数量级；
[3] 较本机数据的传输时间，来自远程计算机数据的传输时间较长（相比前两个原因，这个是主要原因，可参考后续程序例子）。
利用IO并发时主要是为了提升IO利用率——当某路数据中断时去处理另一路数据。
3 IO同步并发 && IO异步并发
IO同步指一个CPU并发单位会阻塞等待IO缓冲区可用并完成对IO的读/写。
IO异步指若IO缓冲区可用则完成对IO的读/写；若IO缓冲区不可用则跳过IO读/写 继续调度并发单位后续指令运行，待IO缓冲区可用时再完成对其的读/写操作。
3.1 IO同步并发
通过多个CPU并发单位读/写某IO下的多路IO缓冲区可实现IO同步并发。
当无不可用IO缓冲区时，各IO缓冲区由CPU调度CPU并发单位 并发读/写；当 当前CPU并发单位读/写的IO缓冲区不可用时 则调度另一CPU并发单位尝试读/写相应的IO缓冲区。
如由n个线程分别读/写m(m >= n)路IO数据的IO缓冲区。
3.2 IO异步并发
由于CPU与IO的速度差异3，除能通过CPU多并发单位实现IO异步并发外，单CPU并发单位也可以实现异步IO在IO层面的并发。
最直接的方式是让CPU轮询读/写IO缓冲区：
for (io = data_conn->start; ; io = data_conn->next)
read or write current io buffer
create one data connection when data connections are less than max
考虑本机IO利用率满（远程计算机将本机IO填满且都上传到IO缓冲区）的场景，当本机可调度并发量未饱和情况下，轮询方式可以满足异步IO并发的读/写。
4 IO并发——IO异步与协程的搭配
若单CPU并发已能满足当前场景下的IO并发，那么实现目标最友好方式就是“单CPU并发+IO异步”的方式啦4。
对于比“CPU轮询+IO异步”稍复杂的IO并发业务，可尝试在单CPU并发单位内使用协程。协程可在应用程序中创建和调度。除具调度式并发特点外，其还有两个优点：
[1] 对CPU层面并发单位的友好性——其创建和调度开销不在内核上；
[2] 上下文切换开销比线程更小，协程执行效率可接近于3.2中的轮询方式。
额，到此为止，这篇文字个人理解文字的戏也太多了点吧…在接下来的篇章里不如直接写个例子算了。
5 写个小例子吧
用什么写个什么呢？
用python写个请求OneNET视频平台部分HTTP-APIs的例子吧。嗯，这种网络IO场景适合用“IO异步+协程”来实现。
5.1 初定功能
[1] 比较OneNET-Video-Platform-HTTP-APIs 同步请求和异步请求的时间差异；
[2] 在异步情况下保证某些HTTP-APIs请求的先后顺序；
[3] 提供参数配置入口，如命令行参数，配置文件。
5.2 asyncio && aiohttp
此文使用python的asyncio和aiohttp来完成HTTP-API的请求。这两个模块对IO异步的设置、协程实现和调度进行了封装，下一篇文字再探讨协程相关内容吧。
#!/usr/bin/python3
# -*- coding: utf-8 -*-
"""
asyctn.py, namely async coroutine,
wrapper-functions for aiohttp contained.
lxr, 2019.12.15
"""
import json
import asyncio
try:
from aiohttp import ClientSession
except:
exit('you should run following command in your own environment ' +
'to install the third party libraries\npip3(or pip) install -r requirements.txt')
_index = 0
async def _w_method(asyctn_method, url,
hdrs=None, params=None, data=None, timeout=None):
''' wrapper for the asyctn_method,e.g. aiohttp.ClientSession.get '''
async with asyctn_method(url = url,
headers = hdrs,
params = params,
data = data,
timeout = timeout) as resp:
try: return await resp.text(encoding='utf-8')
except Exception as e: print(e)
return None
def _method_mapping(session, method):
''' mapping string 'get', 'post', 'delete', 'put' to session.get,
session.post, session.delete, session.put respectively.
'''
if method == 'get':
return session.get
elif method == 'post':
return session.post
elif method == 'delete':
return session.delete
elif method == 'put':
return session.put
else:
return None
async def w_request(url, method,
hdrs=None, params=None, data=None, timeout=None):
''' wrapper for aiohttp.ClientSession.method(),
make it be coroutine && async
'''
global _index
async with ClientSession() as session:
method = _method_mapping(session, method)
_index += 1
index = _index
try: resp = await _w_method(method, url,
hdrs=hdrs, params=params,
data=data, timeout=timeout)
except Exception as e: print(e);return None
return resp
class AsyCtnLoop():
''' wrapper for asyncio '''
def __init__(self):
try:
self.loop = asyncio.get_event_loop()
except Exception as e:
exit(e)
def w_loop(self, requests):
''' to run ont requests asynchronously by asyncio '''
try:
self.loop.run_until_complete(asyncio.wait(requests))
except Exception as e:
print(e)
else:
print('')
def w_close(self):
self.loop.close()
if __name__ == '__main__':
''' for testing asyctn.py '''
async def w_req_test_f(index):
resp = await w_request('https://mijisou.com/', 'get')
print(f'{index} {resp[0:3] if resp != None else None}')
async def w_req_test_s(index):
resp = await w_request('https://mijisou.com/', 'get')
print(f'{index} {resp[0:3] if resp != None else None}')
requests_f = []
requests_s = []
for i in range(5):
requests_f.append(w_req_test_f(i + 1))
for i in range(5, 10):
requests_s.append(w_req_test_s(i + 1))
asyctn_loop = AsyCtnLoop()
asyctn_loop.w_loop(requests_f)
asyctn_loop.w_loop(requests_s)
asyctn_loop.w_close(
由于asyctn.py包含第三方模块aiohttp，为了给使用者在缺乏aiohttp库时一个友好提示，此文用pipreqs为本python工程生成了requirements.txt。
先看看asyctn.py的运行体验吧。
> python asyctn.py
2 <!DOCTYPE
1 <!DOCTYPE
4 <!DOCTYPE
3 <!DOCTYPE
5 <!DOCTYPE
10 <!DOCTYPE
8 <!DOCTYPE
7 <!DOCTYPE
9 <!DOCTYPE
6 <!DOCTYPE
根据运行体验结果，可以明确两个结果：
[1] 组内API以异步方式执行；
[2] 组间API顺序执行。
5.3 提供配置入口
配置入口以命令行参数形式提供还是以配置文件提供呢？此文特别不擅长了解到他人喜好，为了具有尽量友好的理念，只有两种方式都提供了。
此文选择argparse作为命令行参数解析是因为其是python的内置模块，不涉及第三方模块的管理。
#!/usr/bin/python3
# -*- coding: utf-8 -*-
"""
ontApi_argparser.py,
to parse configurations from command or config file.
lxr, 2019.12.15
"""
import argparse
import json
import sys
import os
class OntApiCmdArgs():
''' to parse the command arguments for
OneNET Video-Platform by argparse module.
instance variables PARSER, ARGS, HAS_CFG, HAS_ARGS,
can be used directlly by outside.
'''
def __init__(self):
self.HAS_CFG = False
self.HAS_ARGS = False
self.PARSER = self.ARGS = None
parser = self.__create_cmdarg_parser()
self.__add_optargs(parser)
args = self.__parse_cmdargs(parser)
self.__default_configurations(args)
self.__update_cfg_depend_on_user(args)
self.__tips_for_logs_output_file(args)
self.ARGS = args
self.PARSER = parser
def __create_cmdarg_parser(self):
ARGS = argparse.ArgumentParser(
description = self._descriprion + '-' * len(self._descriprion),
epilog = self._epilog + '-' * len(self._epilog),
formatter_class = argparse.RawTextHelpFormatter)
return ARGS
def __add_optargs(self, parser):
self.__add_version_optarg(parser)
self.__add_platform_optargs(parser)
self.__add_device_optargs(parser)
self.__add_log_optargs(parser)
self.__add_cfg_optargs(parser)
self.__add_default_optargs(parser)
def __add_version_optarg(self, parser):
parser.add_argument('-v', '--version',
action = 'version',
version = f'{sys.argv[0]} 0.0.1' )
def __add_platform_optargs(self, parser):
group = parser.add_argument_group(self._optarg_desc['platform'])
group.add_argument('-s',
metavar = ' IP',
dest = 'ip',
type = str,
default = self._default_cfg['ip'],
help = self._help['ip'] )
group.add_argument('-p',
metavar = ' PORT',
dest = 'port',
type = int,
default = self._default_cfg['port'],
help = self._help['port'] )
group.add_argument('-hdr',
metavar = 'HEADER',
dest = 'header',
type = str,
default = self._default_cfg['header'],
help = self._help['header'])
def __add_device_optargs(self, parser):
group = parser.add_argument_group(self._optarg_desc['device'])
group.add_argument('-did',
metavar = 'DEVICE_ID',
dest = 'did',
type = int,
default = self._default_cfg['did'],
help = self._help['did'] )
group.add_argument('-pid',
metavar = 'PRODUCT_ID',
dest = 'pid',
type = int,
default = self._default_cfg['pid'],
help = self._help['pid'] )
def __add_log_optargs(self, parser):
def str2bool(_str):
return True if _str.lower() not in ['false', '0'] else False
group = parser.add_argument_group(self._optarg_desc['log'])
group.add_argument('-req',
metavar = '0/1',
dest = 'req',
type = str2bool,
default = self._default_cfg['req'],
help = self._help['req'] )
group.add_argument('-rep',
metavar = '0/1',
dest = 'rep',
type = str2bool,
default = self._default_cfg['rep'],
help = self._help['rep'] )
group.add_argument('-jsi',
metavar = 'NR',
dest = 'jsi',
type = int,
default = self._default_cfg['jsi'],
choices = range(1, 8),
help = self._help['jsi'] )
group.add_argument('--log',
metavar = 'LOGFILE',
type = argparse.FileType('w', encoding='UTF-8'),
default = self._default_cfg['log'],
help = self._help['log'] )
def __add_cfg_optargs(self, parser):
group = parser.add_argument_group(self._optarg_desc['cfg'])
group.add_argument('--cfg',
metavar = 'CONFIGFILE',
type = str,
default = self._default_cfg['cfg'],
help = self._help['cfg'])
def __add_default_optargs(self, parser):
group = parser.add_argument_group(self._optarg_desc['dft'])
group.add_argument('-dft', '--default',
action = 'store_true',
help = self._help['dft'])
def __parse_cmdargs(self, parser):
try: return parser.parse_args()
except IOError as e: exit(e)
def __tips_for_logs_output_file(self, args):
if args.__dict__['log'].name != self._default_cfg['log'].name:
tips = f"\nplease to check the * {args.__dict__['log'].name} * for results"
print(tips, '-' * len(tips), sep='\n')
def __default_configurations(self, args):
if args.default:
_default_cfg_dict = {
'IP': self._default_cfg['ip'],
'PORT': self._default_cfg['port'],
'HEADER': self._default_cfg['header'],
'DEVICE_ID': self._default_cfg['did'],
'PRODUCT_ID': self._default_cfg['pid'],
'req': self._default_cfg['req'],
'rep': self._default_cfg['rep'],
'jsi': self._default_cfg['jsi'],
'log': self._default_cfg['log'].name,
'cfg': self._default_cfg['cfg'],
}
print(json.dumps(_default_cfg_dict, indent=4))
dft_tips = 'the default configurations here you are\n'
exit(dft_tips + '-' * len(dft_tips)) if 2 == len(sys.argv) else None
def __update_cfg_depend_on_user(self, args):
if 1 < len(sys.argv):
self.HAS_ARGS = True
yon = input(f'update the command arguments to the * {args.cfg} * (y/n)?')
if yon.lower() in ('y', 'yes'):
self.__update_cfg(args.cfg, args.__dict__)
if os.path.exists(args.cfg):
self.HAS_CFG = True
def __update_cfg(self, cfg, cfg_v):
v = {
'ip': cfg_v['ip'],
'port': cfg_v['port'],
'header': cfg_v['header'],
'pid': cfg_v['pid'],
'did': cfg_v['did'],
'req': cfg_v['req'],
'rep': cfg_v['rep'],
'jsi': cfg_v['jsi'],
'log': cfg_v['log'].name,
'cfg': cfg_v['cfg']
}
with open(cfg, mode='w', encoding='utf-8') as f:
try: f.write(json.dumps(v, indent=4))
except Exception as e: exit(e)
print(f"the command arguments have been updated to the * {cfg_v['cfg']} *")
''' the default configurations of OneNET-Video-Platform '''
_default_cfg = {
'ip': 'api.heclouds.com',
'port': 80,
'header': 'api-key:IqvnbbtJWUN23SHQRGFyfW=8ICc=',
'pid': 174585,
'did': 527776306,
'req': True,
'rep': False,
'jsi': 1,
'log': sys.stdout,
'cfg': 'ontApiTest_config.json'
}
_help = {
'ip': "set API's ip to be IP",
'port': "set API's header to be HEADER",
'header': "set API's header to be HEADER",
'did': "set API's device id to be DEVICE_ID",
'pid': "set API's product id to be PRODUCT_ID",
'req': 'set flag which decides whether to show requests',
'rep': 'set flag which decides whether to show responses',
'jsi': 'set NR-indent to show responses in json format',
'log': 'set LOGFILE to show the log of this program',
'cfg': 'set config file to be CONFIGFILE',
'dft': 'show the default configurations',
}
_optarg_desc = {
'platform': 'optional arguments on OneNET Video-Platform',
'device': 'optional arguments on device',
'log': 'optional arguments on log',
'cfg': 'optional arguments on config file',
'dft': 'optional arguments on default',
}
_descriprion = f'{sys.argv[0]} arguments manual\n'
_epilog = 'those arguments can be configured in CONFIGFILE statically too\n'
if __name__ == '__main__':
''' for testing ontApi_argparser.py '''
cmdargs = OntApiCmdArgs()
cmdargs.ARGS.log.write(f'{cmdargs.ARGS.__dict__}')
cmdargs.ARGS.log.close
根据ontApi_argparser.py的运行体验看看其大概能做些什么吧。
[1] help
> python ontApi_argparser.py -h
usage: ontApi_argparser.py [-h] [-v] [-s IP] [-p PORT] [-hdr HEADER]
[-did DEVICE_ID] [-pid PRODUCT_ID] [-req 0/1]
[-rep 0/1] [-jsi NR] [--log LOGFILE]
[--cfg CONFIGFILE] [-dft]
ontApi_argparser.py arguments manual
--------------------------------------
optional arguments:
-h, --help show this help message and exit
-v, --version show program's version number and exit
optional arguments on OneNET Video-Platform:
-s IP set API's ip to be IP
-p PORT set API's header to be HEADER
-hdr HEADER set API's header to be HEADER
optional arguments on device:
-did DEVICE_ID set API's device id to be DEVICE_ID
-pid PRODUCT_ID set API's product id to be PRODUCT_ID
optional arguments on log:
-req 0/1 set flag which decides whether to show requests
-rep 0/1 set flag which decides whether to show responses
-jsi NR set NR-indent to show responses in json format
--log LOGFILE set LOGFILE to show the log of this program
optional arguments on config file:
--cfg CONFIGFILE set config file to be CONFIGFILE
optional arguments on default:
-dft, --default show the default configurations
those arguments can be configured in CONFIGFILE statically too
---------------------------------------------------------------
[2] 查看默认配置
> python ontApi_argparser.py --default
{
"IP": "api.heclouds.com",
"PORT": 80,
"HEADER": "api-key:IqvnbbtJWUN23SHQRGFyfW=8ICc=",
"DEVICE_ID": 527776306,
"PRODUCT_ID": 174585,
"req": true,
"rep": false,
"jsi": 1,
"log": "" ,
"cfg": "ontApiTest_config.json"
}
the default configurations here you are
----------------------------------------
[3] 同步新配置到配置文件中
ontApi_argparser.py将默认配置内置在argparse中，且默认不生成配置文件（曾有人挑剔说配置文件是尾巴）。
若习惯或不得已要使用配置文件，可通过以下两种方式生成得到配置文件：
[1] 使用–default查看配置文件名和内容格式，然后手动配置；
[2] 在命令行中指定某配置时同步配置到配置文件中（推荐）。
> python ontApi_argparser.py -req 0
update the command arguments to the * ontApiTest_config.json * (y/n)?y
the command arguments have been updated to the * ontApiTest_config.json *
[4] 生成日志文件
除help外内容外，程序打印内容可输出到指定的日志文件中。
> python ontApi_argparser.py --log output.log
update the command arguments to the * ontApiTest_config.json * (y/n)?y
the command arguments have been updated to the * ontApiTest_config.json *
please to check the * output.log * for results
-----------------------------------------------
5.4 抽取HTTP-API通用部分
为了后期更方便的编写各个API请求，前期先观察下有没有通用部分吧，免得增大重复工作量占比。
#!/usr/bin/python3
# -*- coding: utf-8 -*-
"""
ontApi_comm.py,
wrapper for common code for OneNET-Video-Platform-HTTP-APIs.
lxr, 2019.12.15
"""
import json
import ontApi_argparser as _argparser
class Common():
''' instance variables
CMD the information of commad argument-parsing
LOG output file of log
CFGN the dict of command arguments
'''
def __init__(self):
self.CMD = _argparser.OntApiCmdArgs()
self.LOG = self.CMD.ARGS.log
self.CFGN = self.__get_cfgn()
def __get_cfgn(self):
if (not self.CMD.HAS_ARGS) and self.CMD.HAS_CFG:
cfgn = self.__get_cfgn_from_cfg(self.CMD.ARGS.cfg)
else: cfgn = self.__get_cfgn_from_cmd(self.CMD.ARGS)
self.__check_cfgn_if_valid(cfgn)
return cfgn
def __get_cfgn_from_cfg(self, cfg):
with open(cfg, mode='r', encoding='utf-8') as f:
try: return json.loads(f.read())
except Exception as e: exit(f'{cfg}: {e}')
def __get_cfgn_from_cmd(self, args):
return args.__dict__
@staticmethod
def __check_cfgn_if_valid(cfgn):
necessary_kyes = {'ip', 'port', 'header', 'pid', 'did'}
if not necessary_kyes < set(cfgn.keys()):
exit(f"three are no necessary fields in {cfgn['cfg']}")
# the other checks
''' to get configuration by below method besides by CFGN '''
def get_url_comm_part(self):
addr, port = self.CFGN['ip'], self.CFGN['port']
if not any(prtcl in addr for prtcl in ['http://', 'https://']):
addr = 'http://' + addr
return '%s:%d/%s' % (addr, port, 'ipc/video')
''' to get the configuration by following method '''
def get_header(self):
return self.CFGN['header']
def get_product_id(self):
return self.CFGN['pid']
def get_device_id(self):
return self.CFGN['did']
def get_request_showing_flag(self):
return self.CFGN['req'] if 'req' in self.CFGN.keys() else True
def get_response_showing_flag(self):
return self.CFGN['rep'] if 'rep' in self.CFGN.keys() else False
def get_response_json_indent(self):
return self.CFGN['jsi'] if 'jsi' in self.CFGN.keys() else 1
if __name__ == '__main__':
''' for testing ontApi_comm.py '''
comm = Common()
comm.LOG.write(f'{comm.CFGN}')
comm.LOG.close()
ontApi_comm.py具ontApi_argparser.py所有功能，以类方法对外提供配置信息。
简单运行体验下。
> python ontApi_comm.py
{'ip': 'api.heclouds.com', 'port': 80, 'header': 'api-key:IqvnbbtJWUN23SHQRGFyfW=8ICc=', 'did': 527776306, 'pid': 174585, 'req': True, 'rep': False, 'jsi': 1, 'log': <_io.TextIOWrapper name='' mode='w' encoding='utf-8'>, 'cfg': 'ontApiTest_config.json', 'default': False}
5.5 包装API && 异步请求
在正式编写HTTP请求API之前再写一些包装函数。
def _show_log(index, url, resp, dtm, comm):
''' to show some information according to configurations '''
if comm.get_request_showing_flag():
comm.LOG.write(f'{index} {url} {"%8.2f" % (dtm)}ms\n')
indent = comm.get_response_json_indent()
indent = None if indent == 1 else indent
json_flag = 1
try:
resp_dict = json.loads(resp, encoding='utf-8')
if resp_dict['errno'] != 0 and resp_dict['errno'] != 200:
comm.LOG.write(json.dumps(resp_dict, indent=indent) + '\n\n')
return None
except Exception as e:
json_flag = 0
resp = resp if resp != None else 'None'
comm.LOG.write('none-json response: \n' + resp + '\n\n')
if comm.get_response_showing_flag() and json_flag:
comm.LOG.write(json.dumps(json.loads(resp), indent=indent) + '\n\n')
_time_sync_total = 0
def _log_decorator(api_fn):
''' one decorator for OneNET-Video-Platform-API to show logs '''
async def _w(index, comm):
global _time_sync_total
ltm = time.time_ns()
url, resp = await api_fn(index, comm)
dtm = (time.time_ns() - ltm) / 1.0e+6
_time_sync_total += dtm
_show_log(index, url, resp, dtm, comm)
return _w
def _get_url_header(url_path, comm):
''' wrapper for the whole url and headers for API '''
url = comm.get_url_comm_part() + url_path
hdrs = {comm.get_header().split(':', )[0] : comm.get_header().split(':', )[1]}
return url, hdrs
随便找一组APIs实现一下，看看如何使用之前的包装函数来进一步实现HTTP-API请求。
''' 通道管理 '''
@_log_decorator
async def channel_query(index, comm):
url, hdrs = _get_url_header('/device/QryChannel', comm)
params = {'device_id': comm.get_device_id()}
resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
return url, resp
_channle_test_id = 55 + random.randint(0, 10)
@_log_decorator
async def channel_delete(index, comm):
url, hdrs = _get_url_header('/device/DelChannel', comm)
params = {'device_id': comm.get_device_id(),
'channel_id': _channle_test_id
}
resp = await asyctn.w_request(url, 'delete', hdrs=hdrs, params=params)
return url, resp
@_log_decorator
async def channle_add(index, comm):
url, hdrs = _get_url_header('/device/AddChannel', comm)
params = {'product_id': comm.get_product_id(),
'device_id': comm.get_device_id(),
'channel_id': _channle_test_id,
'title': 'addchannel',
'desc': 'addchanneltest'
}
resp = await asyctn.w_request(url, 'post', hdrs=hdrs, params=params)
return url, resp
@_log_decorator
async def channel_modify(index, comm):
url, hdrs = _get_url_header('/device/ModifyChannel', comm)
params = {'device_id': comm.get_device_id(),
'channel_id': _channle_test_id,
'title': 'titlemodifytest',
'desc': 'descmodifytest'
}
resp = await asyctn.w_request(url, 'post', hdrs=hdrs, params=params)
return url, resp
后续APIs都是这么封装啦，来看下完整实现。
#!/usr/bin/python3
# -*- coding: utf-8 -*-
"""
ontApi_main.py,
the main routines of async && coroutine example.
lxr, 2019.12.15
"""
import json
import time
import random
import asyctn
import ontApi_comm as _comm
from platform import python_version
'''
some wrappers for APIs of OneNET-Video-Platform below,
none special ticks.
'''
def _check_python_version():
''' just provide friendly tips to the best of my ability '''
cv = python_version()
tv = '3.7'
if cv < tv:
exit(f"current python version is {cv}, python version can't be less than {tv}\n")
def _apis_done_tips(sync_time, async_time, comm):
tips = 'time saved about %8.2fms from syncio to asyncio-coroutine' % \
(sync_time - async_time)
comm.LOG.write( 'syncio total time: %8.2fms' % sync_time +
f'\nasyncio-coroutine total time: %8.2fms' % async_time +
f'\n{tips}' +
f'\n{"-" * (round(len(tips)/2)-2)}done{"-" * round((len(tips)/2)-2)}')
def _show_log(index, url, resp, dtm, comm):
''' to show some information according to configurations '''
if comm.get_request_showing_flag():
comm.LOG.write(f'{index} {url} {"%8.2f" % (dtm)}ms\n')
indent = comm.get_response_json_indent()
indent = None if indent == 1 else indent
json_flag = 1
try:
resp_dict = json.loads(resp, encoding='utf-8')
if resp_dict['errno'] != 0 and resp_dict['errno'] != 200:
comm.LOG.write(json.dumps(resp_dict, indent=indent) + '\n\n')
return None
except Exception as e:
json_flag = 0
resp = resp if resp != None else 'None'
comm.LOG.write('none-json response: \n' + resp + '\n\n')
if comm.get_response_showing_flag() and json_flag:
comm.LOG.write(json.dumps(json.loads(resp), indent=indent) + '\n\n')
_time_sync_total = 0
def _log_decorator(api_fn):
''' one decorator for OneNET-Video-Platform-API to show logs '''
async def _w(index, comm):
global _time_sync_total
ltm = time.time_ns()
url, resp = await api_fn(index, comm)
dtm = (time.time_ns() - ltm) / 1.0e+6
_time_sync_total += dtm
_show_log(index, url, resp, dtm, comm)
return _w
def _get_url_header(url_path, comm):
''' wrapper for the whole url and headers for API '''
url = comm.get_url_comm_part() + url_path
hdrs = {comm.get_header().split(':', )[0] : comm.get_header().split(':', )[1]}
return url, hdrs
'''
the code on partial(easier to call) APIs of OneNET-Video-Platform,
none special tricks.
'''
''' 通道管理 '''
@_log_decorator
async def channel_query(index, comm):
url, hdrs = _get_url_header('/device/QryChannel', comm)
params = {'device_id': comm.get_device_id()}
resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
return url, resp
_channle_test_id = 55 + random.randint(0, 10)
@_log_decorator
async def channel_delete(index, comm):
url, hdrs = _get_url_header('/device/DelChannel', comm)
params = {'device_id': comm.get_device_id(),
'channel_id': _channle_test_id
}
resp = await asyctn.w_request(url, 'delete', hdrs=hdrs, params=params)
return url, resp
@_log_decorator
async def channle_add(index, comm):
url, hdrs = _get_url_header('/device/AddChannel', comm)
params = {'product_id': comm.get_product_id(),
'device_id': comm.get_device_id(),
'channel_id': _channle_test_id,
'title': 'addchannel',
'desc': 'addchanneltest'
}
resp = await asyctn.w_request(url, 'post', hdrs=hdrs, params=params)
return url, resp
@_log_decorator
async def channel_modify(index, comm):
url, hdrs = _get_url_header('/device/ModifyChannel', comm)
params = {'device_id': comm.get_device_id(),
'channel_id': _channle_test_id,
'title': 'titlemodifytest',
'desc': 'descmodifytest'
}
resp = await asyctn.w_request(url, 'post', hdrs=hdrs, params=params)
return url, resp
''' 设备在线状态查询 '''
@_log_decorator
async def device_online_status_query(index, comm):
url, hdrs = _get_url_header('/device/QryDevStatus', comm)
params = {'devIds': comm.get_device_id()}
resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
return url, resp
''' 命令下发 '''
@_log_decorator
async def cmmd_send(index, comm):
url, hdrs = _get_url_header('/cmds', comm)
params = {'device_id': comm.get_device_id(),
'qos': 1,
'type': 0
}
data = {'type': 'video',
'cmdId': 6,
'cmd':{
'channel_id': 1,
'level': 1
}
}
resp = await asyctn.w_request(url, 'post',
hdrs=hdrs, params=params, data=data)
return url, resp
''' 透传信息查询 '''
@_log_decorator
async def device_msg_query(index, comm):
url, hdrs = _get_url_header('/device/QryDevMsg', comm)
params = {'device_id': comm.get_device_id(),
'begin_time': '2019-12-07 19:35:00',
'end_time': '2020-12-07 19:35:00'
}
resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
return url, resp
''' 拉流地址 '''
_player_address_url_path = '/play_address'
@_log_decorator
async def player_address_rtmp_get(index, comm):
url, hdrs = _get_url_header(_player_address_url_path, comm)
params = {'device_id': comm.get_device_id(),
'channel_id': 1,
'protocol_type': 0,
}
resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
return url, resp
@_log_decorator
async def player_address_hls_get(index, comm):
url, hdrs = _get_url_header(_player_address_url_path, comm)
params = {'device_id': comm.get_device_id(),
'channel_id': 1,
'protocol_type': 1,
}
resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
return url, resp
''' 平台图片相关接口 '''
_picture_test_chid = 1
_picture_test_format = 'jpg'
_picture_test_name = 'ontApiPictureTest'
_picture_fp = 'test_file/' + _picture_test_name + '.' + _picture_test_format
@_log_decorator
async def picture_upload(index, comm):
url, hdrs = _get_url_header('/picture/upload', comm)
params = {'device_id': comm.get_device_id(),
'channel_id': _picture_test_chid,
'format': _picture_test_format,
'name': _picture_test_name,
'desc': 'pictureuploadtest'}
with open(_picture_fp, mode='rb') as pic:
try: data = pic.read()
except Exception as e: exit(f'{_picture_fp}: {e}')
resp = await asyctn.w_request(url, 'put',
hdrs=hdrs, params=params, data=data)
return url, resp
@_log_decorator
async def picture_list_get(index, comm):
url, hdrs = _get_url_header('/picture/get_list', comm)
params = {'device_id': comm.get_device_id(),
'channel_id': _picture_test_chid,
'page_start': 1,
'page_size': 10
}
resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
return url, resp
@_log_decorator
async def picture_info_get(index, comm):
url, hdrs = _get_url_header('/picture/get_info', comm)
params = {'device_id': comm.get_device_id(),
'channel_id': _picture_test_chid,
'name': _picture_test_name
}
resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
return url, resp
@_log_decorator
async def picture_delete(index, comm):
url, hdrs = _get_url_header('/picture/delete_picture', comm)
params = {'device_id': comm.get_device_id(),
'channel_id': _picture_test_chid,
'name': _picture_test_name
}
resp = await asyctn.w_request(url, 'delete', hdrs=hdrs, params=params)
return url, resp
''' 引导机地址 '''
@_log_decorator
async def boot_addr_get(index, comm):
url = comm.get_url_comm_part() + '/boot_address'
resp = await asyctn.w_request(url, 'get')
return url, resp
''' 设备数据透传 '''
@_log_decorator
async def device_data_ttrans(index, comm):
url, hdrs = _get_url_header('/dev_active', comm)
params = {'device_id': comm.get_device_id()}
data = 'd' * 127;
resp = await asyctn.w_request(url, 'post',
hdrs=hdrs, params=params, data=data)
return url, resp
''' 点播相关接口 '''
_vod_test_chid = 1
@_log_decorator
async def vod_list_get(index, comm):
url, hdrs = _get_url_header('/vod/get_video_list', comm)
params = {'device_id': comm.get_device_id(),
'channel_id': _vod_test_chid,
'page_start': 1,
'page_size': 10
}
resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
return url, resp
''' 公共对象存储 '''
_filename = 'fileTest.dat'
_file_fp = 'test_file/' + _filename
@_log_decorator
async def file_upload(index, comm):
url, hdrs = _get_url_header('/file/upload', comm)
params = {'product_id': comm.get_product_id(),
'name': _filename,
'desc': 'fileuploadtest'
}
with open(_file_fp, mode='rb') as f:
try: data = f.read()
except Exception as e: exit(f'{_file_fp}: {e}')
resp = await asyctn.w_request(url, 'put',
hdrs=hdrs, params=params, data=data)
return url, resp
@_log_decorator
async def file_download(index, comm):
url, hdrs = _get_url_header('/file/download', comm)
params = {'product_id': comm.get_product_id(),
'name': _filename
}
resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
return url, resp
@_log_decorator
async def file_list(index, comm):
url, hdrs = _get_url_header('/file/list', comm)
params = {'product_id': comm.get_product_id(),
'name': _filename
}
resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
return url, resp
@_log_decorator
async def file_delete(index, comm):
url, hdrs = _get_url_header('/file/delete', comm)
params = {'product_id': comm.get_product_id(),
'name': _filename
}
resp = await asyctn.w_request(url, 'delete', hdrs=hdrs, params=params)
return url, resp
''' 运营-内部和外部使用 '''
@_log_decorator
async def stat_onlinelist_get(index, comm):
url, hdrs = _get_url_header('/stat/getonlinelist', comm)
params = {'productid': comm.get_product_id(),
'deviceid': comm.get_device_id(),
'protocol_type': 'rtmp',
}
resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
return url, resp
_stat_test_chid = 1
@_log_decorator
async def stat_pushlist_get(index, comm):
url, hdrs = _get_url_header('/stat/getpushlist', comm)
params = {'productid': comm.get_product_id(),
'deviceid': comm.get_device_id(),
'channelid': _stat_test_chid,
}
resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
return url, resp
''' 存储管理配置接口 '''
_storage_test_chid = 1
@_log_decorator
async def storage_info_query(index, comm):
url, hdrs = _get_url_header('/storage/product_scheme_info', comm)
params = {'product_id': comm.get_product_id(),
'device_id': comm.get_device_id(),
'channel_id': _storage_test_chid,
'page_start': 1,
'page_size': 10
}
resp = await asyctn.w_request(url, 'get', hdrs=hdrs, params=params)
return url, resp
if __name__ == '__main__':
''' for request of OneNET-Video-Platform-APIs running '''
_check_python_version()
comm = _comm.Common()
# request first
apis_first = [
channle_add(1, comm),
cmmd_send (2, comm),
file_upload(3, comm),
boot_addr_get (4, comm),
picture_upload(5, comm),
]
# request then
apis_second = [
file_list (6, comm),
file_download(7, comm),
picture_list_get(8, comm),
picture_info_get(9, comm),
channel_modify (10, comm),
device_msg_query(11, comm),
device_online_status_query(12, comm),
]
# request finally
apis_third = [
player_address_rtmp_get(13, comm),
player_address_hls_get (14, comm),
vod_list_get (15, comm),
file_delete (16, comm),
channel_delete(17, comm),
picture_delete(18, comm),
device_data_ttrans (20, comm),
stat_pushlist_get (21, comm),
stat_onlinelist_get(22, comm),
storage_info_query (23, comm),
]
time_async_start = time.time_ns()
asyctn_loop = asyctn.AsyCtnLoop()
asyctn_loop.w_loop(apis_first)
asyctn_loop.w_loop(apis_second)
asyctn_loop.w_loop(apis_third)
asyctn_loop.w_close()
time_async_end = time.time_ns()
_apis_done_tips(_time_sync_total, (time_async_end - time_async_start) / 1.0e+6, comm
6 例子运行体验
> python ontApi_main.py
4 http://api.heclouds.com:80/ipc/video/boot_address 64.88ms
2 http://api.heclouds.com:80/ipc/video/cmds 140.33ms
{"errno": 3, "error": "device not online"}
1 http://api.heclouds.com:80/ipc/video/device/AddChannel 70.10ms
3 http://api.heclouds.com:80/ipc/video/file/upload 75.15ms
5 http://api.heclouds.com:80/ipc/video/picture/upload 95.29ms
9 http://api.heclouds.com:80/ipc/video/picture/get_info 32.76ms
8 http://api.heclouds.com:80/ipc/video/picture/get_list 32.76ms
10 http://api.heclouds.com:80/ipc/video/device/ModifyChannel 32.76ms
6 http://api.heclouds.com:80/ipc/video/file/list 32.76ms
7 http://api.heclouds.com:80/ipc/video/file/download 32.76ms
none-json response:
oh.... filedownload response is not the unform json format
11 http://api.heclouds.com:80/ipc/video/device/QryDevMsg 32.76ms
12 http://api.heclouds.com:80/ipc/video/device/QryDevStatus 40.20ms
13 http://api.heclouds.com:80/ipc/video/play_address 37.36ms
{"errno": 4, "error": "device not online"}
20 http://api.heclouds.com:80/ipc/video/dev_active 37.36ms
{"errno": 3, "error": "device not online"}
17 http://api.heclouds.com:80/ipc/video/device/DelChannel 37.36ms
14 http://api.heclouds.com:80/ipc/video/play_address 37.36ms
{"errno": 4, "error": "device not online"}
18 http://api.heclouds.com:80/ipc/video/picture/delete_picture 37.36ms
23 http://api.heclouds.com:80/ipc/video/storage/product_scheme_info 37.36ms
16 http://api.heclouds.com:80/ipc/video/file/delete 37.36ms
15 http://api.heclouds.com:80/ipc/video/vod/get_video_list 37.36ms
21 http://api.heclouds.com:80/ipc/video/stat/getpushlist 69.62ms
22 http://api.heclouds.com:80/ipc/video/stat/getonlinelist 67.59ms
syncio total time: 1118.56ms
asyncio-coroutine total time: 277.38ms
time saved about 841.18ms from syncio to asyncio-coroutine
----------------------------done----------------------------
ontApi_main.py拥有ontApi_comm.py的功能，其将 默认打印请求url和异常响应。
[1, 5], [6, 12], [13, 23]三组API各组以先后顺序执行，组内API异步执行。这可以满足有先后执行顺序需求的APIs。
另外可以看出，各APIs在一个线程中以同步方式执行完毕时约耗1s，以异步方式执行约耗0.3s。由于此处只有23个APIs，所以同步和异步的时间差异体现不大。在一定范围内，随着API数增加，同步与异步之间时间差也会增加。
根据此次运行体验，一个OneNET-Video-HTTP-APIs请求平均会耗 1118.56ms / 23 ≈ 48.6ms。Linux0.1.1中，一个进程的时间片为150ms。所以…可尝试为您的并发单位争取一下类似的时间片段。
IO驱动程序，如Linux的IO驱动程序。 ↩︎
CPU并发单位，如已形成软件概念的进程，线程。 ↩︎
此文没有在特定计算机中验证过CPU与IO速度差异。只可借助《CSAPP》中对存储器件速度差异描述体现一下此处提到的差异了：CPU访问内部寄存器最快只需1个CPU周期，访问SRAM（cache）需几个CPU周期，访问主存需几十到几百个CPU周期，访问SSD/HHD会打千万级CPU周期。更何况是网络IO呢。 ↩︎
进程或线程会增加CPU层面的创建和调度开销；随着IO并发量增大，进程和线程更容易达到计算机资源上限。可在更复杂的业务场景考虑多种并发方式的结合。 ↩︎