流畅的Python书中的协程部分版本太低，而且讲的比较少，这次根据Python3标准库书中实例来学习记录asyncio的使用。
asyncio模块提供了使用次饿成构建并发应用的工具。threading模块通过应用线程并发，mutilprocessing使用系统进程实现并发，asyncio则使用一个单线程单进程的方法来实现并发，应用的各个部分会彼此合作，在最优的时刻显式地切换任务。
asyncio提供的框架以一个事件循环(event loop)为中心，这是一个首类对象，负责高效地处理I/O事件、系统事件、和应用的上下文切换。
await 都可以直接激活运行协程或future、task
但future与task里买包含了更多的属性，有
'add_done_callback', 'all_tasks', 'cancel', 'cancelled',
'current_task', 'done', 'exception', 'get_loop', 'get_stack',
'print_stack', 'remove_done_callback', 'result', 'set_exception', 'set_result'
所以future与task的功能更加加强，可以使用ensure_future或者loop.create_task将协程装饰成task
task时future的子集，也可以直接通过asyncio.Future创建future
就现在本人的了解，一般的io应用中，还是以多线程使用为主，自己在写协程并发的时候，多个协程之间，无法有效的设置条件交出控制权。
唯一能应用的包，就一个aiohttp，如果我想用另外的包实现协程，基本无法做到，然而协程asyncio包就像流畅的Python书中所说，大部分在讲概念和API，
由于协程的让出控制权让出，能适配的包很多，所以一般在示例中只能通过asyncio.sleep来模拟单携程让出控制权
也只能希望后面能有更加丰富的包来配合协程，或者等我哪一天成为高手，写出能配合协程的包。
2、利用协程合作完成多任务
import asyncio
async def coroutine():
print('in coroutine')
# 定义事件循环
event_loop = asyncio.get_event_loop()
try:
print('starting coroutine')
coro = coroutine()
print('entering event loop')
# 运行协程
event_loop.run_until_complete(coro)
finally:
print('closing event loop')
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/asyncio_coroutine.py starting coroutine entering event loop in coroutine closing event loop Process finished with exit code 0
先通过get_event_loop创建一个默认的事件循环，run_until_complete方法驱动里面的协程，最后关闭事件循环。
从协程返回值
import asyncio
# 有返回值
async def coroutine():
print('in coroutine')
return 'result'
# 定义事件循环
event_loop = asyncio.get_event_loop()
try:
print('starting coroutine')
coro = coroutine()
print('entering event loop')
# 运行协程,获取值
return_value = event_loop.run_until_complete(coro)
print(f'it returned: {return_value!r}')
finally:
print('closing event loop')
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/asyncio_coroutine_return.py starting coroutine entering event loop in coroutine it returned: 'result' closing event loop Process finished with exit code 0
串链协程
import asyncio
async def outer():
print('in outer')
print('waiting for result1')
# 接收phase1()协程产生的值
result1 = await phase1()
print('waiting for result2')
# 接收phase2()协程产生的值
result2 = await phase2(result1)
return result1, result2
async def phase1():
print('in parse1')
return 'result1'
async def phase2(arg):
print('in phase2')
return 'result2 derived from {}'.format(arg)
event_loop = asyncio.get_event_loop()
try:
return_value = event_loop.run_until_complete(outer())
print(f'return_value:{return_value!r}')
finally:
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/asyncio_coroutine_chain.py
in outer
waiting for result1
in parse1
waiting for result2
in phase2
return_value:('result1', 'result2 derived from result1')
Process finished with exit code 0
生成器而不是协程
Python3.5开始
async 代替了@asyncio.coroutine
await 代替了 yield from
本人觉得await 还是yield from更加直观。
3、调度常规函数调用
这个使用的话，需要把事件循环传递放入协程中，协程中通过事件循环的方法去激活需要调用的函数。
在传入事件循环的协程里面必须设置asyncio.seleep，要不然协程不会让出控制权，事件循环根本无法激活调用函数。
import asyncio
from functools import partial
def callback(arg, *, kwarg='default'):
print(f'callback invoked with {arg} and {kwarg}')
async def main(loop):
print('registering callbacks')
# 调用函数,只能传入一个参数，多参数传入调用partial
loop.call_soon(callback, 1)
# 通过partial传入关键字参数
wrapped = partial(callback, kwarg='not default')
loop.call_soon(wrapped, 2)
await asyncio.sleep(.1)
event_loop = asyncio.get_event_loop()
try:
print('entering event loop')
event_loop.run_until_complete(main(event_loop))
finally:
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/asyncio_call_soon.py entering event loop registering callbacks callback invoked with 1 and default callback invoked with 2 and not default Process finished with exit code 0
用Delay调度回调
call_soon是直接调用，call_later()第一个参数可以传入延后的事件，单位为秒，第二个参数为function
在传入事件循环的协程里面必须设置asyncio.seleep，且大于最迟的调用函数时间，要不然协程不会让出控制权，事件循环根本无法激活调用函数。
import asyncio
from functools import partial
def callback(arg):
print(f'callback invoked {arg}')
async def main(loop):
print('registering callbacks')
# 调用函数,第一个参数传入时间
loop.call_later(0.2, callback, 1)
loop.call_later(0.1, callback, 2)
# 这个很重要
await asyncio.sleep(.21)
event_loop = asyncio.get_event_loop()
try:
print('entering event loop')
event_loop.run_until_complete(main(event_loop))
finally:
print('closing event loop')
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/asyncio_call_later.py entering event loop registering callbacks callback invoked 2 callback invoked 1 closing event loop Process finished with exit code 0
在指定事件内调度一个回调
实现这个目的的循环依赖的是一个所谓的单调时钟，用loop.time()生成，激活用call_at
import asyncio
import time
def callback(n, loop):
print(f'callback {n} invoked at {loop.time()}')
async def main(loop):
# 运行结果来看，这now是从0开始的
now = loop.time()
print(f'clock time: {time.time()}')
print(f'loop time {now}')
print('registering callbacks')
# 加0.2秒
loop.call_at(now + .2, callback, 1, loop)
# 加0.1 秒
loop.call_at(now + .1, callback, 2, loop)
# 马上开始
loop.call_soon(callback, 3, loop)
await asyncio.sleep(1)
event_loop = asyncio.get_event_loop()
try:
print('entering event loop')
event_loop.run_until_complete(main(event_loop))
finally:
print('closing event loop')
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/asynvio_call_at.py entering event loop clock time: 1579366496.336677 loop time 0.086557153 registering callbacks callback 3 invoked at 0.086701756 callback 2 invoked at 0.189241196 callback 1 invoked at 0.29037571 closing event loop Process finished with exit code 0
4、异步地生成结果。
Future表示还未完成的工作结果。事件循环可以通过监视一个Future对象的状态来指示它已经完成，从而允许应用的一部分等待另一部分完成一些工作。
从我的理解与操作来看，如果future获取函数中的值，需要通过event_loop.calll_soon调用函数，函数中传入future，通过future.set_result方法保存参数。
import asyncio
def mark_done(future, result):
print('setting future result to {!r}'.format(result))
future.set_result(result)
event_loop = asyncio.get_event_loop()
try:
# 创建future对象
all_done = asyncio.Future()
print('scheduling mark_done')
# 调度函数，函数中传入future
event_loop.call_soon(mark_done, all_done, 'the result')
print('entering event_loop')
# 驱动future对象，并获取值。
result = event_loop.run_until_complete(all_done)
print('returned result: {!r}'.format(result))
finally:
print('closing event_loop')
event_loop.close()
print('future result: {!r}'.format(all_done.result()))
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/asyncio_future_event_loop.py scheduling mark_done entering event_loop setting future result to 'the result' returned result: 'the result' closing event_loop future result: 'the result' Process finished with exit code 0
一般我们获取future的值可以通过await
实际感受，这种写法更加让人理解。
import asyncio
def mark_down(future, result):
print('setting future result to {!r}'.format(result))
future.set_result(result)
async def main(loop):
# 这种逻辑比较清楚，创建协程，协程里面创建future，带入函数中。
all_done = asyncio.Future()
print('scheduling mark done')
loop.call_soon(mark_down, all_done, 'the result')
# 通过await获取future的值
result = await all_done
print('returned result: {!r}'.format(result))
return result
event_loop = asyncio.get_event_loop()
try:
# 驱动协程，并获取协程的返回值。
result = event_loop.run_until_complete(main(event_loop))
print(f'last result is {result!r}')
finally:
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/asyncio_future_await.py scheduling mark done setting future result to 'the result' returned result: 'the result' last result is 'the result' Process finished with exit code 0
Future回调
这个更像是通过future对象传递参数给函数，好比生成器里面的send
很多时候，把await看成yiled from 更加能够让我理解。
import asyncio
import functools
def callback(future, n):
print('{}: future done: {}'.format(n, future.result()))
async def register_callbacks(all_done):
print('register callbacks on future')
all_done.add_done_callback(functools.partial(callback, n=1))
all_done.add_done_callback(functools.partial(callback, n=2))
async def main(all_done):
# 预激回调协程
await register_callbacks(all_done)
print('setting result of future')
# 传入参数
all_done.set_result('the result')
event_loop = asyncio.get_event_loop()
try:
all_done = asyncio.Future()
# 启动协程
event_loop.run_until_complete(main(all_done))
finally:
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/asyncio_future_callback.py register callbacks on future setting result of future 1: future done: the result 2: future done: the result Process finished with exit code 0
5、并发的执行任务
任务的启动可以直接通过await来启动，或者在多个任务之间，有任务await asyncio.sleep，那其他任务马上就可以拿到控制权。
任务是与事件循环交互的主要途径之一。任务可以包装成协程，并跟踪协程何时完成。由于任务是Future的子类，所以其他协程可以等待任务，而且每个任务可以有一个结果，在它完成之后可以获取这个结果
启动一个任务
要启动一个任务，可以使用时间循环.create_task()创建一个Task实例。
import asyncio
async def task_func():
print('in task_func')
return 'the result'
async def main(loop):
print('creating task')
# 创建一个任务
task = loop.create_task(task_func())
# task = [loop.create_task(task_func()) for i in range(1000)]
print('waiting for {!r}'.format(task))
# 直接运行任务，并取回返回值，一般用await取回返回值
return_value = await task
print('task_completed {!r}'.format(task))
print('return value: {!r}'.format(return_value))
event_loop = asyncio.get_event_loop()
try:
event_loop.run_until_complete(main(event_loop))
finally:
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/asyncio_create_task.py creating task waiting for> in task_func task_completed result='the result'> return value: 'the result' Process finished with exit code 0
从上面可以看出，task运行与没运行的状态是不一样的。
取消一个任务
import asyncio
async def task_func():
print('in task_func')
return 'the result'
async def main(loop):
print('creating task')
task = loop.create_task(task_func())
# 取消任务
print('canceling task')
task.cancel()
print('canceled task {!r}'.format(task))
try:
await task
# 获取错误
except asyncio.CancelledError:
print('caught error from canceled task')
else:
print('task result: {!r}'.format(task.result()))
event_loop = asyncio.get_event_loop()
try:
event_loop.run_until_complete(main(event_loop))
finally:
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/asyncio_cancel_task.py creating task canceling task canceled task> caught error from canceled task Process finished with exit code 0
并发取消一个正在执行的任务。
import asyncio
async def task_func():
print('in task_func, sleeping')
try:
# 让出控制权，执行loop.call_soon
await asyncio.sleep(1)
except asyncio.CancelledError:
print('task_func was canceled')
raise
return 'the result'
def task_canceller(t):
print('in task_canceller')
t.cancel()
# print(t)
print('canceled the task')
async def main(loop):
print('create task')
task = loop.create_task(task_func())
# 在协程中并不是第一时间执行，需要等待控制权。
loop.call_soon(task_canceller, task)
print(f'task status is: {task} ')
try:
# 执行 task，让出控制权，给call_soon
res = await task
print(res,'is here')
except asyncio.CancelledError:
print('main() also sees task as canceled')
event_loop = asyncio.get_event_loop()
try:
event_loop.run_until_complete(main(event_loop))
finally:
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/asyncio_cancel_task2.py create task task status is:> in task_func, sleeping in task_canceller canceled the task task_func was canceled main() also sees task as canceled Process finished with exit code 0
从协程创建任务。
import asyncio
# await不仅可以启动协程，还可以启动任务。
async def wrapper():
print('wrapped')
return 'result'
async def inner(task):
print('inner: starting')
print('inner:waiting for {!r}'.format(task))
# 启动task任务
result = await task
print('inner: task returned {!r}'.format(result))
return result
async def starter():
print('starter: creating task')
# 创建任务
task = asyncio.ensure_future(wrapper())
print('starter:waiting for inner')
# 启动inner协程
result = await inner(task)
print('starter: inner returned')
return result
event_loop = asyncio.get_event_loop()
try:
print('entering event loop')
result = event_loop.run_until_complete(starter())
print('last res is {}'.format(result))
finally:
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/asyncio_ensure_future.py entering event loop starter: creating task starter:waiting for inner inner: starting inner:waiting for> wrapped inner: task returned 'result' starter: inner returned last res is result Process finished with exit code 0
6、组合协程和控制结构
一系列协程之间的线性控制流用内置关键字await可以很容易的管理。更复制的结构可能允许一个协程等待多个其他协程并行完成，可以使用asyncio中的工具创建这些更复杂的结构。
等待多个携程
import asyncio
async def phase(i):
print('in phase {}.'.format(i))
await asyncio.sleep(0.1 * i)
print('done with phase {}'.format(i))
return 'phase {} result'.format(i)
async def main(num_phase):
print('starting main')
# 携程队列
phases = [
phase(i)
for i in range(num_phase)
]
print('wait for phases to complete')
# 接收携程队列，complete适完成的，pending是未完成的
completed, pending = await asyncio.wait(phases)
# 从完成的携程里面取值
results = [t.result() for t in completed]
print('results: {!r}'.format(results))
event_loop = asyncio.get_event_loop()
try:
event_loop.run_until_complete(main(3))
finally:
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/asyncio_wait.py starting main wait for phases to complete in phase 1. in phase 2. in phase 0. done with phase 0 done with phase 1 done with phase 2 results: ['phase 2 result', 'phase 0 result', 'phase 1 result'] Process finished with exit code 0
在wait内设定一个超时值，超过这个事件pending里面保留未完成的协程
import asyncio
async def phase(i):
print('in phase {}'.format(i))
try:
await asyncio.sleep(0.1 * i)
except asyncio.CancelledError:
print('phase {} canceled'.format(i))
raise
else:
print('done with phase {}'.format(i))
return 'phase {} result'.format(i)
async def main(num_phases):
print('starting main')
phases = [
phase(i)
for i in range(num_phases)
]
print('waiting 0.1 for phase to complete')
# 设定超时时间，为完成的携程进入pending
completed, pending = await asyncio.wait(phases, timeout=0.1)
print('{} completed and {} pending'.format(len(completed), len(pending)))
# 把未完成的协程取消了
if pending:
print('canceling tasks')
[t.cancel() for t in pending]
print('exiting main')
event_loop = asyncio.get_event_loop()
try:
event_loop.run_until_complete(main(3))
finally:
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/asyncio_wait_timeout.py starting main waiting 0.1 for phase to complete in phase 1 in phase 2 in phase 0 done with phase 0 1 completed and 2 pending canceling tasks exiting main phase 1 canceled phase 2 canceled Process finished with exit code 0
从协程收集结果
import asyncio
async def phase1():
print('in phase1')
await asyncio.sleep(2)
print('done with phase1')
return 'phases1 result'
async def phase2():
print('in phase2')
await asyncio.sleep(1)
print('done with phase2')
return 'phase2 result'
async def main():
print('starting main')
print('waiting for phases to complete')
# 这个直接收集返回值，按照传入的协程顺序排列
result = await asyncio.gather(phase1(), phase2())
print('results: {!r}'.format(result))
event_loop = asyncio.get_event_loop()
try:
event_loop.run_until_complete(main())
finally:
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/async_gather.py starting main waiting for phases to complete in phase1 in phase2 done with phase2 done with phase1 results: ['phases1 result', 'phase2 result'] Process finished with exit code 0
后台操作完成时进行处理。
运用了as_completed方法，那个完成早返回哪个协程
import asyncio
async def phase(i):
print('in phase {}'.format(i))
await asyncio.sleep(.5 - (.1 * i))
print('done with phase {}'.format(i))
return 'phase {} result'.format(i)
async def main(num_phases):
print('starting main')
phases = [
phase(i)
for i in range(num_phases)
]
print('waiting for phases to completed')
results = []
# 参数里面可以填写超时事件
for next_to_complete in asyncio.as_completed(phases):
# 协程没有result属性，future才有
answer = await next_to_complete
print('received answer {!r}'.format(answer))
results.append(answer)
print('results: {!r}'.format(results))
return results
event_loop = asyncio.get_event_loop()
try:
event_loop.run_until_complete(main(3))
finally:
event_loop.close()
/usr/local/bin/python3.7 /Users/shijianzhong/study/t_asyncio/async_gather.py starting main waiting for phases to complete in phase1 in phase2 done with phase2 done with phase1 results: ['phases1 result', 'phase2 result'] Process finished with exit code 0
7、同步原语
待续