python3协程并发执行

近期在做一个python小工具, 其中用到了并发, 所以来学习总结一下.

先看下面的代码:

import asyncio
import time

async def do_some_work(x):
	print('now:', time.time())
	time.sleep(x)
	return x, 'Done after {}s'.format(x)

start = time.time()
loop = asyncio.get_event_loop()
tasks = [
	asyncio.ensure_future(do_some_work(1)),
	asyncio.ensure_future(do_some_work(2)),
]

dones, _ = loop.run_until_complete(asyncio.wait(tasks))
for r in dones:
	print(r.result())

print("used", time.time() - start)

实际输出结果类似这样:

now: 1597230486.0245202
now: 1597230487.0251734
(2, 'Done after 2s')
(1, 'Done after 1s')
used 3.00112247467041

总共的时间为3秒.

这个run_until_complete我以为是和Promise.all()类似的功能, 结果不是.

那么如果让上述的多个task并发呢?

import asyncio
import time
from concurrent.futures import ThreadPoolExecutor

def do_some_work(x):
	print('now:', time.time())
	time.sleep(x)
	return x, 'Done after {}s'.format(x)

# 这个是关键
async def main(loop, x):
	executor = ThreadPoolExecutor()
	return await loop.run_in_executor(executor, do_some_work, x)

start = time.time()
loop = asyncio.get_event_loop()
tasks = [
	asyncio.ensure_future(main(loop, 1)),
	asyncio.ensure_future(main(loop, 2)),
]

dones, _ = loop.run_until_complete(asyncio.wait(tasks))
for r in dones:
	print(r.result())

print("used", time.time() - start)

输出类似于:

now: 1597231215.1245425
now: 1597231215.1247368
(2, 'Done after 2s')
(1, 'Done after 1s')
used 2.002629518508911 

到这一步, 我们还可以控制并发:

import asyncio
import time
from concurrent.futures import ThreadPoolExecutor


def do_some_work(x):
   print('now:', time.time())
   time.sleep(x)
   return x, 'Done after {}s'.format(x)

async def main(loop, semaphore, x):
   async with semaphore:
   	executor = ThreadPoolExecutor()
   	return await loop.run_in_executor(executor, do_some_work, x)

start = time.time()

loop = asyncio.get_event_loop()
semaphore = asyncio.Semaphore(2) # 控制并发

tasks = [asyncio.ensure_future(main(loop, semaphore, 1)) for i in range(6) ]

dones, _ = loop.run_until_complete(asyncio.wait(tasks))
for r in dones:
   print(r.result())

print("used", time.time() - start)

上述代码输出类似:

now: 1597231573.4504874
now: 1597231573.4508233
now: 1597231574.452158
now: 1597231574.452473
now: 1597231575.4529798
now: 1597231575.4539447
(1, 'Done after 1s')
(1, 'Done after 1s')
(1, 'Done after 1s')
(1, 'Done after 1s')
(1, 'Done after 1s')
(1, 'Done after 1s')
used 3.0052316188812256

可以看到, 开始是两个输出一次(并发数为2: semaphore = asyncio.Semaphore(2)).

(完)