python多线程：concurrent.futures模块

• Executor和Future

      concurrent.futures模块的基础是Exectuor，Executor是一个抽象类，ThreadPoolExecutor和ProcessPoolExecutor是其非常有用的两个子类。Future可以把它理解为一个在未来完成的操作，有异步编程的概念。

1.示例1：ThreadPoolExecutor

from concurrent.futures import ThreadPoolExecutor
import time
val_list = ['One', 'Two', 'Three', 'Four', 'Five']

def print_num(num):
    print(num)
# 线程池中的线程数量
executor = ThreadPoolExecutor(max_workers=3)

for num in val_list:
    thread = executor.submit(print_num, num)
    thread.done()

time.sleep(3)
print('主线程')

2.示例2：ProcessPoolExecutor

与多线程不同，调用多进程时，必须要__name__ == '__main__':下

from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import ProcessPoolExecutor

import time
val_list = ['One', 'Two', 'Three', 'Four', 'Five']

def print_num(num):
    print(num)


executor = ProcessPoolExecutor(max_workers=3)


if __name__ == '__main__':
    # 多进程要放在这里
    for num in val_list:
        thread = executor.submit(print_num, num)
        thread.done()

    time.sleep(3)
    print('主线程')

3.示例3：map()函数

from concurrent.futures import ThreadPoolExecutor
import time
val_list = ['One', 'Two', 'Three', 'Four', 'Five']

def print_num(num):
    print(num)


executor = ThreadPoolExecutor(max_workers=3)

''' 
for num in val_list:
    thread = executor.submit(print_num, num)
    thread.done()
'''
# map()方法依次调用print_num方法，print_num()参数依次调用val_list中每个元素
# 代码看起来更加简洁
executor.map(print_num, val_list)

time.sleep(3)
print('主线程')

 4.示例4：wait()函数

wait方法接会返回一个tuple(元组)，tuple中包含两个set(集合)，一个是completed(已完成的)另外一个是uncompleted(未完成的)。使用wait方法的一个优势就是获得更大的自由度，它接收三个参数FIRST_COMPLETED, FIRST_EXCEPTION 和ALL_COMPLETE，默认设置为ALL_COMPLETED。

如果采用默认的ALL_COMPLETED，程序会阻塞直到线程池里面的所有任务都完成，再执行主线程：
如果采用FIRST_COMPLETED参数，程序并不会等到线程池里面所有的任务都完成。

from concurrent.futures import ThreadPoolExecutor, wait, as_completed
import time
val_list = ['One', 'Two', 'Three', 'Four', 'Five']

def print_num(num):
    print(num)
    time.sleep(3)


executor = ThreadPoolExecutor(max_workers=3)

threadList = []
for num in val_list:
    thread = executor.submit(print_num, num)
    threadList.append(thread)
    thread.done()

print(wait(threadList))
# print(wait(threadList, return_when='FIRST_COMPLETED'))

print('主线程')

5.with方法

import json
from concurrent import futures
import requests
import datetime

headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3679.0 Safari/537.36'}


def getpicture(url):
    filename ='picture' + datetime.datetime.now().strftime('%Y-%m-%d-%H-%M-%S') + '.jpg'
    with requests.get(url, headers=headers) as resp:
        with open(filename, 'wb') as fd:  # 打开文件
            fd.write(resp.content)  # 存入数据块

def picture(request):

    '''
        测试时使用urls = request
    '''

    urls = json.loads(request.body)
    # urls = request
    with futures.ThreadPoolExecutor(15) as exector:
        for future in exector.map(getpicture, urls):
            pass
    return request.getRequestURL

if __name__=='__main__':

    url = ["https://upload-images.jianshu.io/upload_images/9327653-16f3d677e967386d.jpg?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp"]
    urls = []
    for x in range(0, 100):
        urls += url
    ret = picture(urls)
    print(ret)

6.异常处理

 

from concurrent.futures import ThreadPoolExecutor, wait
import time
def print_work(val):
    try:
        print('threadname:' + str(val))
        ret = val/0
        time.sleep(val)
    except ZeroDivisionError as e:
        print('print_work:ZeroDivisionError')
    finally:
        ret = 0
    return ret

def thread_pool_callback(worker):
    
    worker_exception = worker.exception() # 检查线程是否发生异常，如果发生异常，可以将异常信息打印出来或者写入
                                          # 其他地方，等待处理
    if worker_exception:
        print("Worker return exception: {}".format(worker_exception))

vallist = [1, 2, 3, 4, 5, 6]

threadList = []
with ThreadPoolExecutor(max_workers=5) as executor:
    for x in vallist:
        thread = executor.submit(print_work, x)   #返回一个future对象
        # 回调函数add_done_callback，其参数是一个函数，线程执行完毕后会自动调用
        # thread_pool_callback(thread)
        thread.add_done_callback(thread_pool_callback) 
        threadList.append(thread)
        pass
wait(threadList)

print('主线程')


'''
在以上的代码中，我们在线程执行过程中加入了异常处理的代码，同时也使用回调函数获取
线程的异常结果，从执行代码的结果中可以看出，只在线程中获取了异常信息，
当我们把线程中的异常处理代码去掉之后就可以看到回调函数中获取的异常信息。
执行结果：
threadname:1
print_work:ZeroDivisionError
threadname:2
print_work:ZeroDivisionError
threadname:3
print_work:ZeroDivisionError
threadname:4
print_work:ZeroDivisionError
threadname:5
print_work:ZeroDivisionError
threadname:6
print_work:ZeroDivisionError
主线程
'''