并发编程-进程

并发编程-进程

进程创建启动

python提供了multiprocessing模块来支持多进程

• multiprocessing.Process(target=task, args=(arg,))用于创建进程

Process类相关方法

• start() 启动进程
• join() 等待进程结束

启动子线程

【注意】线程启动代码块要放在__name__ == __main__下

方式一：

import multiprocessing
import time

def task(name: str, count: int):
    print(f"{name} - step 1\n", end="")
    result = 0
    for i in range(count):
        result += i + 1
    time.sleep(5)
    print(f"{name}- end with {result}")
    
def start_process1():
    name = "p1"
    count = 100
    process = multiprocessing.Process(target=task, args=(name, count))
    process.start()

    process.join()
    print("main process over")
    
if __name__ == "__main__":
    start_process1()

运行结果：

PS F:\Code\python_code\high_python\python_advance\multiprocesses> python .\multiprocesses_demo.py
p1 - step 1
p1- end with 5050
main process over

查看进程信息：一条进程是python用来做资源监控的，所以有两条。

PS F:\Code\python_code\high_python> tasklist.exe|findstr "python"
python.exe                   19588 Console                    2     11,436 K
python.exe                   14300 Console                    2     11,644 K

方式二：

import multiprocessing
import time

def task(name: str, count: int):
    print(f"{name} - step 1\n", end="")
    result = 0
    for i in range(count):
        result += i + 1
    time.sleep(5)
    print(f"{name}- end with {result}")
    
    
def statr_process2():
    args_list = [("a", 100), ("b", 99), ("c", 98)]
    processes = [
        multiprocessing.Process(target=task, args=(name, count)) for name, count in args_list
    ]
    
    for p in processes:
        p.start()
    
    for p in processes:
        p.join()
    
# 创建进程
if __name__ == "__main__":
    statr_process2()

运行结果：

PS F:\Code\python_code\high_python\python_advance\multiprocesses> python .\multiprocesses_demo.py
a - step 1
b - step 1
c - step 1
a- end with 5050
b- end with 4950
c- end with 4851

检查进程信息：

PS F:\Code\python_code\high_python> tasklist.exe|findstr "python"
python.exe                   15016 Console                    2     11,576 K
python.exe                   13820 Console                    2     11,520 K
python.exe                    3676 Console                    2     11,632 K
python.exe                    3080 Console                    2     11,624 K

进程池

• 进程的创建和销毁比起线程更加耗费资源
• 频繁的创建和销毁线程对系统的性能影响更大
• 进程池是python提供的便于进程管理和提高性能的工具

使用进程池

python提供了两个类concurrent.futures.ProcessPoolExecutor和Future

ProcessPoolExecutor相关方法

• submit() # 启动一个线程任务，返回一个Future类的对象
• map() # 相同任务，不同参数，启动多进程，返回可迭代对象，包含任务执行结果
• shutdown() # 停止进程

Future相关方法

• result() # 任务执行结果
• exception() # 执行出现的异常

下载图片案例

from concurrent.futures import ProcessPoolExecutor
from urllib.request import urlopen, Request
import os


headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36",
    "content-type": "Content-Type: image/jpeg"
}
def download_img(url: str):
    site_url = Request(url, headers=headers)

    with urlopen(site_url) as web_file:
        img_data = web_file.read()
        
    if not img_data:
        raise Exception(f"Error: can not load the image from {url}")
    
    file_name = os.path.basename(url)
    
    with open(file_name, "wb") as f:
        f.write(img_data)
        
    return "Download image successfully, {}".format(url)


urls = [
    "https://img0.bdstatic.com/static/searchresult/img/baseimg3_4f26a23.png",
    # "..."
]

def main():
    with ProcessPoolExecutor() as ececutor:
        results = ececutor.map(download_img, urls)
        for r in results:
            print(r)
if __name__ == "__main__":
    main()

运行结果：

PS F:\Code\python_code\high_python\python_advance\multiprocesses> python .\multiprocess_pool.py
Download image successfully, https://img0.bdstatic.com/static/searchresult/img/baseimg3_4f26a23.png