爬虫：IP代理池的四个模块的介绍以及各个模块的实现（下）

代理池分为4个模块：存储模块、获取模块、检测模块、接口模块。
 

模块	作用
存储模块	存储模块使用Redis的有序集合，用来做dialing的去重和状态表示，同时它也是中心模块和基础模块，将其他模块串连起来
获取模块	获取模块定时从代理网站获取代理，将获取的代理传递给存储模块，并保存到数据库
检测模块	检测模块定时从存储模块获取所有代理，并对代理进行检测，根据不同的检测结果对代理设置不同的标识
接口模块	接口模块通过Web API接通的服务接口，接口通过连接数据库并通过Web形式返回可用的代理

检测模块的实现：

from multiprocessing import Process
import time
from api import app
from getter import Getter
from tester import Tester

TESTER_CYCLE = 20
GETTER_CYCLE = 20
TESTER_ENABLED = True
GETTER_ENABLED = True
API_ENABLED = True


class Scheduler():
    """调度模块"""
    def schedule_tester(self, cycle=TESTER_CYCLE):
        """
        定时测试代理
        :param cycle:
        :return:
        """
        tester = Tester()
        while True:
            print("测试器开始运行")
            tester.run()
            time.sleep(cycle)

    def schedule_getter(self, cycle=GETTER_CYCLE):
        """
        定时获取代理
        :param cycle:
        :return:
        """
        getter = Getter()
        while True:
            print("开始抓取代理")
            getter.run()
            time.sleep(cycle)

    def schedule_api(self):
        """
        开启api
        :return:
        """
        app.run()

    def run(self):
        print("代理池开始运行")
        if TESTER_ENABLED:
            test_process = Process(target=self.schedule_tester)
            test_process.start()

        if GETTER_ENABLED:
            getter_process = Process(target=self.schedule_getter)
            getter_process.start()

        if API_ENABLED:
            api_process = Process(target=self.schedule_api)
            api_process.start()


if __name__ == '__main__':
    a = Scheduler()
    a.run()

API模块的实现：

from flask import Flask, g
from db import RedisClient


__all__ = ['app']
app = Flask(__name__)


def get_conn():
    if not hasattr(g, 'redis'):
        g.redis = RedisClient()
    return g.redis


@app.route("/")
def index():
    return '
Welcom to Proxy Pool System 
'


@app.route('/random')
def get_proxy():
    """
    获取随机可用代理
    :return: 随机代理
    """
    conn = get_conn()
    return conn.random()


@app.route("/count")
def get_counts():
    """
    获取代理池总量
    :return: 代理池总量
    """
    conn = get_conn()
    return str(conn.count())


if __name__ == '__main__':
    app.run()

调度模块的实现：

from multiprocessing import Process
import time
from api import app
from getter import Getter
from tester import Tester

TESTER_CYCLE = 20
GETTER_CYCLE = 20
TESTER_ENABLED = True
GETTER_ENABLED = True
API_ENABLED = True


class Scheduler():
    """调度模块"""
    def schedule_tester(self, cycle=TESTER_CYCLE):
        """
        定时测试代理
        :param cycle:
        :return:
        """
        tester = Tester()
        while True:
            print("测试器开始运行")
            tester.run()
            time.sleep(cycle)

    def schedule_getter(self, cycle=GETTER_CYCLE):
        """
        定时获取代理
        :param cycle:
        :return:
        """
        getter = Getter()
        while True:
            print("开始抓取代理")
            getter.run()
            time.sleep(cycle)

    def schedule_api(self):
        """
        开启api
        :return:
        """
        app.run()

    def run(self):
        print("代理池开始运行")
        if TESTER_ENABLED:
            test_process = Process(target=self.schedule_tester)
            test_process.start()

        if GETTER_ENABLED:
            getter_process = Process(target=self.schedule_getter)
            getter_process.start()

        if API_ENABLED:
            api_process = Process(target=self.schedule_api)
            api_process.start()


if __name__ == '__main__':
    a = Scheduler()
    a.run()

获取模块、存储模块、公共模块见

爬虫：IP代理池的四个模块的介绍以及各个模块的实现（下）
https://blog.csdn.net/weixin_40576010/article/details/89366179
参考github : https://github.com/Python3WebSpider/ProxyPool