Zeromq 源码全解析(3)

ZMQ任务分发设计

经典的线程池

经典的线程池多数是抢占式

在主线程上有个任务池,当有任务进入时,唤醒一个线程进行任务执行,以下是一个简单线程池的代码

//首先是一个线程数据,用来控制线程的
struct threadData {
    std::mutex mtx_;                            //互斥量
    std::condition_variable cond_;              //条件变量
    bool is_shutdown_ = false;                  //是否关机
    std::queue> tasks_;   //具体执行任务
};

class ThreadPool {
public:
    ThreadPool(size_t thread_cnt_):m_data(new threadData){
        //创建指定数量的线程
        for(size_t _i = 0 ; _i  _lk(m_data->mtx_);
                for(;;){
                    //判断当前任务池中是否有任务
                    if(!m_data->tasks_.empty()){
                        //如果任务队列中有任务,则拿出任务执行
                        auto _func = std::move(m_data->tasks_.front());
                        //将取出的任务弹出
                        m_data->tasks_.pop();
                        //解锁
                        _lk.unlock();
                        //任务执行
                        _func();
                        //上锁
                        _lk.lock();
                    }else if(m_data->is_shutdown_){
                        //如果判断到关机信号,则直接退出
                        break;
                    }else{
                        //等待条件变量唤醒
                        m_data->cond_.wait(_lk);
                    }
                }
            }).detach();//从主线程中分离
        }
    }
    template
    void execute(TASK&& task_){
        //先上锁
        std::lock_guard _lk(m_data->mtx_);
        //放入任务
        m_data->tasks_.emplace(std::forward(task_));
        //唤醒一个线程
        m_data->cond_.notify_one();
    }

    ~ThreadPool(){
        //先上锁
        if(m_data){
            {
                std::lock_guard _lk(m_data->mtx_);
                m_data->is_shutdown_ = true;
            }
            m_data->cond_.notify_all();
        }

    }
private:
    std::shared_ptr m_data;
};

task_pool

task_a

task_b

task_c

Thread_a

Thread_b

Thread_c

用流程图来描述线程池的任务执行框架,可以明显发现,a,b,c线程可能会对task_pool进行同时访问,这个时候解决办法就是对task_pool进行加锁处理,来觉接条件竞态问题.
同时也不难发现,在多个线程传入任务和多个线程抢占任务时,都需要对task_pool进行加锁访问,也就是一个任务从发布到处理,进行了两次加锁操作.

单向加锁线程池

当然Zmq也会遇到这个问题,Zmq是支持多线程的,且是线程安全的,那Zmq是如何优化上述的线程池框架的呢?

说来也简单,Zmq在每一个线程上都有一个MailBox_t的结构,可以先简单理解成我们举例中的task_pool结构,也就是每一个线程都拥有了自己的任务池,在获取任务时就不用进行加锁了,当然往任务池里添加任务时,还是需要加锁操作的,还是有可能会有多个线程同时进行任务的添加

task_a_pool

task_b_pool

task_c_pool

task_a

task_b

task_c

Thread_a

Thread_b

Thread_c

Zmq的MailBox就是在该种思路上的扩展设计,当然也没那么简单,当pool_a和pool_b执行相同任务时,就可能发生对任务执行资源的抢占,比如一个全局唯一id的生成,同事执行时,如果没有在id生成函数编写时就进行线程安全设计,就会产生未定义的结果,然后线程安全函数编写是困难的,且频繁的上锁解锁相对于非线程函数是效率不高的,所以,在框架设计上,zmq就解决了这种问题.

MailBox结构并不像线程池一般直接存放匿名函数,存放的是一个命令结构,

struct command_t
{
    //  Object to process the command.
    zmq::object_t *destination;
    
    enum type_t
    {
        stop,
        plug,
        own,
        attach,
        bind,
        activate_read,
        activate_write,
        hiccup,
        pipe_term,
        pipe_term_ack,
        pipe_hwm,
        term_req,
        term,
        term_ack,
        term_endpoint,
        reap,
        reaped,
        inproc_connected,
        pipe_peer_stats,
        pipe_stats_publish,
        done
    } type;

    union args_t
    {
        //  Sent to I/O thread to let it know that it should
        //  terminate itself.
        struct
        {
        } stop;

        //  Sent to I/O object to make it register with its I/O thread.
        struct
        {
        } plug;

        //  Sent to socket to let it know about the newly created object.
        struct
        {
            zmq::own_t *object;
        } own;

        //  Attach the engine to the session. If engine is NULL, it informs
        //  session that the connection have failed.
        struct
        {
            struct i_engine *engine;
        } attach;

        //  Sent from session to socket to establish pipe(s) between them.
        //  Caller have used inc_seqnum beforehand sending the command.
        struct
        {
            zmq::pipe_t *pipe;
        } bind;

        //  Sent by pipe writer to inform dormant pipe reader that there
        //  are messages in the pipe.
        struct
        {
        } activate_read;

        //  Sent by pipe reader to inform pipe writer about how many
        //  messages it has read so far.
        struct
        {
            uint64_t msgs_read;
        } activate_write;

        //  Sent by pipe reader to writer after creating a new inpipe.
        //  The parameter is actually of type pipe_t::upipe_t, however,
        //  its definition is private so we'll have to do with void*.
        struct
        {
            void *pipe;
        } hiccup;

        //  Sent by pipe reader to pipe writer to ask it to terminate
        //  its end of the pipe.
        struct
        {
        } pipe_term;

        //  Pipe writer acknowledges pipe_term command.
        struct
        {
        } pipe_term_ack;

        //  Sent by one of pipe to another part for modify hwm
        struct
        {
            int inhwm;
            int outhwm;
        } pipe_hwm;

        //  Sent by I/O object ot the socket to request the shutdown of
        //  the I/O object.
        struct
        {
            zmq::own_t *object;
        } term_req;

        //  Sent by socket to I/O object to start its shutdown.
        struct
        {
            int linger;
        } term;

        //  Sent by I/O object to the socket to acknowledge it has
        //  shut down.
        struct
        {
        } term_ack;

        //  Sent by session_base (I/O thread) to socket (application thread)
        //  to ask to disconnect the endpoint.
        struct
        {
            std::string *endpoint;
        } term_endpoint;

        //  Transfers the ownership of the closed socket
        //  to the reaper thread.
        struct
        {
            zmq::socket_base_t *socket;
        } reap;

        //  Closed socket notifies the reaper that it's already deallocated.
        struct
        {
        } reaped;

        //  Send application-side pipe count and ask to send monitor event
        struct
        {
            uint64_t queue_count;
            zmq::own_t *socket_base;
            endpoint_uri_pair_t *endpoint_pair;
        } pipe_peer_stats;

        //  Collate application thread and I/O thread pipe counts and endpoints
        //  and send as event
        struct
        {
            uint64_t outbound_queue_count;
            uint64_t inbound_queue_count;
            endpoint_uri_pair_t *endpoint_pair;
        } pipe_stats_publish;

        //  Sent by reaper thread to the term thread when all the sockets
        //  are successfully deallocated.
        struct
        {
        } done;

    } args;
};

该结构存放目标对象的指针,然后本次操作的动作,以及参数结构.

在zmq中,每一个对象都会归属一个线程中,所有对象拥有一些基本接口,对应命令结构中的操作指令,所有对象的调用都需要通过命令执行,因为调用时无法知道该对象所属的线程,轻易跨线程调用将是一场灾难.
在该框架下,调用对象通知的是主线程,告诉主线程我要通知某一个对象执行一件事,也就是在执行对象所属线程的mailbox插入一条命令,在下一个循环时,对象所属的线程会从该mailbox中进行执行操作.而一个对象的执行只在一个线程中执行,也就是转变成了单线程操作,也就不需要考虑线程安全事宜了.
当然,这里可能还会有负载均衡的问题,如果多个线程拥有的对象数量有差异,肯定也会造成执行效率降低,所以zmq也在这方面做了相应的处理,在对象进行线程分配时,就进行了负载的判断.

mailbox_a

mailbox_b

mailbox_c

Thread_a_

command_a

Thread_b_

Thread_c_

command_b

command_c

Thread_a

Thread_b

Thread_c