[置顶] zeromq源代码分析6-2------REQ和REP

本文我们讲一下req和rep这对zeromq的socket。

这是一个经典的Request-Reply的例子。

代码也是很简单:

//
// Hello World server
// Binds REP socket to tcp://*:5555
// Expects "Hello" from client, replies with "World"
//
#include <zmq.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>

int main (void)
{
void *context = zmq_init (1); // 创建上下文, 初始化一个io_thread

// Socket to talk to clients
void *responder = zmq_socket (context, ZMQ_REP); // 创建REP类型的socket
zmq_bind (responder, "tcp://*:5555"); // 绑定到端口并且在io_thread中accept连接

while (1) {
// Wait for next request from client
zmq_msg_t request; // 创建消息结构
zmq_msg_init (&request); // 初始化空的消息
zmq_recv (responder, &request, 0); // 从管道中接收消息
printf ("Received Hello\n");
zmq_msg_close (&request); // 销毁消息

// Do some 'work'
sleep (1);

// Send reply back to client
zmq_msg_t reply; // 创建reply消息的结构
zmq_msg_init_size (&reply, 5); // 初始化5个字节的消息来容纳“World”
memcpy (zmq_msg_data (&reply), "World", 5); // 拷贝到消息中
zmq_send (responder, &reply, 0); // 发送消息到管道，等待io_thread从管道中读取后发送
zmq_msg_close (&reply);
}
// We never get here but if we did, this would be how we end
zmq_close (responder);
zmq_term (context);
return 0;
}

//
// Hello World client
// Connects REQ socket to tcp://localhost:5555
// Sends "Hello" to server, expects "World" back
//
#include <zmq.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>

int main (void)
{
void *context = zmq_init (1); // 创建上下文, 初始化一个io_thread

// Socket to talk to server
printf ("Connecting to hello world server…\n");
void *requester = zmq_socket (context, ZMQ_REQ);
zmq_connect (requester, "tcp://localhost:5555"); // 在io_thread中连接到端点

int request_nbr;
for (request_nbr = 0; request_nbr != 10; request_nbr++) { // 发送10次
zmq_msg_t request; // 建立request的消息结构
zmq_msg_init_size (&request, 5); // 初始化request消息为5个字节
memcpy (zmq_msg_data (&request), "Hello", 5); // 设置request消息内容为"Hello"
printf ("Sending Hello %d…\n", request_nbr);
zmq_send (requester, &request, 0); // 发送消息到管道，等待io_thread从管道中读取后发送
zmq_msg_close (&request); // 销毁request消息

zmq_msg_t reply; // 建立reply的消息结构
zmq_msg_init (&reply); // 初始化reply消息为0字节的空消息
zmq_recv (requester, &reply, 0); // 从管道中接收消息
printf ("Received World %d\n", request_nbr);
zmq_msg_close (&reply); // 销毁reply消息
}
zmq_close (requester);
zmq_term (context);
return 0;
}

从以前几篇blog的分析你应该明白以下几点:

1. 创建上下文的时候会创建指定数目的io_thread，这儿是1个。该io_thread采用reactor模式，会借助于poller不断地去轮询可读可写事件。

2. 最开始bind socket的时候会去选择一个io_thread来调用accept()，当有连接被accept后会互相交换对端的identity。

3. 交换完identity后，io_thread会建立一个session，并且attach到两个管道。socket调用send()和recv()的时候会和相应的管道交互。

4. io_thread当poller轮询到可读事件的时候，就会将数据写入到管道中或者有可写事件的时候，就会从管道中读数据。

ok，这些如果不明白的话，请回过头去结合源代码看前面的文章，如果还是不懂可以联系我。

接下去转入正题，我们来看今天的主角REQ和REP:

首先我们来介绍一下req和rep的消息封装的格式:

从前面的博文中我们已经了解了zeromq中multipart message，而zeromq中每一种socket在发送和接受的时候会封装相应的消息，在消息头部加上相应的消息头，这些消息头的消息的flag就是ZMQ_MSG_MORE，通过和后面的原始msg组成了一个multipart message。

以下就是REQ和REP的消息封装:

1. REQ发送消息:

原始消息:

REQ封装后的消息:

可以看到REQ发送的时候会加上一个0字节的message part。

我们看一下源代码:

int zmq::req_t::xsend (zmq_msg_t *msg_, int flags_)
{
    //  If we've sent a request and we still haven't got the reply,
    //  we can't send another request.
    if (receiving_reply) {
        errno = EFSM;
        return -1;
    }

    //  First part of the request is empty message part (stack bottom).
    if (message_begins) { // 加上empty message part的prefix
        zmq_msg_t prefix;
        int rc = zmq_msg_init (&prefix);
        zmq_assert (rc == 0);
        prefix.flags |= ZMQ_MSG_MORE;
        rc = xreq_t::xsend (&prefix, flags_);
        if (rc != 0)
            return rc;
        message_begins = false;
    }

    bool more = msg_->flags & ZMQ_MSG_MORE;

    int rc = xreq_t::xsend (msg_, flags_); // 发送消息
    if (rc != 0)
        return rc;

    //  If the request was fully sent, flip the FSM into reply-receiving state.
    if (!more) {
        receiving_reply = true;
        message_begins = true;
    }

    return 0;
}

我们先不管xreq_t::xsend(2)函数的细节，我们会在后面的章节会讲，这儿就先理解成发送消息，那么这边的代码就很容易理解了，就是加上在原消息前面加上empty message part发送。此外还有通过receiving_reply和message_begins的这两个flag来控制消息发送和接受。

2. REP接收消息:

REP接收消息的时候会一直读消息的每一个part，知道找到emtpy message part，并且把empty message part上面的消息头(包括empty message part)，发送到相应的输出的管道中，来作reply消息的头。

源代码如下:

int zmq::rep_t::xrecv (zmq_msg_t *msg_, int flags_)
{
    //  If we are in middle of sending a reply, we cannot receive next request.
    if (sending_reply) {
        errno = EFSM;
        return -1;
    }

    if (request_begins) { 

        //  Copy the backtrace stack to the reply pipe.
        bool bottom = false;
        while (!bottom) { // 读取消息，直到找到empty message part

            //  TODO: What if request can be read but reply pipe is not
            //  ready for writing?

            //  Get next part of the backtrace stack.
            int rc = xrep_t::xrecv (msg_, flags_);
            if (rc != 0)
                return rc;

            if ((msg_->flags & ZMQ_MSG_MORE)) {
                //  Empty message part delimits the traceback stack.
                bottom = (zmq_msg_size (msg_) == 0); // 检测是否是empty message part

                //  Push it to the reply pipe.
                rc = xrep_t::xsend (msg_, flags_); // 发送消息头到reply管道作为reply消息的消息头
                zmq_assert (rc == 0);
            }
            else {
                //  If the traceback stack is malformed, discard anything
                //  already sent to pipe (we're at end of invalid message).
                rc = xrep_t::rollback (); // 消息封装格式invalid，rollback消息。
                zmq_assert (rc == 0);
            }
        }

        request_begins = false;
    }

    //  Now the routing info is safely stored. Get the first part
    //  of the message payload and exit.
    int rc = xrep_t::xrecv (msg_, flags_); // 接收消息
    if (rc != 0)
        return rc;

    //  If whole request is read, flip the FSM to reply-sending state.
    if (!(msg_->flags & ZMQ_MSG_MORE)) {
        sending_reply = true;
        request_begins = true;
    }

    return 0;
}

然而这边有一个细节，就是因为消息发送是间接通过管道异步发送的，而一个REP可能会有多个REQ，于是attach多个管道，所以这儿会根据每个管道的identity来标示区分管道，在REP接收消息的时候就要获得对端的identity，这样reply消息才能放入上文提到过的相应的reply管道中。

看代码可能容易懂点:

int zmq::xrep_t::xrecv (zmq_msg_t *msg_, int flags_)
{
    //  If there is a prefetched message, return it.
    if (prefetched) {
        zmq_msg_move (msg_, &prefetched_msg);
        more_in = msg_->flags & ZMQ_MSG_MORE;
        prefetched = false;
        return 0;
    }

    //  Deallocate old content of the message.
    zmq_msg_close (msg_);

    //  If we are in the middle of reading a message, just grab next part of it.
    if (more_in) {
        zmq_assert (inpipes [current_in].active);
        bool fetched = inpipes [current_in].reader->read (msg_);
        zmq_assert (fetched);
        more_in = msg_->flags & ZMQ_MSG_MORE;
        if (!more_in) {
            current_in++;
            if (current_in >= inpipes.size ())
                current_in = 0;
        }
        return 0;
    }

    //  Round-robin over the pipes to get the next message.
    for (int count = inpipes.size (); count != 0; count--) {

        //  Try to fetch new message.
        if (inpipes [current_in].active)
            prefetched = inpipes [current_in].reader->read (&prefetched_msg);

        //  If we have a message, create a prefix and return it to the caller.
        if (prefetched) {
            int rc = zmq_msg_init_size (msg_,
                inpipes [current_in].identity.size ());
            zmq_assert (rc == 0);
            memcpy (zmq_msg_data (msg_), inpipes [current_in].identity.data (),
                zmq_msg_size (msg_));
            msg_->flags |= ZMQ_MSG_MORE;
            return 0;
        }

        //  If me don't have a message, mark the pipe as passive and
        //  move to next pipe.
        inpipes [current_in].active = false;
        current_in++;
        if (current_in >= inpipes.size ())
            current_in = 0;
    }

    //  No message is available. Initialise the output parameter
    //  to be a 0-byte message.
    zmq_msg_init (msg_);
    errno = EAGAIN;
    return -1;
}

其中红色字体部分就是一开始收到消息头，就现在的例子来说就是empty message part的时候，会添加identity前缀到消息中返回给caller。

然后收到该消息以后会发送到管道中, 调用xreq_t::xsend(2):

int zmq::xrep_t::xsend (zmq_msg_t *msg_, int flags_)
{
    //  If this is the first part of the message it's the identity of the
    //  peer to send the message to.
    if (!more_out) {
        zmq_assert (!current_out);

        //  If we have malformed message (prefix with no subsequent message)
        //  then just silently ignore it.
        if (msg_->flags & ZMQ_MSG_MORE) {

            more_out = true;

            //  Find the pipe associated with the identity stored in the prefix.
            //  If there's no such pipe just silently ignore the message.
            blob_t identity ((unsigned char*) zmq_msg_data (msg_),
                zmq_msg_size (msg_));
            outpipes_t::iterator it = outpipes.find (identity);

            if (it != outpipes.end ()) {
                current_out = it->second.writer;
                zmq_msg_t empty;
                int rc = zmq_msg_init (&empty);
                zmq_assert (rc == 0);
                if (!current_out->check_write (&empty)) {
                    it->second.active = false;
                    more_out = false;
                    current_out = NULL;
                    rc = zmq_msg_close (&empty);
                    zmq_assert (rc == 0);
                    errno = EAGAIN;
                    return -1;
                }
                rc = zmq_msg_close (&empty);
                zmq_assert (rc == 0);
            }
        }

        int rc = zmq_msg_close (msg_);
        zmq_assert (rc == 0);
        rc = zmq_msg_init (msg_);
        zmq_assert (rc == 0);
        return 0;
    }

    //  Check whether this is the last part of the message.
    more_out = msg_->flags & ZMQ_MSG_MORE;

    //  Push the message into the pipe. If there's no out pipe, just drop it.
    if (current_out) {
        bool ok = current_out->write (msg_);
        zmq_assert (ok);
        if (!more_out) {
            current_out->flush ();
            current_out = NULL;
        }
    }
    else {
        int rc = zmq_msg_close (msg_);
        zmq_assert (rc == 0);
    }

    //  Detach the message from the data buffer.
    int rc = zmq_msg_init (msg_);
    zmq_assert (rc == 0);

    return 0;
}

1. 这儿就会根据identity去选择相应的输出管道发送消息。

2. 但是注意identity的前缀消息部分是不会发送到输出管道中的，只是使用check_write(1)检测管道。

3. REP发送消息:

从上面在REP接收消息的分而析的时候已经知道会将消息头，在这儿是Frame1和Frame2写入到reply的管道中。

因此在REP发送的时候就只要加入Frame 3数据部分发送到reply管道中去就ok了。

我们看一下源代码:

int zmq::rep_t::xsend (zmq_msg_t *msg_, int flags_)
{
    //  If we are in the middle of receiving a request, we cannot send reply.
    if (!sending_reply) {
        errno = EFSM;
        return -1;
    }

    bool more = (msg_->flags & ZMQ_MSG_MORE);

    //  Push message to the reply pipe.
    int rc = xrep_t::xsend (msg_, flags_); // 发送消息到reply pipe中
    if (rc != 0)
        return rc;

    //  If the reply is complete flip the FSM back to request receiving state.
    if (!more)
        sending_reply = false;

    return 0;
}

而xrep_t::xsend(2)部分在REP接收消息的时候已经看过了，对于消息体会写入相应的管道中, 并且在本条消息全部写完后flush管道。

xrep_t::xsend(2):
...
 //  Push the message into the pipe. If there's no out pipe, just drop it.
    if (current_out) {
        bool ok = current_out->write (msg_);
        zmq_assert (ok);
        if (!more_out) {
            current_out->flush ();
            current_out = NULL;
        }
    }
...

4. REQ接收消息:

REQ接收消息的时候就会收消息， 而消息的第一个部分应该是empty message part，然后再把后面的消息收取返回给caller。

代码如下:

int zmq::req_t::xrecv (zmq_msg_t *msg_, int flags_)
{
    //  If request wasn't send, we can't wait for reply.
    if (!receiving_reply) {
        errno = EFSM;
        return -1;
    }

    //  First part of the reply should be empty message part (stack bottom).
    if (message_begins) { // empty message part处理
        int rc = xreq_t::xrecv (msg_, flags_);
        if (rc != 0)
            return rc;
        zmq_assert (msg_->flags & ZMQ_MSG_MORE);
        zmq_assert (zmq_msg_size (msg_) == 0);
        message_begins = false;
    }

    int rc = xreq_t::xrecv (msg_, flags_); // 接收消息
    if (rc != 0)
        return rc;

    //  If the reply is fully received, flip the FSM into request-sending state.
    if (!(msg_->flags & ZMQ_MSG_MORE)) {
        receiving_reply = false;
        message_begins = true;
    }

    return 0;
}

总结:

本文通过一个简单的REQ-REP例子并结合源代码分析了zeromq中Request-Reply的相关流程。

这边还没有讲xreq::xsend()和xreq::xrecv()函数，但是其实简单地说就是发送的时候会有load-balance去选择一个对端，而接收的时候会有fair-queueing去选择一个对端先处理。这两个策略从概念上来说类似，都是轮流选择。后面在讲ROUTER和DEALER的时候我们会分析相关代码实现，而你会发现ZMQ_DEALER就是用xreq实现的。

下一篇我们会讲ROUTER和DEALER。敬请期待！希望有兴趣的朋友可以和我联系，一起学习。 [email protected]