NSQ源码分析（1）-nsqd消息的生产

NSQ通过topic区分不同的消息队列，每个topic具有不同的channel，同一个topic下的每一个消息会被广播到每个channel中。

消息从生产者到消费者之路

nsq同时支持HTTP协议和TCP协议，客户端可以通过tcp经过特定的协议发布一个消息到nsq的指定topic，或者通过http协议的指定接口。

我们先来看一条消息由客户端发布到NSQ的topic会发生什么。

从topic到channel

下面是简单的流程图：

Alt text

无论是http还是tcp调用，都会调用nsqd/topic.go/Topic.PutMessage方法。内部会把它放入memoryMsgChan这个Buffered Channel。buffer的大小由配置设定，超过了buffer大小的消息会写入backend，即diskq。
至此，put消息的同步操作完成，剩下的工作由这个topic的协程异步完成，这个协程执行nsqd/topic.go/Topic.messagePump方法。这个方法的源码如下：

// messagePump从memoryMsgChan或者diskq里拿出message，并转发到这个topic下的每个Channel之中。
func (t *Topic) messagePump() {    
    var msg *Message
    var buf []byte
    var err error
    var chans []*Channel
    var memoryMsgChan chan *Message
    var backendChan chan []byte

    t.RLock()
    for _, c := range t.channelMap {
        chans = append(chans, c)
    }
    t.RUnlock()

    if len(chans) > 0 {
        memoryMsgChan = t.memoryMsgChan
        backendChan = t.backend.ReadChan()
    }

    for {
        select {
        case msg = <-memoryMsgChan:
        case buf = <-backendChan:
            msg, err = decodeMessage(buf)
            if err != nil {
                t.ctx.nsqd.logf(LOG_ERROR, "failed to decode message - %s", err)
                continue
            }
        case <-t.channelUpdateChan: //topic channels update
            chans = chans[:0]
            t.RLock()
            for _, c := range t.channelMap {
                chans = append(chans, c)
            }
            t.RUnlock()
            if len(chans) == 0 || t.IsPaused() {
                memoryMsgChan = nil
                backendChan = nil
            } else {
                memoryMsgChan = t.memoryMsgChan
                backendChan = t.backend.ReadChan()
            }
            continue
        case pause := <-t.pauseChan:
            if pause || len(chans) == 0 {
                memoryMsgChan = nil
                backendChan = nil
            } else {
                memoryMsgChan = t.memoryMsgChan
                backendChan = t.backend.ReadChan()
            }
            continue
        case <-t.exitChan:
            goto exit
        }
        
        //遍历所有订阅topic的channel
        for i, channel := range chans {
            chanMsg := msg
            // 除了第一个channel，都需要复制message，每个channel需要unique的消息
            if i > 0 {
                chanMsg = NewMessage(msg.ID, msg.Body)
                chanMsg.Timestamp = msg.Timestamp
                chanMsg.deferred = msg.deferred
            }
            if chanMsg.deferred != 0 {
                channel.PutMessageDeferred(chanMsg, chanMsg.deferred)
                continue
            }
            err := channel.PutMessage(chanMsg)
            if err != nil {
                t.ctx.nsqd.logf(LOG_ERROR,
                    "TOPIC(%s) ERROR: failed to put msg(%s) to channel(%s) - %s",
                    t.name, msg.ID, channel.name, err)
            }
        }
    }

exit:
    t.ctx.nsqd.logf(LOG_INFO, "TOPIC(%s): closing ... messagePump", t.name)
}

这段代码非常简单，但是这部分异步的操作不同于许多传统语言的实现，比如放到线程池里去执行一段代码。

NSQ的这种方式在高并发的环境下并没有加很多的锁，而是通过channel和单协程操作关键数据结构的方式实现。channel实现协程间的通信，每一个数据结构对象（需要高并发操作的一组相关数据）都会在创建之初启动一个维护协程（messagePump），负责用select监听其它协程发给这组结构的消息（包含需要对数据进行的操作），并在无竞争的情况下操作这组数据。这样的操作串行了所有对共享数据的所有操作，避免大量使用锁。需要注意的是，在这里，这些对数据的串行操作都是读写数据结构，还有写到其它channel做通信之类的操作，应当要避免特别耗时的计算或者同步的IO，否则会造成channel的阻塞。

这也是golang下并发开发的一种比较常见的范式，golang推荐的同步方式是通信，而不是共享内存，这种范式也是这种思想的体现。详细可以看Effective Go - Concurrency这部分怎么说：

Share by communicating
Concurrent programming is a large topic and there is space only for some Go-specific highlights here.
Concurrent programming in many environments is made difficult by the subtleties required to implement correct access to shared variables. Go encourages a different approach in which shared values are passed around on channels and, in fact, never actively shared by separate threads of execution. Only one goroutine has access to the value at any given time. Data races cannot occur, by design. To encourage this way of thinking we have reduced it to a slogan:
Do not communicate by sharing memory; instead, share memory by communicating.
This approach can be taken too far. Reference counts may be best done by putting a mutex around an integer variable, for instance. But as a high-level approach, using channels to control access makes it easier to write clear, correct programs.
One way to think about this model is to consider a typical single-threaded program running on one CPU. It has no need for synchronization primitives. Now run another such instance; it too needs no synchronization. Now let those two communicate; if the communication is the synchronizer, there's still no need for other synchronization. Unix pipelines, for example, fit this model perfectly. Although Go's approach to concurrency originates in Hoare's Communicating Sequential Processes (CSP), it can also be seen as a type-safe generalization of Unix pipes.

我们也可以看到，NSQ代码也会用到锁，那么什么时候用锁，什么时候用channel呢？最简单的原则就是，哪种用起来自然就用哪一种，哪种简单用哪种，哪种效率高用哪种。