聊聊storm的direct grouping

序

本文主要研究一下storm的direct grouping

direct grouping

direct grouping是一种特殊的grouping，它是由上游的producer直接指定下游哪个task去接收它发射出来的tuple。direct grouping的使用有如下几个步骤：

1、上游在prepare方法保存下游bolt的taskId列表

public class SentenceDirectBolt extends BaseRichBolt {

    private static final Logger LOGGER = LoggerFactory.getLogger(SentenceDirectBolt.class);

    private OutputCollector collector;

    private List taskIds;

    private int numCounterTasks;

    public void prepare(Map config, TopologyContext context, OutputCollector collector) {
        this.collector = collector;
        //NOTE 1 这里要取到下游的bolt的taskId，用于emitDirect时指定taskId
        this.taskIds = context.getComponentTasks("count-bolt");
        this.numCounterTasks = taskIds.size();
    }
    //......
}

这里保存了下游的bolt的taskId列表，用于emitDirect时选择taskId

2、上游在declareOutputFields使用declareStream声明streamId

public class SentenceDirectBolt extends BaseRichBolt {
    //......
    public void declareOutputFields(OutputFieldsDeclarer declarer) {
        declarer.declare(new Fields("word"));
        //NOTE 2 这里要通过declareStream声明direct stream，并指定streamId
        declarer.declareStream("directStreamDemo1",true,new Fields("word"));
        declarer.declareStream("directStreamDemo2",true,new Fields("word"));
    }
}

这里声明了两个streamId，一个是directStreamDemo1，一个是directStreamDemo2

3、上游采用emitDirect指定下游taskId及streamId

public class SentenceDirectBolt extends BaseRichBolt {
    //......
    public void execute(Tuple tuple) {
        String sentence = tuple.getStringByField("sentence");
        String[] words = sentence.split(" ");
        for(String word : words){
            int targetTaskId = getWordCountTaskId(word);
            LOGGER.info("word:{} choose taskId:{}",word,targetTaskId);
            // NOTE 3 这里指定发送给下游bolt的哪个taskId，同时指定streamId
            if(targetTaskId % 2 == 0){
                this.collector.emitDirect(targetTaskId,"directStreamDemo1",new Values(word));
            }else{
                this.collector.emitDirect(targetTaskId,"directStreamDemo2",new Values(word));
            }
        }
        this.collector.ack(tuple);
    }
}

这里使用emitDirect(int taskId, String streamId, List