JStorm平台搭建

简易版可以参考
http://www.jstorm.io/QuickStart/Deploy/Standalone.html

下载源代码

在github下下载JStorm 2.2.1的源代码。因为Storm on yarn模块在编译时报错而我们又不需要，删除该模块

mvn package assembly:assembly

会在工程目录下生成一个文件，打包在服务器上解压缩，按照上述步骤执行即可。

******************************************
The jstorm client can only be run from within a release. You appear to be trying to run the client from a checkout of JStorm's source code.

You can download a JStorm release
******************************************

出现上述问题是因为我们的目录下缺少RELEASE文件。放一个进去就好。
请不要忘记修改storm.yaml文件。

########### These MUST be filled in for a storm configuration
 storm.zookeeper.servers:
     - "10.237.65.11"
     - "10.237.65.12"
     - "10.237.65.13"

 storm.zookeeper.root: "/your_cluster_name"

否则在集群启动topo时可能会报告

Connection timed out for connection string (localhost:2181) and timeout (15000) / elapsed (16540)
shade.storm.org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss

启动命令为

##Nimbus
nohup jstorm nimbus & 
##Supervisor
nohup jstorm supervisor >/dev/null 2>&1 &

压测

我们使用了下面的压测代码。测试Storm的极限在哪里。

public class PressureMain {
    private static class PressureSpout extends BaseRichSpout{
        private SpoutOutputCollector collector;
        public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
            this.collector = collector;
        }
        public void nextTuple() {
            this.collector.emit(new Values("test-msg"));
        }
        public void declareOutputFields(OutputFieldsDeclarer declarer) {
            declarer.declare(new Fields("str"));
        }
    }

    private static class PressureBolt extends BaseBasicBolt{
        public void execute(Tuple input, BasicOutputCollector collector) {
            System.out.println("RECEIVE");
        }
        public void declareOutputFields(OutputFieldsDeclarer declarer) {

        }
    }

    public static void main(String[] args) {
        TopologyBuilder tp = new TopologyBuilder();
        tp.setSpout("PressureSpout",  new PressureSpout());
        tp.setBolt("PressureBolt", new PressureBolt(),2).shuffleGrouping("PressureSpout");
        Config conf = new Config();
        conf.setNumWorkers(4);
        conf.setNumAckers(1);
        conf.put("spout.single.thread", true);
        conf.setMessageTimeoutSecs(60);
        conf.setMaxSpoutPending(20);
        //提交job
        if (args != null && args.length > 0) {
            try {
                StormSubmitter.submitTopology(args[0], conf, tp.createTopology());
            }catch (Exception e){
                e.printStackTrace();
            }
        } else {
            LocalCluster cluster = new LocalCluster();
            conf.setMaxTaskParallelism(1);
            cluster.submitTopology("KafkaStormDemo", conf, tp.createTopology());
        }
        System.out.println("GDL finish");
    }
}

测试结果显示我们的集群能扛住百万级别的TPS。Storm可能会产生堆积。

Topo处理速度测试

我们接下来测试最慢的处理速度。