storm集群任务迁移问题平滑迁移

最近想把老storm集群上的部分任务，迁移到新搭建的storm集群上。

BrokerHosts hosts = new ZkHosts(Constants.ZOOKEEPER_HOSTS);

虽然都是从一个kafka集群取数，但新老storm集群对应的ZK地址不一样，迁移过去后消息消费没有续上。

老storm集群上运行的任务，在zk集群上的topic消费偏移量一直在更新：

[zk: localhost(CONNECTED) 21] get /ad_***@***/online-1/partition_9 {"topology":{"id":"c412eaff-42c4-4bc6-baf0-5fff4633be03","name":"ad_***"},"offset":1148171,"partition":9,"broker":{"host":"***","port":9092},"topic":"ad_***"} cZxid = 0x1a9a2a6758 ctime = Wed Jun 1514:24:58CST 2016 mZxid = 0x1e001f6e2f mtime = Tue Nov 1518:57:22CST 2016 pZxid = 0x1a9a2a6758 cversion = 0 dataVersion = 836004 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 204 numChildren = 0 [zk: localhost(CONNECTED) 22] get /ad_***@***/online-1/partition_9 {"topology":{"id":"c412eaff-42c4-4bc6-baf0-5fff4633be03","name":"ad_***"},"offset":1148177,"partition":9,"broker":{"host":"***","port":9092},"topic":"ad_***"} cZxid = 0x1a9a2a6758 ctime = Wed Jun 1514:24:58CST 2016 mZxid = 0x1e00202046 mtime = Tue Nov 1518:58:38CST 2016 pZxid = 0x1a9a2a6758 cversion = 0 dataVersion = 836009 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 204 numChildren = 0

 

转移到新storm集群上的任务，在zk上的偏移量，一直没更新过：

[zk: localhost(CONNECTED) 29] get /***_log_hz@***/OnLine-2/partition_9 {"topology":{"id":"aeef3c11-4f58-4baa-9f25-cae3bf25e7d2","name":"***_log_hz@***"},"offset":4860271981,"partition":9,"broker":{"host":"10.***","port":9092},"topic":"***_log"} cZxid = 0x1c0cc06551 ctime = Thu Oct 2011:30:10CST 2016 mZxid = 0x1c49d95a71 mtime = Wed Nov 0916:47:03CST 2016 pZxid = 0x1c0cc06551 cversion = 0 dataVersion = 513318 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 199 numChildren = 0

 

从kafkaspout源码中看到更新偏移量的函数：

publicvoidcommit() { ... _state.writeJSON(committedPath(), data); ... }

_state的初始化在：

Map stateConf = newHashMap(conf); List zkServers = _spoutConfig.zkServers; if(zkServers == null) { zkServers = (List) conf.get(Config.STORM_ZOOKEEPER_SERVERS); } Integer zkPort = _spoutConfig.zkPort; if(zkPort == null) { zkPort = ((Number) conf.get(Config.STORM_ZOOKEEPER_PORT)).intValue(); } stateConf.put(Config.TRANSACTIONAL_ZOOKEEPER_SERVERS, zkServers); stateConf.put(Config.TRANSACTIONAL_ZOOKEEPER_PORT, zkPort); stateConf.put(Config.TRANSACTIONAL_ZOOKEEPER_ROOT, _spoutConfig.zkRoot); _state = newZkState(stateConf);

 

而_spoutConfig.zkServers并没有被主动赋值过，因此取的是Config.STORM_ZOOKEEPER_SERVERS配置项，即新storm集群用的zk地址：storm.zookeeper.servers
在此zk集群上，没有创建storm任务的zk路径，偏移量也没有更新。

因此，在代码中将zkServers变量赋值为原来zookeeper的地址，看是否会更新偏移量:

为了防止生效后，kafkaspout读取到几天前的偏移量，下面将groupid修改成了OnLine-3，可以看到offset开始更新了：

[zk: localhost(CONNECTED) 35] get /***_log_hz@***/OnLine-3/partition_9 {"topology":{"id":"f67da6a9-d136-449e-b00c-ed14eefe6a9c","name":"***_log_hz@***"},"offset":5957447236,"partition":9,"broker":{"host":"10.***","port":9092},"topic":"***_log"} cZxid = 0x1e0050ca16 ctime = Tue Nov 1520:30:32CST 2016 mZxid = 0x1e00516f84 mtime = Tue Nov 1520:31:43CST 2016 pZxid = 0x1e0050ca16 cversion = 0 dataVersion = 34 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 199 numChildren = 0 [zk: localhost(CONNECTED) 36] get /***_log_hz@***/OnLine-3/partition_9 {"topology":{"id":"f67da6a9-d136-449e-b00c-ed14eefe6a9c","name":"***_log_hz@***"},"offset":5957454675,"partition":9,"broker":{"host":"10.***","port":9092},"topic":"***_log"} cZxid = 0x1e0050ca16 ctime = Tue Nov 1520:30:32CST 2016 mZxid = 0x1e00517378 mtime = Tue Nov 1520:31:45CST 2016 pZxid = 0x1e0050ca16 cversion = 0 dataVersion = 35 aclVersion = 0 ephemeralOwner = 0x0 dataLength = 199 numChildren = 0

因此，在老storm集群上跑的任务，往新storm集群上迁移时，只要保证kafkaspout的zk路径不变，并设置好zk地址，就能平滑迁移。