kafka集群扩容(Topic迁移)
文章新地址
文章开始前推荐使用 kafka-manager 工具，可以实时查看kafka同步状态，broker实时读写流量，topic分区情况等信息，提供重选最优leader等操作。
kafka的集群扩容实际上就是把 topic 的 partition 移动到新加的集群上。
生成 topic 移动 json文件有两种方式:
通过
--topics-to-move-json-file和--broker-list批量生成新的topic分区信息，然后根据该信息执行转移操作。手动写要移动的topic信息，更灵活，但是在大量 topic 和 partition 的情况下非常繁琐并且容易出错。
1. 启动新节点
将原节点上的 kafka 目录通过 scp 命令拷贝到新节点，只需要修改配置文件中的 broker_id 和 ip 地址，然后依次启动 kafka 服务。
2. 移动topic
移动 topicA , topicB
sudo vim topics-to-move.json
{"topics": [{"topic": "topicA"},
{"topic": "topicB"}],
"version":1
}
将上述 topic 移动到 broker 3,4,5上，用 generate 命令生成 partition 分配 json 串。
bin/kafka-reassign-partitions.sh --zookeeper zk1:2181
--topics-to-move-json-file topics-to-move.json --broker-list "3,4,5" --generate
将看到如下：
Current partition replica assignment
{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[0,2]},
{"topic":"topicA","partition":1,"replicas":[1,0]},
{"topic":"topicB","partition":1,"replicas":[0,1]},
{"topic":"topicA","partition":0,"replicas":[2,1]},
{"topic":"topicB","partition":0,"replicas":[2,0]},
{"topic":"topicB","partition":2,"replicas":[1,2]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[3,4]},
{"topic":"topicA","partition":1,"replicas":[5,3]},
{"topic":"topicB","partition":1,"replicas":[3,4]},
{"topic":"topicA","partition":0,"replicas":[4,5]},
{"topic":"topicB","partition":0,"replicas":[5,3]},
{"topic":"topicB","partition":2,"replicas":[4,5]}]}
将 Proposed partition reassignment configuration 下方的数据写入 expand-cluster-reassignment.json
上文说的第二种方式其实就是手动写该文件，而不用命令生成的。
vim expand-cluster-reassignment.json
{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[3,4]},
{"topic":"topicA","partition":1,"replicas":[5,3]},
{"topic":"topicB","partition":1,"replicas":[3,4]},
{"topic":"topicA","partition":0,"replicas":[4,5]},
{"topic":"topicB","partition":0,"replicas":[5,3]},
{"topic":"topicB","partition":2,"replicas":[4,5]}]}
用 execute 命令开始正式执行，将会把上述2个topic按 expand-cluster-reassignment 中的设置进行分配，该过程可能会影响其它topic的读取（延时）
bin/kafka-reassign-partitions.sh --zookeeper zk1:2181
--reassignment-json-file expand-cluster-reassignment.json --execute
执行状态查询
查询执行状态
bin/kafka-reassign-partitions.sh --zookeeper zk1:2181
--reassignment-json-file expand-cluster-reassignment.json --verify
主要有几种状态
Reassignment of partition [topicA,4] is still in progress # 转移中
Reassignment of partition [topicB,2] completed successfully # 转移结束
如果你看到一大堆的 ERROR 信息：
ERROR: Assigned replicas (0,5,1,2,3,4) don't match the list of replicas for reassignment (5,3,4) for partition [topicA,3]
Reassignment of partition [topicA,3] failed
不要方，不要关闭节点，过会再看看就正常了。