[bigdata] kafka基本命令 -- 迁移topic partition到指定的broker

版本 0.9.2

创建topic

bin/kafka-topics.sh --create --topic topic_name --partition 6 --replication-factor 1 -zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181

开启console consumer查看消息

bin/kafka-console-consumer.sh --topic rt_live_pcweb -zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181 --from-begin

改变topic的partition数， kafka topic的partition数只能增加，不能减少。

[root@ip-10-27-100-100 kafka]# bin/kafka-topics.sh --zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181 --alter --topic rt_live_pcweb --partitions 8

WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected

Error while executing topic command The number of partitions for a topic can only be increased

kafka.admin.AdminOperationException: The number of partitions for a topic can only be increased

at kafka.admin.AdminUtils$.addPartitions(AdminUtils.scala:98)

at kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:109)

at kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:93)

at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

at kafka.admin.TopicCommand$.alterTopic(TopicCommand.scala:93)

at kafka.admin.TopicCommand$.main(TopicCommand.scala:52)

at kafka.admin.TopicCommand.main(TopicCommand.scala)

 

迁移topic的partition到指定的broker上

1. 将需要迁移的topic按如下格式写成json文件

[root@ip-10-27-111-107 kafka]# cat topics-to-move.json 

{"topics":

     [{"topic": "rt_live_mobile_new"}],

     "version":1

}

2. 执行如下命令，其中 0，1，2，3 为要迁移到的broker id

[root@ip-10-27-111-107 kafka]# bin/kafka-reassign-partitions.sh --zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2,3" --generate

Current partition replica assignment

 

{"version":1,"partitions":[{"topic":"rt_live_mobile_new","partition":2,"replicas":[4]},{"topic":"rt_live_mobile_new","partition":1,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":6,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":4,"replicas":[6]},{"topic":"rt_live_mobile_new","partition":8,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":7,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":11,"replicas":[5]},{"topic":"rt_live_mobile_new","partition":3,"replicas":[5]},{"topic":"rt_live_mobile_new","partition":0,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":10,"replicas":[4]},{"topic":"rt_live_mobile_new","partition":5,"replicas":[7]},{"topic":"rt_live_mobile_new","partition":9,"replicas":[3]}]}

Proposed partition reassignment configuration

 

{"version":1,"partitions":[{"topic":"rt_live_mobile_new","partition":2,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":6,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":1,"replicas":[3]},{"topic":"rt_live_mobile_new","partition":8,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":4,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":7,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":11,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":10,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":0,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":3,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":9,"replicas":[3]},{"topic":"rt_live_mobile_new","partition":5,"replicas":[3]}]}

3. 将上面命令的输出（红色部门）存入json文件reassignment.json中

4. 执行如下命令，即将partition都迁移至broker 0，1，2，3 服务器上。

[root@ip-10-27-111-107 kafka]# bin/kafka-reassign-partitions.sh --zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181 --reassignment-json-file reassignment.json --execute

Current partition replica assignment

 

{"version":1,"partitions":[{"topic":"rt_live_mobile_new","partition":2,"replicas":[4]},{"topic":"rt_live_mobile_new","partition":1,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":6,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":4,"replicas":[6]},{"topic":"rt_live_mobile_new","partition":8,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":7,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":11,"replicas":[5]},{"topic":"rt_live_mobile_new","partition":3,"replicas":[5]},{"topic":"rt_live_mobile_new","partition":0,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":10,"replicas":[4]},{"topic":"rt_live_mobile_new","partition":5,"replicas":[7]},{"topic":"rt_live_mobile_new","partition":9,"replicas":[3]}]}

 

Save this to use as the --reassignment-json-file option during rollback

Successfully started reassignment of partitions {"version":1,"partitions":[{"topic":"rt_live_mobile_new","partition":2,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":4,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":8,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":1,"replicas":[3]},{"topic":"rt_live_mobile_new","partition":6,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":7,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":11,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":3,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":0,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":10,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":5,"replicas":[3]},{"topic":"rt_live_mobile_new","partition":9,"replicas":[3]}]}

 

5.隔一会之后（partition迁移需要一段时间），执行describe查看，partition已经迁移成功

[root@ip-10-27-111-107 kafka]# bin/kafka-topics.sh --topic rt_live_mobile_new --describe -zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181

Topic:rt_live_mobile_new PartitionCount:12 ReplicationFactor:1 Configs:

Topic: rt_live_mobile_new Partition: 0 Leader: 2 Replicas: 2 Isr: 2

Topic: rt_live_mobile_new Partition: 1 Leader: 3 Replicas: 3 Isr: 3

Topic: rt_live_mobile_new Partition: 2 Leader: 0 Replicas: 0 Isr: 0

Topic: rt_live_mobile_new Partition: 3 Leader: 1 Replicas: 1 Isr: 1

Topic: rt_live_mobile_new Partition: 4 Leader: 2 Replicas: 2 Isr: 2

Topic: rt_live_mobile_new Partition: 5 Leader: 3 Replicas: 3 Isr: 3

Topic: rt_live_mobile_new Partition: 6 Leader: 0 Replicas: 0 Isr: 0

Topic: rt_live_mobile_new Partition: 7 Leader: 1 Replicas: 1 Isr: 1

Topic: rt_live_mobile_new Partition: 8 Leader: 2 Replicas: 2 Isr: 2

Topic: rt_live_mobile_new Partition: 9 Leader: 3 Replicas: 3 Isr: 3

Topic: rt_live_mobile_new Partition: 10 Leader: 0 Replicas: 0 Isr: 0

Topic: rt_live_mobile_new Partition: 11 Leader: 1 Replicas: 1 Isr: 1