kafka重新分配partition
摘要
当kafka集群扩容时，新加入的节点可以很平滑的加入到集群中。但是原来创建的topic并不会将partition数据均匀的分配到这个节点上，只有新创建的topic才会分配到新的节点上。这时就可以用kafka的kafka-reassign-partitions.sh工具来进行手动平均分配。
This command moves topic partitions between replicas.
Option Description
------ -----------
--broker-list <brokerlist> The list of brokers to which the
partitions need to be reassigned in
the form "0,1,2". This is required
if --topics-to-move-json-file is
used to generate reassignment
configuration
--disable-rack-aware Disable rack aware replica assignment
--execute Kick off the reassignment as specified
by the --reassignment-json-file
option.
--generate Generate a candidate partition
reassignment configuration. Note
that this only generates a candidate
assignment, it does not execute it.
--reassignment-json-file <manual The JSON file with the partition
assignment json file path> reassignment configurationThe format
to use is -
{"partitions":
[{"topic": "foo",
"partition": 1,
"replicas": [1,2,3] }],
"version":1
}
--throttle <Long: throttle> The movement of partitions will be
throttled to this value (bytes/sec).
Rerunning with this option, whilst a
rebalance is in progress, will alter
the throttle value. The throttle
rate should be at least 1 KB/s.
(default: -1)
--topics-to-move-json-file <topics to Generate a reassignment configuration
reassign json file path> to move the partitions of the
specified topics to the list of
brokers specified by the --broker-
list option. The format to use is -
{"topics":
[{"topic": "foo"},{"topic": "foo1"}],
"version":1
}
--verify Verify if the reassignment completed
as specified by the --reassignment-
json-file option. If there is a
throttle engaged for the replicas
specified, and the rebalance has
completed, the throttle will be
removed
--zookeeper <urls> REQUIRED: The connection string for
the zookeeper connection in the form
host:port. Multiple URLS can be
given to allow fail-over.
1.配置文件
使用kafka-reassign-partitions.sh工具前，必须先手动写一个配置文件，告诉工具你要分配哪些topic数据。
{
"topics": [
{"topic": "test"}
],
"version": 1
}
如果要移动多个topic数据可以输入多个。
2.生成分配计划
./kafka-reassign-partitions.sh --zookeeper druid1:2181 --topics-to-move-json-file topic.json --broker-list "0,1,2,4" --generate
broker-list参数为brokerid，并不是broker的服务地址，执行完毕之后会生成类似下面的数据。
[root@druid1 bin]# ./kafka-reassign-partitions.sh --zookeeper druid1:2181 --topics-to-move-json-file topic.json --broker-list "0,1,2,4" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test","partition":11,"replicas":[0,1,2]},{"topic":"test","partition":4,"replicas":[1,0,2]},{"topic":"test","partition":13,"replicas":[2,0,1]},{"topic":"test","partition":1,"replicas":[2,0,1]},{"topic":"test","partition":8,"replicas":[1,2,4,0]},{"topic":"test","partition":10,"replicas":[4,0,1,2]},{"topic":"test","partition":5,"replicas":[2,1,4,0]},{"topic":"test","partition":2,"replicas":[4,1,2,0]},{"topic":"test","partition":15,"replicas":[0,2,4,1]},{"topic":"test","partition":9,"replicas":[2,4,0,1]},{"topic":"test","partition":14,"replicas":[4,1,2,0]},{"topic":"test","partition":7,"replicas":[0,4,1,2]},{"topic":"test","partition":12,"replicas":[1,4,0,2]},{"topic":"test","partition":6,"replicas":[4,2,0,1]},{"topic":"test","partition":0,"replicas":[1,4,0,2]},{"topic":"test","partition":3,"replicas":[0,2,4,1]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"test","partition":11,"replicas":[2,0,1]},{"topic":"test","partition":4,"replicas":[1,2,0]},{"topic":"test","partition":13,"replicas":[1,0,2]},{"topic":"test","partition":1,"replicas":[1,0,2]},{"topic":"test","partition":8,"replicas":[2,1,0]},{"topic":"test","partition":10,"replicas":[1,2,0]},{"topic":"test","partition":5,"replicas":[2,0,1]},{"topic":"test","partition":2,"replicas":[2,1,0]},{"topic":"test","partition":15,"replicas":[0,1,2]},{"topic":"test","partition":9,"replicas":[0,1,2]},{"topic":"test","partition":14,"replicas":[2,1,0]},{"topic":"test","partition":7,"replicas":[1,0,2]},{"topic":"test","partition":12,"replicas":[0,2,1]},{"topic":"test","partition":6,"replicas":[0,2,1]},{"topic":"test","partition":0,"replicas":[0,2,1]},{"topic":"test","partition":3,"replicas":[0,1,2]}]}
上面一行是topic当前partition的分配规则，下面一行是按照上面命令执行后程序自动分配的分配计划，可以根据机器实际资源情况手动调整，将确认后的分配规则写到一个文件里面，我这里命名为move.json
3.执行分配
./kafka-reassign-partitions.sh --zookeeper druid1:2181 --reassignment-json-file move.json --throttle 31457280 --execute
–throttle 为了避免迁移过程中影响kafka正常使用，这个参数可以设置迁移的流量速度，单位是byte。
4.验证/查看进度
./kafka-reassign-partitions.sh --zookeeper druid1:2181 --reassignment-json-file move.json --verify
可以查看迁移进度