《Apache Kafka 实战》笔记 - 7.5.2 分区重分配

目标

实践分区的重新分配，先搭建3个节点（broker1,broker2,broker3），在其上创建2个topic，然后新加入2个节点，重新分配这2个topic的分区。

准备环境

搭建一个3个节点的集群，然后创建两个测试 topic：foo1、foo2，它们都是3个分区，副本因子都是2：

bin/kafka-topics.sh --create --zookeeper localhost:2181 --partitions 3 --replication-factor 2 --topic foo1

bin/kafka-topics.sh --create --zookeeper localhost:2181 --partitions 3 --replication-factor 2 --topic foo2

# 查看
bin/kafka-topics.sh  --zookeeper localhost:2181 --describe  --topic foo1

bin/kafka-topics.sh  --zookeeper localhost:2181 --describe  --topic foo2

使用自动生成的分配方案

创建一个json文件，指明要分区重分配的topic列表：

echo '{"topics": [{"topic": "foo1"}, {"topic": "foo2"}], "version":1}' > topics-to-move.json

启动2个新的节点，broker4 和 broker5。

生成分配方案：

bin/kafka-reassign-partitions.sh  --zookeeper localhost:2181 --topics-to-move-json-file topics-to-move.json --broker-list "4,5" --generate

输出信息：

Current partition replica assignment
{
    "version": 1,
    "partitions": [{
        "topic": "foo1",
        "partition": 1,
        "replicas": [2, 3],
        "log_dirs": ["any", "any"]
    }, {
        "topic": "foo2",
        "partition": 2,
        "replicas": [2, 3],
        "log_dirs": ["any", "any"]
    }, {
        "topic": "foo2",
        "partition": 0,
        "replicas": [3, 1],
        "log_dirs": ["any", "any"]
    }, {
        "topic": "foo2",
        "partition": 1,
        "replicas": [1, 2],
        "log_dirs": ["any", "any"]
    }, {
        "topic": "foo1",
        "partition": 0,
        "replicas": [1, 2],
        "log_dirs": ["any", "any"]
    }, {
        "topic": "foo1",
        "partition": 2,
        "replicas": [3, 1],
        "log_dirs": ["any", "any"]
    }]
}

Proposed partition reassignment configuration
{
    "version": 1,
    "partitions": [{
        "topic": "foo1",
        "partition": 0,
        "replicas": [4, 5],
        "log_dirs": ["any", "any"]
    }, {
        "topic": "foo2",
        "partition": 2,
        "replicas": [5, 4],
        "log_dirs": ["any", "any"]
    }, {
        "topic": "foo1",
        "partition": 2,
        "replicas": [4, 5],
        "log_dirs": ["any", "any"]
    }, {
        "topic": "foo2",
        "partition": 1,
        "replicas": [4, 5],
        "log_dirs": ["any", "any"]
    }, {
        "topic": "foo2",
        "partition": 0,
        "replicas": [5, 4],
        "log_dirs": ["any", "any"]
    }, {
        "topic": "foo1",
        "partition": 1,
        "replicas": [5, 4],
        "log_dirs": ["any", "any"]
    }]
}

上面是当前的分区状况，下面是针对新节点的分区方案，把下面的方案保存成 expand-cluster-reassignment.json，然后执行命令进行重分配：

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --execute

验证：

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --verify

# 输出信息：
Status of partition reassignment:
Reassignment of partition foo1-1 completed successfully
Reassignment of partition foo2-2 completed successfully
Reassignment of partition foo2-0 completed successfully
Reassignment of partition foo2-1 completed successfully
Reassignment of partition foo1-0 completed successfully
Reassignment of partition foo1-2 completed successfully

自己定义分配方案

例如把foo1分区0的两个副本搬移到broker4和broker5上，同时把foo2分区1的两个副本搬移到broker2和broker3上，方案：

# custom-reassignment.json

{
    "version": 1,
    "partitions": [{
        "topic": "foo1",
        "partition": 0,
        "replicas": [4, 5]
    }, {
        "topic": "foo2",
        "partition": 1,
        "replicas": [2, 3]
    }]
}

执行分配命令：

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --execute

验证：

bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --verify