【Kafka-分区增加副本】Kafka分区增加副本
【Kafka-分区增加副本】Kafka分区增加副本
- 1）前言
- 2）创建测试topic
- 3）增加副本
- 4）附录
1）前言
Kafka 中 topic 的每个分区可以设置多个副本。如果副本数为1，当该分区副本的 leader 节点宕机后，会导致该分区不可用。故需要设置多副本来保证可用性。
实际项目中，存在项目初期创建了副本数为1的 topic，但是后期又需要扩大副本数的场景。通常不能直接删除 topic 重建，可以通过如下操作实现。
2）创建测试topic
（1）创建副本为1的 topic：
kafka-topics --bootstrap-server kafka01:9092 --create --replication-factor 1 --partitions 3 --topic test_topic
（2）查看topic信息：
kafka-topics --describe --bootstrap-server kafka01:9092 --topic test_topic
输出（可以看出3个分区，各自都是只有一个副本。当 leader 对应节点挂掉后，分区就不可用了）：
Topic:test_topic PartitionCount:3 ReplicationFactor:1 Configs:
Topic: test_topic Partition: 0 Leader: 364 Replicas: 364 Isr: 364
Topic: test_topic Partition: 1 Leader: 365 Replicas: 365 Isr: 365
Topic: test_topic Partition: 2 Leader: 366 Replicas: 366 Isr: 366
3）增加副本
使用官方自带的kafka-reassign-partitions.sh脚本实现。该脚本用来移动分区的副本位置。除了可以实现增加副本，还可以实现将分区的副本移动到指定机器上。该脚本的 help 见附录。
以下步骤实现将 test_topic 的各个分区增加为三副本。
（1）制定分区及副本分配策略：
创建 json 文件，写入如下内容：
填写说明：
- topic 为 topic 名称
- partition 为分区号
- replicas 为 broker id 的数组
{
"version": 1,
"partitions": [
{
"topic": "test_topic",
"partition": 0,
"replicas": [
364,
365,
366
]
},
{
"topic": "test_topic",
"partition": 1,
"replicas": [
364,
365,
366
]
},
{
"topic": "test_topic",
"partition": 2,
"replicas": [
364,
365,
366
]
}
]
}
（2）执行扩副本操作（本例中创建的 json 文件为 test_topic.json）：
1、执行如下命令：
kafka-reassign-partitions --bootstrap-server kafka01:9092 --reassignment-json-file test_topic.json --execute
输出内容（这是个异步操作，执行命令后开始执行扩副本。输出内容中有之前的副本策略，可以保存下来，用于回滚）：
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test_topic","partition":2,"replicas":[366],"log_dirs":["any"]},{"topic":"test_topic","partition":1,"replicas":[365],"log_dirs":["any"]},{"topic":"test_topic","partition":0,"replicas":[364],"log_dirs":["any"]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.
2、查看执行进度（如果 topic 数据量小很快就会完成添加副本的操作，如果数据量过大那么会在后台执行一段时间）：
kafka-reassign-partitions --bootstrap-server kafka01:9092 --reassignment-json-file test_topic.json --verify
出现如下内容说明执行完成：
Status of partition reassignment:
Reassignment of partition test_topic-0 completed successfully
Reassignment of partition test_topic-1 completed successfully
Reassignment of partition test_topic-2 completed successfully
3、验证是否成功
kafka-topics --describe --bootstrap-server kafka01:9092 --topic test_topic
输出（从输出中可以看出每个分区的副本都变成了3个）：
Topic:test_topic PartitionCount:3 ReplicationFactor:3 Configs:
Topic: test_topic Partition: 0 Leader: 364 Replicas: 364,365,366 Isr: 364,366,365
Topic: test_topic Partition: 1 Leader: 365 Replicas: 364,365,366 Isr: 365,366,364
Topic: test_topic Partition: 2 Leader: 366 Replicas: 364,365,366 Isr: 366,365,364
4）附录
# kafka-reassign-partitions
This tool helps to moves topic partitions between replicas.
Option Description
------ -----------
--bootstrap-server <String: Server(s) the server(s) to use for
to use for bootstrapping> bootstrapping. REQUIRED if an
absolute path of the log directory
is specified for any replica in the
reassignment json file
--broker-list <String: brokerlist> The list of brokers to which the
partitions need to be reassigned in
the form "0,1,2". This is required
if --topics-to-move-json-file is
used to generate reassignment
configuration
--command-config <String: Admin client Property file containing configs to be
property file> passed to Admin Client.
--disable-rack-aware Disable rack aware replica assignment
--execute Kick off the reassignment as specified
by the --reassignment-json-file
option.
--generate Generate a candidate partition
reassignment configuration. Note
that this only generates a candidate
assignment, it does not execute it.
--help Print usage information.
--reassignment-json-file <String: The JSON file with the partition
manual assignment json file path> reassignment configurationThe format
to use is -
{"partitions":
[{"topic": "foo",
"partition": 1,
"replicas": [1,2,3],
"log_dirs": ["dir1","dir2","dir3"]
}],
"version":1
}
Note that "log_dirs" is optional. When
it is specified, its length must
equal the length of the replicas
list. The value in this list can be
either "any" or the absolution path
of the log directory on the broker.
If absolute log directory path is
specified, the replica will be moved
to the specified log directory on
the broker.
--replica-alter-log-dirs-throttle The movement of replicas between log
<Long: replicaAlterLogDirsThrottle> directories on the same broker will
be throttled to this value
(bytes/sec). Rerunning with this
option, whilst a rebalance is in
progress, will alter the throttle
value. The throttle rate should be
at least 1 KB/s. (default: -1)
--throttle <Long: throttle> The movement of partitions between
brokers will be throttled to this
value (bytes/sec). Rerunning with
this option, whilst a rebalance is
in progress, will alter the throttle
value. The throttle rate should be
at least 1 KB/s. (default: -1)
--timeout <Long: timeout> The maximum time in ms allowed to wait
for partition reassignment execution
to be successfully initiated
(default: 10000)
--topics-to-move-json-file <String: Generate a reassignment configuration
topics to reassign json file path> to move the partitions of the
specified topics to the list of
brokers specified by the --broker-
list option. The format to use is -
{"topics":
[{"topic": "foo"},{"topic": "foo1"}],
"version":1
}
--verify Verify if the reassignment completed
as specified by the --reassignment-
json-file option. If there is a
throttle engaged for the replicas
specified, and the rebalance has
completed, the throttle will be
removed
--zookeeper <String: urls> REQUIRED: The connection string for
the zookeeper connection in the form
host:port. Multiple URLS can be
given to allow fail-over.