topic list
brokers spread: Spread : As you can see in the source code again app/kafka/manager/model/ActorModel.scala#L400, brokers spread is the percentage of brokers in the cluster that has partitions for the given topic.
Example: 3 brokers share a topic that has 2 partitions, so 66% of the brokers have partitions for this topic
对于该topic的partition分布情况,只使用到66%的broker。
brokers skew: Skewed : As you can see in the source code app/kafka/manager/model/ActorModel.scala#L380, a broker is skewed if its number of partitions is greater that the average of partitions per broker on the given topic.
Example: 2 brokers share 4 partitions, if one of them has 3 partitions, it is skewed (3 > 2)
brokers Leader skew :Percentage of brokers having more partitions as leader than the average
topic view
consumer group
-- 直接平衡所有的topic
./bin/kafka-preferred-replica-election.sh --zookeeper 172.16.30.13:2181/kafkanew
-- 也可以指定某个topic的平衡,先创建一个test_two_leader json文件。
{"version":1,
"partitions":[{"topic":"dev_infra_andre_test_two","partition":0}]}
./bin/kafka-preferred-replica-election.sh --zookeeper 172.16.30.13:2181/kafkanew -path-to-json-file test_two_leader
--- 重启broker日志
[2017-08-03 14:13:51,240] INFO [ReplicaFetcherThread-0-3], Starting (kafka.server.ReplicaFetcherThread)
[2017-08-03 14:13:51,241] INFO [ReplicaFetcherManager on broker 5] Added fetcher for partitions List([[dev_infra_andre_test_two,0], initOffset 127425 to broker BrokerEndPoint(3,172.16.30.13,9095)] , [[dev_infra_andre_test_two,2], initOffset 130260 to broker BrokerEndPoint(4,172.16.30.13,9096)] , [[ci_online_pay_log,0], initOffset 0 to broker BrokerEndPoint(3,172.16.30.13,9095)] , [[ci_login,0], initOffset 0 to broker BrokerEndPoint(3,172.16.30.13,9095)] ) (kafka.server.ReplicaFetcherManager)
[2017-08-03 14:14:53,197] INFO [ReplicaFetcherManager on broker 5] Removed fetcher for partitions [dev_infra_andre_test_two,2],[ci_login,0],[ci_online_pay_log,0] (kafka.server.ReplicaFetcherManager)
[2017-08-03 14:14:53,248] INFO [ReplicaFetcherThread-0-4], Shutting down (kafka.server.ReplicaFetcherThread)
[2017-08-03 14:14:53,251] INFO [ReplicaFetcherThread-0-4], Stopped (kafka.server.ReplicaFetcherThread)
[2017-08-03 14:14:53,252] INFO [ReplicaFetcherThread-0-4], Shutdown completed (kafka.server.ReplicaFetcherThread)
点击kafka-manager header上面的Preferred Replica Election,点击Run Preferred Replica Election 按钮也能触发。
测试环境的例子,先关闭broker5的auto.leader.rebalance.enable=false
1.重启broker 5前,topic的partition分布情况
2.重启broker 5后,topic的partition分布情况
3 执行上面的Preferred Replica Election,后leader关系被调整。
--- 创建分配json
> cat topics-to-move.json
{"topics": [{"topic": "foo1"},
{"topic": "foo2"}],
"version":1
}
--- 生成一个候选任务,候选任务expand-cluster-reassignment.json。
> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --topics-to-move-json-file topics-to-move.json --broker-list "5,6" --generate
Current partition replica assignment
{"version":1,
"partitions":[{"topic":"foo1","partition":2,"replicas":[1,2]},
{"topic":"foo1","partition":0,"replicas":[3,4]},
{"topic":"foo2","partition":2,"replicas":[1,2]},
{"topic":"foo2","partition":0,"replicas":[3,4]},
{"topic":"foo1","partition":1,"replicas":[2,3]},
{"topic":"foo2","partition":1,"replicas":[2,3]}]
}
Proposed partition reassignment configuration
{"version":1,
"partitions":[{"topic":"foo1","partition":2,"replicas":[5,6]},
{"topic":"foo1","partition":0,"replicas":[5,6]},
{"topic":"foo2","partition":2,"replicas":[5,6]},
{"topic":"foo2","partition":0,"replicas":[5,6]},
{"topic":"foo1","partition":1,"replicas":[5,6]},
{"topic":"foo2","partition":1,"replicas":[5,6]}]
}
------- 实际执行该分配任务,
> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --execute
----- 校验任务的执行情况:
> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file expand-cluster-reassignment.json --verify
Status of partition reassignment:
Reassignment of partition [foo1,0] completed successfully
Reassignment of partition [foo1,1] is in progress
Reassignment of partition [foo1,2] is in progress
Reassignment of partition [foo2,0] completed successfully
Reassignment of partition [foo2,1] completed successfully
Reassignment of partition [foo2,2] completed successfully
1.例子移动topic foo1的partition 0移到 5,6 broker上,移动topic foo2的partition 1移动到brokers 2,3
> cat custom-reassignment.json
{"version":1,"partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},{"topic":"foo2","partition":1,"replicas":[2,3]}]}
> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file custom-reassignment.json --execute
Current partition replica assignment
{"version":1,
"partitions":[{"topic":"foo1","partition":0,"replicas":[1,2]},
{"topic":"foo2","partition":1,"replicas":[3,4]}]
}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":1,
"partitions":[{"topic":"foo1","partition":0,"replicas":[5,6]},
{"topic":"foo2","partition":1,"replicas":[2,3]}]
}
1.例子:为topic foo的partition 0把复制从1个调整为3个节点,以前节点只在broker5上,现在将增加brokers 6,7.
-- 构建json文件
> cat increase-replication-factor.json
{"version":1,
"partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]}
--- 执行
> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment
{"version":1,
"partitions":[{"topic":"foo","partition":0,"replicas":[5]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":1,
"partitions":[{"topic":"foo","partition":0,"replicas":[5,6,7]}]}
-- 查询result
> bin/kafka-topics.sh --zookeeper localhost:2181 --topic foo --describe
Topic:foo PartitionCount:1 ReplicationFactor:3 Configs:
Topic: foo Partition: 0 Leader: 5 Replicas: 5,6,7 Isr: 5,6,7