Kafka架构

目录

• 环境搭建

• 水平扩展

  • Producer

  • Comsumer

• 高可用性

  • ZooKeeper

  • Replicas

• 数据一致

环境搭建

wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.3.0/kafka_2.12-2.3.0.tgz

tar xf kafka_2.12-2.3.0.tgz && cd kafka_2.12-2.3.0

bin/zookeeper-server-start.sh -daemon config/zookeeper.properties

cp config/server.properties config/server0.properties
# broker.id=0
# listeners=PLAINTEXT://:9092
# log.dirs=/tmp/kafka-logs0
bin/kafka-server-start.sh -daemon config/server0.properties

cp config/server.properties config/server1.properties
# broker.id=1
# listeners=PLAINTEXT://:9093
# log.dirs=/tmp/kafka-logs1
bin/kafka-server-start.sh -daemon config/server1.properties

cp config/server.properties config/server2.properties
# broker.id=2
# listeners=PLAINTEXT://:9094
# log.dirs=/tmp/kafka-logs2
bin/kafka-server-start.sh -daemon config/server2.properties

jps

bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic topic-demo --partitions 3 --replication-factor 3

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9093,localhost:9094 -topic topic-demo
# hello kafka

bin/kafka-console-producer.sh --broker-list localhost:9092,localhost:9093,localhost:9094 --topic topic-demo
# >hello kafka

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic topic-demo
# Topic: topic-demo PartitionCount:3    ReplicationFactor:3 Configs:
# Topic: topic-demo Partition: 0    Leader: 2   Replicas: 2,1,0 Isr: 2,1,0
# Topic: topic-demo Partition: 1    Leader: 0   Replicas: 0,2,1 Isr: 0,2,1
# Topic: topic-demo Partition: 2    Leader: 1   Replicas: 1,0,2 Isr: 1,0,2

水平扩展

Producer

• 扩展Topic => Multi Partition

image.png

• Broker Count MUST >= Partition Factor

Comsumer

• 扩展Consume => Comsumer Group

image.png

• Consumer Count SHOULD <= Partition Factor

高可用性

ZooKeeper

• ZooKeeper Cluster

• Starting from 0.9, we are removing all the Zookeeper dependency from the clients. However, the brokers will continue to be heavily depend on Zookeeper for

Server failure detection

Data partitioning

In-sync data replication

Replicas

image.png

数据一致

image.png

• AR(所有副本) = ISR(和leader同步的副本) + OSR(未和leader同步的副本)

• 不是完全同步 => 因为性能较差 & 也不是单纯异步 => 因为一致性较差

参考

• Kafka入门

• Kafka 数据可靠性深度解读

• 消息中间件选型分析：从 Kafka 与 RabbitMQ 的对比看全局

• What happens if Zookeeper fails completely?