kafka集群部署，多节点多broker
环境
| 主机名 |
IP 地址 |
安装路径 |
系统 |
| 135.36.247.91 |
135.36.247.91 |
/board/opt/kafka_2.11-1.0.0 |
CentOS Linux release 7.3.1611 (Core) |
| 135.36.247.92 |
135.36.247.92 |
||
| 135.36.247.93 |
135.36.247.93 |
Kafka下载
http://archive.apache.org/dist/kafka/1.0.0/kafka_2.12-1.0.0.tgz
解压：tar -zxf kafka_2.12-1.0.0.tgz
一. zookeeper配置
1. 文件zookeeper.properties，各节点一致
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/board/opt/kafka_2.11-1.0.0/data
clientPort=2182
server.1=135.36.247.91:2889:3889
server.2=135.36.247.92:2889:3889
server.3=135.36.247.93:2889:3889
2. 各节点分别创建server-id
echo 1 > /board/opt/kafka_2.11-1.0.0/data/myid
echo 2 > /board/opt/kafka_2.11-1.0.0/data/myid
echo 3 > /board/opt/kafka_2.11-1.0.0/data/myid
3.启动zookeeper
# /board/opt/kafka_2.11-1.0.0/bin/zookeeper-server-start.sh -daemon /board/opt/kafka_2.11-1.0.0/config/zookeeper.properties
4.查看zookeeper集群状态
echo stat | nc 135.36.247.91 2182 | grep Mode
Mode: follower
echo stat | nc 135.36.247.92 2182 | grep Mode
Mode: follower
echo stat | nc 135.36.247.93 2182 | grep Mode
Mode: follower
二. kafka配置
1.各节点配置文件server.properties
broker.id=0
listeners=PLAINTEXT://135.36.247.91:9092
advertised.listeners=PLAINTEXT://135.36.247.91:9092
log.dirs=/board/opt/kafka_2.11-1.0.0/data
zookeeper.connect=135.36.247.91:2182,135.36.247.92:2182,135.36.247.93:2182
broker.id=1
listeners=PLAINTEXT://135.36.247.92:9092
advertised.listeners=PLAINTEXT://135.36.247.92:9092
log.dirs=/board/opt/kafka_2.11-1.0.0/data
zookeeper.connect=135.36.247.91:2182,135.36.247.92:2182,135.36.247.93:2182
broker.id=2
listeners=PLAINTEXT://135.36.247.93:9092
advertised.listeners=PLAINTEXT://135.36.247.93:9092
log.dirs=/board/opt/kafka_2.11-1.0.0/data
zookeeper.connect=135.36.247.91:2182,135.36.247.92:2182,135.36.247.93:2182
2. 启动kafka
# /board/opt/kafka_2.11-1.0.0/bin/kafka-server-start.sh -daemon /board/opt/kafka_2.11-1.0.0/config/server.properties
3. 查看集群状态
# echo dump | nc 135.36.247.91 2182 | grep broker
三 测试
1. 创建topic
# ./kafka-topics.sh --zookeeper 135.36.247.91:2182,135.36.247.92:2182,135.36.247.93:2182 --create --partitions 3 --replication-factor 3 --topic test-topic
Created topic "test-topic".
# ./kafka-topics.sh --zookeeper 135.36.247.91:2182,135.36.247.92:2182,135.36.247.93:2182 --describe --topic test-topic
Topic:test-topic PartitionCount:3 ReplicationFactor:3 Configs:
Topic: test-topic Partition: 0 Leader: 1 Replicas: 1,2,0 Isr: 1,2,0
Topic: test-topic Partition: 1 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1
Topic: test-topic Partition: 2 Leader: 0 Replicas: 0,1,2 Isr: 0,1,2
2. 生产者生产数据
# ./kafka-console-producer.sh --broker-list 135.36.247.91:9092,135.36.247.92:9092,135.36.247.93:9092 --topic test-topic >Hello >This is my fisrt Kafka !
3. 消费者消费数据
# ./kafka-console-consumer.sh --bootstrap-server bootstrap-servers=135.36.247.91:9092,135.36.247.92:9092,135.36.247.93:9092 --topic test-topic --from-beginning Hello This is my fisrt Kafka !