Kafka + Zookeeper集群搭建

Kafka + Zookeeper集群搭建

2022-03-20 2022-04-04 约 630 字 预计阅读 2 分钟

新版本的kafka可以不依赖zookeeper，不过目前处于测试阶段，且不支持zookeeper迁移到新本，所以在生产环境还是建议等一等。本文使用的是kafka_2.13-2.8.1.tgz

zookeeper相关的文档可以参考https://zookeeper.apache.org/doc/r3.8.0/zookeeperStarted.html

本次我们把kafka安装到/data/目录

首先需要安装java环境

apt install openjdk-8-jdk-headless 

配置zookeeper

首先需要绑定hosts

这一步可选，不绑定的话后面的配置文件里面就需要写ip地址，会不灵活，后续如果有变动也会比较麻烦

vim /etc/hosts

10.0.0.31 kafka1 zoo1
10.0.0.32 kafka2 zoo2
10.0.0.33 kafka3 zoo3

修改zookeeper配置

vim config/zookeeper.properties

dataDir=/data/kafka/zookeeper_data
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
# Disable the adminserver by default to avoid port conflicts.
# Set the port to something non-conflicting if choosing to enable this
admin.enableServer=false
# admin.serverPort=8080

tickTime=2000
initLimit=5
syncLimit=2
server.1=zoo1:2888:3888
server.2=zoo2:2888:3888
server.3=zoo3:2888:3888

创建数据目录和集群ID

myid 集群内不能重复的，每台机器设置成不一样的。

mkdir zookeeper_data echo 1 > zookeeper_data/myid 

启动zookeeper

cd /data/kafka bin/zookeeper-server-start.sh config/zookeeper.properties 

创建systemd文件(可选)

vim /etc/systemd/system/zookeeper.service

[Unit] Description=Zookeeper service After=network.target [Service] Type=simple User=root Group=root ExecStart=/data/kafka/bin/zookeeper-server-start.sh /data/kafka/config/zookeeper.properties ExecStop=/data/kafka/bin/zookeeper-server-stop.sh Restart=on-failure [Install] WantedBy=multi-user.target 

配置kafka

修改配置文件

vim config/server.properties

broker.id=1
listeners=PLAINTEXT://kafka1:9092
zookeeper.connect=zoo1:2181,zoo2:2181,zoo3:2181

启动kafka

bin/kafka-server-start.sh config/server.properties 

如果报错： kafka.common.InconsistentClusterIdException: The Cluster ID C4wRULTzSGqNoEAInvubIw doesn’t match stored clusterId Some(eA5rD8rZSUm3EXr2glib2w) in meta.properties. The broker is trying tojoin the wrong cluster. Configured zookeeper.connect may be wrong.

这个时候需要删除kafka的log目录，让程序重新生成

创建systemd配置文件

vim /etc/systemd/system/kafka.service

[Unit] Description=Apache Kafka server (broker) After=network.target zookeeper.service [Service] Type=simple User=root Group=root ExecStart=/data/kafka/bin/kafka-server-start.sh /data/kafka/config/server.properties ExecStop=/data/kafka/bin/kafka-server-stop.sh Restart=on-failure [Install] WantedBy=multi-user.target 

测试和使用

cd /data/kafka # 创建topic bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic testtopic # 查看topic list bin/kafka-topics.sh --zookeeper localhost:2181 --list # 控制台生产消息 bin/kafka-console-producer.sh --bootstrap-server kafka2:9092 --topic testtopic # 控制台消费消息 bin/kafka-console-consumer.sh --bootstrap-server kafka3:9092 --topic testtopic --from-beginning