Kafka集群部署

0 准备

0.1 ZooKeeper

https://blog.csdn.net/weixin_43724577/article/details/134829362

0.2 xsync

https://blog.csdn.net/weixin_43724577/article/details/134872115

1 集群规划

centos701	centos702	centos704
ZooKeeper	ZooKeeper	ZooKeeper
Kafka	Kafka	Kafka

2 下载

官网

https://kafka.apache.org/

https://kafka.apache.org/downloads

上面两个文件中的 2.12 /2.13 分别表示编译 Kafka 的 Scala 语言版本，后面的 3.6.1 是 Kafka 的版本 。

3 解压二进制包

tar -zxvf /tmp/kafka_2.12-3.6.1.tgz -C /opt/module/

4 修改配置文件

vim /opt/module/kafka_2.12-3.6.1/config/server.properties

broker.id=701
log.dirs=/opt/module/kafka_2.12-3.6.1/datas
zookeeper.connect=centos701:2181,centos702:2181,centos704:2181/kafka

5 分发安装包

xsync /opt/module/kafka_2.12-3.6.1/

6 其他节点修改配置文件

[root@centos702 ~]# vim /opt/module/kafka_2.12-3.6.1/config/server.properties
broker.id=702
[root@centos704 ~]# vim /opt/module/kafka_2.12-3.6.1/config/server.properties
broker.id=704

7 配置环境变量

vim /etc/profile.d/my_env.sh

增加如下：

#KAFKA_HOME
export KAFKA_HOME=/opt/module/kafka_2.12-3.6.1
export PATH=$PATH:$KAFKA_HOME/bin

分发my_env.sh

xsync /etc/profile.d/my_env.sh

每个节点刷新环境变量

source /etc/profile

8 Kafka集群启停脚本

vim /bin/kafka.sh

#!/bin/bash

hosts="centos701 centos702 centos704"
case $1 in 
"start"){
    for i in $hosts
    do
        echo ---------- Kafka $i 启动 ----------
        ssh $i "/opt/module/kafka_2.12-3.6.1/bin/kafka-server-start.sh -daemon /opt/module/kafka_2.12-3.6.1/config/server.properties"
    done
};;
"stop"){
    for i in $hosts
    do
        echo ---------- Kafka $i 停止 ----------
        ssh $i "/opt/module/kafka_2.12-3.6.1/bin/kafka-server-stop.sh"
    done
};;
esac

添加执行权限

chmod +x /bin/kafka.sh

9 Kafka，启动！

先启动ZooKeeper，再启动Kafka

zk.sh start
kafka.sh start

[root@centos701 ~]# zk.sh start
---------- zookeeper centos701 启动 ----------
ZooKeeper JMX enabled by default
Using config: /opt/module/apache-zookeeper-3.9.1-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
---------- zookeeper centos702 启动 ----------
ZooKeeper JMX enabled by default
Using config: /opt/module/apache-zookeeper-3.9.1-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
---------- zookeeper centos704 启动 ----------
ZooKeeper JMX enabled by default
Using config: /opt/module/apache-zookeeper-3.9.1-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
[root@centos701 ~]# kafka.sh start
---------- Kafka centos701 启动 ----------
---------- Kafka centos702 启动 ----------
---------- Kafka centos704 启动 ----------

jps

[root@centos701 ~]# xcall.sh jps
--------- centos701 ----------
2816 Jps
2647 Kafka
2155 QuorumPeerMain
--------- centos702 ----------
2586 Kafka
2108 QuorumPeerMain
2733 Jps
--------- centos704 ----------
2531 Kafka
2680 Jps
2060 QuorumPeerMain

注意：
停止Kafka集群时，一定要等Kafka所有节点进程全部停止后再停止ZooKeeper集群。因为Zookeeper集群当中记录着Kafka集群相关信息，ZooKeeper集群一旦先停止，Kafka集群就没有办法再获取停止进程的信息，只能手动杀死Kafka进程了。