Linux 搭建Kafka集群，最新教程，细到极致
大家好呀，今天给大家带来的是，最新版kafka集群的安装教程，希望给小伙伴们一点小小的帮助。
注意：提前安装好jdk, Jdk安装教程
1、准备安装包，Kafka官网下载
2、kafka安装需要zk,注意：kafka安装版本和zk版本要一致，下载以后，先解压kafka，进入libs，查看依赖的zk版本再下载,切记切记！！！zookeeper官网下载
[root@VM-4-2-centos jdk]# java -version
java version "1.8.0_333"
Java(TM) SE Runtime Environment (build 1.8.0_333-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.333-b02, mixed mode)
3、先解压zk，然后重命名,同时配置环境变量
[root@VM-4-2-centos kafka]# tar -zxvf apache-zookeeper-3.6.3-bin.tar.gz
[root@VM-4-2-centos kafka]# mv apache-zookeeper-3.6.3-bin/ zookeeper-3.6.3
echo 'export ZOOKEEPER_HOME=/opt/kafka/zookeeper/zookeeper-3.6.3' >> /etc/profile
echo 'export PATH=$PATH:$ZOOKEEPER_HOME/bin' >> /etc/profile
source /etc/profile
3A、新建一个zookeeper文件夹，然后移进去
[root@VM-4-2-centos kafka]# mkdir zookeeper
[root@VM-4-2-centos kafka]# mv zookeeper-3.6.3 ./zookeeper/
5、修改conf文件夹配置
[root@VM-4-2-centos kafka]# cd zookeeper/zookeeper-3.6.3/conf
[root@VM-4-2-centos zookeeper-3.6.3]# mv zoo_sample.cfg zoo.cfg
[root@VM-4-2-centos conf]# vi zoo.cfg
注意:server 设置broker Id的服务地址，本机ip一定要用0.0.0.0代替
server.1=0.0.0.0:2888:3888，1是我们data/myid,2888是服务端通信端口，3888服务端之间选举端口，云服务器开放2181 2888 3888端口
dataDir=/opt/kafka/zookeeper/zookeeper-3.6.3/data
dataLogDir=/opt/kafka/zookeeper/zookeeper-3.6.3/log
server.1=0.0.0.0:2888:3888
server.2=tencent02:2888:3888
server.3=tencent03:2888:3888
6、创建data 、log文件夹，然后进入data,
添加内容：1 ，其他两台主机分别配置 2和3
[root@VM-4-2-centos conf]# cd ..
[root@VM-4-2-centos zookeeper-3.6.3]# mkdir data && mkdir log
[root@VM-12-13-centos zookeeper-3.6.3]# cd data
echo 1 > myid
7、上去，然后启动，第二台、第三台服务器，请重复3-7步骤
[root@VM-4-2-centos data]# cd ..
[root@VM-4-2-centos zookeeper-3.6.3]# ./bin/zkServer.sh start
8、查看zk是否启动成功，没有问题
[root@VM-4-2-centos zookeeper-3.6.3]# jps
92838 QuorumPeerMain
94502 Jps
[root@VM-4-2-centos zookeeper-3.6.3]# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1741/sshd
tcp6 0 0 :::2181 :::* LISTEN 92838/java
tcp6 0 0 :::3888 :::* LISTEN 92838/java
tcp6 0 0 :::8080 :::* LISTEN 92838/java
tcp6 0 0 :::39381 :::* LISTEN 92838/java
tcp6 0 0 :::22 :::* LISTEN 1741/sshd
9、查看zk集群状态
[root@VM-12-13-centos logs]# cd /opt/kafka/zookeeper/zookeeper-3.6.3
[root@VM-12-13-centos zookeeper-3.6.3]# ./bin/zkServer.sh status ./conf/zoo.cfg
ZooKeeper JMX enabled by default
Using config: ./conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader
切换到第二台服务器：
[root@VM-4-12-centos kafka]# cd /opt/kafka/zookeeper/zookeeper-3.6.3
[root@VM-4-12-centos zookeeper-3.6.3]# ./bin/zkServer.sh status ./conf/zoo.cfg
ZooKeeper JMX enabled by default
Using config: ./conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
10、接下来是正式环节，搭建kafka集群
[root@VM-4-2-centos zookeeper-3.6.3]# cd /opt/kafka/
[root@VM-4-2-centos kafka]# ll
total 96716
-rw-r--r-- 1 root root 12516362 May 31 17:33 apache-zookeeper-3.6.3-bin.tar.gz
-rw-r--r-- 1 root root 86512293 Apr 10 00:46 kafka_2.13-3.0.1.tgz
drwxr-xr-x 3 root root 4096 Jun 1 07:39 zookeeper
11、解压kafka,然后重命名
[root@VM-4-2-centos kafka]# tar -zxvf kafka_2.13-3.0.1.tgz
[root@VM-4-2-centos kafka]# mv kafka_2.13-3.0.1 kafka
12、进入kakfa,创建log文件夹
[root@VM-4-2-centos kafka]# cd kafka/
[root@VM-4-12-centos kafka]# mkdir log
[root@VM-4-12-centos kafka]# ll
total 68
drwxr-xr-x 3 root root 4096 Mar 4 00:20 bin
drwxr-xr-x 3 root root 4096 Mar 4 00:20 config
drwxr-xr-x 2 root root 4096 Jun 1 08:26 libs
-rw-r--r-- 1 root root 14587 Mar 4 00:15 LICENSE
drwxr-xr-x 2 root root 4096 Mar 4 00:20 licenses
drwxr-xr-x 2 root root 4096 Jun 1 08:34 log
-rw-r--r-- 1 root root 28184 Mar 4 00:15 NOTICE
drwxr-xr-x 2 root root 4096 Mar 4 00:20 site-docs
13、修改kafka配置，broker.id 不同服务器 分别为1 、2 、3
[root@VM-4-12-centos kafka]# vi config/server.properties
broker.id=1
listeners=PLAINTEXT://:9092
advertised.listeners=PLAINTEXT://101.35.245.191:9092
log.dirs=/opt/kafka/kafka/log
# topic 在当前broker上的分片个数，与broker保持一致
num.partitions=3
# 设置zookeeper集群地址与端口如下：
zookeeper.connect=tencent01:2181,tencent02:2181,tencent03:2181
14、启动kafka集群
[root@VM-4-12-centos kafka]# ./bin/kafka-server-start.sh -daemon ./config/server.properties &
15、查看，没有问题
[root@VM-4-2-centos kafka]# jps
112519 Jps
112361 Kafka
109642 QuorumPeerMain
[root@VM-4-12-centos kafka]# lsof -i:9092
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 32026 root 134u IPv6 180770 0t0 TCP *:XmlIpcRegSvc (LISTEN)
java 32026 root 153u IPv6 182221 0t0 TCP VM-4-12-centos:nimhub->tencent02:XmlIpcRegSvc (ESTABLISHED)
java 32026 root 154u IPv6 180774 0t0 TCP VM-4-12-centos:XmlIpcRegSvc->tencent02:nimhub (ESTABLISHED)
java 32026 root 158u IPv6 181065 0t0 TCP VM-4-12-centos:51164->tencent01:XmlIpcRegSvc (ESTABLISHED)
16、然后我们连接上kafka，新建一个topic
–replication-factor 2 两个副本
–partitions 1 一个分区
–topic test 一个test主题
[root@VM-4-2-centos kafka]# cd /opt/kafka/kafka/
[root@VM-4-2-centos kafka]# ./bin/kafka-topics.sh --create --replication-factor 2 --partitions 1 --topic test --bootstrap-server 101.35.245.191:9092
17、切换到第二台服务器，查看主题是否创建成功
[root@VM-4-12-centos zookeeper-3.6.3]# cd /opt/kafka/kafka/
[root@VM-4-12-centos kafka]# ./bin/kafka-topics.sh --list --bootstrap-server 101.35.245.191:9092
test
18、启动生产者
[root@VM-4-2-centos kafka]# ./bin/kafka-console-producer.sh --broker-list tencent01:9092 --topic test
19、启动消费者，第二台和第三台服务器
./bin/kafka-console-consumer.sh --bootstrap-server tencent02:9092 --topic test
./bin/kafka-console-consumer.sh --bootstrap-server tencent03:9092 --topic test
20、生产者发送消息，接收者接收消息，成功！
21、至此，整个kafka集群搭建完毕，最后，各位小伙伴们，麻烦给老哥一个点赞、关注、收藏三连好吗，你的支持是老哥更新最大的动力，谢谢！