Linux搭建Kafka集群

Kafka是由Apache软件基金会开发的一个开源流处理平台，由Scala和Java编写。Kafka是一种高吞吐量的分布式发布订阅消息系统，它可以处理消费者在网站中的所有动作流数据。
####一.下载

JDK8，点击进入下载
Kafka2.4.0，下载，新特性MirrorMaker 2.0，[KAFKA-8634] - Update ZooKeeper to 3.5.5
ZooKeeper3.5.5，下载带-bin的apache-zookeeper-3.5.5-bin.tar.gz
####二.安装
2.1 修改主机域名(参考)，设置机器间互信(不是必须，参考)

hostnamectl set-hostname kafka1
cat /etc/hostname

vi /etc/hosts
192.168.0.101 kafka1
192.168.0.102 kafka2
192.168.0.103 kafka3
reboot

2.2安装JDK
将文件拷贝到任意目录，解压；在/usr目录下创建java目录，然后将jdk文件夹移动到java目录

tar -zxvf jdk-8u144-linux-x64.tar.gz
mkdir /usr/java
mv jdk1.8.0_144/ /usr/java

配置环境变量：修改/etc/profile，在末尾添加java信息

export JAVA_HOME=/usr/java/jdk1.8.0_144
export JRE_HOME=${JAVA_HOME}/jre 
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib 
export PATH=${JAVA_HOME}/bin:$PATH

重新加载profile，查看Java安装情况

[root@kafka1 test]# source /etc/profile
[root@kafka1 test]# java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

2.3 安装ZooKeeper(首先安装)
解压zk安装包，修改配置文件

tar -zxvf apache-zookeeper-3.5.5-bin.tar.gz 
cd apache-zookeeper-3.5.5-bin/conf/
vi zoo.cfg
添加
dataDir=/data/app/zookeeper/zkdata
server.1=192.168.0.101:2888:3888
server.2=192.168.0.102:2888:3888
server.3=192.168.0.103:2888:3888
设置myid文件，在dataDir目录下，并且要求 myid中的值要与server.X中设置的X相一致, 注意1后面的空格
echo '1' > /data/app/zookeeper/zkdata/myid   

启动zk

进去zookeeper目录
启动ZK服务:       sh bin/zkServer.sh start
查看ZK服务状态:    sh bin/zkServer.sh status
停止ZK服务:       sh bin/zkServer.sh stop
重启ZK服务:       sh bin/zkServer.sh restart

三台分别配置启动后，查看启动结果（1个leader、2个follower）

[root@kafka1 zookeeper-3.5.5]# sh bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /data/app/zookeeper/zookeeper-3.5.5/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

2.4 安装Kafka
解压Kafka安装包，修改配置文件

tar -zxvf kafka_2.12-2.4.0.tgz 
vi server.properties 
broker.id=1
listeners=PLAINTEXT://192.168.0.101:9092
log.dirs=/data/app/kafka/kafka_log
zookeeper.connect=192.168.0.101:2181,192.168.0.102:2181,192.168.0.103:2181

后台启动Kafka

nohup ./bin/kafka-server-start.sh config/server.properties 1>/dev/null 2>&1 &

查看Kafk集群运行情况

[root@kafka1 kafka_2.12-2.4.0]# jps
30530 Jps
30122 Kafka
29439 QuorumPeerMain（zookeeper）

创建Topic、消费Topic测试

创建topic
./bin/kafka-topics.sh --create --bootstrap-server 192.168.0.101:9092 --replication-factor 1 --partitions 2 --topic test-topic1
命令行生产数据
./bin/kafka-console-producer.sh --broker-list 192.168.0.101:9092 --topic test-topic1
命令行从头消费
./bin/kafka-console-consumer.sh --bootstrap-server 192.168.0.101:9092  --topic test-topic1 --from-beginning
重置指定消费offset(待补齐参数测试)
Reset offsets to a specific offset.
./bin/kafka-consumer-groups.sh --help
./bin/kafka-consumer-groups.sh --to-offset
查看所有topics
./bin/kafka-topics.sh --list --bootstrap-server 192.168.0.101:9092
删除topic
./bin/kafka-topics.sh --delete  --bootstrap-server 192.168.0.101:9092 --topic test-topic1
删除所有消费组
./bin/kafka-consumer-groups.sh --bootstrap-server 192.168.0.101:9092 --delete --all-groups 

自定义监控Kafka消费进度

举例5秒获取一次消费进度
[root@kafka1 kafka_monitor]# cat kafka_monitor.sh 
!/bin/sh
customer=${1}
while [ true ]; do
/bin/sleep 5
date "+%Y-%m-%d %H:%M:%S" >> monitor.log
sh /data/app/kafka/kafka_2.12-2.4.0/bin/kafka-consumer-groups.sh --describe --bootstrap-server 192.168.0.101:9092 --group $customer >> monitor.log
done
返回结果案例：
2020-08-27 13:00:42
GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
customer-6      A.test-topic11  0          337500          339700          2200            consumer-1-f1474c11-4aeb-402b-b423-df0355f135cb /192.168.0.107  consumer-1
customer-6      A.test-topic11  1          337500          339700          2200            consumer-1-f1474c11-4aeb-402b-b423-df0355f135cb /192.168.0.107  consumer-1
2020-08-27 13:00:46
GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
customer-6      A.test-topic11  0          407500          409600          2100            consumer-1-f1474c11-4aeb-402b-b423-df0355f135cb /192.168.0.107  consumer-1
customer-6      A.test-topic11  1          407500          409599          2099            consumer-1-f1474c11-4aeb-402b-b423-df0355f135cb /192.168.0.107  consumer-1
2020-08-27 13:00:50
GROUP           TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
customer-6      A.test-topic11  0          477500          479488          1988            consumer-1-f1474c11-4aeb-402b-b423-df0355f135cb /192.168.0.107  consumer-1
customer-6      A.test-topic11  1          477500          479450          1950            consumer-1-f1474c11-4aeb-402b-b423-df0355f135cb /192.168.0.107  consumer-1

停止运行Kafka

[root@kafka1 kafka_2.12-2.4.0]#./bin/kafka-server-stop.sh
或
[root@kafka1 kafka_2.12-2.4.0]# kill -9 30122

以上!