Kafka 高可用集群搭建

前言

kafka集群搭建需要在已经安装Java JDK 和Zookeeper的基础上。

如未安装JDK和Zookeeper，请参考如下博客：

JDK安装：https://blog.csdn.net/lichenglong33/article/details/99636768

Zookeeper 集群安装 ：https://blog.csdn.net/lichenglong33/article/details/103894582

准备工作

操作系统：centos 7；

JDK：1.8

安装包：kafka_2.11-2.4.0.tgz

192.168.16.6 master
192.168.16.4 slave1
192.168.16.5 slave2

Kafka集群安装

 第一步：到官网下载对应版本的kafka安装包。

http://kafka.apache.org/downloads

第二步：将下载的安装包上传并解压到/home/kafka 目录下（目录如不存在则新建 mkdir kafka），命令如下

[root@master software]# rz -y
[root@master software]# tar -xzvf ./kafka_2.11-2.4.0.tgz -C /home/kafka/

第三步：修改配置文件（server.properties）

broker.id=1
listeners=PLAINTEXT://master:9092
advertised.listeners=PLAINTEXT://master:9092
log.dirs=/home/kafka/logs
zookeeper.connect=master:2181,slave1:2181,slave2:2181

第四步：将master配置好的kafka复制到slave1和slave2机器上

[root@master home]# scp -r /home/kafka/ root@slave1:/home/
[root@master home]# scp -r /home/kafka/ root@slave2:/home/

第五步：修改broker.id

   slave1机器上的修改为broker.id=2 ；    slave2机器上的修改为broker.id=3

第六步：后台启动

[root@master software]# cd /home/kafka/kafka_2.11-2.4.0/
[root@master kafka_2.11-2.4.0]# ./bin/kafka-server-start.sh -daemon ./config/server.properties

第七步：验证启动是否成功

[root@master kafka_2.11-2.4.0]# ps -ef|grep kafka
root      7586     1 13 18:47 pts/0    00:00:05 /java/jdk1.8.0_161/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Xloggc:/home/kafka/kafka_2.11-2.4.0/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/home/kafka/kafka_2.11-2.4.0/bin/../logs -Dlog4j.configuration=file:./bin/../config/log4j.properties -cp .:/java/jdk1.8.0_161/lib/dt.jar:/java/jdk1.8.0_161/lib/tools.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/activation-1.1.1.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/aopalliance-repackaged-2.5.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/argparse4j-0.7.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/audience-annotations-0.5.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/commons-cli-1.4.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/commons-lang3-3.8.1.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/connect-api-2.4.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/connect-basic-auth-extension-2.4.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/connect-file-2.4.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/connect-json-2.4.0.jar:/home/kafkakafka_2.11-2.4.0/bin/../libs/connect-mirror-2.4.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/connect-mirror-client-2.4.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/connect-runtime-2.4.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/connect-transforms-2.4.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/guava-20.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/hk2-api-2.5.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/hk2-locator-2.5.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/hk2-utils-2.5.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jackson-annotations-2.10.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jackson-core-2.10.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jackson-databind-2.10.0.jar:/home/kafk/kafka_2.11-2.4.0/bin/../libs/jackson-dataformat-csv-2.10.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jackson-datatype-jdk8-2.10.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jackson-module-paranamer-2.10.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jackson-module-scala_2.11-2.10.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jakarta.activation-api-1.2.1.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jakarta.annotation-api-1.3.4.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jakarta.inject-2.5.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/javassist-3.22.0-CR2.jar:/home/kafk/kafka_2.11-2.4.0/bin/../libs/javax.servlet-api-3.1.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/javax.ws.rs-api-2.1.1.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jaxb-api-2.3.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jersey-client-2.28.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jersey-common-2.28.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jersey-container-servlet-2.28.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jersey-container-servlet-core-2.28.jar:/home/kafkakafka_2.11-2.4.0/bin/../libs/jersey-hk2-2.28.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jersey-media-jaxb-2.28.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jersey-server-2.28.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jetty-client-9.4.20.v20190813.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jetty-continuation-9.4.20.v20190813.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jetty-http-9.4.20.v20190813.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jetty-io-9.4.20.v20190813.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jetty-security-9.4.20.v20190813.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jetty-server-9.4.20.v20190813.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jetty-servlet-9.4.20.v20190813.jar:/home/kafk/kafka_2.11-2.4.0/bin/../libs/jetty-servlets-9.4.20.v20190813.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jetty-util-9.4.20.v20190813.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/jopt-simple-5.0.4.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/kafka_2.11-2.4.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/kafka_2.11-2.4.0-sources.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/kafka-clients-2.4.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/kafka-log4j-appender-2.4.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/kafka-streams-2.4.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/kafka-streams-examples-2.4.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/kafka-streams-scala_2.11-2.4.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/kafka-streams-test-utils-2.4.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/kafka-tools-2.4.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/log4j-1.2.17.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/lz4-java-1.6.0.jar:/home/kafka/kafk_2.11-2.4.0/bin/../libs/maven-artifact-3.6.1.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/metrics-core-2.2.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/netty-buffer-4.1.42.Final.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/netty-codec-4.1.42.Final.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/netty-common-4.1.42.Final.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/netty-handler-4.1.42.Final.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/netty-resolver-4.1.42.Final.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/netty-transport-4.1.42.Final.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/netty-transport-native-epoll-4.1.42.Final.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/netty-transport-native-unix-common-4.1.42.Final.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/paranamer-2.8.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/plexus-utils-3.2.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/reflections-0.9.11.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/rocksdbjni-5.18.3.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/scala-collection-compat_2.11-2.1.2.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/scala-java8-compat_2.11-0.9.0.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/scala-library-2.11.12.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/scala-logging_2.11-3.9.2.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/scala-reflect-2.11.12.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/slf4j-api-1.7.28.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/slf4j-log4j12-1.7.28.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/snappy-java-1.1.7.3.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/validation-api-2.0.1.Final.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/zookeeper-3.5.6.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/zookeeper-jute-3.5.6.jar:/home/kafka/kafka_2.11-2.4.0/bin/../libs/zstd-jni-1.4.3-1.jar kafka.Kafka ./config/server.properties
root      7654  1225  0 18:48 pts/0    00:00:00 grep --color=auto kafka

如果进程存在则表示启动成功。