Kafka集群配置文档

拓扑结构介绍：

192.168.43.60 kafka01

192.168.43.61 kafka02

192.168.43.62 kafka03

OS : CentOS Linux release 7.7.1908 (Core)

JAVA : jdk-8u162-linux-x64.tar.gz

Kafka : kafka_2.11-2.0.0 .tgz

Zookeeper : zookeeper-3.4.12.tar.gz

一、系统基本配置（所有节点执行）

1.基本软件安装

[root@kafka01 ~]# yum install -y wget curl vim net-tools ntp

2.关闭SELINUX及防火墙及/etc/hosts修改

[root@kafka01 ~]# grep '^SELINUX' /etc/selinux/config

SELINUX=disabled

SELINUXTYPE=targeted

[root@kafka01 ~]# systemctl status firewalld

● firewalld.service - firewalld - dynamic firewall daemon

Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)

Active: inactive (dead)

Docs: man:firewalld(1)

[root@kafka01 ~]# cat /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.43.60 kafka01

192.168.43.61 kafka02

192.168.43.62 kafka03

[root@kafka01 ~]#

3.开启NTP服务

[root@kafka01 ~]# grep '^server' /etc/ntp.conf

server ntp.aliyun.com

[root@kafka01 ~]# systemctl enable ntpd

[root@kafka01 ~]# systemctl start ntpd

[root@kafka01 ~]#

二、基础环境的配置（所有节点执行）

1.jdk1.8的安装

[root@kafka01 opt]# cd /opt

[root@kafka01 opt]# ls

jdk-8u162-linux-x64.tar.gz kafka_2.11-2.0.0 .tgz zookeeper-3.4.12.tar.gz

[root@kafka01 opt]# tar -xf jdk-8u162-linux-x64.tar.gz

[root@kafka01 opt]# ls

jdk1.8.0_162 kafka_2.11-2.0.0 .tgz

jdk-8u162-linux-x64.tar.gz zookeeper-3.4.12.tar.gz

[root@kafka01 opt]# ls jdk1.8.0_162/

bin javafx-src.zip man THIRDPARTYLICENSEREADME-JAVAFX.txt

db lib release

include LICENSE src.zip

[root@kafka01 opt]# mv jdk1.8.0_162 /usr/local/

[root@kafka01 opt]# ls

jdk-8u162-linux-x64.tar.gz kafka_2.11-2.0.0 .tgz zookeeper-3.4.12.tar.gz

[root@kafka01 opt]# vim /etc/profile

[root@kafka01 opt]# tail -n 4 /etc/profile

export JAVA_HOME=/usr/local/jdk1.8.0_162

export JRE_HOME=/usr/local/jdk1.8.0_162/jre

export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JRE_HOME/lib

export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin

[root@kafka01 opt]# source /etc/profile

[root@kafka01 opt]# java -version

java version "1.8.0_162"

Java(TM) SE Runtime Environment (build 1.8.0_162-b12)

Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)

[root@kafka01 opt]#

2.zookeeper的配置

[root@kafka01 opt]# tar -xf zookeeper-3.4.12.tar.gz

[root@kafka01 opt]# ls

jdk-8u162-linux-x64.tar.gz zookeeper-3.4.12

kafka_2.11-2.0.0 .tgz zookeeper-3.4.12.tar.gz

[root@kafka01 opt]# mv zookeeper-3.4.12 /usr/local/

[root@kafka01 opt]# vim /etc/profile

[root@kafka01 opt]# tail -n 2 /etc/profile

export ZK_HOME=/usr/local/zookeeper-3.4.12

export PATH=$ZK_HOME/bin:$PATH

[root@kafka01 opt]# source /etc/profile

[root@kafka01 opt]# cp /usr/local/zookeeper-3.4.12/conf/zoo_sample.cfg /usr/local/zookeeper-3.4.12/conf/zoo.cfg

[root@kafka01 opt]#

[root@kafka01 opt]# vim /usr/local/zookeeper-3.4.12/conf/zoo.cfg

[root@kafka01 opt]# tail -n 4 /usr/local/zookeeper-3.4.12/conf/zoo.cfg

dataDir=/usr/local/zookeeper-3.4.12/data

server.1=192.168.43.60:2888:3888

server.2=192.168.43.61:2888:3888

server.3=192.168.43.62:2888:3888

[root@kafka01 opt]#

[root@kafka01 opt]# mkdir /usr/local/zookeeper-3.4.12/data

--------------------------------------------------

说明：zookeeper三个端口作用：

2181：对cline端提供服务

2888：集群内机器通讯使用（Leader监听此端口）

3888：选举leader时机器之间通信使用

--------------------------------------------------

3.创建myid文件，在myid文件中添加本机的 server ID

kafka01上：

[root@kafka01 opt]# echo "1" > /usr/local/zookeeper-3.4.12/data/myid

[root@kafka01 opt]#

kafka02上：

[root@kafka02 opt]# echo "2" > /usr/local/zookeeper-3.4.12/data/myid

[root@kafka02 opt]#

kafka03上：

[root@kafka03 opt]# echo "3" > /usr/local/zookeeper-3.4.12/data/myid

[root@kafka03 opt]#