Linux搭建kafka集群并测试

1 准备3台机器

hostname	ip
server1	192.168.16.151
server2	192.168.16.152
server3	192.168.16.153

1.1host配置

#以server1为例
127.0.0.1       server1
192.168.16.151  server1
192.168.16.152  server2
192.168.16.153  server3

2 zookeeper配置

参考上篇博客 安装Zookeeper集群

3 安装kafka

3.1 设置环境变量

解压kafka后添加环境变量

#修改环境变量
sudo vim /etc/profile

export JAVA_HOME=/usr/lib/java/jdk1.8.0_181
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH

export SCALA_HOME=/usr/lib/scala/scala-2.11.8
export PATH=${SCALA_HOME}/bin:$PATH

export ZOOKEEPER_HOME=/usr/local/programs/zookeeper-3.4.13
export PATH=${ZOOKEEPER_HOME}/bin:$PATH

export KAFKA_HOME=/usr/local/programs/kafka_2.11-2.1.0
export PATH=${KAFKA_HOME}/bin:$PATH

#让环境变量生效
source /etc/profile

3.2 修改改$KAFKA_HOME/config 目录下的server.properties文件

在三台机器上分别修改,这里用server1举例

#这个1就是和zookeeper的myid文件的1对应
broker.id=1
#填写3台机器的地址,中间用逗号隔开
zookeeper.connect=server1:2181,server2:2181,server3:2181
#host.name是当前机器的地址
host.name=192.168.16.151

3.4 启动zookeeper和kafka

在3台机器上分别启动

#启动zookeeper
zkServer.sh start

#启动kafka
kafka-server-start.sh -daemon /usr/local/programs/kafka_2.11-2.1.0/config/server.properties 

#查看进程
jps

3.5 测试

学习参考博客

#在leader上创建主题
kafka-topics.sh --create --zookeeper server1:2181,server2:2181,server3:2181 --replication-factor 3 --partitions 3 --topic WinterTop

#查看主题List
kafka-topics.sh --list --zookeeper server1:2181,server2:2181,server3:2181

#在leader上发送消息
kafka-console-producer.sh --broker-list server1:9092,server2:9092,server3:9092 --topic WinterTop

#在follower上消费消息
kafka-console-consumer.sh --bootstrap-server server1:9092,server2:9092,server3:9092  --from-beginning --topic WinterTop