kafka安装文档

一、环境准备

1、JDK1.8的环境

2、创建kafka用户并进入其家目录

3、下载安装包

4、解压并进入目录

[root@node1 ~]# java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)
[root@node1 ~]# useradd -m kafka ; cd /home/kafka
[root@node1 kafka]# wget http://mirrors.hust.edu.cn/apache/kafka/2.0.0/kafka_2.11-2.0.0.tgz
100%[=======================>] 55,751,827  1.51MB/s   in 38s 
[root@node1 kafka]# tar xf kafka_2.11-2.0.0.tgz && mv kafka_2.11-2.0.0 kafka && cd kafka

二、安装模式选择

1、为每个broker准备一份配置文件

2、分别修改每个配置文件信息

• broker.id=分别为0、1、2
• listeners = PLAINTEXT://localhost:端口号分别为9092、9093、9094
• log.dirs=分别为/tmp/kafka-logs、/tmp/kafka-logs1、/tmp/kafka-log2

3、启动自带zookeeper服务

4、依次启动多个broker

[root@node1 kafka]# cp config/server.properties config/server1.properties 
[root@node1 kafka]# cp config/server.properties config/server2.properties 
[root@node1 kafka]# vim config/server.properties
[root@node1 kafka]# vim config/server1.properties 
[root@node1 kafka]# vim config/server2.properties 
[root@node1 kafka]# ./bin/zookeeper-server-start.sh -daemon config/zookeeper.properties 
[root@node1 kafka]# ./bin/kafka-server-start.sh -daemon config/server.properties 
[root@node1 kafka]# netstat -anpl | grep 9092
tcp6       0      0 127.0.0.1:9092          :::*                    LISTEN      57283/java          
tcp6       0      0 127.0.0.1:38886         127.0.0.1:9092          ESTABLISHED 57283/java          
tcp6       0      0 127.0.0.1:9092          127.0.0.1:38886         ESTABLISHED 57283/java          
[root@node1 kafka]# ./bin/kafka-server-start.sh -daemon config/server1.properties 
[root@node1 kafka]# netstat -anpl | grep 9093
tcp6       0      0 127.0.0.1:9093          :::*                    LISTEN      57626/java          
[root@node1 kafka]# ./bin/kafka-server-start.sh -daemon config/server2.properties 
[root@node1 kafka]# netstat -anpl | grep 9094
tcp6       0      0 127.0.0.1:9094          :::*                    LISTEN      57966/java          
tcp6       0      0 127.0.0.1:35228         127.0.0.1:9094          ESTABLISHED 57283/java          
tcp6       0      0 127.0.0.1:9094          127.0.0.1:35228         ESTABLISHED 57966/java 

 三、kafka集群

前提：kafka集群所有主机都要完成环境准备

1、修改所有主机的server.properties文件

• broker.id=分别为0、1、2
• listeners = PLAINTEXT://localhost:端口号分别为9092、9093、9094
• zookeeper.connect=192.168.88.10:2181,192.168.88.12:2181,192.168.88.13:2181

注意：所有主机zookeeper.connect值相同，ip:2181，中间用英文逗号隔开

2、启动所有主机的zookeeper服务

3、启动所有主机的kafka服务

注意：nodex表示所有主机都要执行，node1/2/3只需要在其中一个节点执行

[root@nodex kafka]# vim config/server.properties
[root@nodex kafka]# ./bin/zookeeper-server-start.sh -daemon config/zookeeper.properties && netstat -anpl | grep 2181
tcp6       0      0 :::2181                 :::*                    LISTEN      59624/java 
[root@node1 kafka]# ./bin/kafka-server-start.sh -daemon config/server.properties && netstat -anpl | grep 9092
tcp6       0      0 127.0.0.1:9092          :::*                    LISTEN      59940/java          
tcp6       0      0 127.0.0.1:9092          127.0.0.1:38916         ESTABLISHED 59940/java          
tcp6       0      0 127.0.0.1:38916         127.0.0.1:9092          ESTABLISHED 59940/java 
[root@node2 kafka]# ./bin/kafka-server-start.sh -daemon config/server.properties && netstat -anpl | grep 9093
tcp6       0      0 127.0.0.1:9093          :::*                    LISTEN      42376/java          
tcp6       0      0 127.0.0.1:39436         127.0.0.1:9093          ESTABLISHED 42376/java          
tcp6       0      0 127.0.0.1:9093          127.0.0.1:39436         ESTABLISHED 42376/java 
[root@node3 kafka]# ./bin/kafka-server-start.sh -daemon config/server.properties && netstat -anpl | grep 9094
tcp6       0      0 127.0.0.1:9094          :::*                    LISTEN      41731/java          
tcp6       0      0 127.0.0.1:9094          127.0.0.1:55364         ESTABLISHED 41731/java          
tcp6       0      0 127.0.0.1:55364         127.0.0.1:9094          ESTABLISHED 41731/java 

四、验证测试

1、创建并查看topic

[root@node1 kafka]# ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".
[root@node1 kafka]# ./bin/kafka-topics.sh --list --zookeeper localhost:2181
test

2、生产消息

[root@node1 kafka]# ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>pre send!!
>

3、消费消息

[root@node1 kafka]# ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
pre send!!