Kafka-3.3.4摒弃zookeeper集群部署操作文档
一、集群规划
kafka从2.8版本开始，就允许用户在不需要zookeeper的情况下运行。真正全面摒弃zookeeper的版本是3.0，在后面的版本kafka使用了新的元数据管理方式kraft,提高了Kafka的可扩展性、可用性和性能。
下载地址：Apache Kafka
这里下载kafka_2.13-3.4.0.tgz版本，自带有支持raft模式
| 主机名称 | IP | process.roles | node.id |
|---|---|---|---|
| master | 192.168.20.101 | broker,controller | 1 |
| salve1 | 192.168.20.102 | broker,controller | 2 |
| salve2 | 192.168.20.103 | broker,controller | 3 |
二、集群部署
将安装包上传到/opt/software目录下,解压到安装目录/opt/moudle/
[appview@master software]$ll -rw-rw-r-- 1 appview appview 106290956 5月 2 08:26 kafka_2.13-3.4.0.tgz [appview@master software]$ tar -zxvf kafka_2.13-3.4.0.tgz -C /opt/moudle/ [appview@master moudle]$ mv kafka_2.13-3.4.0 kafka-3.4.0 [appview@master kafka-3.4.0]$ cd config/kraft/ [appview@master kraft]$ ll 总用量 24 -rw-r--r-- 1 appview appview 6136 2月 1 2023 broker.properties -rw-r--r-- 1 appview appview 5765 2月 1 2023 controller.properties -rw-r--r-- 1 appview appview 6340 2月 1 2023 server.properties [appview@master kraft]$ pwd /opt/moudle/kafka-3.4.0/config/kraft [appview@master kraft]$ vim server.properties
下面的文件是我的配置
只需要修改下面这几个配置成相应的ip地址以及node.id
node.id=1 （##修改） ### node.id=2 ### node.id=3 对应我们的集群规划 [email protected]:9093,[email protected]:9093,[email protected]:9093 listeners=PLAINTEXT://192.168.20.101:9092,CONTROLLER://192.168.20.101:9093 （##修改） advertised.listeners=PLAINTEXT://192.168.20.166:9092 （##修改） log.dirs=/tmp/kraft-combined-logs （可选择）
配置文件server.properties
# Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # This configuration file is intended for use in KRaft mode, where # Apache ZooKeeper is not present. See config/kraft/README.md for details. # ############################# Server Basics ############################# # The role of this server. Setting this puts us in KRaft mode process.roles=broker,controller # The node id associated with this instance's roles node.id=1 # The connect string for the controller quorum [email protected]:9093,[email protected]:9093,[email protected]:9093 ############################# Socket Server Settings ############################# # The address the socket server listens on. # Combined nodes (i.e. those with `process.roles=broker,controller`) must list the controller listener here at a minimum. # If the broker listener is not defined, the default listener will use a host name that is equal to the value of java.net.InetAddress.getCanonicalHostName(), # with PLAINTEXT listener name, and port 9092. # FORMAT: # listeners = listener_name://host_name:port # EXAMPLE: # listeners = PLAINTEXT://your.host.name:9092 listeners=PLAINTEXT://192.168.20.101:9092,CONTROLLER://192.168.20.101:9093 # Name of listener used for communication between brokers. inter.broker.listener.name=PLAINTEXT # Listener name, hostname and port the broker will advertise to clients. # If not set, it uses the value for "listeners". advertised.listeners=PLAINTEXT://192.168.20.101:9092 # A comma-separated list of the names of the listeners used by the controller. # If no explicit mapping set in `listener.security.protocol.map`, default will be using PLAINTEXT protocol # This is required if running in KRaft mode. controller.listener.names=CONTROLLER # Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL # The number of threads that the server uses for receiving requests from the network and sending responses to the network num.network.threads=3 # The number of threads that the server uses for processing requests, which may include disk I/O num.io.threads=8 # The send buffer (SO_SNDBUF) used by the socket server socket.send.buffer.bytes=102400 # The receive buffer (SO_RCVBUF) used by the socket server socket.receive.buffer.bytes=102400 # The maximum size of a request that the socket server will accept (protection against OOM) socket.request.max.bytes=104857600 ############################# Log Basics ############################# # A comma separated list of directories under which to store log files log.dirs=/tmp/kraft-combined-logs # The default number of log partitions per topic. More partitions allow greater # parallelism for consumption, but this will also result in more files across # the brokers. num.partitions=1 # The number of threads per data directory to be used for log recovery at startup and flushing at shutdown. # This value is recommended to be increased for installations with data dirs located in RAID array. num.recovery.threads.per.data.dir=1 ############################# Internal Topic Settings ############################# # The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state" # For anything other than development testing, a value greater than 1 is recommended to ensure availability such as 3. offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 ############################# Log Flush Policy ############################# # Messages are immediately written to the filesystem but by default we only fsync() to sync # the OS cache lazily. The following configurations control the flush of data to disk. # There are a few important trade-offs here: # 1. Durability: Unflushed data may be lost if you are not using replication. # 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush. # 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks. # The settings below allow one to configure the flush policy to flush data after a period of time or # every N messages (or both). This can be done globally and overridden on a per-topic basis. # The number of messages to accept before forcing a flush of data to disk #log.flush.interval.messages=10000 # The maximum amount of time a message can sit in a log before we force a flush #log.flush.interval.ms=1000 ############################# Log Retention Policy ############################# # The following configurations control the disposal of log segments. The policy can # be set to delete segments after a period of time, or after a given size has accumulated. # A segment will be deleted whenever *either* of these criteria are met. Deletion always happens # from the end of the log. # The minimum age of a log file to be eligible for deletion due to age log.retention.hours=168 # A size-based retention policy for logs. Segments are pruned from the log unless the remaining # segments drop below log.retention.bytes. Functions independently of log.retention.hours. #log.retention.bytes=1073741824 # The maximum size of a log segment file. When this size is reached a new log segment will be created. log.segment.bytes=1073741824 # The interval at which log segments are checked to see if they can be deleted according # to the retention policies log.retention.check.interval.ms=300000
配置环境变量
[appview@master ~]$ vim .bashrc [appview@master ~]$ pwd /home/appview [appview@master ~]$ ll /home/appview/.bashrc -rw-r--r-- 1 appview appview 868 4月 13 22:35 /home/appview/.bashrc #### 添加以下内容 #KAFKA_HOME export KAFKA_HOME=/opt/moudle/kafka-3.4.0 export PATH=$PATH:$KAFKA_HOME/bin #### source以下使环境变量生效 [appview@master ~]$ source /home/appview/.bashrc
分发安装包，没有分发脚本的可以scp过去，注意：分发之后记得配置其他机器的环境变量以及修改配置
[appview@master ~]$ xsync /opt/moudle/kafka-3.4.0/
三、集群ID初始化
切换到kafka-3.3.4家目录下执行以下命令生成集群唯一的ID，在其中一个节点执行即可
[appview@master kafka-3.4.0]$ ./bin/kafka-storage.sh random-uuid rBEaIR-6Qx2xLfHh95nFCA [appview@master kafka-3.4.0]$
指定集群id进行集群初始化，每个节点都执行
[appview@master kafka-3.4.0]$ ./bin/kafka-storage.sh format -t rBEaIR-6Qx2xLfHh95nFCA -c ./config/kraft/server.properties Formatting /opt/moudle/kafka-3.4.0/logs/kraft-combined-logs with metadata.version 3.4-IV0. [appview@master kafka-3.4.0]$
四、启动集群
在每个部署的节点执行以下启动命令，jps查看进程，三个节点都已经有进程了
[appview@master kafka-3.4.0]$ ./bin/kafka-server-start.sh -daemon ./config/kraft/server.properties [appview@master kafka-3.4.0]$ jps 13577 Jps 13502 Kafka [appview@master kafka-3.4.0]$ [appview@salve1 kafka-3.4.0]$ ./bin/kafka-server-start.sh -daemon ./config/kraft/server.properties [appview@salve1 kafka-3.4.0]$ jps 12772 Jps 12669 Kafka [appview@salve1 kafka-3.4.0]$ [appview@salve2 kafka-3.4.0]$ ./bin/kafka-server-start.sh -daemon ./config/kraft/server.properties [appview@salve2 kafka-3.4.0]$ jps 6611 Jps 6591 Kafka [appview@salve2 kafka-3.4.0]$
关闭集群命令
[appview@salve2 kafka-3.4.0]$ bin/kafka-server-stop.sh
五、集群验证
使用以下命令测试集群是否可用，如果和下面的一样就说明我们安装可以了
###### 列出所有主题 [appview@master kafka-3.4.0]$ bin/kafka-topics.sh --list --bootstrap-server master:9092 [appview@master kafka-3.4.0]$ ###### 创建主题 [appview@master kafka-3.4.0]$ bin/kafka-topics.sh --create --bootstrap-server master:9092 --replication-factor 1 --partitions 1 --topic hello-world Created topic hello-world. [appview@master kafka-3.4.0]$ #### 启动生产者 [appview@master kafka-3.4.0]$ bin/kafka-console-producer.sh --bootstrap-server master:9092 --topic hello-world > > >how are you ? >i am fine. > #### 启动salve1消费者验证 [appview@salve1 kafka-3.4.0]$ bin/kafka-console-consumer.sh --bootstrap-server salve1:9092 --topic hello-world --from-beginning how are you ? i am fine. #### 启动salve2消费者验证 [appview@salve2 kafka-3.4.0]$ bin/kafka-console-consumer.sh --bootstrap-server salve2:9092 --topic hello-world --from-beginning how are you ? i am fine. asdasdsafswfwqfqwgwg
六、结语
最近忙得天昏地暗的，没时间继续更新文档，kafka集群是我在实际项目中部署了多次总结的，其中最近的一个项目中测试环境部署了5套kafka集群，生产环境部署了2套，今天有空在自己的虚拟机上安装部署并把每个操作记录后分享的