kafka集群的使用
前提条件
三台Linux虚拟机
安装好kafka集群，可参考 安装kafka集群
使用kafka
Kafka常用命令
启动集群
启动zk
[hadoop@node2 kafka]$ zk.sh start ---------- zookeeper node2 启动 ------------ ZooKeeper JMX enabled by default Using config: /home/hadoop/soft/zookeeper-3.5.7/bin/../conf/zoo.cfg Starting zookeeper ... STARTED ---------- zookeeper node3 启动 ------------ ZooKeeper JMX enabled by default Using config: /home/hadoop/soft/zookeeper-3.5.7/bin/../conf/zoo.cfg Starting zookeeper ... STARTED ---------- zookeeper node4 启动 ------------ ZooKeeper JMX enabled by default Using config: /home/hadoop/soft/zookeeper-3.5.7/bin/../conf/zoo.cfg Starting zookeeper ... STARTED
启动kafka
[hadoop@node2 kafka]$ kf.sh start --------启动 node2 Kafka------- --------启动 node3 Kafka------- --------启动 node4 Kafka-------
jps查看进程
[hadoop@node2 kafka]$ jps 3267 Kafka 2868 QuorumPeerMain 3354 Jps
查看topic
[hadoop@node2 kafka]$ kafka-topics.sh --zookeeper node2:2181/kafka --list first
创建topic
[hadoop@node2 kafka]$ kafka-topics.sh --zookeeper node2:2181,node3:2181,node4:2181/kafka --create --replication-factor 1 --partitions 1 --topic topic_log
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic topic_log.
查看zk里的topic
[hadoop@node3 ~]$ zkCli.sh [zk: localhost:2181(CONNECTED) 3] ls -R /kafka /kafka /kafka/admin /kafka/brokers /kafka/cluster /kafka/config /kafka/consumers /kafka/controller /kafka/controller_epoch /kafka/isr_change_notification /kafka/latest_producer_id_block /kafka/log_dir_event_notification /kafka/admin/delete_topics /kafka/brokers/ids /kafka/brokers/seqid /kafka/brokers/topics /kafka/brokers/ids/0 /kafka/brokers/ids/1 /kafka/brokers/ids/2 /kafka/brokers/topics/first /kafka/brokers/topics/topic_log /kafka/brokers/topics/first/partitions /kafka/brokers/topics/first/partitions/0 /kafka/brokers/topics/first/partitions/0/state /kafka/brokers/topics/topic_log/partitions /kafka/brokers/topics/topic_log/partitions/0 /kafka/brokers/topics/topic_log/partitions/0/state /kafka/cluster/id /kafka/config/brokers /kafka/config/changes /kafka/config/clients /kafka/config/topics /kafka/config/users /kafka/config/topics/first /kafka/config/topics/topic_log
kafka生产消息
[hadoop@node2 kafka]$ kafka-console-producer.sh --broker-list node2:9092 --topic topic_log >hello world >hello ha
kafka消费消息
打开一个新的node2窗口
[hadoop@node2 kafka]$ kafka-console-consumer.sh --bootstrap-server node2:9092 --from-beginning --topic topic_log hello world hello ha
--from-beginning：会把主题中以往所有的数据都读取出来。根据业务场景选择是否增加该配置。
删除topic
[hadoop@node2 kafka]$ kafka-topics.sh --delete --zookeeper node2:2181,node3:2181,node4:2181/kafka --topic topic_log Topic topic_log is marked for deletion. Note: This will have no impact if delete.topic.enable is not set to true.
Kafka压力测试
用Kafka官方自带的脚本，对Kafka进行压测。Kafka压测时，可以查看到哪个地方出现了瓶颈（CPU，内存，网络IO）。一般都是网络IO达到瓶颈。
kafka-consumer-perf-test.sh
kafka-producer-perf-test.sh
Kafka Producer压力测试
（1）在/opt/module/kafka/bin目录下面有这两个文件。我们来测试一下
[hadoop@node2 kafka]$ kafka-producer-perf-test.sh --topic test --record-size 100 --num-records 100000 --throughput -1 --producer-props bootstrap.servers=node2:9092,node2:9092,node4:9092
[2022-04-07 10:40:11,991] WARN [Producer clientId=producer-1] Error while fetching metadata with correlation id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
100000 records sent, 94073.377234 records/sec (8.97 MB/sec), 317.07 ms avg latency, 451.00 ms max latency, 320 ms 50th, 440 ms 95th, 446 ms 99th, 450 ms 99.9th.
test会自动创建出来
[hadoop@node2 kafka]$ bin/kafka-topics.sh --zookeeper node2:2181/kafka --list __consumer_offsets first test topic_log
Kafka Consumer压力测试
Consumer的测试，如果这四个指标（IO，CPU，内存，网络）都不能改变，考虑增加分区数来提升性能。
[hadoop@node2 kafka]$ kafka-consumer-perf-test.sh --broker-list node2:9092,node3:9092,node4:9092 --topic test --fetch-size 10000 --messages 10000000 --threads 1 start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec WARNING: Exiting before consuming the expected number of messages: timeout (10000 ms) exceeded. You can use the --timeout option to increase the timeout. 2022-04-07 10:46:58:881, 2022-04-07 10:47:09:972, 9.5367, 0.8599, 100000, 9016.3195, 1649299619086, -1649299607995, -0.0000, -0.0001
用csv格式查看输出结果
查看kafa本地数据
[hadoop@node2 ~]$ cd $KAFKA_HOME/ [hadoop@node2 kafka]$ ls bin config data libs LICENSE logs NOTICE site-docs [hadoop@node2 kafka]$ cd data/
[hadoop@node2 data]$ ls cleaner-offset-checkpoint __consumer_offsets-24 __consumer_offsets-39 first-0 __consumer_offsets-0 __consumer_offsets-27 __consumer_offsets-42 log-start-offset-checkpoint __consumer_offsets-12 __consumer_offsets-3 __consumer_offsets-45 meta.properties __consumer_offsets-15 __consumer_offsets-30 __consumer_offsets-48 recovery-point-offset-checkpoint __consumer_offsets-18 __consumer_offsets-33 __consumer_offsets-6 replication-offset-checkpoint __consumer_offsets-21 __consumer_offsets-36 __consumer_offsets-9 test-0 [hadoop@node3 data]$ ls cleaner-offset-checkpoint __consumer_offsets-22 __consumer_offsets-4 first-0 __consumer_offsets-1 __consumer_offsets-25 __consumer_offsets-40 log-start-offset-checkpoint __consumer_offsets-10 __consumer_offsets-28 __consumer_offsets-43 meta.properties __consumer_offsets-13 __consumer_offsets-31 __consumer_offsets-46 recovery-point-offset-checkpoint __consumer_offsets-16 __consumer_offsets-34 __consumer_offsets-49 replication-offset-checkpoint __consumer_offsets-19 __consumer_offsets-37 __consumer_offsets-7 topic_log-0 [hadoop@node4 data]$ ls cleaner-offset-checkpoint __consumer_offsets-23 __consumer_offsets-41 log-start-offset-checkpoint __consumer_offsets-11 __consumer_offsets-26 __consumer_offsets-44 meta.properties __consumer_offsets-14 __consumer_offsets-29 __consumer_offsets-47 recovery-point-offset-checkpoint __consumer_offsets-17 __consumer_offsets-32 __consumer_offsets-5 replication-offset-checkpoint __consumer_offsets-2 __consumer_offsets-35 __consumer_offsets-8 __consumer_offsets-20 __consumer_offsets-38 first-0
创建多个分区的topic
kafka-topics.sh --create --zookeeper node2:2181 --topic testmultipart --partitions 3 --replication-factor 1 创建失败 [hadoop@node2 config]$ kafka-topics.sh --create --zookeeper node2:2181 --topic testmultipart --partitions 3 --replication-factor 1 Error while executing topic command : Replication factor: 1 larger than available brokers: 0. [2022-04-07 12:47:24,185] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 1 larger than available brokers: 0. (kafka.admin.TopicCommand$) 失败原因： zk的地址要写成zookeeper.connect设置的地址 zookeeper.connect=node2:2181,node3:2181,node4:2181/kafka 创建成功 [hadoop@node2 config]$ kafka-topics.sh --create --zookeeper node2:2181/kafka --topic testmultipart --partitions 3 --replication-factor 1 Created topic testmultipart. 查看本地数据目录，topic有3个分区，每台机器各存放一个分区 看到 node2 有testmultipart-1 看到 node3 有testmultipart-2 看到 node4 有testmultipart-0 [hadoop@node2 data]$ ls cleaner-offset-checkpoint __consumer_offsets-27 __consumer_offsets-45 recovery-point-offset-checkpoint __consumer_offsets-0 __consumer_offsets-3 __consumer_offsets-48 replication-offset-checkpoint __consumer_offsets-12 __consumer_offsets-30 __consumer_offsets-6 test-0 __consumer_offsets-15 __consumer_offsets-33 __consumer_offsets-9 testmultipart-1 __consumer_offsets-18 __consumer_offsets-36 first-0 __consumer_offsets-21 __consumer_offsets-39 log-start-offset-checkpoint __consumer_offsets-24 __consumer_offsets-42 meta.properties [hadoop@node3 data]$ ls cleaner-offset-checkpoint __consumer_offsets-25 __consumer_offsets-43 recovery-point-offset-checkpoint __consumer_offsets-1 __consumer_offsets-28 __consumer_offsets-46 replication-offset-checkpoint __consumer_offsets-10 __consumer_offsets-31 __consumer_offsets-49 testmultipart-2 __consumer_offsets-13 __consumer_offsets-34 __consumer_offsets-7 topic_log-0 __consumer_offsets-16 __consumer_offsets-37 first-0 __consumer_offsets-19 __consumer_offsets-4 log-start-offset-checkpoint __consumer_offsets-22 __consumer_offsets-40 meta.properties [hadoop@node4 data]$ ls cleaner-offset-checkpoint __consumer_offsets-23 __consumer_offsets-41 log-start-offset-checkpoint __consumer_offsets-11 __consumer_offsets-26 __consumer_offsets-44 meta.properties __consumer_offsets-14 __consumer_offsets-29 __consumer_offsets-47 recovery-point-offset-checkpoint __consumer_offsets-17 __consumer_offsets-32 __consumer_offsets-5 replication-offset-checkpoint __consumer_offsets-2 __consumer_offsets-35 __consumer_offsets-8 testmultipart-0 __consumer_offsets-20 __consumer_offsets-38 first-0
查看topic的描述
[hadoop@node2 data]$ kafka-topics.sh --describe --topic testmultipart --zookeeper node2:2181/kafka
Topic: testmultipart PartitionCount: 3 ReplicationFactor: 1 Configs:
Topic: testmultipart Partition: 0 Leader: 2 Replicas: 2 Isr: 2
Topic: testmultipart Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Topic: testmultipart Partition: 2 Leader: 1 Replicas: 1 Isr: 1
isr: in sync replicas
创建多个分区多个副本的topic，并查看topic的描述
[hadoop@node2 data]$ kafka-topics.sh --create --zookeeper node2:2181/kafka --topic testmultipart2 --partitions 3 --replication-factor 2
Created topic testmultipart2.
[hadoop@node2 data]$ kafka-topics.sh --describe --topic testmultipart2 --zookeeper node2:2181/kafka
Topic: testmultipart2 PartitionCount: 3 ReplicationFactor: 2 Configs:
Topic: testmultipart2 Partition: 0 Leader: 1 Replicas: 1,0 Isr: 1,0
Topic: testmultipart2 Partition: 1 Leader: 2 Replicas: 2,1 Isr: 2,1
Topic: testmultipart2 Partition: 2 Leader: 0 Replicas: 0,2 Isr: 0,2
以上数据看出，broker 0（Leader 0） 有副本数0和2； broker 1 有副本数1和0；broker 2 有副本数2和1，验证如下：
查看本地数据
[hadoop@node2 data]$ ls cleaner-offset-checkpoint __consumer_offsets-27 __consumer_offsets-45 recovery-point-offset-checkpoint __consumer_offsets-0 __consumer_offsets-3 __consumer_offsets-48 replication-offset-checkpoint __consumer_offsets-12 __consumer_offsets-30 __consumer_offsets-6 test-0 __consumer_offsets-15 __consumer_offsets-33 __consumer_offsets-9 testmultipart-1 __consumer_offsets-18 __consumer_offsets-36 first-0 testmultipart2-0 __consumer_offsets-21 __consumer_offsets-39 log-start-offset-checkpoint testmultipart2-2 __consumer_offsets-24 __consumer_offsets-42 meta.properties [hadoop@node3 data]$ ls cleaner-offset-checkpoint __consumer_offsets-25 __consumer_offsets-43 recovery-point-offset-checkpoint __consumer_offsets-1 __consumer_offsets-28 __consumer_offsets-46 replication-offset-checkpoint __consumer_offsets-10 __consumer_offsets-31 __consumer_offsets-49 testmultipart-2 __consumer_offsets-13 __consumer_offsets-34 __consumer_offsets-7 testmultipart2-0 __consumer_offsets-16 __consumer_offsets-37 first-0 testmultipart2-1 __consumer_offsets-19 __consumer_offsets-4 log-start-offset-checkpoint topic_log-0 __consumer_offsets-22 __consumer_offsets-40 meta.properties [hadoop@node4 data]$ ls cleaner-offset-checkpoint __consumer_offsets-32 log-start-offset-checkpoint __consumer_offsets-11 __consumer_offsets-35 meta.properties __consumer_offsets-14 __consumer_offsets-38 recovery-point-offset-checkpoint __consumer_offsets-17 __consumer_offsets-41 replication-offset-checkpoint __consumer_offsets-2 __consumer_offsets-44 testmultipart-0 __consumer_offsets-20 __consumer_offsets-47 testmultipart2-1 __consumer_offsets-23 __consumer_offsets-5 testmultipart2-2 __consumer_offsets-26 __consumer_offsets-8 __consumer_offsets-29 first-0
完成！enjoy it！