Kafka常用命令(带SASL权限版)

前言

一般需要使用的Kafka原生命令多位于bin目录，其下使用较多的脚本如下

• kafka-topics.sh
• kafka-console-producer.sh
• kafka-console-consumer.sh
• kafka-acls.sh
• kafka-consumer-groups.sh
• kafka-server-start.sh
• kafka-server-stop.sh

Kafka Orders

• Topic相关(创建/查询/删除/修改分片/平衡leader)

# 查询
./kafka-topics.sh --list --zookeeper 192.168.100.85:2181,192.168.100.85:2182

# 详情查询
./kafka-topics.sh --zookeeper 192.168.100.85:2181,192.168.100.85:2182 --describe --topic topic-sean

# 创建(3分片 复制2份 注意:replication-factor个数包括本来的个数)
./kafka-topics.sh --create --zookeeper 192.168.100.85:2181 --replication-factor 2 --partitions 3 --topic topic-sean

# 修改分片个数(只能增加，不能删除。删除需要进行迁移。)
./kafka-topics.sh --zookeeper 192.168.100.85:2181 --alter --topic topic-sean --partitions 12

# 删除 (注意 1.如果有权限需要把权限删除/2.如果需要完全删除需要开启server.properties删除属性)
./kafka-topics.sh --zookeeper 192.168.100.85:2181 --delete --topic topic-sean

# 平衡leader（注意平衡后需要使用查询命令查看集群状况）
./kafka-preferred-replica-election.sh --zookeeper zk_host:port/chroot

• 生成/消费(原始)

# 生产者
./kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic topic-sean
# 消费者
./kafka-console-consumer.sh --zookeeper 127.0.0.1:2181 --topic topic-sean --from-beginning

# 新生产者（支持0.9版本+） 
./kafka-console-producer.sh --broker-list localhost:9092 --topic topic-sean --producer.config config/producer.properties 

## 新消费者（支持0.9版本+） 
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic-sean --new-consumer --from-beginning --consumer.config config/consumer.properties 

## 高级点的用法(具体到某个分片)
./kafka-simple-consumer-shell.sh --brist localhost:9092 --topic topic-sean --partition 0 --offset 1234 --max-messages 10

• 生成/消费(sasl权限控制)

# sasl consumer
./kafka-console-consumer.sh  --bootstrap-server 192.168.100.87:9093,192.168.100.88:9093,192.168.100.89:9093  --topic bmc-events-topic-dev --consumer.config ../config/consumer.properties --new-consumer

# sasl producer
./kafka-console-producer.sh --broker-list 192.168.100.87:9093  --topic bmc-events-topic-dev --producer.config ../config/consumer.properties

• sasl 权限

# 增加权限
./kafka-acls.sh --authorizer-properties zookeeper.connect=192.168.100.85:2181 --add --allow-principal User:alice --operation Read --operation Write --topic bmc-events-topic-test

# 查询权限
./kafka-acls.sh --authorizer-properties zookeeper.connect=192.168.100.85:2181 --list --topic sean-security

# 删除权限
./kafka-acls.sh --authorizer-properties zookeeper.connect=192.168.100.85:2181 --remove --allow-principal User:alice --allow-principal User:Alice  --operation Read --operation Write --topic bmc-events-topic-dev

• 生成/消费(offset详情查询与统计)

# 无权限版本查询(ConsumerOffsetChecker)
./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper 192.168.100.85:2181 --group test --topic hello-sean

# 无权限版本查询(GetOffsetShell)
./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 192.168.100.87:9093 --topic hello-sean --time -1

# 含权限版本查询
./kafka-consumer-groups.sh --bootstrap-server 192.168.100.87:9093,192.168.100.88:9093,192.168.100.89:9093 --command-config ../config/consumer.properties --list

# 查询单个consumer group
./kafka-consumer-groups.sh --bootstrap-server 192.168.100.87:9093 --command-config ../config/consumer.properties --describe --group 'console-consumer-93774'

# 查询单个consumer group(含统计)
 ./kafka-consumer-groups.sh --bootstrap-server 192.168.100.87:9093 --command-config ../config/consumer.properties --describe --group 'console-consumer-21925' | awk ' {sum += $3};END {print sum}'

# 使用脚本新生产一个consumer group(为了方便查询)
./kafka-console-consumer.sh  --bootstrap-server 192.168.100.87:9093,192.168.100.88:9093,192.168.100.89:9093  --topic hello-sean --consumer.config ../config/consumer.properties --new-consumer

./kafka-consumer-groups.sh --bootstrap-server 192.168.100.87:9093 --command-config ../config/consumer.properties --describe --group 'event_processor_group'

./kafka-topics.sh --list --zookeeper 192.168.100.85:2181

./kafka-consumer-groups.sh --bootstrap-server 192.168.100.87:9093,192.168.100.88:9093,192.168.100.89:9093 --command-config ../config/consumer.properties --describe --group 'console-consumer-67854'| awk ' {sum += $3};END {print sum}'

./kafka-consumer-groups.sh --bootstrap-server 192.168.100.87:9093,192.168.100.88:9093,192.168.100.89:9093 --command-config ../config/consumer.properties --describe --group 'console-consumer-67854'| awk ' {sum += $4};END {print sum}'

---

Reference

[1]. JIRA-KAFKA-3151(SASL)
[2]. kafka命令大全(orchome.com)
[3]. Kafka命令行常用命令说明
[4]. Kafka常用命令(csdn.yanxml)
[5]. Kafka常用命令行总结

---

Others

bin目录下的所有脚本，分析作用如下：

# version kafka-1.0.0
# kakfa 相关
connect-standalone.sh         
connect-distributed.sh   

# consumer     
kafka-consumer-groups.sh   
kafka-consumer-perf-test.sh          
kafka-console-consumer.sh     
kafka-simple-consumer-shell.sh  

# producer
kafka-console-producer.sh   
kafka-producer-perf-test.sh          

# broker 启/停  
kafka-server-stop.sh            
kafka-server-start.sh  
kafka-broker-api-versions.sh  

# topic
kafka-topics.sh                     

# acls 权限
kafka-acls.sh                 

# others

# kafka config 配置
kafka-configs.sh              

# 重新分片 partition                   
kafka-reassign-partitions.sh

# reset kafka streaming    
kafka-streams-application-reset.sh  

kafka-replay-log-producer.sh    
kafka-delete-records.sh              
kafka-replica-verification.sh   
kafka-verifiable-consumer.sh        
kafka-log-dirs.sh                    
kafka-verifiable-producer.sh
kafka-mirror-maker.sh                
trogdor.sh
kafka-preferred-replica-election.sh  

# run-class
kafka-run-class.sh              

# 内置ZooKeeper相关
zookeeper-shell.sh
zookeeper-server-start.sh
zookeeper-server-stop.sh
zookeeper-security-migration.sh

# windows脚本 文件夹
windows/