查看命令
- 查看主题
[root@host-192-168-2-165 ~]# kafka-topics.sh --list --zookeeper=10.143.200.13:24002,10.143.200.14:24002,10.143.200.15:24002/kafka
__consumer_offsets
__default_metrics
example-metric1
iot-water
jf-alert
jf-device
jf-pic-visitor
jf-visitor
sensor-connector-data
sensor-device-added-data
sensor-device-deleted-data
sensor-parsed-basedata
sensor-parsed-over-threshold
- 查看kafka特定topic的详情，使用
--topic与--describe参数
parition数量、replica因子以及每个partition的leader、replica信息
[root@host-192-168-2-165 ~]# kafka-topics.sh --zookeeper=10.143.200.13:24002,10.143.200.14:24002,10.143.200.15:24002/kafka --topic iot-water --describe
Topic:iot-water PartitionCount:6 ReplicationFactor:2 Configs:
Topic: iot-water Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2
Topic: iot-water Partition: 1 Leader: 2 Replicas: 2,1 Isr: 2,1
Topic: iot-water Partition: 2 Leader: 1 Replicas: 1,2 Isr: 1,2
Topic: iot-water Partition: 3 Leader: 2 Replicas: 2,1 Isr: 2,1
Topic: iot-water Partition: 4 Leader: 1 Replicas: 1,2 Isr: 1,2
Topic: iot-water Partition: 5 Leader: 2 Replicas: 2,1 Isr: 2,1
- 查看topic订阅用户
[root@host-192-168-2-165 ~]# kafka-acls.sh --authorizer-properties zookeeper.connect=10.143.200.13:24002,10.143.200.14:24002,10.143.200.15:24002/kafka --list --topic iot-water
Current ACLs for resource `Topic:iot-water`:
User:huanshui has Allow permission for operations: Read from hosts: *
User:huanshui has Allow permission for operations: Describe from hosts: *
- 增加消费端
kafka-acls.sh --authorizer-properties zookeeper.connect=10.143.200.13:24002,10.143.200.14:24002,10.143.200.15:24002/kafka \
--add --allow-principal User:kafka_dfgx \
--consumer --topic jf-device --group ioc-device-dfgx
- 查看消费者组
[root@host-192-168-2-165 ~]# kafka-consumer-groups.sh --list --bootstrap-server 10.143.200.19:21007,10.143.200.20:21007 --command-config /opt/hadoopclient/Kafka/kafka/config/consumer.properties
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
iot-water-huanshui
parser-connector-data
group_iot_huawei004
iot-persistence
operation-connector-data
iot-push
group_jf_huawei0010
iot-check
- 查看消费者组情况(旧版本
kafka.tools.ConsumerOffsetChecker 新版本：kafka-consumer-groups.sh)
TOPIC: 该消费者组消费的是哪些topic；
PARTITION: 表示该消费者消费的是哪些分区；
CURRENT-OFFSET: 表示消费者组最新消费的位移值, 此值在消费过程中是变化的；
LOG-END-OFFSET: 表示topic所有分区当前的日志终端位移值
LAG: 表示滞后进度, 此值为LOG-END-OFFSET 与 CURRENT-OFFSET的差值, 代表的是滞后情况, 此值越大表示滞后严重；
[root@host-192-168-2-165 ~]# kafka-consumer-groups.sh --bootstrap-server 10.143.200.19:21007,10.143.200.20:21007 --command-config /opt/hadoopclient/Kafka/kafka/config/consumer.properties --group iot-water-huanshui --describe
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
Consumer group 'iot-water-huanshui' has no active members.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
iot-water 2 29954 32836 2882 - - -
iot-water 0 22395 24365 1970 - - -
iot-water 3 8613 8875 262 - - -
iot-water 4 10246 10508 262 - - -
iot-water 5 34561 37277 2716 - - -
iot-water 1 15281 16142 861 - - -
[root@host-192-168-2-165 ~]# kafka-consumer-groups.sh --new-consumer --bootstrap-server 10.143.200.19:21007,10.143.200.20:21007 --command-config /opt/hadoopclient/Kafka/kafka/config/consumer.properties --group iot-water-huanshui --describe
Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
iot-water 3 716413 716413 0 consumer-1-df445e3a-d106-4b54-a138-a4b06e55fc23 /10.143.11.70 consumer-1
iot-water 4 785438 785439 1 consumer-1-df445e3a-d106-4b54-a138-a4b06e55fc23 /10.143.11.70 consumer-1
iot-water 5 382630 382630 0 consumer-1-df445e3a-d106-4b54-a138-a4b06e55fc23 /10.143.11.70 consumer-1
iot-water 0 797089 797089 0 consumer-1-a55a13ef-3131-4c70-a653-ebaf6377a144 /10.143.11.74 consumer-1
iot-water 1 630126 630126 0 consumer-1-a55a13ef-3131-4c70-a653-ebaf6377a144 /10.143.11.74 consumer-1
iot-water 2 344468 344468 0 consumer-1-a55a13ef-3131-4c70-a653-ebaf6377a144 /10.143.11.74 consumer-1
创建命令
- 创建主题
kafka-topics.sh --create --topic test-demo --partitions 6 --replication-factor 2 --zookeeper=10.143.200.13:24002,10.143.200.14:24002,10.143.200.15:24002/kafka
- 查看topic权限
kafka-acls.sh --authorizer-properties zookeeper.connect=10.143.200.13:24002,10.143.200.14:24002,10.143.200.15:24002/kafka --list --topic test-demo
- 添加生产者
kafka-acls.sh --authorizer-properties zookeeper.connect=10.143.200.13:24002,10.143.200.14:24002,10.143.200.15:24002/kafka --add --allow-principal User:test --producer --topic test-demo
[root@host-192-168-2-165 ~]# kafka-acls.sh --authorizer-properties zookeeper.connect=10.143.200.13:24002,10.143.200.14:24002,10.143.200.15:24002/kafka --add --allow-principal User:test --producer --topic test-demo
Adding ACLs for resource `Topic:test-demo`:
User:test has Allow permission for operations: Write from hosts: *
User:test has Allow permission for operations: Describe from hosts: *
Adding ACLs for resource `Cluster:kafka-cluster`:
User:test has Allow permission for operations: Create from hosts: *
Current ACLs for resource `Topic:test-demo`:
User:test has Allow permission for operations: Write from hosts: *
User:test has Allow permission for operations: Describe from hosts: *
[root@host-192-168-2-165 ~]# kafka-acls.sh --authorizer-properties zookeeper.connect=10.143.200.13:24002,10.143.200.14:24002,10.143.200.15:24002/kafka --list --topic test-demo
Current ACLs for resource `Topic:test-demo`:
User:test has Allow permission for operations: Write from hosts: *
User:test has Allow permission for operations: Describe from hosts: *
