kafka常用命令

KFKA版本： kafka_2.11-2.2.0

添加环境变量：

vi ~/.bash_profile

PATH=$PATH:$HOME/bin:/app/kafka_2.11-2.2.0/bin

export PATH

kafka在bin目录下提供的一些工具：

kafka-acls.sh                        kafka-preferred-replica-election.sh
kafka-broker-api-versions.sh         kafka-producer-perf-test.sh
kafka-configs.sh                     kafka-reassign-partitions.sh
kafka-console-consumer.sh            kafka-replica-verification.sh
kafka-console-producer.sh            kafka-run-class.sh
kafka-consumer-groups.sh             kafka-server-start.sh
kafka-consumer-perf-test.sh          kafka-server-stop.sh
kafka-delegation-tokens.sh           kafka-streams-application-reset.sh
kafka-delete-records.sh              kafka-topics.sh
kafka-dump-log.sh                    kafka-verifiable-consumer.sh
kafka-log-dirs.sh                    kafka-verifiable-producer.sh
kafka-mirror-maker.sh 

---

1.启停文件

启停文件内容

启动文件：
[root@kafka bin]# cat kafka-server-start.sh
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one or more

....

COMMAND=$1
case $COMMAND in
  -daemon)
    EXTRA_ARGS="-daemon "$EXTRA_ARGS
    shift
    ;;
  *)
    ;;
esac

exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"
[root@kafka bin]# 


停止文件：
[root@kafka bin]# cat kafka-server-stop.sh 
#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
...
PIDS=$(ps ax | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk '{print $1}')

if [ -z "$PIDS" ]; then
  echo "No kafka server to stop"
  exit 1
else
  kill -s $SIGNAL $PIDS
fi

可以看到stop里使用的kill命令

pid获取是通过ps进行的， 但是由于操作系统对于ps的长度有限制，有时候ps并不能正确获取到pid， 从而无法通过该命令脚本停止，用户可以根据自己的环境进行修改

---

2.topic 主题

# kafka-topics.sh --help
This tool helps to create, delete, describe, or change a topic.
Option                                   Description                            
------                                   -----------                            
--alter                                  Alter the number of partitions,        
                                           replica assignment, and/or           
                                           configuration for the topic.         
--bootstrap-server                               to. In case of providing this, a     
                                           direct Zookeeper connection won't be 
                                           required.                            
--command-config                     passed to Admin Client. This is used 
  ....
  
 --under-replicated-partitions            if set when describing topics, only    
                                           show under replicated partitions     
--zookeeper               DEPRECATED, The connection string for  
                                           the zookeeper connection in the form 
                                           host:port. Multiple hosts can be     
                                           given to allow fail-over.

查看：

# kafka-topics.sh --list --zookeeper 127.0.0.1:2181
__consumer_offsets
mtest

查看topic信息

kafka-topics.sh --describe --zookeeper 127.0.0.1:2181 --topic mtest
Topic:mtest     PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: mtest    Partition: 0    Leader: 0       Replicas: 0     Isr: 0

ISR： 同步的副本数量

创建topic

[root@kafka ~]# kafka-topics.sh --create --topic test0 -zookeeper 127.0.0.1:2181 --partitions 2 --replication-factor 1
Created topic test0.

[root@kafka ~]# kafka-topics.sh --list --zookeeper 127.0.0.1:2181
__consumer_offsets
mtest
test0
[root@kafka ~]# kafka-topics.sh --describe --zookeeper 127.0.0.1:2181 --topic test0
Topic:test0     PartitionCount:2        ReplicationFactor:1     Configs:
        Topic: test0    Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: test0    Partition: 1    Leader: 0       Replicas: 0     Isr: 0
[root@kafka ~]# 

partitions 分区数量

replication-factor 副本数量

kafka存储消息的日志文件：

server配置文件：

# A comma separated list of directories under which to store log files
log.dirs=/tmp/kafka-logs

查看一个分区：

[root@kafka mtest-0]# ll
total 28
-rw-r--r-- 1 root root 10485760 Mar 12 02:15 00000000000000000011.index
-rw-r--r-- 1 root root    10168 Mar 12 04:45 00000000000000000011.log
-rw-r--r-- 1 root root       10 Mar 11 08:26 00000000000000000011.snapshot
-rw-r--r-- 1 root root 10485756 Mar 12 02:15 00000000000000000011.timeindex
-rw-r--r-- 1 root root        9 Mar 11 08:26 leader-epoch-checkpoint
[root@kafka mtest-0]# pwd
/tmp/kafka-logs/mtest-0

拥有log文件，index文件，timeindex文件等

添加分区（不允许减少分区）

[root@kafka mtest-0]# kafka-topics.sh --alter --topic test0 -zookeeper 127.0.0.1:2181 --partitions 3
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
[root@kafka mtest-0]# 
[root@kafka mtest-0]# kafka-topics.sh --describe --zookeeper 127.0.0.1:2181 --topic test0
Topic:test0     PartitionCount:3        ReplicationFactor:1     Configs:
        Topic: test0    Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: test0    Partition: 1    Leader: 0       Replicas: 0     Isr: 0
        Topic: test0    Partition: 2    Leader: 0       Replicas: 0     Isr: 0

删除topic：

[root@kafka mtest-0]# kafka-topics.sh --list --zookeeper 127.0.0.1:2181
__consumer_offsets
mtest
test0
test1
[root@kafka mtest-0]# 
[root@kafka mtest-0]# kafka-topics.sh --delete --topic test1 -zookeeper 127.0.0.1:2181
Topic test1 is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
[root@kafka mtest-0]# kafka-topics.sh --list --zookeeper 127.0.0.1:2181
__consumer_offsets
mtest
test0
[root@kafka mtest-0]# 

获取偏移量：

# kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 127.0.0.1:9092 --topic test0 --time -1
test0:0:0
test0:1:0

消息堆积信息：

[root@kafka mtest-0]# kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --list
console-consumer-57275

[root@kafka mtest-0]# kafka-consumer-groups.sh --bootstrap-server 127.0.0.1:9092 --describe --group console-consumer-57275 

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
test0           0          -               6               -               consumer-1-05e24e75-7d36-480d-964a-c16905a7923a /192.168.137.51 consumer-1
test0           1          -               6               -               consumer-1-05e24e75-7d36-480d-964a-c16905a7923a /192.168.137.51 consumer-1
test0           2          -               5               -               consumer-1-05e24e75-7d36-480d-964a-c16905a7923a /192.168.137.51 consumer-1
[root@kafka mtest-0]# 

LAG 为堆积消息