【Kafka】性能测试命令

提示：一直在用kafka但是对它的性能一直没有啥认识，趁着这次优化算法中台的算法效率问题，梳理一下这块。

---

kafka是个消息中间件，现在我们要搞清楚它的性能，离开硬件资源配置说性能都是xjb整。

---

1.硬件资源

cpu:8h memery:16G disk:1.5T

cpu 查看资源

cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c #查看cpu型号

8 Intel® Xeon® Gold 5218 CPU @ 2.30GHz

硬盘信息
安装工具smartmontools

yum -y install smartmontools

查看磁盘型号：smartctl --all /dev/sda

[root@localhost bin]# smartctl --all /dev/sda
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-3.10.0-957.el7.x86_64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               VMware
Product:              Virtual disk
Revision:             2.0
Compliance:           SPC-4
User Capacity:        1,649,267,441,664 bytes [1.64 TB]
Logical block size:   512 bytes
LU is fully provisioned
Rotation Rate:        15000 rpm
Device type:          disk
Local Time is:        Wed Dec 28 19:27:45 2022 CST
SMART support is:     Unavailable - device lacks SMART capability.

=== START OF READ SMART DATA SECTION ===
Current Drive Temperature:     0 C
Drive Trip Temperature:        0 C

Error Counter logging not supported

Device does not support Self Test logging

命令指引看这里:https://blog.51cto.com/u_15127621/4055708

2.测试

测试脚本：
kafka-producer-perf-test --topic test --num-records 100 --record-size 1 --throughput 100 --producer-props bootstrap.servers=localhost:9092

localhost 不好使的话，换成127.0.0.1 或者 实际ip， 和config的配置有关。

kafka 被运维装到了这里，里面的shell脚本如下面：

[root@localhost bin]# pwd
/usr/local/kafka/bin

[root@localhost bin]# ls -alh
总用量 464K
drwxr-xr-x. 3 root root 4.0K 10月 27 08:53 .
drwxr-xr-x. 8 root root  117 12月 19 11:04 ..
-rwxr-xr-x. 1 root root 1.4K 9月  14 2021 connect-distributed.sh
-rwxr-xr-x. 1 root root 1.4K 9月  14 2021 connect-mirror-maker.sh
-rwxr-xr-x. 1 root root 1.4K 9月  14 2021 connect-standalone.sh
-rwxr-xr-x. 1 root root  861 9月  14 2021 kafka-acls.sh
-rwxr-xr-x. 1 root root  873 9月  14 2021 kafka-broker-api-versions.sh
-rwxr-xr-x. 1 root root  860 9月  14 2021 kafka-cluster.sh
-rwxr-xr-x. 1 root root  864 9月  14 2021 kafka-configs.sh
-rwxr-xr-x. 1 root root  945 9月  14 2021 kafka-console-consumer.sh
-rwxr-xr-x. 1 root root  944 9月  14 2021 kafka-console-producer.sh
-rwxr-xr-x. 1 root root  871 9月  14 2021 kafka-consumer-groups.sh
-rwxr-xr-x. 1 root root  948 9月  14 2021 kafka-consumer-perf-test.sh
-rwxr-xr-x. 1 root root  871 9月  14 2021 kafka-delegation-tokens.sh
-rwxr-xr-x. 1 root root  869 9月  14 2021 kafka-delete-records.sh
-rwxr-xr-x. 1 root root  866 9月  14 2021 kafka-dump-log.sh
-rwxr-xr-x. 1 root root  863 9月  14 2021 kafka-features.sh
-rwxr-xr-x. 1 root root  870 9月  14 2021 kafka-leader-election.sh
-rwxr-xr-x. 1 root root  863 9月  14 2021 kafka-log-dirs.sh
-rwxr-xr-x. 1 root root  873 9月  14 2021 kafka-metadata-shell.sh
-rwxr-xr-x. 1 root root  862 9月  14 2021 kafka-mirror-maker.sh
-rwxr-xr-x. 1 root root  886 9月  14 2021 kafka-preferred-replica-election.sh
-rwxr-xr-x. 1 root root  959 9月  14 2021 kafka-producer-perf-test.sh
-rwxr-xr-x. 1 root root  874 9月  14 2021 kafka-reassign-partitions.sh
-rwxr-xr-x. 1 root root  874 9月  14 2021 kafka-replica-verification.sh
-rwxr-xr-x. 1 root root  11K 9月  14 2021 kafka-run-class.sh
-rwxr-xr-x. 1 root root 1.4K 9月  14 2021 kafka-server-start.sh
-rwxr-xr-x. 1 root root 1.4K 9月  14 2021 kafka-server-stop.sh
-rwxr-xr-x. 1 root root  860 9月  14 2021 kafka-storage.sh
-rwxr-xr-x. 1 root root  945 9月  14 2021 kafka-streams-application-reset.sh
-rwxr-xr-x. 1 root root  863 9月  14 2021 kafka-topics.sh
-rwxr-xr-x. 1 root root  958 9月  14 2021 kafka-verifiable-consumer.sh
-rwxr-xr-x. 1 root root  958 9月  14 2021 kafka-verifiable-producer.sh
-rw-------. 1 root root 304K 12月 28 18:56 nohup.out
-rwxr-xr-x. 1 root root 1.7K 9月  14 2021 trogdor.sh
drwxr-xr-x. 2 root root 4.0K 9月  14 2021 windows
-rwxr-xr-x. 1 root root  867 9月  14 2021 zookeeper-security-migration.sh
-rwxr-xr-x. 1 root root 1.4K 9月  14 2021 zookeeper-server-start.sh
-rwxr-xr-x. 1 root root 1.4K 9月  14 2021 zookeeper-server-stop.sh
-rwxr-xr-x. 1 root root 1019 9月  14 2021 zookeeper-shell.sh

测试命令 sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1 --throughput 100 --producer-props bootstrap.servers=10.10.90.210:9092

[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1 --throughput 100 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 99.601594 records/sec (0.00 MB/sec), 8.77 ms avg latency, 341.00 ms max latency, 1 ms 50th, 14 ms 95th, 341 ms 99th, 341 ms 99.9th.

num-records，消息数量
record-size，单条消息的大小，单位为字节
throughput，多少秒一条消息，-1为没有间歇

[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 267.379679 records/sec (0.00 MB/sec), 18.74 ms avg latency, 367.00 ms max latency, 15 ms 50th, 18 ms 95th, 367 ms 99th, 367 ms 99.9th.

可以看到单点kafka， 消息没有间歇的前提下 1kb 一条数据，可以发 267条/秒

[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1 --throughput 100 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 99.601594 records/sec (0.00 MB/sec), 8.77 ms avg latency, 341.00 ms max latency, 1 ms 50th, 14 ms 95th, 341 ms 99th, 341 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 267.379679 records/sec (0.00 MB/sec), 18.74 ms avg latency, 367.00 ms max latency, 15 ms 50th, 18 ms 95th, 367 ms 99th, 367 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 10 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 280.112045 records/sec (0.00 MB/sec), 15.42 ms avg latency, 349.00 ms max latency, 12 ms 50th, 14 ms 95th, 349 ms 99th, 349 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 10 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 267.379679 records/sec (0.00 MB/sec), 16.51 ms avg latency, 367.00 ms max latency, 13 ms 50th, 15 ms 95th, 367 ms 99th, 367 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 10 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 285.714286 records/sec (0.00 MB/sec), 17.30 ms avg latency, 344.00 ms max latency, 14 ms 50th, 16 ms 95th, 344 ms 99th, 344 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 100 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 275.482094 records/sec (0.03 MB/sec), 16.33 ms avg latency, 356.00 ms max latency, 13 ms 50th, 15 ms 95th, 356 ms 99th, 356 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 100 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 264.550265 records/sec (0.03 MB/sec), 15.85 ms avg latency, 371.00 ms max latency, 12 ms 50th, 14 ms 95th, 371 ms 99th, 371 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1000 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 268.096515 records/sec (0.26 MB/sec), 20.13 ms avg latency, 360.00 ms max latency, 17 ms 50th, 18 ms 95th, 360 ms 99th, 360 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 10000 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 246.305419 records/sec (2.35 MB/sec), 45.08 ms avg latency, 344.00 ms max latency, 44 ms 50th, 60 ms 95th, 344 ms 99th, 344 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 100000 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 229.885057 records/sec (21.92 MB/sec), 60.76 ms avg latency, 353.00 ms max latency, 58 ms 50th, 79 ms 95th, 353 ms 99th, 353 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1000000 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 138.121547 records/sec (131.72 MB/sec), 115.93 ms avg latency, 353.00 ms max latency, 122 ms 50th, 137 ms 95th, 353 ms 99th, 353 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1000000 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 142.857143 records/sec (136.24 MB/sec), 105.84 ms avg latency, 339.00 ms max latency, 121 ms 50th, 141 ms 95th, 339 ms 99th, 339 ms 99.9th.

从以上脚本可以看出，仅仅是修改了 record-size，也就是消息体的大小，当1,10,100,1000,10000kb的时候，都是250records/s 以上，max latency延时则从10毫秒到了几十毫秒。
当增加到100000kb （100MB），1000000kb（1G）的时候，显著下降为100+records/s 延时也增加到100毫秒以上，但正常情况下也没有人会发100MB，或者 1G每条消息的数据吧。

2018年我在搞直播系统的时候，当时的答题回传答案用的kafka，那时候看的二手技术博客说单条消息大于10kb的时候，kafka性能会显著下降，现在看不是那么回事。眼见为实，到底咋样，拉出来溜溜！

3.总结

单体kafka， 8h 16G 的服务器，1kb每条消息， kafka的承载量是接近300条消息/s，延时在10毫秒，
随之，单体消息的数据量增加，延时在增加，但1kb----10MB的数据，消息承载量没有显著下降从280–240，平均延时延时从10毫秒到50毫秒