kafka压力测试步骤
kafka的性能测试，对于CPU和内存要求不是特别大 最主要的还是磁盘和网络的性能。
生产
kafka本身提供了用于生产者性能测试的命令. kafka-producer-perf-test.sh
我们向一个只有1个分区，1个副本的主题 yache中发送100万条消息,并且每条消息大小为1024B，生产者对应的ack机制是1.
kafka-producer-perf-test.sh --topic yache --num-records 1000000 --record-size 1024 --throughput -1 --producer-props bootstrap.servers=node02:6667 ack=1 --print-metrics
- num-records 表示指定发送消息的总条数。
- record-size 表示每条消息的字节数
- producer-props 表示指定生产者的配置
- throughput 表示进行限流控制 <0表示不限流控制
- print-metrics 表示测试完成之后会打印很多指标信息.
[hadoop@node02 bin]$ kafka-producer-perf-test.sh --topic yache --num-records 1000000 --record-size 1024 --throughput -1 --producer-props bootstrap.servers=node02:6667 ack=1 --print-metrics
[2020-07-30 23:32:23,324] WARN The configuration 'ack' was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig)
125611 records sent, 25117.2 records/sec (24.53 MB/sec), 966.3 ms avg latency, 1522.0 max latency.
192300 records sent, 38460.0 records/sec (37.56 MB/sec), 803.2 ms avg latency, 949.0 max latency.
202320 records sent, 40464.0 records/sec (39.52 MB/sec), 764.4 ms avg latency, 880.0 max latency.
188640 records sent, 37728.0 records/sec (36.84 MB/sec), 808.6 ms avg latency, 987.0 max latency.
214455 records sent, 42891.0 records/sec (41.89 MB/sec), 722.0 ms avg latency, 854.0 max latency.
1000000 records sent, 38067.684343 records/sec (37.18 MB/sec), 780.93 ms avg latency, 1522.00 ms max latency, 769 ms 50th, 983 ms 95th, 1367 ms 99th, 1508 ms 99.9th.
Metric Name Value
app-info:commit-id:{client-id=producer-1} : 4243d589e2b33433
app-info:version:{client-id=producer-1} : 2.0.0.3.1.4.0-315
kafka-metrics-count:count:{client-id=producer-1} : 94.000
producer-metrics:batch-size-avg:{client-id=producer-1} : 15555.923
producer-metrics:batch-size-max:{client-id=producer-1} : 15556.000
producer-metrics:batch-split-rate:{client-id=producer-1} : 0.000
producer-metrics:batch-split-total:{client-id=producer-1} : 0.000
producer-metrics:buffer-available-bytes:{client-id=producer-1} : 33554432.000
producer-metrics:buffer-exhausted-rate:{client-id=producer-1} : 0.000
producer-metrics:buffer-exhausted-total:{client-id=producer-1} : 0.000
producer-metrics:buffer-total-bytes:{client-id=producer-1} : 33554432.000
producer-metrics:bufferpool-wait-ratio:{client-id=producer-1} : 0.398
producer-metrics:bufferpool-wait-time-total:{client-id=producer-1} : 22058025495.000
producer-metrics:compression-rate-avg:{client-id=producer-1} : 1.000
producer-metrics:connection-close-rate:{client-id=producer-1} : 0.000
producer-metrics:connection-close-total:{client-id=producer-1} : 0.000
producer-metrics:connection-count:{client-id=producer-1} : 2.000
producer-metrics:connection-creation-rate:{client-id=producer-1} : 0.036
producer-metrics:connection-creation-total:{client-id=producer-1} : 2.000
producer-metrics:failed-authentication-rate:{client-id=producer-1} : 0.000
producer-metrics:failed-authentication-total:{client-id=producer-1} : 0.000
producer-metrics:incoming-byte-rate:{client-id=producer-1} : 62992.425
producer-metrics:incoming-byte-total:{client-id=producer-1} : 3534001.000
producer-metrics:io-ratio:{client-id=producer-1} : 0.022
producer-metrics:io-time-ns-avg:{client-id=producer-1} : 15171.919
producer-metrics:io-wait-ratio:{client-id=producer-1} : 0.351
producer-metrics:io-wait-time-ns-avg:{client-id=producer-1} : 243222.740
producer-metrics:io-waittime-total:{client-id=producer-1} : 19786413110.000
producer-metrics:iotime-total:{client-id=producer-1} : 1234250744.000
producer-metrics:metadata-age:{client-id=producer-1} : 26.087
producer-metrics:network-io-rate:{client-id=producer-1} : 2376.615
producer-metrics:network-io-total:{client-id=producer-1} : 1044267457.000
producer-metrics:outgoing-byte-rate:{client-id=producer-1} : 18550075.859
producer-metrics:outgoing-byte-total:{client-id=producer-1} : 1040733456.000
producer-metrics:produce-throttle-time-avg:{client-id=producer-1} : 0.000
producer-metrics:produce-throttle-time-max:{client-id=producer-1} : 0.000
producer-metrics:record-error-rate:{client-id=producer-1} : 0.000
producer-metrics:record-error-total:{client-id=producer-1} : 0.000
producer-metrics:record-queue-time-avg:{client-id=producer-1} : 778.984
producer-metrics:record-queue-time-max:{client-id=producer-1} : 1521.000
producer-metrics:record-retry-rate:{client-id=producer-1} : 0.000
producer-metrics:record-retry-total:{client-id=producer-1} : 0.000
producer-metrics:record-send-rate:{client-id=producer-1} : 17859.375
producer-metrics:record-send-total:{client-id=producer-1} : 1000000.000
producer-metrics:record-size-avg:{client-id=producer-1} : 1110.000
producer-metrics:record-size-max:{client-id=producer-1} : 1110.000
producer-metrics:records-per-request-avg:{client-id=producer-1} : 15.000
producer-metrics:request-latency-avg:{client-id=producer-1} : 1.932
producer-metrics:request-latency-max:{client-id=producer-1} : 100.000
producer-metrics:request-rate:{client-id=producer-1} : 1188.329
producer-metrics:request-size-avg:{client-id=producer-1} : 15610.221
producer-metrics:request-size-max:{client-id=producer-1} : 15611.000
producer-metrics:request-total:{client-id=producer-1} : 1040733456.000
producer-metrics:requests-in-flight:{client-id=producer-1} : 0.000
producer-metrics:response-rate:{client-id=producer-1} : 1188.371
producer-metrics:response-total:{client-id=producer-1} : 3534001.000
producer-metrics:select-rate:{client-id=producer-1} : 1444.802
producer-metrics:select-total:{client-id=producer-1} : 19786413110.000
producer-metrics:successful-authentication-rate:{client-id=producer-1} : 0.000
producer-metrics:successful-authentication-total:{client-id=producer-1} : 0.000
producer-metrics:waiting-threads:{client-id=producer-1} : 0.000
producer-node-metrics:incoming-byte-rate:{client-id=producer-1, node-id=node--1} : 6.738
producer-node-metrics:incoming-byte-rate:{client-id=producer-1, node-id=node-1001} : 63102.664
producer-node-metrics:incoming-byte-total:{client-id=producer-1, node-id=node--1} : 378.000
producer-node-metrics:incoming-byte-total:{client-id=producer-1, node-id=node-1001} : 3533623.000
producer-node-metrics:outgoing-byte-rate:{client-id=producer-1, node-id=node--1} : 1.069
producer-node-metrics:outgoing-byte-rate:{client-id=producer-1, node-id=node-1001} : 18584193.068
producer-node-metrics:outgoing-byte-total:{client-id=producer-1, node-id=node--1} : 60.000
producer-node-metrics:outgoing-byte-total:{client-id=producer-1, node-id=node-1001} : 1040733396.000
producer-node-metrics:request-latency-avg:{client-id=producer-1, node-id=node--1} : 0.000
producer-node-metrics:request-latency-avg:{client-id=producer-1, node-id=node-1001} : 1.932
producer-node-metrics:request-latency-max:{client-id=producer-1, node-id=node--1} : -Infinity
producer-node-metrics:request-latency-max:{client-id=producer-1, node-id=node-1001} : 100.000
producer-node-metrics:request-rate:{client-id=producer-1, node-id=node--1} : 0.036
producer-node-metrics:request-rate:{client-id=producer-1, node-id=node-1001} : 1190.457
producer-node-metrics:request-size-avg:{client-id=producer-1, node-id=node--1} : 30.000
producer-node-metrics:request-size-avg:{client-id=producer-1, node-id=node-1001} : 15610.689
producer-node-metrics:request-size-max:{client-id=producer-1, node-id=node--1} : 36.000
producer-node-metrics:request-size-max:{client-id=producer-1, node-id=node-1001} : 15611.000
producer-node-metrics:request-total:{client-id=producer-1, node-id=node--1} : 60.000
producer-node-metrics:request-total:{client-id=producer-1, node-id=node-1001} : 1040733396.000
producer-node-metrics:response-rate:{client-id=producer-1, node-id=node--1} : 0.036
producer-node-metrics:response-rate:{client-id=producer-1, node-id=node-1001} : 1190.543
producer-node-metrics:response-total:{client-id=producer-1, node-id=node--1} : 378.000
producer-node-metrics:response-total:{client-id=producer-1, node-id=node-1001} : 3533623.000
producer-topic-metrics:byte-rate:{client-id=producer-1, topic=yache} : 18522024.736
producer-topic-metrics:byte-total:{client-id=producer-1, topic=yache} : 1037066687.000
producer-topic-metrics:compression-rate:{client-id=producer-1, topic=yache} : 1.000
producer-topic-metrics:record-error-rate:{client-id=producer-1, topic=yache} : 0.000
producer-topic-metrics:record-error-total:{client-id=producer-1, topic=yache} : 0.000
producer-topic-metrics:record-retry-rate:{client-id=producer-1, topic=yache} : 0.000
producer-topic-metrics:record-retry-total:{client-id=producer-1, topic=yache} : 0.000
producer-topic-metrics:record-send-rate:{client-id=producer-1, topic=yache} : 17860.013
producer-topic-metrics:record-send-total:{client-id=producer-1, topic=yache} : 1000000.000
发送的消息总数 record sent : 1000000
以每秒发送的消息数来统计吞吐量 records/sec : 38067
每秒发送的消息大小来统计吞吐量 括号中的MB/sec : 37MB/s
消息处理的平均耗时 avg latency ： 780ms
消息处理的最大耗时 max latency . 1522ms
50th,95th,99th,99.9th 分别表示 50%，95%，99%，99.9%的消息处理耗时
消费
kafka-consumer-perf-test.sh 来进行对消费者的测试。
kafka-consumer-perf-test.sh --topic yache --messages 1000000 --broker-list node02:6667 --print-metrics
[hadoop@node02 bin]$ kafka-consumer-perf-test.sh --topic yache --messages 1000000 --broker-list node02:6667
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-07-30 23:43:42:933, 2020-07-30 23:43:51:900, 977.0264, 108.9580, 1000475, 111572.9899, 3068, 5899, 165.6258, 169600.7798
[hadoop@node02 bin]$ kafka-consumer-perf-test.sh --topic yache --messages 1000000 --broker-list node02:6667 --print-metrics
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-07-30 23:44:49:390, 2020-07-30 23:44:56:665, 977.0264, 134.2992, 1000475, 137522.3368, 3031, 4244, 230.2136, 235738.6899Metric Name Value
consumer-coordinator-metrics:assigned-partitions:{client-id=consumer-1} : 1.000
consumer-coordinator-metrics:commit-latency-avg:{client-id=consumer-1} : 3.000
consumer-coordinator-metrics:commit-latency-max:{client-id=consumer-1} : 3.000
consumer-coordinator-metrics:commit-rate:{client-id=consumer-1} : 0.033
consumer-coordinator-metrics:commit-total:{client-id=consumer-1} : 3.000
consumer-coordinator-metrics:heartbeat-rate:{client-id=consumer-1} : 0.032
consumer-coordinator-metrics:heartbeat-response-time-max:{client-id=consumer-1} : 2.000
consumer-coordinator-metrics:heartbeat-total:{client-id=consumer-1} : 2.000
consumer-coordinator-metrics:join-rate:{client-id=consumer-1} : 0.029
consumer-coordinator-metrics:join-time-avg:{client-id=consumer-1} : 3012.000
consumer-coordinator-metrics:join-time-max:{client-id=consumer-1} : 3012.000
consumer-coordinator-metrics:join-total:{client-id=consumer-1} : 3012.000
consumer-coordinator-metrics:last-heartbeat-seconds-ago:{client-id=consumer-1} : 1.000
consumer-coordinator-metrics:sync-rate:{client-id=consumer-1} : 0.029
consumer-coordinator-metrics:sync-time-avg:{client-id=consumer-1} : 15.000
consumer-coordinator-metrics:sync-time-max:{client-id=consumer-1} : 15.000
consumer-coordinator-metrics:sync-total:{client-id=consumer-1} : 15.000
consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=consumer-1, topic=yache} : 30449631.596
consumer-fetch-manager-metrics:bytes-consumed-rate:{client-id=consumer-1} : 30449631.596
consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=consumer-1, topic=yache} : 1033490946.000
consumer-fetch-manager-metrics:bytes-consumed-total:{client-id=consumer-1} : 1033490946.000
consumer-fetch-manager-metrics:fetch-latency-avg:{client-id=consumer-1} : 3.244
consumer-fetch-manager-metrics:fetch-latency-max:{client-id=consumer-1} : 32.000
consumer-fetch-manager-metrics:fetch-rate:{client-id=consumer-1} : 29.292
consumer-fetch-manager-metrics:fetch-size-avg:{client-id=consumer-1, topic=yache} : 1037641.512
consumer-fetch-manager-metrics:fetch-size-avg:{client-id=consumer-1} : 1037641.512
consumer-fetch-manager-metrics:fetch-size-max:{client-id=consumer-1, topic=yache} : 1038179.000
consumer-fetch-manager-metrics:fetch-size-max:{client-id=consumer-1} : 1038179.000
consumer-fetch-manager-metrics:fetch-throttle-time-avg:{client-id=consumer-1} : 0.000
consumer-fetch-manager-metrics:fetch-throttle-time-max:{client-id=consumer-1} : 0.000
consumer-fetch-manager-metrics:fetch-total:{client-id=consumer-1} : 3231.000
consumer-fetch-manager-metrics:records-consumed-rate:{client-id=consumer-1, topic=yache} : 29477.755
consumer-fetch-manager-metrics:records-consumed-rate:{client-id=consumer-1} : 29476.886
consumer-fetch-manager-metrics:records-consumed-total:{client-id=consumer-1, topic=yache} : 1000475.000
consumer-fetch-manager-metrics:records-consumed-total:{client-id=consumer-1} : 1000475.000
consumer-fetch-manager-metrics:records-lag-avg:{client-id=consumer-1, topic=yache, partition=0} : 1520691.500
consumer-fetch-manager-metrics:records-lag-max:{client-id=consumer-1, topic=yache, partition=0} : 2020679.000
consumer-fetch-manager-metrics:records-lag-max:{client-id=consumer-1} : 2020679.000
consumer-fetch-manager-metrics:records-lag:{client-id=consumer-1, topic=yache, partition=0} : 1020704.000
consumer-fetch-manager-metrics:records-lead-avg:{client-id=consumer-1, topic=yache, partition=0} : 500487.500
consumer-fetch-manager-metrics:records-lead-min:{client-id=consumer-1, topic=yache, partition=0} : 500.000
consumer-fetch-manager-metrics:records-lead-min:{client-id=consumer-1} : 500.000
consumer-fetch-manager-metrics:records-lead:{client-id=consumer-1, topic=yache, partition=0} : 1000475.000
consumer-fetch-manager-metrics:records-per-request-avg:{client-id=consumer-1, topic=yache} : 1004.493
consumer-fetch-manager-metrics:records-per-request-avg:{client-id=consumer-1} : 1004.493
kafka-metrics-count:count:{client-id=consumer-1} : 46.000
输出结果分析：
起始运行时间 start.time：2020-07-30 23:44:49:390
结束运行时间 end.time ：2020-07-30 23:44:56:665
消费的消息总量 data.consumed.in.MB ：977.0264 （MB）
按字节大小计算的消费吞吐量 MB.sec ： 134.2992 （MB/s）
消费的消息总数 data.consumed.in.nMsg ： 1000475
按消息个数计算的吞吐量 nMsg.sec ： 137522.3368
在平衡的时间 rebalance.time.ms ： 3031ms
拉取消息的持续时间 fetch.time.ms : 4244ms
每秒拉取消息的字节大小 fetch.MB.sec : 230.2136 MB/s
每秒拉取消息的个数 fetch.nMsg.sec : 235738.6899
磁盘性能：
我们可以通过 watch iostat -x 来监听磁盘的IO情况。如果%util 达到100%，说明达到了磁盘的瓶颈。
或者kafka的每秒读写速度，达到了磁盘读写速度的极限，也说明到了磁盘的瓶颈。
网卡性能:
千兆网卡 1000Mb/s 换算： 1B=8bit, 1000/8 = 125MB/s. 所以传输速度到达125MB/s的时候是网卡的瓶颈