在典型的Kubernetes部署中，所有到Kubernetes服务的流量都流经ingress。Ingress代理从Internet到后端服务的流量。这样，Ingres就在您提高性能的关键路径上。有多种基准测试和衡量性能的方法。
衡量代理性能的最常见方法可能是原始吞吐量。在这种类型的测试中，通过代理发送的通信量不断增加，并且可以测量代理可以处理的最大通信量。一个典型的衡量标准是衡量每秒请求数（RPS）的性能。
本文主要测试Ambassador(1.4)，contour(1.2.1) 和某云Ingress负载均衡器(openresty)。
测试
测试环境说明
- 后端负载是通过Deployment部署的2个nginx Pod。每个Pod的资源限制为1c1G。
- 所有的负载均衡器均打开了access log。
- 分别创建三个ingress，对应的域名为contour.xx.me, ambassador.xx.me, cloud.xx.me。其中敏感域名，故用xx表示。
测试工具以及场景说明
- 测试工具我们选择wrk。
- 场景分为50线程并发，100线程并发，200线程并发，500线程并发，1000线程并发。
- 所有测试均在本地，压测时长30s。
- 所有结果都只看聚合报告，重点查看吞吐量、时间和错误数。
- 测试均不作任何优化。
测试对比
长连接
测试脚本类似如下：
wrk -c50 -d30s -t8 --latency http://ambassador.xx.mecontour 50并发测试结果：
8 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 85.89ms 4.14ms 120.42ms 82.95%
Req/Sec 69.81 12.34 121.00 76.56%
16723 requests in 30.09s, 13.60MB read
Requests/sec: 555.83
Transfer/sec: 463.02KB某云LB 50并发测试结果：
8 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 86.68ms 6.45ms 261.78ms 91.00%
Req/Sec 69.15 12.77 121.00 85.59%
16571 requests in 30.09s, 13.29MB read
Requests/sec: 550.65
Transfer/sec: 452.25KBambassador 50并发测试结果：
8 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 84.62ms 5.53ms 364.52ms 91.04%
Req/Sec 70.89 12.17 120.00 65.52%
16983 requests in 30.08s, 13.82MB read
Requests/sec: 564.62
Transfer/sec: 470.33KBcontour 100并发测试结果：
8 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 95.72ms 63.15ms 1.29s 97.76%
Req/Sec 133.17 24.09 200.00 80.35%
31820 requests in 30.10s, 25.89MB read
Requests/sec: 1057.09
Transfer/sec: 0.86MB某云LB 100并发测试结果：
8 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 89.44ms 15.96ms 351.22ms 96.39%
Req/Sec 134.97 23.39 210.00 80.29%
32254 requests in 30.10s, 25.87MB read
Requests/sec: 1071.58
Transfer/sec: 0.86MBambassador 100并发测试结果：
8 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 87.71ms 7.64ms 366.18ms 92.17%
Req/Sec 137.07 18.80 222.00 66.43%
32778 requests in 30.10s, 26.66MB read
Requests/sec: 1089.01
Transfer/sec: 0.89MBcontour 200并发测试结果：
8 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 183.34ms 117.76ms 1.30s 77.01%
Req/Sec 171.04 100.51 330.00 55.31%
33234 requests in 30.02s, 27.04MB read
Requests/sec: 1107.02
Transfer/sec: 0.90MB某云LB 200并发测试结果：
8 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 219.83ms 209.10ms 1.99s 87.46%
Req/Sec 161.85 84.62 330.00 60.94%
33204 requests in 30.10s, 26.63MB read
Socket errors: connect 0, read 0, write 0, timeout 14
Requests/sec: 1103.28
Transfer/sec: 0.88MBambassador 200并发测试结果：
8 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 187.48ms 126.93ms 1.94s 76.47%
Req/Sec 168.35 98.32 330.00 55.38%
33163 requests in 30.10s, 26.98MB read
Requests/sec: 1101.68
Transfer/sec: 0.90MBcontour 500并发测试结果：
8 threads and 500 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 344.10ms 301.84ms 2.00s 92.89%
Req/Sec 195.18 149.87 515.00 50.52%
33180 requests in 30.09s, 26.99MB read
Socket errors: connect 0, read 0, write 0, timeout 641
Requests/sec: 1102.55
Transfer/sec: 0.90MB某云LB 500并发测试结果：
8 threads and 500 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 333.72ms 360.42ms 1.99s 84.15%
Req/Sec 156.28 105.29 450.00 55.70%
33003 requests in 30.05s, 26.47MB read
Socket errors: connect 0, read 0, write 0, timeout 1867
Requests/sec: 1098.37
Transfer/sec: 0.88MBambassador 500并发测试结果：
8 threads and 500 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 356.51ms 309.38ms 2.00s 92.74%
Req/Sec 205.37 158.66 545.00 50.94%
33722 requests in 30.09s, 27.43MB read
Socket errors: connect 0, read 0, write 0, timeout 779
Requests/sec: 1120.86
Transfer/sec: 0.91MBcontour 1000并发测试结果：
8 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 382.13ms 379.83ms 2.00s 89.32%
Req/Sec 194.26 151.76 620.00 52.61%
Latency Distribution
50% 407.73ms
75% 453.65ms
90% 843.94ms
99% 1.93s
33442 requests in 30.10s, 27.20MB read
Socket errors: connect 0, read 0, write 0, timeout 2226
Requests/sec: 1111.04
Transfer/sec: 0.90MB某云lb 1000并发测试结果：
8 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 333.49ms 406.57ms 2.00s 85.78%
Req/Sec 169.41 122.66 464.00 53.97%
Latency Distribution
50% 94.30ms
75% 417.93ms
90% 945.75ms
99% 1.85s
32689 requests in 30.06s, 26.22MB read
Socket errors: connect 0, read 0, write 0, timeout 4376
Requests/sec: 1087.60
Transfer/sec: 0.87MBambasador 1000并发测试结果：
8 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 382.19ms 355.99ms 2.00s 90.02%
Req/Sec 196.49 158.81 610.00 53.32%
Latency Distribution
50% 415.31ms
75% 462.48ms
90% 669.19ms
99% 1.91s
33373 requests in 30.05s, 27.15MB read
Socket errors: connect 0, read 0, write 0, timeout 1746
Requests/sec: 1110.70
Transfer/sec: 0.90MB短连接
测试脚本类似如下：
wrk -c50 -d30s -t8 -H “Connection:Close” --latency http://ambassador.xx.mecontour 50并发测试结果：
8 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 86.01ms 5.47ms 259.47ms 91.16%
Req/Sec 69.72 12.40 111.00 74.87%
Latency Distribution
50% 84.72ms
75% 87.08ms
90% 90.96ms
99% 102.92ms
16711 requests in 30.09s, 13.59MB read
Requests/sec: 555.34
Transfer/sec: 462.60KB某云LB 50并发测试结果：
8 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 84.97ms 5.16ms 188.45ms 90.26%
Req/Sec 70.58 12.86 121.00 76.57%
Latency Distribution
50% 83.81ms
75% 85.90ms
90% 89.82ms
99% 103.37ms
16894 requests in 30.08s, 13.55MB read
Requests/sec: 561.67
Transfer/sec: 461.30KBambassador 50并发测试结果：
8 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 85.79ms 5.05ms 137.31ms 84.17%
Req/Sec 69.87 12.85 120.00 75.89%
Latency Distribution
50% 84.45ms
75% 87.41ms
90% 91.62ms
99% 104.99ms
16744 requests in 30.09s, 13.62MB read
Requests/sec: 556.51
Transfer/sec: 463.57KBcontour 100并发测试结果：
8 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 87.12ms 4.88ms 271.63ms 87.72%
Req/Sec 137.94 18.71 212.00 66.89%
Latency Distribution
50% 86.13ms
75% 88.48ms
90% 91.66ms
99% 103.89ms
32978 requests in 30.10s, 26.83MB read
Requests/sec: 1095.53
Transfer/sec: 0.89MB某云LB 100并发测试结果：
8 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 86.71ms 5.37ms 368.70ms 87.74%
Req/Sec 138.64 17.96 212.00 65.24%
Latency Distribution
50% 85.64ms
75% 88.27ms
90% 91.84ms
99% 103.92ms
33138 requests in 30.08s, 26.58MB read
Requests/sec: 1101.84
Transfer/sec: 0.88MBambassador 100并发测试结果：
8 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 88.00ms 19.61ms 461.19ms 98.08%
Req/Sec 138.57 20.44 210.00 69.63%
Latency Distribution
50% 85.21ms
75% 87.84ms
90% 91.99ms
99% 116.29ms
32974 requests in 30.10s, 26.82MB read
Requests/sec: 1095.61
Transfer/sec: 0.89MBcontour 200并发测试结果：
8 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 184.28ms 121.72ms 1.50s 76.78%
Req/Sec 164.99 101.34 340.00 53.49%
Latency Distribution
50% 99.14ms
75% 211.76ms
90% 380.26ms
99% 397.00ms
33565 requests in 30.10s, 27.30MB read
Requests/sec: 1114.97
Transfer/sec: 0.91MB某云LB 200并发测试结果：
8 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 217.14ms 203.96ms 1.83s 87.42%
Req/Sec 165.24 87.48 350.00 61.82%
Latency Distribution
50% 98.69ms
75% 364.03ms
90% 549.69ms
99% 996.06ms
33211 requests in 30.05s, 26.64MB read
Socket errors: connect 0, read 0, write 0, timeout 28
Requests/sec: 1105.09
Transfer/sec: 0.89MBambassador 200并发测试结果：
8 threads and 200 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 185.02ms 121.04ms 1.49s 76.35%
Req/Sec 169.18 103.22 343.00 57.33%
Latency Distribution
50% 101.96ms
75% 214.25ms
90% 380.10ms
99% 401.77ms
33254 requests in 30.08s, 27.05MB read
Requests/sec: 1105.48
Transfer/sec: 0.90MBcontour 500并发测试结果：
8 threads and 500 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 350.32ms 297.08ms 2.00s 93.25%
Req/Sec 195.72 154.55 535.00 49.55%
Latency Distribution
50% 414.74ms
75% 442.70ms
90% 508.65ms
99% 1.54s
33482 requests in 30.11s, 27.24MB read
Socket errors: connect 0, read 0, write 0, timeout 594
Requests/sec: 1112.14
Transfer/sec: 0.90MB某云LB 500并发测试结果：
8 threads and 500 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 345.04ms 363.97ms 2.00s 84.27%
Req/Sec 159.28 105.05 504.00 59.37%
Latency Distribution
50% 109.52ms
75% 501.37ms
90% 877.03ms
99% 1.49s
33158 requests in 30.10s, 26.59MB read
Socket errors: connect 0, read 0, write 0, timeout 1338
Requests/sec: 1101.55
Transfer/sec: 0.88MBambassador 500并发测试结果：
8 threads and 500 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 369.84ms 314.10ms 2.00s 92.19%
Req/Sec 201.26 160.23 555.00 50.29%
Latency Distribution
50% 417.94ms
75% 453.18ms
90% 554.53ms
99% 1.53s
33641 requests in 30.09s, 27.37MB read
Socket errors: connect 0, read 0, write 0, timeout 673
Requests/sec: 1117.89
Transfer/sec: 0.91MBcontour 1000并发测试结果：
8 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 404.53ms 388.22ms 2.00s 87.93%
Req/Sec 193.97 151.51 560.00 51.62%
Latency Distribution
50% 415.37ms
75% 477.34ms
90% 925.15ms
99% 1.92s
33820 requests in 30.10s, 27.51MB read
Socket errors: connect 0, read 0, write 0, timeout 2098
Requests/sec: 1123.48
Transfer/sec: 0.91MB某云LB 1000并发测试结果：
8 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 349.84ms 419.86ms 2.00s 83.71%
Req/Sec 160.99 116.65 474.00 52.72%
Latency Distribution
50% 95.68ms
75% 452.18ms
90% 1.02s
99% 1.80s
32723 requests in 30.09s, 26.25MB read
Socket errors: connect 0, read 0, write 0, timeout 4088
Requests/sec: 1087.53
Transfer/sec: 0.87MBambassador 1000并发测试结果：
8 threads and 1000 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 386.74ms 367.84ms 2.00s 89.83%
Req/Sec 198.21 157.69 600.00 52.61%
Latency Distribution
50% 410.62ms
75% 467.75ms
90% 821.71ms
99% 1.90s
33668 requests in 30.10s, 27.39MB read
Socket errors: connect 0, read 0, write 0, timeout 2162
Requests/sec: 1118.44
Transfer/sec: 0.91MB总结
本质上ambassador 和contour 都是基于envoy的ingress 控制器。某云7层负载均衡器基于openresty。
- 无论是长连接场景还是短连接场景，ambassador 表现都不错。
- contour和某云lb相差不大。在地并发下，某云lb性能更好，在高并发下，contour 错误较少。不过，错误的多少和超时时间有很大关系。
- 客户端在生产环境最好设置超时时间。快速失败，对整体链路是一个很好的保护。
接下来，说两个个人观点：
1）ambasador和contour 表现差距，验证了优化的重要性，contour使用了官方的envoy镜像。而ambasador使用了定制的镜像。
但是contour的架构如下：
ambassador架构图如下：
对比两张架构图可以看出：
contour 的控制层和数据层是分离的，这样可以选择数据层daemonset或是deloyment部署。更加灵活。而ambasador 控制层和数据层单个pod部署，在对边界ingress 扩缩的时候，控制层也会扩缩。这样增加了对apiserver的压力。
可以得出，contour的架构更灵活。
但是ambassador 支持的功能比较丰富，auth和限流等。
总结就是，ambasador是个网关，顺便做了ingress的活。而contour 是个ingress，正在不断完善apigateway的功能。
2） 尽管k8s社区已经意识到了ingress 功能的不足，ingress v2版本已经在路上了，但是功能性还是比较弱。而且社区和各家公有云对于ingress v2的支持还需要一定时间。
ingress 是解决k8s南北向流量的重要方式，我们可能不仅仅是需要一个引流功能而已，对于生产业务，我们更倾向于网关。不仅可以起到ingress 的作用，同时能支持灰度，影子流量，grpc，熔断，限流等服务治理的功能。
比如kong ingress controller。而基于envoy的新兴ingress controller--contour，ambasaador，gloo更是如此。
PS：本文只是简单使用wrk测试，没有覆盖所有场景。