在Kubernetes上对Ambassador,Contour和Nginx性能进行基准测试

在Kubernetes上对Ambassador,Contour和Nginx性能进行基准测试_第1张图片

在典型的Kubernetes部署中,所有到Kubernetes服务的流量都流经ingress。Ingress代理从Internet到后端服务的流量。这样,Ingres就在您提高性能的关键路径上。有多种基准测试和衡量性能的方法。

衡量代理性能的最常见方法可能是原始吞吐量。在这种类型的测试中,通过代理发送的通信量不断增加,并且可以测量代理可以处理的最大通信量。一个典型的衡量标准是衡量每秒请求数(RPS)的性能。

本文主要测试Ambassador(1.4),contour(1.2.1) 和某云Ingress负载均衡器(openresty)。

测试

测试环境说明

  • 后端负载是通过Deployment部署的2个nginx Pod。每个Pod的资源限制为1c1G。
  • 所有的负载均衡器均打开了access log。
  • 分别创建三个ingress,对应的域名为contour.xx.me, ambassador.xx.me, cloud.xx.me。其中敏感域名,故用xx表示。

测试工具以及场景说明

  • 测试工具我们选择wrk。
  • 场景分为50线程并发,100线程并发,200线程并发,500线程并发,1000线程并发。
  • 所有测试均在本地,压测时长30s。
  • 所有结果都只看聚合报告,重点查看吞吐量、时间和错误数。
  • 测试均不作任何优化。

测试对比

长连接

测试脚本类似如下:

wrk -c50 -d30s -t8 --latency  http://ambassador.xx.me

contour 50并发测试结果:

 8 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    85.89ms    4.14ms 120.42ms   82.95%
    Req/Sec    69.81     12.34   121.00     76.56%
  16723 requests in 30.09s, 13.60MB read
Requests/sec:    555.83
Transfer/sec:    463.02KB

某云LB 50并发测试结果:

8 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    86.68ms    6.45ms 261.78ms   91.00%
    Req/Sec    69.15     12.77   121.00     85.59%
  16571 requests in 30.09s, 13.29MB read
Requests/sec:    550.65
Transfer/sec:    452.25KB

ambassador 50并发测试结果:

 8 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    84.62ms    5.53ms 364.52ms   91.04%
    Req/Sec    70.89     12.17   120.00     65.52%
  16983 requests in 30.08s, 13.82MB read
Requests/sec:    564.62
Transfer/sec:    470.33KB

contour 100并发测试结果:

 8 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    95.72ms   63.15ms   1.29s    97.76%
    Req/Sec   133.17     24.09   200.00     80.35%
  31820 requests in 30.10s, 25.89MB read
Requests/sec:   1057.09
Transfer/sec:      0.86MB

某云LB 100并发测试结果:

 8 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    89.44ms   15.96ms 351.22ms   96.39%
    Req/Sec   134.97     23.39   210.00     80.29%
  32254 requests in 30.10s, 25.87MB read
Requests/sec:   1071.58
Transfer/sec:      0.86MB

ambassador 100并发测试结果:

 8 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    87.71ms    7.64ms 366.18ms   92.17%
    Req/Sec   137.07     18.80   222.00     66.43%
  32778 requests in 30.10s, 26.66MB read
Requests/sec:   1089.01
Transfer/sec:      0.89MB

contour 200并发测试结果:

 8 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   183.34ms  117.76ms   1.30s    77.01%
    Req/Sec   171.04    100.51   330.00     55.31%
  33234 requests in 30.02s, 27.04MB read
Requests/sec:   1107.02
Transfer/sec:      0.90MB

某云LB 200并发测试结果:

 8 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   219.83ms  209.10ms   1.99s    87.46%
    Req/Sec   161.85     84.62   330.00     60.94%
  33204 requests in 30.10s, 26.63MB read
  Socket errors: connect 0, read 0, write 0, timeout 14
Requests/sec:   1103.28
Transfer/sec:      0.88MB

ambassador 200并发测试结果:

 8 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   187.48ms  126.93ms   1.94s    76.47%
    Req/Sec   168.35     98.32   330.00     55.38%
  33163 requests in 30.10s, 26.98MB read
Requests/sec:   1101.68
Transfer/sec:      0.90MB

contour 500并发测试结果:

 8 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   344.10ms  301.84ms   2.00s    92.89%
    Req/Sec   195.18    149.87   515.00     50.52%
  33180 requests in 30.09s, 26.99MB read
  Socket errors: connect 0, read 0, write 0, timeout 641
Requests/sec:   1102.55
Transfer/sec:      0.90MB

某云LB 500并发测试结果:

 8 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   333.72ms  360.42ms   1.99s    84.15%
    Req/Sec   156.28    105.29   450.00     55.70%
  33003 requests in 30.05s, 26.47MB read
  Socket errors: connect 0, read 0, write 0, timeout 1867
Requests/sec:   1098.37
Transfer/sec:      0.88MB

ambassador 500并发测试结果:

 8 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   356.51ms  309.38ms   2.00s    92.74%
    Req/Sec   205.37    158.66   545.00     50.94%
  33722 requests in 30.09s, 27.43MB read
  Socket errors: connect 0, read 0, write 0, timeout 779
Requests/sec:   1120.86
Transfer/sec:      0.91MB

contour 1000并发测试结果:

  8 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   382.13ms  379.83ms   2.00s    89.32%
    Req/Sec   194.26    151.76   620.00     52.61%
  Latency Distribution
     50%  407.73ms
     75%  453.65ms
     90%  843.94ms
     99%    1.93s
  33442 requests in 30.10s, 27.20MB read
  Socket errors: connect 0, read 0, write 0, timeout 2226
Requests/sec:   1111.04
Transfer/sec:      0.90MB

某云lb 1000并发测试结果:

  8 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   333.49ms  406.57ms   2.00s    85.78%
    Req/Sec   169.41    122.66   464.00     53.97%
  Latency Distribution
     50%   94.30ms
     75%  417.93ms
     90%  945.75ms
     99%    1.85s
  32689 requests in 30.06s, 26.22MB read
  Socket errors: connect 0, read 0, write 0, timeout 4376
Requests/sec:   1087.60
Transfer/sec:      0.87MB

ambasador 1000并发测试结果:

  8 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   382.19ms  355.99ms   2.00s    90.02%
    Req/Sec   196.49    158.81   610.00     53.32%
  Latency Distribution
     50%  415.31ms
     75%  462.48ms
     90%  669.19ms
     99%    1.91s
  33373 requests in 30.05s, 27.15MB read
  Socket errors: connect 0, read 0, write 0, timeout 1746
Requests/sec:   1110.70
Transfer/sec:      0.90MB

短连接

测试脚本类似如下:

wrk -c50 -d30s -t8 -H “Connection:Close”  --latency  http://ambassador.xx.me

contour 50并发测试结果:

  8 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    86.01ms    5.47ms 259.47ms   91.16%
    Req/Sec    69.72     12.40   111.00     74.87%
  Latency Distribution
     50%   84.72ms
     75%   87.08ms
     90%   90.96ms
     99%  102.92ms
  16711 requests in 30.09s, 13.59MB read
Requests/sec:    555.34
Transfer/sec:    462.60KB

某云LB 50并发测试结果:

  8 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    84.97ms    5.16ms 188.45ms   90.26%
    Req/Sec    70.58     12.86   121.00     76.57%
  Latency Distribution
     50%   83.81ms
     75%   85.90ms
     90%   89.82ms
     99%  103.37ms
  16894 requests in 30.08s, 13.55MB read
Requests/sec:    561.67
Transfer/sec:    461.30KB

ambassador 50并发测试结果:

  8 threads and 50 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    85.79ms    5.05ms 137.31ms   84.17%
    Req/Sec    69.87     12.85   120.00     75.89%
  Latency Distribution
     50%   84.45ms
     75%   87.41ms
     90%   91.62ms
     99%  104.99ms
  16744 requests in 30.09s, 13.62MB read
Requests/sec:    556.51
Transfer/sec:    463.57KB

contour 100并发测试结果:

  8 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    87.12ms    4.88ms 271.63ms   87.72%
    Req/Sec   137.94     18.71   212.00     66.89%
  Latency Distribution
     50%   86.13ms
     75%   88.48ms
     90%   91.66ms
     99%  103.89ms
  32978 requests in 30.10s, 26.83MB read
Requests/sec:   1095.53
Transfer/sec:      0.89MB

某云LB 100并发测试结果:

  8 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    86.71ms    5.37ms 368.70ms   87.74%
    Req/Sec   138.64     17.96   212.00     65.24%
  Latency Distribution
     50%   85.64ms
     75%   88.27ms
     90%   91.84ms
     99%  103.92ms
  33138 requests in 30.08s, 26.58MB read
Requests/sec:   1101.84
Transfer/sec:      0.88MB

ambassador 100并发测试结果:

  8 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    88.00ms   19.61ms 461.19ms   98.08%
    Req/Sec   138.57     20.44   210.00     69.63%
  Latency Distribution
     50%   85.21ms
     75%   87.84ms
     90%   91.99ms
     99%  116.29ms
  32974 requests in 30.10s, 26.82MB read
Requests/sec:   1095.61
Transfer/sec:      0.89MB

contour 200并发测试结果:

  8 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   184.28ms  121.72ms   1.50s    76.78%
    Req/Sec   164.99    101.34   340.00     53.49%
  Latency Distribution
     50%   99.14ms
     75%  211.76ms
     90%  380.26ms
     99%  397.00ms
  33565 requests in 30.10s, 27.30MB read
Requests/sec:   1114.97
Transfer/sec:      0.91MB

某云LB 200并发测试结果:

  8 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   217.14ms  203.96ms   1.83s    87.42%
    Req/Sec   165.24     87.48   350.00     61.82%
  Latency Distribution
     50%   98.69ms
     75%  364.03ms
     90%  549.69ms
     99%  996.06ms
  33211 requests in 30.05s, 26.64MB read
  Socket errors: connect 0, read 0, write 0, timeout 28
Requests/sec:   1105.09
Transfer/sec:      0.89MB

ambassador 200并发测试结果:

  8 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   185.02ms  121.04ms   1.49s    76.35%
    Req/Sec   169.18    103.22   343.00     57.33%
  Latency Distribution
     50%  101.96ms
     75%  214.25ms
     90%  380.10ms
     99%  401.77ms
  33254 requests in 30.08s, 27.05MB read
Requests/sec:   1105.48
Transfer/sec:      0.90MB

contour 500并发测试结果:

 8 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   350.32ms  297.08ms   2.00s    93.25%
    Req/Sec   195.72    154.55   535.00     49.55%
  Latency Distribution
     50%  414.74ms
     75%  442.70ms
     90%  508.65ms
     99%    1.54s
  33482 requests in 30.11s, 27.24MB read
  Socket errors: connect 0, read 0, write 0, timeout 594
Requests/sec:   1112.14
Transfer/sec:      0.90MB

某云LB 500并发测试结果:

  8 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   345.04ms  363.97ms   2.00s    84.27%
    Req/Sec   159.28    105.05   504.00     59.37%
  Latency Distribution
     50%  109.52ms
     75%  501.37ms
     90%  877.03ms
     99%    1.49s
  33158 requests in 30.10s, 26.59MB read
  Socket errors: connect 0, read 0, write 0, timeout 1338
Requests/sec:   1101.55
Transfer/sec:      0.88MB

ambassador 500并发测试结果:

  8 threads and 500 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   369.84ms  314.10ms   2.00s    92.19%
    Req/Sec   201.26    160.23   555.00     50.29%
  Latency Distribution
     50%  417.94ms
     75%  453.18ms
     90%  554.53ms
     99%    1.53s
  33641 requests in 30.09s, 27.37MB read
  Socket errors: connect 0, read 0, write 0, timeout 673
Requests/sec:   1117.89
Transfer/sec:      0.91MB

contour 1000并发测试结果:

 8 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   404.53ms  388.22ms   2.00s    87.93%
    Req/Sec   193.97    151.51   560.00     51.62%
  Latency Distribution
     50%  415.37ms
     75%  477.34ms
     90%  925.15ms
     99%    1.92s
  33820 requests in 30.10s, 27.51MB read
  Socket errors: connect 0, read 0, write 0, timeout 2098
Requests/sec:   1123.48
Transfer/sec:      0.91MB

某云LB 1000并发测试结果:

 8 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   349.84ms  419.86ms   2.00s    83.71%
    Req/Sec   160.99    116.65   474.00     52.72%
  Latency Distribution
     50%   95.68ms
     75%  452.18ms
     90%    1.02s
     99%    1.80s
  32723 requests in 30.09s, 26.25MB read
  Socket errors: connect 0, read 0, write 0, timeout 4088
Requests/sec:   1087.53
Transfer/sec:      0.87MB

ambassador 1000并发测试结果:

 8 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   386.74ms  367.84ms   2.00s    89.83%
    Req/Sec   198.21    157.69   600.00     52.61%
  Latency Distribution
     50%  410.62ms
     75%  467.75ms
     90%  821.71ms
     99%    1.90s
  33668 requests in 30.10s, 27.39MB read
  Socket errors: connect 0, read 0, write 0, timeout 2162
Requests/sec:   1118.44
Transfer/sec:      0.91MB

总结

本质上ambassador 和contour 都是基于envoy的ingress 控制器。某云7层负载均衡器基于openresty。

  • 无论是长连接场景还是短连接场景,ambassador 表现都不错。
  • contour和某云lb相差不大。在地并发下,某云lb性能更好,在高并发下,contour 错误较少。不过,错误的多少和超时时间有很大关系。
  • 客户端在生产环境最好设置超时时间。快速失败,对整体链路是一个很好的保护。

接下来,说两个个人观点:

1)ambasador和contour 表现差距,验证了优化的重要性,contour使用了官方的envoy镜像。而ambasador使用了定制的镜像。

但是contour的架构如下:

在Kubernetes上对Ambassador,Contour和Nginx性能进行基准测试_第2张图片

ambassador架构图如下:

在Kubernetes上对Ambassador,Contour和Nginx性能进行基准测试_第3张图片

对比两张架构图可以看出:

contour 的控制层和数据层是分离的,这样可以选择数据层daemonset或是deloyment部署。更加灵活。而ambasador 控制层和数据层单个pod部署,在对边界ingress 扩缩的时候,控制层也会扩缩。这样增加了对apiserver的压力。

可以得出,contour的架构更灵活。

但是ambassador 支持的功能比较丰富,auth和限流等。

总结就是,ambasador是个网关,顺便做了ingress的活。而contour 是个ingress,正在不断完善apigateway的功能。

2) 尽管k8s社区已经意识到了ingress 功能的不足,ingress v2版本已经在路上了,但是功能性还是比较弱。而且社区和各家公有云对于ingress v2的支持还需要一定时间。

ingress 是解决k8s南北向流量的重要方式,我们可能不仅仅是需要一个引流功能而已,对于生产业务,我们更倾向于网关。不仅可以起到ingress 的作用,同时能支持灰度,影子流量,grpc,熔断,限流等服务治理的功能。

比如kong ingress controller。而基于envoy的新兴ingress controller--contour,ambasaador,gloo更是如此。

PS:本文只是简单使用wrk测试,没有覆盖所有场景。

你可能感兴趣的:(kubernetes,k8s,ingress)