k8s集群ingress controller压测记录

前言

这两天老大让做k8s测试集群内ingress controller的压测报告，记录下。ingress-controller中nginx结构为1master4worker，结构如下：

短连接测试

最开始使用jmeter，配置如下：

四个线程组（其实和单线程组的效果一致），从1000开始增加，得到的CPU使用率结果如下：

当请求次数超过9500次之后，CPU负载不再增加，在结果树中可以看到大量失败返回：

可以知道是由大量短连接导致的timewait过多，无法建立新连接，在http头中开启长连接，发现依然有这个问题，查阅资料后发现,jmeter会主动去关闭连接，于是使用HttpClient进行测试，最开始使用单连接进行调用，结果发现请求数量到4000以上，CPU使用率不再提高：

于是使用4000个连接开启测试：

测试源码如下：

import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHeaders;
import org.apache.http.HttpResponse;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.client.methods.RequestBuilder;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicHeader;
import org.apache.http.util.EntityUtils;

import java.io.IOException;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.CountDownLatch;

public class testCPU {
    static int count = 0;
    static int num = 0;
    static int requestPerConnection = 16;
    static int totalRequest = requestPerConnection * 4000;
    public static void send(CloseableHttpClient httpclient, HttpUriRequest request, CountDownLatch countDownLatch) throws IOException{

        // Create a custom response handler
        ResponseHandler responseHandler = (HttpResponse response) -> {
                HttpEntity entity = response.getEntity();
                if (response.getStatusLine().getStatusCode() != 200) {
                    count++;
                }else {
                    num++;
                }
                return entity != null ? EntityUtils.toString(entity) : null;
        };
        String responseBody = httpclient.execute(request, responseHandler);
        countDownLatch.countDown();
    }

    public static void main(String[] args) throws InterruptedException, IOException {
        CountDownLatch countDownLatch = new CountDownLatch(totalRequest);
        for (int i = 0; i < 4000; i++) {
            // create custom http headers for httpclient
            List
 defaultHeaders = Arrays.asList(
                    new BasicHeader("X-Default-Header", "default header httpclient"));
            //创建一个HttpClient对象
            CloseableHttpClient httpclient = HttpClients
                    .custom()
                    .setDefaultHeaders(defaultHeaders)
                    .build();
            HttpUriRequest request = RequestBuilder.get()
                    .setUri("http://xiaoxingchen-demo.com:30102/index")
                    .setHeader(HttpHeaders.CONTENT_TYPE, "application/json")
                    .setHeader(HttpHeaders.CONNECTION, "Keep-Alive")
                    .setHeader(HttpHeaders.CACHE_CONTROL, "no-cache")
                    .build();

            Runnable runnable = () -> {
                try {
                    send(httpclient, request, countDownLatch);
                } catch (IOException e) {
                    e.printStackTrace();
                }
            };
            for (int j = 0; j < requestPerConnection; j ++) {
                Thread thread = new Thread(runnable);
                thread.start();
            }
        }
        countDownLatch.await();
        System.out.println("成功请求个数：" + num + "个");
        System.out.println("成功请求个数：" + + count + "个");

    }
}

负载又没有冲上去，因为用了countdownlatch计算相应的成功率，其实逻辑上还是单线程的调用，于是去掉countdownlatch：

import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHeaders;
import org.apache.http.HttpResponse;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.client.methods.RequestBuilder;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicHeader;
import org.apache.http.util.EntityUtils;

import java.io.IOException;
import java.util.Arrays;
import java.util.List;

public class testCPU {
    static int requestPerConnection = 20;
    public static void send(CloseableHttpClient httpclient, HttpUriRequest request) throws IOException{

        // Create a custom response handler
        ResponseHandler responseHandler = (HttpResponse response) -> {
                HttpEntity entity = response.getEntity();
                if (response.getStatusLine().getStatusCode() != 200) {
                    count++;
                }else {
                    num++;
                }
                return entity != null ? EntityUtils.toString(entity) : null;
        };
        String responseBody = httpclient.execute(request, responseHandler);
    }

    public static void main(String[] args) throws InterruptedException, IOException {
        for (int i = 0; i < 8000; i++) {
            // create custom http headers for httpclient
            List
 defaultHeaders = Arrays.asList(
                    new BasicHeader("X-Default-Header", "default header httpclient"));
            //创建一个HttpClient对象
            CloseableHttpClient httpclient = HttpClients
                    .custom()
                    .setDefaultHeaders(defaultHeaders)
                    .build();
            HttpUriRequest request = RequestBuilder.get()
                    .setUri("http://xiaoxingchen-demo.com:30102/index")
                    .setHeader(HttpHeaders.CONTENT_TYPE, "application/json")
                    .setHeader(HttpHeaders.CONNECTION, "Keep-Alive")
                    .setHeader(HttpHeaders.CACHE_CONTROL, "no-cache")
                    .build();

            Runnable runnable = () -> {
                try {
                    send(httpclient, request);
                } catch (IOException e) {
                    e.printStackTrace();
                }
            };
            for (int j = 0; j < requestPerConnection; j ++) {
                Thread thread = new Thread(runnable);
                thread.start();
            }
        }
    }
}

测试结果如下：

负载冲到了百分之二十后又冲不上去了，并出现大量连接超时，查看pod负载和连接数如下：

可以看到测试的tomcat服务已经打满了，但是nginx服务负载还没冲上去，被主管一语道破，用ngnix去压tomcat是不合理的！临近双11，机器紧张，节后再申请10台机器部署服务继续测试~~~~

节后再更，先做DB负载均衡服务的压测