OkHttp工具类在微服高并发场景中问题实践总结
OkHttp工具类在微服高并发场景中问题实践总结
问题场景
- 我的应用是一个中间业务应用XXApp，一个交易请求进来需要依赖下游应用，采用http协议通讯方式，需要调用3-4次下游请求。
- 老XXApp在生产上有大量Close_wait问题，（可以请求交易没有及时关闭，会造成系统有限资源port被占用完）导致“too many open files” SocketExcepiton.
- 我改成以前常用的OKhttp3，有个高并发场景的应用Utils类。但是让我没想到的在压力测试环境下20qps，XXApp系统不能连接后端服务，异常表现是大量的“time out”。
- 开始判断是后端服务能力不够或者是业务服务线程数不够，
OKHttp的部分代码
private static final OkHttpClient OK_HTTP_CLIENT =new OkHttpClient().newBuilder().connectTimeout(5,TimeUnit.SECONDS).build();
/**
* POST请求，参数为json格式。
*
* @param url
* @param json
* @return Optional
*/
public static String post(String url, String json, int timeout) throws Exception {
long start = System.currentTimeMillis();
try {
RequestBody body = RequestBody.create(JSON, json);
Request request = new Request.Builder().url(url).post(body).build();
return getHttpClient(timeout).newCall(request).execute().body().string();
} catch (Exception e) {
throw e;
} finally {
log.info("request url {} ,total time {} ms", url, (System.currentTimeMillis() - start));
}
}
public static OkHttpClient getHttpClient(int timeout) {
if (timeout == 0 || DEFAULT_TIME_OUT == timeout) {
return OK_HTTP_CLIENT;
} else {
OkHttpClient okHttpClient = new OkHttpClient.Builder().connectTimeout(timeout, TimeUnit.SECONDS).readTimeout(timeout, TimeUnit.SECONDS).writeTimeout(timeout, TimeUnit.SECONDS).build();
return okHttpClient;
}
}
- 为了支持不同业务调用需要不同timeout，提供自定义okHttpClient与默认的OK_HTTP_CLIENT。
- XXApp在业务调用的时候采用不同接口自定义timeout时间。
压测现象分析
-
top -H -P 进程ID， 查看进程的线程数，发现totalThread数上15000多，还在不停的增加，这就有些奇怪了，以前高并发应用也没有这么高的线程数。
-
netstat -n | awk ‘/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}’ ； 发现也有CLOSE_WAIT的连接状态在增长。
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-CGSrfyGC-1582808024865)(./image/close_wait_issue.jpg)]
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-I65oGUiM-1582808024867)(./image/many_conn.jpg)]
3. 开始怀疑是不是每个请求都创建okHttpClient实例造成线程数量增加。然后在关闭连接过程造成CLOSE_WAIT数量波动在3000-4000多。
4. 查询一下http协议1.1 keep-alive长连接应用可以减少tcp 连接，关闭的时间。OKHttp也有一个ConnectionPool类来支持并发请求，查看builder类，果然给okHttpClient创建一个默认
5. 如果每次都要创建okHttpClient的确不很好提升性能，所有先改成默认OK_HTTP_CLIENT实例试下，压测效果瞬间提升到100qps。 进程的线程数也稳定在500左右，CLOSE_WAIT问题也没有了.
再次压测现象分析
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-Mp6JHBlo-1582808024867)(./image/ok-conn-count.jpg)]
-
各种压测指标，监控指标都正常了。
-
继续分析源码来总结，查看OkHttpClients源码
* Factory for {@linkplain Call calls}, which can be used to send HTTP requests and read their * responses. * *OkHttpClients should be shared
* *OkHttp performs best when you create a single {@code OkHttpClient} instance and reuse it for * all of your HTTP calls. This is because each client holds its own connection pool and thread * pools. Reusing connections and threads reduces latency and saves memory. Conversely, creating a * client for each request wastes resources on idle pools. * *
Use {@code new OkHttpClient()} to create a shared instance with the default settings: *
{@code * * // The singleton HTTP client. * public final OkHttpClient client = new OkHttpClient(); * }* -
已经明确描述希望OkHttpClient是一个单例，因为它有自己连接池与线程池。也解释为什么不停调用会thread增加。
总结
- 我在构建一个utils的时候，除参考网络上的代码例子，还应该参考官方example，及api的comments。
https://github.com/square/okhttp/blob/master/okhttp/src/test/java/okhttp3/OkHttpClientTest.java - 关注点
- 注意测试工具的timeout问题，一定要写单元测试。
- 注意工具的连接关闭问题，参考官方demo是最好的。netstat 观察
- 了解工具调用连接结构，方便调优，已解决在高并发场景下的调优。
修改后的OKHttp3Utils3
import lombok.extern.slf4j.Slf4j;
import okhttp3.*;
import okhttp3.internal.Util;
import okio.BufferedSink;
import okio.Okio;
import okio.Source;
import java.io.IOException;
import java.io.InputStream;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.TimeUnit;
@Slf4j
public final class OKHttp3Utils3 {
public static int DEFAULT_TIME_OUT = 10;
/**
* 全局实例可以保持http1.1 连接复用，线程池复用， 减少tcp的网络连接，关闭，
* 如果每次一个请求，在高并发下，thread增多到1W，close_wait持续增加到6k。
*/
private static final OkHttpClient OK_HTTP_CLIENT = new OkHttpClient.Builder()
.connectionPool(new ConnectionPool(50, 5, TimeUnit.MINUTES))
.connectTimeout(10, TimeUnit.SECONDS).readTimeout(10, TimeUnit.SECONDS).writeTimeout(10, TimeUnit.SECONDS).build();
private static final MediaType JSON = MediaType.parse("application/json; charset=utf-8");
private static final MediaType FORM_TYPE = MediaType.parse("application/x-www-form-urlencoded; charset=UTF-8");
/**
* 不同timeout的连接池
*/
public static ConcurrentHashMap cacheClients = new ConcurrentHashMap();
public static OkHttpClient getHttpClient(int timeout) {
if (timeout == 0 || DEFAULT_TIME_OUT == timeout) {
return OK_HTTP_CLIENT;
} else {
OkHttpClient okHttpClient = cacheClients.get(timeout);
if (okHttpClient == null) {
return syncCreateClient(timeout);
}
return okHttpClient;
}
}
private static synchronized OkHttpClient syncCreateClient(int timeout) {
OkHttpClient okHttpClient;
okHttpClient = cacheClients.get(timeout);
if (okHttpClient != null) {
return okHttpClient;
}
okHttpClient = new OkHttpClient.Builder().connectTimeout(timeout, TimeUnit.SECONDS).readTimeout(timeout, TimeUnit.SECONDS).writeTimeout(timeout, TimeUnit.SECONDS).build();
cacheClients.put(timeout, okHttpClient);
return okHttpClient;
}
/**
* GET请求
*
* @param url
* @return Optional
*/
public static String get(String url, int timeout) throws Exception {
Request request = new Request.Builder().url(url)
.build();
return getHttpClient(timeout).newCall(request).execute().body().string();
}
public static String get(String url) throws Exception {
return get(url, 0);
}
/**
* POST请求，参数为json格式。
*
* @param url
* @param json
* @return Optional
*/
public static String post(String url, String json, int timeout) throws Exception {
long start = System.currentTimeMillis();
try {
RequestBody body = RequestBody.create(JSON, json);
Request request = new Request.Builder().url(url).post(body).build();
return getHttpClient(timeout).newCall(request).execute().body().string();
} catch (Exception e) {
throw e;
} finally {
log.info("request url {} ,total time {} ms", url, (System.currentTimeMillis() - start));
}
}
public static String post(String url, String json) throws Exception {
return post(url, json, 0);
}
public static String postByFormType(String url, String form) throws Exception {
long start = System.currentTimeMillis();
try {
RequestBody body = RequestBody.create(FORM_TYPE, form);
Request request = new Request.Builder().url(url).post(body).build();
return getHttpClient(0).newCall(request).execute().body().string();
} catch (Exception e) {
throw e;
} finally {
log.info("request url {} ,total time {} ms", url, (System.currentTimeMillis() - start));
}
}
/**
* 根据不同的类型和requestbody类型来接续参数
*
* @param url
* @param mediaType
* @param inputStream
* @return
* @throws Exception
*/
public static String post(String url, MediaType mediaType, InputStream inputStream) throws Exception {
RequestBody body = createRequestBody(mediaType, inputStream);
Request request = new Request.Builder().url(url).post(body).build();
return OK_HTTP_CLIENT.newCall(request).execute().body().string();
}
private static RequestBody createRequestBody(final MediaType mediaType, final InputStream inputStream) {
return new RequestBody() {
// @Nullable
@Override
public MediaType contentType() {
return mediaType;
}
@Override
public long contentLength() throws IOException {
try {
return inputStream.available();
} catch (IOException e) {
return 0;
}
}
@Override
public void writeTo(BufferedSink sink) throws IOException {
Source source = null;
try {
source = Okio.source(inputStream);
sink.writeAll(source);
} finally {
Util.closeQuietly(source);
}
}
};
}
}