RestTemplate超时配置引发的宕机事故让经理念叨了一年
想必很多数据中台，会集成很多系统的数据然后进行统一的规划处理，一些业务场景实现中必然少不了对第三方系统的依赖。在一个业务场景中，当用户在线的时候，就需要实时地去请求第三方接口获取相关信息，这是一个对实时性要求比较高的场景。用RestTemplate去请求第三方接口的时候，用了默认配置，没有设置超时时间，也就没有降级处理机制。某一天，第三方服务出现了故障，导致我们请求一直没有得到返回，然后请求线程一直在等待，一个用户每一分钟一个这样的请求，数量慢慢上来，最终不一会就耗尽了机器线程数，应用宕机。
通过下面的例子来进行模拟
用一个延时两分钟才返回的接口来模拟第三方服务
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.RestController;
@RestController
public class CodeController {
@RequestMapping(value = "/getCode", method = RequestMethod.GET)
@ResponseBody
public String getValue() {
long start = System.currentTimeMillis() + (2 * 60 * 1000);
while (true) {
long end = System.currentTimeMillis();
if(end >= start) {
break;
}
}
return "ok";
}
}
ClientController模拟自身服务，通过restTemplate来调用/getCode第三方接口服务
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;
@RestController
public class ClientController {
private static final Logger logger = LoggerFactory.getLogger(ClientController.class);
@Autowired
RestTemplate restTemplate;
@RequestMapping(value = "/getInfo", method = RequestMethod.GET)
public void getInfo() {
logger.info("start");
ResponseEntity forEntity = restTemplate.getForEntity("http://127.0.0.1:8080/getCode", String.class);
logger.info(String.valueOf(forEntity.getStatusCodeValue()));
}
}
注入了一个默认配置的RestTemplate实例
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.web.client.RestTemplate;
@SpringBootApplication
@EnableScheduling
public class ShowApplication {
@Bean
public RestTemplate restTemplate() {
return new RestTemplate();
}
public static void main(String[] args) {
SpringApplication.run(ShowApplication.class, args);
}
}通过如下脚本调用一千次/getInfo接口
for((i=1;i<=1000;i++));
do
curl http://127.0.0.1:8080/getInfo & sleep 0
done
echo "ok" 调用之前的线程数量，线程数使用数在2000左右
执行脚本之后，线程数以千为单位的数量级上升，因为在自身服务和第三方服务都在一个机器上跑着，所以线程数上升2000+
两分钟之后，模拟结束，线程数恢复到之前水平范围
为什么RestTemplate默认不超时呢？
来看一下RestTemplate的默认构造器，在默认构造器里只是对象消息装换对象进行设置，没有找到相关超时设置，属性里也没有。
public class RestTemplate extends InterceptingHttpAccessor implements RestOperations {
private static final boolean shouldIgnoreXml = SpringProperties.getFlag("spring.xml.ignore");
private static final boolean romePresent;
private static final boolean jaxb2Present;
private static final boolean jackson2Present;
private static final boolean jackson2XmlPresent;
private static final boolean jackson2SmilePresent;
private static final boolean jackson2CborPresent;
private static final boolean gsonPresent;
private static final boolean jsonbPresent;
private static final boolean kotlinSerializationJsonPresent;
private final List> messageConverters = new ArrayList<>();
private ResponseErrorHandler errorHandler = new DefaultResponseErrorHandler();
private UriTemplateHandler uriTemplateHandler;
private final ResponseExtractor headersExtractor = new HeadersExtractor();
/**
* Create a new instance of the {@link RestTemplate} using default settings.
* Default {@link HttpMessageConverter HttpMessageConverters} are initialized.
*/
public RestTemplate() {
this.messageConverters.add(new ByteArrayHttpMessageConverter());
this.messageConverters.add(new StringHttpMessageConverter());
this.messageConverters.add(new ResourceHttpMessageConverter(false));
if (!shouldIgnoreXml) {
try {
this.messageConverters.add(new SourceHttpMessageConverter<>());
}
catch (Error err) {
// Ignore when no TransformerFactory implementation is available
}
}
this.messageConverters.add(new AllEncompassingFormHttpMessageConverter());
if (romePresent) {
this.messageConverters.add(new AtomFeedHttpMessageConverter());
this.messageConverters.add(new RssChannelHttpMessageConverter());
}
if (!shouldIgnoreXml) {
if (jackson2XmlPresent) {
this.messageConverters.add(new MappingJackson2XmlHttpMessageConverter());
}
else if (jaxb2Present) {
this.messageConverters.add(new Jaxb2RootElementHttpMessageConverter());
}
}
if (jackson2Present) {
this.messageConverters.add(new MappingJackson2HttpMessageConverter());
}
else if (gsonPresent) {
this.messageConverters.add(new GsonHttpMessageConverter());
}
else if (jsonbPresent) {
this.messageConverters.add(new JsonbHttpMessageConverter());
}
else if (kotlinSerializationJsonPresent) {
this.messageConverters.add(new KotlinSerializationJsonHttpMessageConverter());
}
if (jackson2SmilePresent) {
this.messageConverters.add(new MappingJackson2SmileHttpMessageConverter());
}
if (jackson2CborPresent) {
this.messageConverters.add(new MappingJackson2CborHttpMessageConverter());
}
this.uriTemplateHandler = initUriTemplateHandler();
} 可以RestTemplate发现对RestOperations接口进行了实现，该接口定义了各种请求方法，同时继承了InterceptingHttpAccessor，该类维护着ClientHttpRequestFactory，但是也没有默认实现，这时候再看父类HttpAccessor
public abstract class InterceptingHttpAccessor extends HttpAccessor {
private final List interceptors = new ArrayList<>();
@Nullable
private volatile ClientHttpRequestFactory interceptingRequestFactory;
} HttpAccessor维护着ClientHttpRequestFactory，在ClientHttpRequestFactory中可以看到两个关于超时的配置，connectTimeout = -1，readTimeout = -1，可以看到在准备连接的时候，当这两个参数小于0的时候，是没有对HttpURLConnection进行超时设置的
public abstract class HttpAccessor {
/** Logger available to subclasses. */
protected final Log logger = HttpLogging.forLogName(getClass());
private ClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
}public class SimpleClientHttpRequestFactory implements ClientHttpRequestFactory, AsyncClientHttpRequestFactory {
private static final int DEFAULT_CHUNK_SIZE = 4096;
@Nullable
private Proxy proxy;
private boolean bufferRequestBody = true;
private int chunkSize = DEFAULT_CHUNK_SIZE;
private int connectTimeout = -1;
private int readTimeout = -1;
private boolean outputStreaming = true;
protected void prepareConnection(HttpURLConnection connection, String httpMethod) throws IOException {
if (this.connectTimeout >= 0) {
connection.setConnectTimeout(this.connectTimeout);
}
if (this.readTimeout >= 0) {
connection.setReadTimeout(this.readTimeout);
}
boolean mayWrite =
("POST".equals(httpMethod) || "PUT".equals(httpMethod) ||
"PATCH".equals(httpMethod) || "DELETE".equals(httpMethod));
connection.setDoInput(true);
connection.setInstanceFollowRedirects("GET".equals(httpMethod));
connection.setDoOutput(mayWrite);
connection.setRequestMethod(httpMethod);
}
}然后再看RestTemplate是如何执行请求的，不难发现该类实现的各种请求方法中都是分为三步，1、构造RequestCallback，2、构造ResponseExtractor，3、执行doExecute
@Nullable
protected T doExecute(URI url, @Nullable HttpMethod method, @Nullable RequestCallback requestCallback,
@Nullable ResponseExtractor responseExtractor) throws RestClientException {
Assert.notNull(url, "URI is required");
Assert.notNull(method, "HttpMethod is required");
ClientHttpResponse response = null;
try {
ClientHttpRequest request = createRequest(url, method);
if (requestCallback != null) {
requestCallback.doWithRequest(request);
}
response = request.execute();
handleResponse(url, method, response);
return (responseExtractor != null ? responseExtractor.extractData(response) : null);
}
catch (IOException ex) {
String resource = url.toString();
String query = url.getRawQuery();
resource = (query != null ? resource.substring(0, resource.indexOf('?')) : resource);
throw new ResourceAccessException("I/O error on " + method.name() +
" request for \"" + resource + "\": " + ex.getMessage(), ex);
}
finally {
if (response != null) {
response.close();
}
}
} 在doExecute中通过createRequest方法构造了ClientHttpRequest，createRequest在父类中进行了实现，在该方法通过ClientHttpRequestFactory构造http请求，刚才可以知道的ClientHttpRequestFactory默认实现是SimpleClientHttpRequestFactory
public abstract class HttpAccessor {
protected ClientHttpRequest createRequest(URI url, HttpMethod method) throws IOException {
ClientHttpRequest request = getRequestFactory().createRequest(url, method);
initialize(request);
if (logger.isDebugEnabled()) {
logger.debug("HTTP " + method.name() + " " + url);
}
return request;
}
}而SimpleClientHttpRequestFactory中createRequest方法中调用了刚才所说的prepareConnection，从而没有进行超时设置。
public class SimpleClientHttpRequestFactory implements ClientHttpRequestFactory, AsyncClientHttpRequestFactory {
@Override
public ClientHttpRequest createRequest(URI uri, HttpMethod httpMethod) throws IOException {
HttpURLConnection connection = openConnection(uri.toURL(), this.proxy);
prepareConnection(connection, httpMethod.name());
if (this.bufferRequestBody) {
return new SimpleBufferingClientHttpRequest(connection, this.outputStreaming);
}
else {
return new SimpleStreamingClientHttpRequest(connection, this.chunkSize, this.outputStreaming);
}
}
protected void prepareConnection(HttpURLConnection connection, String httpMethod) throws IOException {
if (this.connectTimeout >= 0) {
connection.setConnectTimeout(this.connectTimeout);
}
if (this.readTimeout >= 0) {
connection.setReadTimeout(this.readTimeout);
}
boolean mayWrite =
("POST".equals(httpMethod) || "PUT".equals(httpMethod) ||
"PATCH".equals(httpMethod) || "DELETE".equals(httpMethod));
connection.setDoInput(true);
connection.setInstanceFollowRedirects("GET".equals(httpMethod));
connection.setDoOutput(mayWrite);
connection.setRequestMethod(httpMethod);
}
}通过源码的review之后，可得出如下的UML类图
通过UML类图可清晰得看出ClientHttpRequestFactory有多个实现，在RestTemplate中默认持有的是SimpleClientHttpRequestFactory对象实例。怎么实现自定义呢？可以看到在RestTemplate中还有另一个构造器：public RestTemplate(ClientHttpRequestFactory requestFactory)，可以传入一个ClientHttpRequestFactory实现类进来，下面例子向spring容器注入了 一个 设置读取和连接超时 时间都为1000ms的HttpComponentsClientHttpRequestFactory实例，并传入RestTemplate的构造器中，这样既可实现一个简单的超时配置import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
@Configuration
public class RestTemplateConfig {
@Bean
public HttpComponentsClientHttpRequestFactory getRequestFactory() {
HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
requestFactory.setConnectTimeout(1000);
requestFactory.setConnectionRequestTimeout(1000);
requestFactory.setReadTimeout(1000);
return requestFactory;
}
}
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.web.client.RestTemplate;
@SpringBootApplication
@EnableScheduling
public class ShowApplication {
@Autowired
HttpComponentsClientHttpRequestFactory httpRequestFactory;
@Bean
public RestTemplate restTemplate() {
return new RestTemplate(httpRequestFactory);
}
public static void main(String[] args) {
SpringApplication.run(ShowApplication.class, args);
}
}