聊聊nginx与tomcat的5xx
序
本文主要讲述一下nginx与tomcat的502、504、503错误及其常见的产生原因。
502
定义
502 Bad Gateway : 作为网关或者代理工作的服务器尝试执行请求时，从上游服务器接收到无效的响应。
常见原因
- 后端服务挂了的情况,直接502
- 后端服务在重启
实例
将后端服务关掉，然后向nginx发送请求后端接口，日志如下：
- access.log
127.0.0.1 - - [22/Dec/2017:20:44:38 +0800] "GET /timeout/long-write HTTP/1.1" 502 537 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36"- error.log
2017/12/22 20:45:12 [error] 1481#0: *3 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 127.0.0.1, server: localhost, request: "GET /timeout/long-write HTTP/1.1", upstream: "http://[::1]:8080/timeout//long-write", host: "localhost:8888"504
定义
504 Gateway Timeout : 作为网关或者代理工作的服务器尝试执行请求时，未能及时从上游服务器（URI标识出的服务器，例如HTTP、FTP、LDAP）或者辅助服务器（例如DNS）收到响应。注意：某些代理服务器在DNS查询超时时会返回400或者500错误
常见原因
- 该接口太耗时，后端服务接收到请求，开始执行，未能在设定时间返回数据给nginx
- 后端服务器整体负载太高，接受到请求之后，由于线程繁忙，未能安排给请求的接口，导致未能在设定时间返回数据给nginx
实例
- 前端返回
504 Gateway Time-out
504 Gateway Time-out
openresty/1.9.15.1
- access.log
192.168.99.1 - - [22/Dec/2017:21:58:20 +0800] "GET /timeout/long-resp HTTP/1.1" 504 591 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36" "-" "-"- error.log
2017/12/22 21:58:20 [error] 5#5: *7 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.99.1, server: , request: "GET /timeout/long-resp HTTP/1.1", upstream: "http://192.168.99.100:8080/timeout//long-resp", host: "192.168.99.100:8686"- nginx.conf
location /timeout/long-resp {
proxy_connect_timeout 30;
proxy_read_timeout 100;
proxy_send_timeout 10;
proxy_pass http://192.168.99.100:8080/timeout/long-resp ;
}- java代码
@GetMapping("/timeout/long-resp")
public String longResp() throws InterruptedException {
TimeUnit.SECONDS.sleep(120);
return "finish";
}服务器接受请求一直没有返回,nginx在等待100秒后报Connection timed out,返回504；但是后端继续执行，在第120秒才执行完。
503(相对少见)
定义
503 Service Unavailable : 表示服务器当前处于暂时不可用状态，无论是有意还是无意，当服务器端处于无法应答的状态时，就会返回该状态码。其中，服务端因维护需要而停止服务属于有意的情况。而当服务器自身负载过高，处于无法响应的状态时，则属于无意的情况。另外，负载均衡器或者web服务器的前置机等这些地方的服务器也有可能返回503.
常见原因
- nginx进行限流，超过限速则返回503
- 后端服务进行常规维护，比如pause tomcat
nginx限流返回503实例
- config
http{
## test 503
limit_conn_zone $binary_remote_addr zone=addr:10m;
server {
listen 8686;
location /timeout {
limit_conn addr 1;
proxy_connect_timeout 30;
proxy_read_timeout 100;
proxy_send_timeout 2;
proxy_pass http://192.168.99.100:8080/timeout/ ;
}
}
}- error.log
2017/12/24 20:58:29 [error] 5#5: *1473 limiting connections by zone "addr", client: 192.168.99.1, server: , request: "GET /timeout/busy HTTP/1.1", host: "192.168.99.100:8686"- access.log
192.168.99.1 - - [24/Dec/2017:20:58:39 +0800] "GET /timeout/busy HTTP/1.1" 503 219 "-" "-" "-" "-"- client
wrk -t12 -c200 -d100s -T60s --latency http://192.168.99.100:8686/timeout/busy
➜ ~ curl -i http://192.168.99.100:8686/timeout/busy
HTTP/1.1 503 Service Temporarily Unavailable
Server: openresty/1.9.15.1
Date: Sun, 24 Dec 2017 12:58:26 GMT
Content-Type: text/html
Content-Length: 219
Connection: keep-alive
503 Service Temporarily Unavailable
503 Service Temporarily Unavailable
openresty/1.9.15.1
tomcat返回503实例
- Http11Processor
tomcat-embed-core-8.5.23-sources.jar!/org/apache/coyote/http11/Http11Processor.java
@Override
public SocketState service(SocketWrapperBase socketWrapper)
throws IOException {
RequestInfo rp = request.getRequestProcessor();
rp.setStage(org.apache.coyote.Constants.STAGE_PARSE);
// Setting up the I/O
setSocketWrapper(socketWrapper);
inputBuffer.init(socketWrapper);
outputBuffer.init(socketWrapper);
// Flags
keepAlive = true;
openSocket = false;
readComplete = true;
boolean keptAlive = false;
SendfileState sendfileState = SendfileState.DONE;
while (!getErrorState().isError() && keepAlive && !isAsync() && upgradeToken == null &&
sendfileState == SendfileState.DONE && !endpoint.isPaused()) {
//......
if (endpoint.isPaused()) {
// 503 - Service unavailable
response.setStatus(503);
setErrorState(ErrorState.CLOSE_CLEAN, null);
} else {
keptAlive = true;
// Set this every time in case limit has been changed via JMX
request.getMimeHeaders().setLimit(endpoint.getMaxHeaderCount());
if (!inputBuffer.parseHeaders()) {
// We've read part of the request, don't recycle it
// instead associate it with the socket
openSocket = true;
readComplete = false;
break;
}
if (!disableUploadTimeout) {
socketWrapper.setReadTimeout(connectionUploadTimeout);
}
}
}
} 只要endpoint的状态是paused，则返回503
- AbstractEndpoint
tomcat-embed-core-8.5.23-sources.jar!/org/apache/tomcat/util/net/AbstractEndpoint.java
/**
* Pause the endpoint, which will stop it accepting new connections.
*/
public void pause() {
if (running && !paused) {
paused = true;
unlockAccept();
getHandler().pause();
}
}
/**
* Resume the endpoint, which will make it start accepting new connections
* again.
*/
public void resume() {
if (running) {
paused = false;
}
}这里是endpoint的pause以及resume方法
- 请求
当请求进入Http11Processor的service方法到执行endpoint.isPaused()方法期间，tomcat被pause了，这个时候，就会返回503，如下：
➜ ~ curl -i http://localhost:8080/demo/test
HTTP/1.1 503
Transfer-Encoding: chunked
Date: Sun, 24 Dec 2017 14:10:16 GMT
Connection: close小结
- 502
通常是后端服务挂了或在重启
- 504
通常是请求的接口执行耗时，亦或是后端服务负载高，执行耗时
- 503
通常是nginx限流或后端服务pause进行维护