这几天被集群并发问题快折腾死了,望哪位高人看下到底是哪里出现了问题。
Apache Server是正常的,各服务器的Tomcat 也是正常的,但当Apache的连接数达到 300左右的时候,JK就会出现如下的异常信息,然后整个系统就崩溃了,这些都是通过JK访问的,就无法打开了。但是各分服务器访问还是正常的。 如果把Apache重启后,访问速度又很顺畅了。所以现在初步判定是 Jk的问题。
JK work.properties的配置如下:
worker.list=loadbalancer,status,tomcat1,tomcat6,tomcat7
worker.tomcat1.port=8009
worker.tomcat1.host=localhost
worker.tomcat1.type=ajp13
worker.tomcat1.lbfactor=2
worker.tomcat5.port=9009
worker.tomcat5.host=localhost
worker.tomcat5.type=ajp13
worker.tomcat5.lbfactor=1
worker.tomcat6.port=8009
worker.tomcat6.host=110.121.211.205
worker.tomcat6.type=ajp13
worker.tomcat6.lbfactor=1
worker.tomcat7.port=7009
worker.tomcat7.host=110.144.0.154
worker.tomcat7.type=ajp13
worker.tomcat7.lbfactor=1
worker.loadbalancer.type=lb
worker.lb_s.retries=3
worker.recovery_options=3
worker.loadbalancer.balance_workers=tomcat1,tomcat7,tomcat6
worker.status.type=status
[Thu Apr 30 14:43:29 2015] [26545:14544] [info] service::jk_lb_worker.c (1098): service failed, worker tomcat1 is in error state
[Thu Apr 30 14:43:29 2015] [26586:14544] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (951): (tomcatTaohandan) can't receive the response message from tomcat, network problems or tomcat (10.144.211.205:8009) is down (errno=104)
[Thu Apr 30 14:43:29 2015] [26586:14544] [error] ajp_get_reply::jk_ajp_common.c (1592): (tomcatTaohandan) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Thu Apr 30 14:43:29 2015] [26586:14544] [info] ajp_service::jk_ajp_common.c (1902): (tomcatTaohandan) receiving from tomcat failed, recoverable operation attempt=0
[Thu Apr 30 14:43:29 2015] [26586:14544] [info] ajp_service::jk_ajp_common.c (1941): (tomcatTaohandan) sending request to tomcat failed, recoverable operation attempt=1
[Thu Apr 30 14:43:30 2015] [26769:14544] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (951): (tomcatTaohandan) can't receive the response message from tomcat, network problems or tomcat (10.144.211.205:8009) is down (errno=104)
[Thu Apr 30 14:43:30 2015] [26769:14544] [error] ajp_get_reply::jk_ajp_common.c (1592): (tomcatTaohandan) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Thu Apr 30 14:43:30 2015] [26769:14544] [info] ajp_service::jk_ajp_common.c (1902): (tomcatTaohandan) receiving from tomcat failed, recoverable operation attempt=1
[Thu Apr 30 14:43:30 2015] [26769:14544] [info] ajp_service::jk_ajp_common.c (1941): (tomcatTaohandan) sending request to tomcat failed, recoverable operation attempt=2
[Thu Apr 30 14:43:30 2015] [26769:14544] [error] ajp_service::jk_ajp_common.c (1953): (tomcatTaohandan) Connecting to tomcat failed. Tomcat is probably not started or is listening on the wrong port
[Thu Apr 30 14:43:30 2015] [26769:14544] [info] service::jk_lb_worker.c (1098): service failed, worker tomcatTaohandan is in error state
[Thu Apr 30 14:43:31 2015] [26620:14544] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (951): (tomcat1) can't receive the response message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=104)
[Thu Apr 30 14:43:31 2015] [26620:14544] [error] ajp_get_reply::jk_ajp_common.c (1592): (tomcat1) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Thu Apr 30 14:43:31 2015] [26620:14544] [info] ajp_service::jk_ajp_common.c (1902): (tomcat1) receiving from tomcat failed, recoverable operation attempt=0
[Thu Apr 30 14:43:31 2015] [26620:14544] [info] ajp_service::jk_ajp_common.c (1941): (tomcat1) sending request to tomcat failed, recoverable operation attempt=1
[Thu Apr 30 14:43:31 2015] [26618:14544] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (951): (tomcatTaohandan) can't receive the response message from tomcat, network problems or tomcat (10.144.211.205:8009) is down (errno=104)
[Thu Apr 30 14:43:31 2015] [26618:14544] [error] ajp_get_reply::jk_ajp_common.c (1592): (tomcatTaohandan) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Thu Apr 30 14:43:31 2015] [26618:14544] [info] ajp_service::jk_ajp_common.c (1902): (tomcatTaohandan) receiving from tomcat failed, recoverable operation attempt=0
[Thu Apr 30 14:43:31 2015] [26618:14544] [info] ajp_service::jk_ajp_common.c (1941): (tomcatTaohandan) sending request to tomcat failed, recoverable operation attempt=1
[Thu Apr 30 14:43:32 2015] [26770:14544] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (951): (tomcatTaohandan) can't receive the response message from tomcat, network problems or tomcat (10.144.211.205:8009) is down (errno=104)
[Thu Apr 30 14:43:32 2015] [26770:14544] [error] ajp_get_reply::jk_ajp_common.c (1592): (tomcatTaohandan) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Thu Apr 30 14:43:32 2015] [26770:14544] [info] ajp_service::jk_ajp_common.c (1902): (tomcatTaohandan) receiving from tomcat failed, recoverable operation attempt=1
[Thu Apr 30 14:43:32 2015] [26770:14544] [info] ajp_service::jk_ajp_common.c (1941): (tomcatTaohandan) sending request to tomcat failed, recoverable operation attempt=2
[Thu Apr 30 14:43:32 2015] [26770:14544] [error] ajp_service::jk_ajp_common.c (1953): (tomcatTaohandan) Connecting to tomcat failed. Tomcat is probably not started or is listening on the wrong port
[Thu Apr 30 14:43:32 2015] [26770:14544] [info] service::jk_lb_worker.c (1098): service failed, worker tomcatTaohandan is in error state
[Thu Apr 30 14:43:33 2015] [25203:14544] [info] ajp_send_request::jk_ajp_common.c (1215): (tomcatchangbao) error sending request. Will try another pooled connection
[Thu Apr 30 14:43:33 2015] [25203:14544] [info] ajp_send_request::jk_ajp_common.c (1241): (tomcatchangbao) all endpoints are disconnected
[Thu Apr 30 14:43:33 2015] [25203:14544] [info] ajp_send_request::jk_ajp_common.c (1244): (tomcatchangbao) increase the backend idle connection timeout or the connection_pool_minsize
[Thu Apr 30 14:43:33 2015] [25203:14544] [info] ajp_service::jk_ajp_common.c (1941): (tomcatchangbao) sending request to tomcat failed, recoverable operation attempt=1
[Thu Apr 30 14:43:33 2015] [25483:14544] [error] ajp_connection_tcp_get_message::jk_ajp_common.c (951): (tomcat1) can't receive the response message from tomcat, network problems or tomcat (127.0.0.1:8009) is down (errno=104)
[Thu Apr 30 14:43:33 2015] [25483:14544] [error] ajp_get_reply::jk_ajp_common.c (1592): (tomcat1) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Thu Apr 30 14:43:33 2015] [25483:14544] [info] ajp_service::jk_ajp_common.c (1902): (tomcat1) receiving from tomcat failed, recoverable operation attempt=1
[Thu Apr 30 14:43:33 2015] [25483:14544] [info] ajp_service::jk_ajp_common.c (1941): (tomcat1) sending request to tomcat failed, recoverable operation attempt=2
[Thu Apr 30 14:43:33 2015] [25483:14544] [error] ajp_service::jk_ajp_common.c (1953): (tomcat1) Connecting to tomcat failed. Tomcat is probably not started or is listening on the wrong port
[Thu Apr 30 14:43:33 2015] [25483:14544] [info] service::jk_lb_worker.c (1098): service failed, worker tomcat1 is in error state