quartz线程阻塞问题

quartz blocked

现象-所有quartz的job(Hour/Minute等失去作用 无相关任何log输出)

jstack 发现:所有的quartz线程都在MinuteJob.execute#PrometheusService.pushgateway

• 最终阻塞在SocketInputStream.socketRead0
• 注意此时线程阻塞 但是状态是runnable
• 因为线程就new/runnable/blocked#synchronized/waiting#Object.wait/TIMED_WAITING#sleep.../TERMINATED
• 所以即使类似调用socket.read时 线程会挂起 但此时状态还是runnable

    "DefaultQuartzScheduler_Worker-9" #40 prio=5 os_prio=0 tid=0x00007f38ecd51800 nid=0x5a5d runnable [0x00007f38c94c8000]
       java.lang.Thread.State: RUNNABLE
            at java.net.SocketInputStream.socketRead0(Native Method)
            at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
            at java.net.SocketInputStream.read(SocketInputStream.java:170)
            at java.net.SocketInputStream.read(SocketInputStream.java:141)
            at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
            at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
            at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
            at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
            at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
            at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
            at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
            at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
            at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
            at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
            at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
            at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
            at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
            at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
            at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
            at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
            at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
            at com.xx.achilles.spurs.common.utils.HttpUtils.doPost(HttpUtils.java:53)
            at com.xx.achilles.spurs.gs.service.PrometheusService.pushgateway(PrometheusService.java:85)
            at com.xx.achilles.spurs.gs.service.PrometheusService.pushOnlineCount(PrometheusService.java:63)
            at com.xx.achilles.spurs.gs.quartz.MinuteJob.execute(MinuteJob.java:46)
            at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
            at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
            - locked <0x00000000805e7928> (a java.lang.Object)
    

• debug看-push的http地址

  • 有时候是connect连接不上 直接走超时了
  • 线上的错误是connect连接上了 但是socket#read超时了 所以应该再加上一个socketTimeout

  try {
            HttpUtils.doPost(url, data, HttpConstants.CONNECT_TIMEOUT, HttpConstants.SO_TIMEOUT);
        } catch (Exception e) {
            DataLogService.getInstance().error("时实采集数据推送异常", e);
        }
  

  • Defines the socket timeout ({@code SO_TIMEOUT}) in milliseconds,which is the timeout for waiting for data or, put differently, a maximum period inactivity between two consecutive data packets
  • 即connectTimeout和socketTimeout都要设置

        *  connectionRequestTimeout:
        *  从连接池中获取连接的超时时间，超过该时间未拿到可用连接，
        *  会抛出org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
        *  connectTimeout:
        *  连接上服务器(握手成功)的时间，超出该时间抛出connect timeout
        *  socketTimeout:
        *  服务器返回数据(response)的时间，超过该时间抛出read timeout