测试环境内存溢出排查过程

  • 基本信息

    客户名称:xxx

    产品名称:

    版本号:

    问题分类:编码问题

    环境类型:实体机

  • 问题现象

        保险公司测试环境爆内存,机器挂掉。总共64g的内存,在没有开始测试tomcat部署war包前内存使用率为25%左右。开始测试tomcat后,内存使用率一直上涨,内存使用率从12-08下午起就维持在98%的水平,知道最后机器down掉,才发现存在的问题。

  • 问题原因

        tomcat执行shutdown后,虽然端口占用被释放,tomcat也已经停止。但tomcat的进程并未被杀死,仍旧占用着一开始分配给他的xmx的jvm内存空间。

        Tomcat是一个Servlet容器,用于部署Serlvet程序(我们通常写的各种Java Web应用本质上就是一个Servlet程序)。也就说,在停止Tomcat时不仅仅需要释放Tomcat进程本身所占用的资源,还需要释放Serlvet程序所占用的资源。而出现“停止Tomcat之后进程依然存在”这种现象的主要原因就是:我们自己写的Java Web应用在Tomcat容器停止时没有正常释放所占用的系统资源,比如:线程池未关闭,输入输出流未关闭等等。这些系统资源没有被完全释放,导致tomcat进程一直存在,且无法被正常回收。

        下面是tomcat的shutdown执行的catalina命令,可以看到是程序是先调用Bootstrap类的方法释放Tomcat进程所占用的资源。再使用kill命令停止Tomcat进程:kill -15 。到在这里使用kill命令发送的信号为SIGTERM(15),kill -15在进程未释放系统资源时不能停止Tomcat进程(如:程序的线程池未关闭)。

eval ""$_RUNJAVA"" $LOGGING_MANAGER $JAVA_OPTS
    -Djava.endorsed.dirs=""$JAVA_ENDORSED_DIRS"" -classpath ""$CLASSPATH""
    -Dcatalina.base=""$CATALINA_BASE""
    -Dcatalina.home=""$CATALINA_HOME""
    -Djava.io.tmpdir=""$CATALINA_TMPDIR""
    org.apache.catalina.startup.Bootstrap "$@" stop
# stop failed. Shutdown port disabled? Try a normal kill.
  if [ $? != 0 ]; then
    if [ ! -z "$CATALINA_PID" ]; then
      echo "The stop command failed. Attempting to signal the process to stop through OS signal."
      kill -15 `cat "$CATALINA_PID"` >/dev/null 2>&1
    fi
  fi
  • 排查过程

        察觉到问题发生时,机器已经down掉了。服务器管理员已经执行了重启的操作,时间点为10:29。因此无法直接得知机器当时的内存占用的情况。只能通过一些命令获取重启前机器的部分信息

1、查看系统日志,查看宕机时内存和cpu的情况

1.1、查看是否有杀内存的情况发生:cat /var/log/messages |grep -i "kill"

        可以看到在宕机时,发生了杀内存的情况,证明在机器出问题之前,内存已经相当吃紧。基本上可以确认问题就出在机器内存满了。

测试环境内存溢出排查过程_第1张图片

1.2、查看历史cpu占用情况:sar -u -f /var/log/sa/sa09

cup看起来没什么异常

测试环境内存溢出排查过程_第2张图片

1.3、查看历史内存:sar -r -f /var/log/sa/sa09

可以看到内存占用一直处在99.45%的超高状态,用户占有率高达181%

测试环境内存溢出排查过程_第3张图片

1.4、再向前看一天 sar -r -f /var/log/sa/sa09

        可以观察到8时50分和9时00分,内存激增10g左右。这个有明显的异常,且内存激增的时间点和tomcat开始测试的时间点吻合。因此怀疑是tomcat导致的问题。

测试环境内存溢出排查过程_第4张图片

2、本地测试,尝试还原问题原因

2.1、启动tomcat容器和容器中的luna-project-hub服务,查看内存占用

测试环境内存溢出排查过程_第5张图片

netstat -tnlp | grep 8455(port)

 ps -ef | grep 2730(pid)

测试环境内存溢出排查过程_第6张图片

2.2、关闭tomcat

执行shutdown命令后,项目已经关闭。

测试环境内存溢出排查过程_第7张图片

重新执行ps -ef | grep 2730,发现进程居然仍旧存在,没有被杀死。

测试环境内存溢出排查过程_第8张图片

2.3、jstack,查看tomcat关闭后,进程中仍旧存在的线程

        发现存在的都是一些ScheduledExecutorService类创建的线程。这些线程没有无法关闭,导致了这次的问题。

[root@localhost bin]# jstack -l 2730
2022-12-09 17:16:57
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.301-b09 mixed mode):
 
"Attach Listener" #215 daemon prio=9 os_prio=0 tid=0x00007f3608001800 nid=0xc4e waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
 
   Locked ownable synchronizers:
        - None
 
"DestroyJavaVM" #214 prio=5 os_prio=0 tid=0x00007f364c5fe800 nid=0xb27 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
 
   Locked ownable synchronizers:
        - None
 
"CACHE_LOAD_POOL1" #184 prio=5 os_prio=0 tid=0x00007f35d829e800 nid=0xbe1 waiting on condition [0x00007f359bfdf000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000b0a627b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.naming.beat.sender" #183 daemon prio=5 os_prio=0 tid=0x00007f35b8008000 nid=0xbe0 waiting on condition [0x00007f359c0e0000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ab99ee30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"DictionaryCacheRefreshThreadPool1" #181 prio=5 os_prio=0 tid=0x00007f35d8296000 nid=0xbde waiting on condition [0x00007f359c1e1000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000b0a8f210> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"Curator-Framework-0" #178 daemon prio=5 os_prio=0 tid=0x00007f35a899f800 nid=0xbdb waiting on condition [0x00007f359c4e2000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000b16411d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:938)
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)
        at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - <0x00000000b16413d8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"localhost-startStop-1-EventThread" #177 daemon prio=5 os_prio=0 tid=0x00007f35a899d000 nid=0xbda waiting on condition [0x00007f359c5e3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000b1647918> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:504)
 
   Locked ownable synchronizers:
        - None
 
"localhost-startStop-1-SendThread(10.60.44.187:2181)" #176 daemon prio=5 os_prio=0 tid=0x00007f35a899b800 nid=0xbd9 runnable [0x00007f359c6e4000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000b1618ba8> (a sun.nio.ch.Util$3)
        - locked <0x00000000b1618b98> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000b16186f0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
 
   Locked ownable synchronizers:
        - None
 
"Curator-ConnectionStateManager-0" #175 daemon prio=5 os_prio=0 tid=0x00007f35a892e800 nid=0xbd8 waiting on condition [0x00007f359c7e5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000b1641938> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:272)
        at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:44)
        at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:133)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - <0x00000000b1641b30> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"TaskCommandCacheRefreshThreadPool1" #173 prio=5 os_prio=0 tid=0x00007f35a81ff000 nid=0xbd6 waiting on condition [0x00007f359c8e6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000aedc4b70> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.naming.beat.sender" #172 daemon prio=5 os_prio=0 tid=0x00007f35a81fd000 nid=0xbd5 waiting on condition [0x00007f359c9e7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ab99ee30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"THREAD_POOL_ALARM_POOL-1" #169 prio=5 os_prio=0 tid=0x00007f35a80fc800 nid=0xbd3 waiting on condition [0x00007f359d7e9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000b0e8b0e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"Timer-3" #162 daemon prio=5 os_prio=0 tid=0x00007f35a8b0b000 nid=0xbcc in Object.wait() [0x00007f359e6ee000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        - locked <0x00000000ad465348> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.naming.push.receiver" #127 daemon prio=5 os_prio=0 tid=0x00007f35a80a7800 nid=0xba9 runnable [0x00007f35a3a19000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainDatagramSocketImpl.receive0(Native Method)
        - locked <0x00000000ab99e588> (a java.net.PlainDatagramSocketImpl)
        at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:149)
        - locked <0x00000000ab99e588> (a java.net.PlainDatagramSocketImpl)
        at java.net.DatagramSocket.receive(DatagramSocket.java:812)
        - locked <0x00000000abe96300> (a java.net.DatagramPacket)
        - locked <0x00000000ab99e548> (a java.net.DatagramSocket)
        at com.alibaba.nacos.client.naming.core.PushReceiver.run(PushReceiver.java:83)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - <0x00000000ab99e2d8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"com.alibaba.nacos.naming.failover" #126 daemon prio=5 os_prio=0 tid=0x00007f35a80a5000 nid=0xba8 waiting on condition [0x00007f35a3b1a000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ab9b4980> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.client.naming.updater" #125 daemon prio=5 os_prio=0 tid=0x00007f35a80a3000 nid=0xba7 waiting on condition [0x00007f35a3c1b000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ab9b3c60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.client.naming.updater" #124 daemon prio=5 os_prio=0 tid=0x00007f35a80a1000 nid=0xba6 waiting on condition [0x00007f35a3d1c000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ab9b3c60> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.naming.client.listener" #123 daemon prio=5 os_prio=0 tid=0x00007f35a809e800 nid=0xba5 waiting on condition [0x00007f35a3e1d000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ab99eb90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at com.alibaba.nacos.client.naming.core.EventDispatcher$Notifier.run(EventDispatcher.java:163)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - <0x00000000ab99e918> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"AsyncAppender-Worker-ConsoleAppender" #117 daemon prio=5 os_prio=0 tid=0x00007f35e5959000 nid=0xb9f waiting on condition [0x00007f35a5722000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000aa244020> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
        at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.client.Worker.fixed-10.60.52.109_8848-insurance-ats-upgrade-urtrust" #114 daemon prio=5 os_prio=0 tid=0x00007f35e59d2000 nid=0xb9c runnable [0x00007f35a501f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000aa250eb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.client.config.security.updater" #113 daemon prio=5 os_prio=0 tid=0x00007f35e59d0000 nid=0xb9b waiting on condition [0x00007f35a5120000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000aa2506b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"Timer-2" #112 daemon prio=5 os_prio=0 tid=0x00007f35e59c9800 nid=0xb9a in Object.wait() [0x00007f35a5421000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x00000000aa251358> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)
 
   Locked ownable synchronizers:
        - None
 
"AsyncAppender-Worker-ConsoleAppender" #108 daemon prio=5 os_prio=0 tid=0x00007f35e5945800 nid=0xb96 waiting on condition [0x00007f35a5b24000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000aa0d74c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
        at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)
 
   Locked ownable synchronizers:
        - None
 
"CACHE_LOAD_POOL1" #100 prio=5 os_prio=0 tid=0x00007f3618022000 nid=0xb8c waiting on condition [0x00007f35a6a27000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a771aee0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.naming.beat.sender" #99 daemon prio=5 os_prio=0 tid=0x00007f35ec004000 nid=0xb8b waiting on condition [0x00007f35a6d28000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a3300ea8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"DictionaryCacheRefreshThreadPool1" #96 prio=5 os_prio=0 tid=0x00007f3618003000 nid=0xb88 waiting on condition [0x00007f35a6e29000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a771abb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"Curator-Framework-0" #93 daemon prio=5 os_prio=0 tid=0x00007f35e7630000 nid=0xb85 waiting on condition [0x00007f35a7ffe000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a83f4b90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:938)
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:66)
        at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:346)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - <0x00000000a83f4d90> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"localhost-startStop-1-EventThread" #92 daemon prio=5 os_prio=0 tid=0x00007f35e70f6000 nid=0xb84 waiting on condition [0x00007f35ac550000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a83f4490> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:504)
 
   Locked ownable synchronizers:
        - None
 
"localhost-startStop-1-SendThread(10.60.44.187:2181)" #91 daemon prio=5 os_prio=0 tid=0x00007f35e70f4800 nid=0xb83 runnable [0x00007f35ac651000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000a83e2b00> (a sun.nio.ch.Util$3)
        - locked <0x00000000a83e2af0> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000000a83e2648> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
 
   Locked ownable synchronizers:
        - None
 
"Curator-ConnectionStateManager-0" #90 daemon prio=5 os_prio=0 tid=0x00007f35e749c800 nid=0xb82 waiting on condition [0x00007f35ac952000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a83f52d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
        at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:272)
        at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:44)
        at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:133)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - <0x00000000a83f54c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"TaskCommandCacheRefreshThreadPool1" #88 prio=5 os_prio=0 tid=0x00007f35e6b33800 nid=0xb80 waiting on condition [0x00007f35aca53000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a48584c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.naming.beat.sender" #86 daemon prio=5 os_prio=0 tid=0x00007f35e6b2f800 nid=0xb7e waiting on condition [0x00007f35acc55000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a3300ea8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"THREAD_POOL_ALARM_POOL-1" #83 prio=5 os_prio=0 tid=0x00007f35e7185800 nid=0xb7c waiting on condition [0x00007f35ad257000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a7f8c2e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"Java2D Disposer" #81 daemon prio=10 os_prio=0 tid=0x00007f35e76f4800 nid=0xb7b in Object.wait() [0x00007f3630892000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000a7a92958> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
        - locked <0x00000000a7a92958> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
        at sun.java2d.Disposer.run(Disposer.java:148)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"Timer-1" #74 daemon prio=5 os_prio=0 tid=0x00007f35e7ab2000 nid=0xb74 in Object.wait() [0x00007f35af408000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000a420e9f0> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:502)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        - locked <0x00000000a420e9f0> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.naming.push.receiver" #39 daemon prio=5 os_prio=0 tid=0x00007f35e6d09800 nid=0xb51 runnable [0x00007f362adfa000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainDatagramSocketImpl.receive0(Native Method)
        - locked <0x00000000a3300670> (a java.net.PlainDatagramSocketImpl)
        at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:149)
        - locked <0x00000000a3300670> (a java.net.PlainDatagramSocketImpl)
        at java.net.DatagramSocket.receive(DatagramSocket.java:812)
        - locked <0x00000000a34f2448> (a java.net.DatagramPacket)
        - locked <0x00000000a3300630> (a java.net.DatagramSocket)
        at com.alibaba.nacos.client.naming.core.PushReceiver.run(PushReceiver.java:83)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - <0x00000000a33001e8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"com.alibaba.nacos.naming.failover" #38 daemon prio=5 os_prio=0 tid=0x00007f35e6d04800 nid=0xb50 waiting on condition [0x00007f362b0fb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a33029e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.client.naming.updater" #37 daemon prio=5 os_prio=0 tid=0x00007f35e6d01000 nid=0xb4f waiting on condition [0x00007f362b1fc000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a3301ed8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.client.naming.updater" #36 daemon prio=5 os_prio=0 tid=0x00007f35e6cff000 nid=0xb4e waiting on condition [0x00007f362b2fd000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a3301ed8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.naming.client.listener" #35 daemon prio=5 os_prio=0 tid=0x00007f35e6cf8800 nid=0xb4d waiting on condition [0x00007f362b3fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a3300c08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at com.alibaba.nacos.client.naming.core.EventDispatcher$Notifier.run(EventDispatcher.java:163)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - <0x00000000a3300918> (a java.util.concurrent.ThreadPoolExecutor$Worker)
 
"AsyncAppender-Worker-ConsoleAppender" #29 daemon prio=5 os_prio=0 tid=0x00007f35e6059800 nid=0xb47 waiting on condition [0x00007f3631f2d000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a20439f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
        at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.client.Worker.fixed-10.60.52.109_8848-insurance-ats-upgrade-urtrust" #26 daemon prio=5 os_prio=0 tid=0x00007f35e6387000 nid=0xb44 runnable [0x00007f3630690000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a20a3b98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"com.alibaba.nacos.client.config.security.updater" #25 daemon prio=5 os_prio=0 tid=0x00007f35e6380800 nid=0xb43 waiting on condition [0x00007f3630791000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a20a3090> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
 
   Locked ownable synchronizers:
        - None
 
"Timer-0" #23 daemon prio=5 os_prio=0 tid=0x00007f35e634c800 nid=0xb41 in Object.wait() [0x00007f3630b93000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x00000000a2856cf0> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)
 
   Locked ownable synchronizers:
        - None
 
"AsyncAppender-Worker-ConsoleAppender" #19 daemon prio=5 os_prio=0 tid=0x00007f35e61a3800 nid=0xb3d waiting on condition [0x00007f3631296000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a1e7a940> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
        at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:289)
 
   Locked ownable synchronizers:
        - None
 
"GC Daemon" #12 daemon prio=2 os_prio=0 tid=0x00007f364c523000 nid=0xb36 in Object.wait() [0x00007f36381c9000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000000a0f796d8> (a sun.misc.GC$LatencyLock)
        at sun.misc.GC$Daemon.run(GC.java:117)
        - locked <0x00000000a0f796d8> (a sun.misc.GC$LatencyLock)
 
   Locked ownable synchronizers:
        - None
 
"AsyncFileHandlerWriter-1259475182" #11 daemon prio=5 os_prio=0 tid=0x00007f364c1b6000 nid=0xb35 waiting on condition [0x00007f3638520000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000a0f796e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
        at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
        at org.apache.juli.AsyncFileHandler$LoggerThread.run(AsyncFileHandler.java:160)
 
   Locked ownable synchronizers:
        - None
 
"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f364c0cd000 nid=0xb33 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
 
   Locked ownable synchronizers:
        - None
 
"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f364c0c2000 nid=0xb32 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
 
   Locked ownable synchronizers:
        - None
 
"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f364c0c0000 nid=0xb31 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
 
   Locked ownable synchronizers:
        - None
 
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f364c0bd000 nid=0xb30 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
 
   Locked ownable synchronizers:
        - None
 
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f364c0bb800 nid=0xb2f runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
 
   Locked ownable synchronizers:
        - None
 
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f364c088800 nid=0xb2e in Object.wait() [0x00007f3639351000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
        - locked <0x00000000a0f79d30> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
 
   Locked ownable synchronizers:
        - None
 
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f364c084000 nid=0xb2d in Object.wait() [0x00007f3639452000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        - locked <0x00000000a0f79d70> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
 
   Locked ownable synchronizers:
        - None
 
"VM Thread" os_prio=0 tid=0x00007f364c07a000 nid=0xb2c runnable
 
"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f364c020800 nid=0xb28 runnable
 
"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f364c022800 nid=0xb29 runnable
 
"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f364c024800 nid=0xb2a runnable
 
"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f364c026000 nid=0xb2b runnable
 
"VM Periodic Task Thread" os_prio=0 tid=0x00007f364c0d2000 nid=0xb34 waiting on condition
 
JNI global references: 3541

2.4、tomcat关闭时的报错信息

        tomcat关闭时,其实有线程无法关闭的报错信息。但当时认为端口已经释放,tomcat程序已经关闭,就没有意识到这个问题。

  • 解决方案

手动关停无法关闭的tomcat的进程:

./shutdown.sh

ps -ef |grep tomcat

kill -9 pid

  • 排查内存问题时可能用到的其他命令

查看当前内存占用高低排序:

ps aux --sort -%mem


查看内存占用最高的10个:

ps aux | head -1;ps aux |grep -v PID |sort -rn -k +4 | head -10


查看当前内存占用:

free

参考文档

排查思路:https://www.cnblogs.com/zcyy/p/16033677.html

tomcat关闭时,nacos线程报错关闭的原因:tomcat无法正常关闭问题分析及解决 - 码农教程

tomcat无法正常关闭的原因:记一次关于tomcat关闭时,清理线程时的警告日志问题排查。_org.apache.catalina.loader.webappclassloaderbase:1-CSDN博客

你可能感兴趣的:(java,tomcat)