java系统运行缓慢，CPU 100%，以及Full GC次数过多问题的排查工具和思路
一.排查思路
推荐参考: https://my.oschina.net/zhangxufeng/blog/3017521
二.排查工具使用
-
寻找服务进程(如2823)
[root@zuo demo]# jps-
查看服务每个内存区域情况
[root@zuo demo]# jmap -heap 2823
Attaching to process ID 5187, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.161-b12
using thread-local object allocation.
Parallel GC with 4 thread(s)
Heap Configuration:
MinHeapFreeRatio = 0
MaxHeapFreeRatio = 100
MaxHeapSize = 1572864000 (1500.0MB)
NewSize = 349175808 (333.0MB)
MaxNewSize = 524288000 (500.0MB)
OldSize = 699400192 (667.0MB)
NewRatio = 2
SurvivorRatio = 8
MetaspaceSize = 21807104 (20.796875MB)
CompressedClassSpaceSize = 1073741824 (1024.0MB)
MaxMetaspaceSize = 17592186044415 MB
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
PS Young Generation
Eden Space:
capacity = 187170816 (178.5MB)
used = 171007032 (163.08501434326172MB)
free = 16163784 (15.414985656738281MB)
91.36415369370404% used
From Space:
capacity = 4718592 (4.5MB)
used = 4278352 (4.0801544189453125MB)
free = 440240 (0.4198455810546875MB)
90.67009819878473% used
To Space:
capacity = 108527616 (103.5MB)
used = 0 (0.0MB)
free = 108527616 (103.5MB)
0.0% used
PS Old Generation
capacity = 1048576000 (1000.0MB)
used = 94086448 (89.72782897949219MB)
free = 954489552 (910.2721710205078MB)
8.97278289794922% used
-
查看该进程对象空间占用排名看有没有业务大对象排在前面
[root@zuo demo]# jmap -histo:live 2823 > aaa.txt
[root@zuo demo]# more aaa.txt
num #instances #bytes class name
----------------------------------------------
1: 189797 20947312 [C
2: 88129 7755352 java.lang.reflect.Method
3: 188451 6030432 java.util.concurrent.ConcurrentHashMap$Node
4: 187165 4491960 java.lang.String
5: 1052 4035808 [J
6: 83755 3350200 java.util.LinkedHashMap$Entry
7: 58419 2804112 org.aspectj.weaver.reflect.ShadowMatchImpl
8: 47515 2608432 [Ljava.lang.Object;
9: 46272 2591232 java.util.LinkedHashMap
10: 13806 2197104 [B
11: 19004 2141400 java.lang.Class
12: 30251 2099096 [Ljava.util.HashMap$Node;
13: 1083 1989080 [Ljava.util.concurrent.ConcurrentHashMap$Node;
14: 58419 1869408 org.aspectj.weaver.patterns.ExposedState
15: 43429 1389728 java.util.HashMap$Node
16: 8325 1323768 [I
17: 42141 1011384 java.util.ArrayList
18: 40578 973872 org.springframework.core.MethodClassKey
19: 52418 838688 java.lang.Object
20: 36074 803976 [Ljava.lang.Class;
21: 17623 563936 java.util.LinkedList
22: 19220 517112 [Ljava.lang.String;
23: 15732 503424 java.lang.ref.WeakReference
24: 5909 425448 org.springframework.core.annotation.AnnotationAttributes-
查看GC/内存占用/cpu占用/死锁检测等[jconsole]
-
查看该进程下所有线程的运行情况[top]
看哪个线程占用内存或者cpu高?
[root@zuo demo]# top -H -p 2823
top - 17:50:39 up 8 days, 4:37, 7 users, load average: 0.04, 0.04, 0.09
Threads: 101 total, 0 running, 101 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.1 us, 0.0 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.1 st
KiB Mem : 14203320 total, 355044 free, 12963976 used, 884300 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 917336 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5187 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:00.04 java
5191 root 20 0 5462568 1.934g 14652 S 0.0 14.3 2:44.18 java
5192 root 20 0 5462568 1.934g 14652 S 0.0 14.3 1:06.53 java
5193 root 20 0 5462568 1.934g 14652 S 0.0 14.3 1:06.70 java
5194 root 20 0 5462568 1.934g 14652 S 0.0 14.3 1:06.32 java
5195 root 20 0 5462568 1.934g 14652 S 0.0 14.3 1:06.41 java
5202 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:07.31 java
5203 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:00.03 java
5204 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:00.25 java
5205 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:00.00 java
5206 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:00.00 java
5207 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:00.00 java
5208 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:53.65 java
5209 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:51.13 java
5210 root 20 0 5462568 1.934g 14652 S 0.0 14.3 0:14.52 java
-
查看该进程下所有线程运行信息[jstack]
看看有没死锁信息;
找找cpu或内存高的进程在做什么?
[root@zuo demo]# jstack 2823 > bbb.txt
[root@zuo demo]# more bbb.txt
2020-07-01 17:53:01
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode):
"Attach Listener" #155 daemon prio=9 os_prio=0 tid=0x00007fd600209000 nid=0x184d waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"RMI TCP Connection(24)-192.168.18.135" #153 daemon prio=5 os_prio=0 tid=0x00007fd5ec013800 nid=0x17fe runnable [0x00007fd5a16e0000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x00000000a79d1e20> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$240/1511977249.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
"RMI TCP Connection(20)-192.168.18.135" #151 daemon prio=5 os_prio=0 tid=0x00007fd5ec011000 nid=0x17fa runnable [0x00007fd5a1fe7000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
- locked <0x00000000a79d63d8> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)该命令是查看进程2823下所有线程此刻的运行信息,但是线程号nid是16进制的,需要通过linux命令进行转换,看占用资源高的进行具体是在运行什么
printf %x 十六进制数 --> 输出十进制
printf %d 十进制数 --> 输出十六进制