服务器CPU飙高问题查找(待续)
CPU飙高
- 背景：
- 百度整理方法如下：
- 使用top查询出 当前对cpu占用最多的进程
- 查询该进程中具体是哪些线程CPU占用高
- 将线程号 转换为 16进制
- 根据线程号查看线程栈信息
- jstat命令查看了内存使用情况
- jmap查看了内存中创建的对象情况（太多直接存文件）
背景：
- 生产上线后的程序在晚上0:00-6:00 时间段发出告警，cpu 使用过高超过97%。
接下来一个邮件让查找原因。 - 环境 服务器：Centos7 64 JDK8 Springboot.jar项目
百度整理方法如下：
注：测试环境服务器
使用top查询出 当前对cpu占用最多的进程
[root@centos7 hcpu]# top
Tasks: 173 total, 1 running, 171 sleeping, 0 stopped, 1 zombie
Cpu(s): 35.9%us, 0.2%sy, 0.0%ni, 63.8%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8174452k total, 8124064k used, 50388k free, 133764k buffers
Swap: 4095992k total, 1848224k used, 2247768k free, 1040192k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15358 root 18 0 2382m 879m 14m S 40.4 11.0 2326:49 java
32186 root 18 0 2226m 593m 13m S 38.1 7.4 33482:06 java
17397 root 18 0 2761m 1.1g 13m S 30.5 13.5 18771:35 java
24831 root 18 0 2395m 765m 14m S 17.9 9.6 7643:31 java
15540 root 18 0 2245m 686m 13m S 15.9 8.6 17014:51 java
3637 root 15 0 12764 1172 844 R 0.3 0.0 0:00.01 top
3647 root 16 0 10252 656 636 S 0.3 0.0 21:41.94 hald-addon-stor
1 root 15 0 10372 588 556 S 0.0 0.0 0:04.54 init
2 root RT -5 0 0 0 S 0.0 0.0 0:02.79 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:06.93 migration/1
5 root 34 19 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/1
6 root RT -5 0 0 0 S 0.0 0.0 0:02.54 migration/2
7 root 34 19 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/2
8 root RT -5 0 0 0 S 0.0 0.0 0:07.03 migration/3
9 root 34 19 0 0 0 S 0.0 0.0 0:00.01 ksoftirqd/3
10 root 10 -5 0 0 0 S 0.0 0.0 0:07.16 events/0
11 root 10 -5 0 0 0 S 0.0 0.0 0:02.14 events/1
12 root 10 -5 0 0 0 S 0.0 0.0 0:01.93 events/2
13 root 10 -5 0 0 0 S 0.0 0.0 0:01.64 events/3
14 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khelper
87 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
94 root 10 -5 0 0 0 S 0.0 0.0 0:03.79 kblockd/0
95 root 10 -5 0 0 0 S 0.0 0.0 0:00.28 kblockd/1
96 root 10 -5 0 0 0 S 0.0 0.0 0:00.38 kblockd/2
97 root 10 -5 0 0 0 S 0.0 0.0 0:00.15 kblockd/3
98 root 19 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
查看top找出 cpu占用高的进程 PID=15358，正是我们的jar项目
15358 root 18 0 2382m 879m 14m S 40.4 11.0 2326:49 java
查询该进程中具体是哪些线程CPU占用高
这里是引用
[root@centos7 hcpu]# top -Hp 15358
top - 18:29:10 up 310 days, 2:12, 4 users, load average: 1.52, 1.62, 1.91
Tasks: 92 total, 0 running, 92 sleeping, 0 stopped, 0 zombie
Cpu(s): 31.0%us, 0.2%sy, 0.0%ni, 68.4%id, 0.4%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8174452k total, 8126988k used, 47464k free, 133828k buffers
Swap: 4095992k total, 1848224k used, 2247768k free, 1043032k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15360 root 16 0 2382m 879m 14m S 10.3 11.0 531:08.49 java
15361 root 16 0 2382m 879m 14m S 4.3 11.0 530:53.63 java
15365 root 15 0 2382m 879m 14m S 4.0 11.0 183:01.24 java
15362 root 16 0 2382m 879m 14m S 2.0 11.0 532:01.80 java
15363 root 15 0 2382m 879m 14m S 1.0 11.0 531:25.98 java
15437 root 15 0 2382m 879m 14m S 0.3 11.0 0:27.72 java
15358 root 18 0 2382m 879m 14m S 0.0 11.0 0:00.00 java
15359 root 18 0 2382m 879m 14m S 0.0 11.0 0:00.26 java
15364 root 15 0 2382m 879m 14m S 0.0 11.0 6:05.53 java
15366 root 15 0 2382m 879m 14m S 0.0 11.0 0:00.07 java
15367 root 15 0 2382m 879m 14m S 0.0 11.0 0:00.08 java
15368 root 15 0 2382m 879m 14m S 0.0 11.0 0:04.32 java
15369 root 15 0 2382m 879m 14m S 0.0 11.0 0:00.00 java
15370 root 15 0 2382m 879m 14m S 0.0 11.0 1:07.27 java
15371 root 15 0 2382m 879m 14m S 0.0 11.0 1:02.25 java
15372 root 15 0 2382m 879m 14m S 0.0 11.0 0:14.96 java
15373 root 18 0 2382m 879m 14m S 0.0 11.0 0:00.00 java
15374 root 15 0 2382m 879m 14m S 0.0 11.0 0:03.55 java
15380 root 15 0 2382m 879m 14m S 0.0 11.0 0:00.95 java
15381 root 15 0 2382m 879m 14m S 0.0 11.0 0:00.21 java
15382 root 15 0 2382m 879m 14m S 0.0 11.0 0:00.00 java
15383 root 15 0 2382m 879m 14m S 0.0 11.0 0:00.01 java
15384 root 15 0 2382m 879m 14m S 0.0 11.0 0:00.00 java
15386 root 15 0 2382m 879m 14m S 0.0 11.0 0:01.49 java
15387 root 15 0 2382m 879m 14m S 0.0 11.0 0:02.42 java
15388 root 15 0 2382m 879m 14m S 0.0 11.0 0:03.06 java
15389 root 15 0 2382m 879m 14m S 0.0 11.0 0:02.08 java
查询出CPU占用比较高的线程号
15360 root 16 0 2382m 879m 14m S 10.3 11.0 531:08.49 java
将线程号 转换为 16进制
[root@centos7 hcpu]# printf "%x\n" 15360
3c00
根据线程号查看线程栈信息
[root@centos7 hcpu]# jstack 15358 | grep 3c00 -A 60
"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x000000000a47f000 nid=0x3c00 runnable
"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x000000000a480800 nid=0x3c01 runnable
"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x000000000a482800 nid=0x3c02 runnable
"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x000000000a484000 nid=0x3c03 runnable
"Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x000000000a4c7000 nid=0x3c04 runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00002b206c04c000 nid=0x3c0e waiting on condition
JNI global references: 438
jstat命令查看了内存使用情况
[root@centos7 hcpu]# jstat -gcutil 15358 1000 5
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
0.00 0.00 70.29 99.91 98.02 95.76 179675 3.865 359085 76249.377 76253.241
0.00 0.00 70.29 99.91 98.02 95.76 179675 3.865 359085 76249.377 76253.241
0.00 0.00 70.30 99.91 98.02 95.76 179676 3.865 359087 76249.805 76253.670
0.00 0.00 70.30 99.91 98.02 95.76 179676 3.865 359087 76249.805 76253.670
0.00 0.00 70.30 99.91 98.02 95.76 179677 3.865 359089 76250.235 76254.100
查看Suvivor space1、Eden Space和Old Space等内存使用情况是否过多，并且Young GC和Full GC运行时间和次数是否非常高
jmap查看了内存中创建的对象情况（太多直接存文件）
[root@centos7 hcpu]# jmap -histo 15358 > 15358.txt
[root@centos7 hcpu]#
num #instances #bytes class name
----------------------------------------------
1: 1082026 131539016 [C
2: 631018 61248960 [B
3: 106121 20041952 [I
4: 671542 16117008 java.lang.String
5: 254020 8128640 java.util.HashMap$Node
6: 106559 6819776 java.net.URL
7: 179234 5735488 org.springframework.boot.loader.util.AsciiBytes
8: 92029 4360224 [Ljava.lang.Object;
9: 33200 3844736 [Ljava.util.HashMap$Node;
10: 52918 3810096 org.springframework.boot.loader.jar.JarURLConnection
11: 41057 3613016 java.lang.reflect.Method
12: 63154 3536624 org.springframework.boot.loader.jar.JarEntryData
13: 53169 2977464 sun.nio.cs.UTF_8$Encoder
14: 57375 2754000 java.util.HashMap
15: 58026 2321040 java.lang.ref.SoftReference
16: 51646 2065840 java.util.TreeMap$Entry
17: 61791 1977312 java.util.concurrent.ConcurrentHashMap$Node
18: 53500 1712000 com.mysql.cj.conf.BooleanProperty
19: 14283 1618832 java.lang.Class
20: 44868 1435776 com.mysql.cj.util.LazyString
21: 41928 1341696 java.util.Hashtable$Entry
22: 52919 1270056 org.springframework.boot.loader.jar.JarURLConnection$JarEntryName
23: 15205 1216400 java.lang.reflect.Constructor
24: 30239 1209560 java.util.LinkedHashMap$Entry
25: 58712 1184568 [Ljava.lang.Class;
26: 35934 1149888 java.io.ObjectStreamClass$WeakClassKey
27: 47472 1139328 java.util.ArrayList
28: 18612 893376 java.nio.HeapByteBuffer
29: 15333 858648 java.util.LinkedHashMap
"15358.txt" 5747L, 497189C