生产问题定位与排查 - CPU占用过高造成系统卡顿
1.用top命令定位哪个进程对CPU的占用过高
命令：top
[root@izbp1c527]# top
top - 13:30:59 up 49 days, 11:18, 1 user, load average: 0.22, 0.13, 0.33
Tasks: 84 total, 2 running, 82 sleeping, 0 stopped, 0 zombie
%Cpu(s): 85.7 us, 14.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 1882080 total, 73448 free, 1182724 used, 625908 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 512980 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16573 root 20 0 2398708 436044 5500 S 96.7 23.2 13274:56 java
1 root 20 0 59952 2716 1336 S 0.0 0.1 30:48.77 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.86 kthreadd
4 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0.0 0.0 0:56.06 ksoftirqd/0
7 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 R 0.0 0.0 18:10.25 rcu_sched
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain
2.用ps命令进一步定位是哪个线程tid引起的CPU占用过高
命令：ps H -eo pid,tid,%cpu | grep ${pid}
或：ps -mp pid -o THREAD,tid,time
[root@izbp1c527]# ps H -eo pid,tid,%cpu,%mem | grep 16573
16573 16573 0.0 23.1
16573 16586 0.0 23.1
16573 16587 0.0 23.1
16573 16588 22.0 23.1 --> 说明该进程对CPU的占用很高
16573 16589 7.2 23.1
16573 16590 0.0 23.1
16573 16591 0.0 23.1
16573 16592 0.0 23.1
3.使用jstack命令查看栈信息
# 1.查看进程的大概的栈信息
命令：jstack ${pid}
[root@izbp1c527]# jstack 16573
4.转换线程编号tid为16进制(计算器或指令转换)
命令：printf "%x\n" ${tid}
[root@izbp1c527]# printf "%x\n" 16588
40cc
5.打印出问题线程的栈信息，从而定位到源码的位置
命令：jstack -F ${pid}|grep ${tid的16进制值} -A60
jstack -F 31067|grep 796f -A60
[root@izbp1c527]# jstack 16573|grep 40cc -A60
"Concurrent Mark-Sweep GC Thread" os_prio=0 tid=0x00007fd72003e800 nid=0x40cc runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007fd7200f2800 nid=0x40d5 waiting on condition
JNI global references: 42846