定位高cpu使用率

1、  使用top查看占用cpu高的进程

2、  对cpu占用高的进程的所有线程进行排序，以17ce集中监控程序为例

发现该进程下有好多的线程，而每个线程占用的资源并不高，将他打印到文件里统计

root@ubuntu:~/admin/17ce#ps H -e -o pid,tid,pcpu,cmd --sort=pcpu |grep 17ce_cli_cen_1 >1.out

root@ubuntu:~/admin/17ce#wc -l 1.out

361 1.out

统计有360个线程，所以这里cpu使用率过高应该是线程数量太多导致。

接下来看一个其他原因的，以调度为例dispatch

root@198-77:~#ps H -e -o pid,tid,pcpu,cmd --sort=pcpu |grep dispatch

 8472 8472  0.0 grep --color=autodispatch

 6805 20305 0.1 ./dispatch -log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4-vmodule=SearchNodes=4 -vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 6885  0.1 ./dispatch-log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 6985  0.1 ./dispatch-log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 21105 0.1 ./dispatch -log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4-vmodule=SearchNodes=4 -vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 6806  0.5 ./dispatch-log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 6833  1.2 ./dispatch -log_dir=./logs-v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 20303 1.4 ./dispatch -log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4-vmodule=SearchNodes=4 -vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 21104 2.1 ./dispatch -log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4-vmodule=SearchNodes=4 -vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 6883  3.5 ./dispatch-log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 6812  4.5 ./dispatch-log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 27636 4.8 ./dispatch -log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4-vmodule=SearchNodes=4 -vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805  6810  5.0 ./dispatch -log_dir=./logs -v 7-vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 20302 5.0 ./dispatch -log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4-vmodule=SearchNodes=4 -vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 21103 5.2 ./dispatch -log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4-vmodule=SearchNodes=4 -vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 20300 5.6 ./dispatch -log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4-vmodule=SearchNodes=4 -vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 21219 5.8 ./dispatch -log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4-vmodule=SearchNodes=4 -vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 6808  6.1 ./dispatch-log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 6807  6.7 ./dispatch-log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 9440  6.8 ./dispatch-log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 6811  6.8 ./dispatch-log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 6805  6.9 ./dispatch-log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 21106 6.9 ./dispatch -log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4-vmodule=SearchNodes=4 -vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 6814  7.0 ./dispatch-log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 6809  7.2 ./dispatch-log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 6884  7.4 ./dispatch-log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 6813  7.5 ./dispatch-log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 20304 7.8 ./dispatch -log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4-vmodule=SearchNodes=4 -vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 6823  8.0 ./dispatch-log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4 -vmodule=SearchNodes=4-vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

 6805 20301 8.2 ./dispatch -log_dir=./logs -v 7 -vmodule=SearchForNewChannel=4-vmodule=SearchNodes=4 -vmodule=RuntimeSpeedTest=4 -SetKey=17ce.mon.*.*

发现每个线程cpu占用都挺高，这里线程20301占用资源8.2%最高

3、  调出调用栈：

如果没有gdb的话安装个，apt-get install gdb

进入gdb环境

root@198-77:~#gdb

GNU gdb(Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04

Copyright(C) 2012 Free Software Foundation, Inc.

LicenseGPLv3+: GNU GPL version 3 or later

This is freesoftware: you are free to change and redistribute it.

There is NOWARRANTY, to the extent permitted by law. Type "show copying"

and"show warranty" for details.

This GDB wasconfigured as "x86_64-linux-gnu".

For bugreporting instructions, please see:

.

(gdb)

attach 命令进行连接该进程

(gdb) attach6805

使用info threads显示所有线程

(gdb) infothreads

  Id  Target Id         Frame

  29  Thread 0x7f956810c700 (LWP 6806) "dispatch" runtime.usleep ()at /root/go/src/runtime/sys_linux_amd64.s:82

  28  Thread 0x7f956790b700 (LWP 6807) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  27  Thread 0x7f9566ce7700 (LWP 6808) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  26  Thread 0x7f95660cd700 (LWP 6809) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  25  Thread 0x7f95658cc700 (LWP 6810) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  24  Thread 0x7f95650cb700 (LWP 6811) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  23  Thread 0x7f95648ca700 (LWP 6812) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  22  Thread 0x7f9557fff700 (LWP 6813) "dispatch" runtime.xchg () at/root/go/src/runtime/asm_amd64.s:531

  21  Thread 0x7f95577fe700 (LWP 6814) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  20  Thread 0x7f9556ffd700 (LWP 6823) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  19  Thread 0x7f95567fc700 (LWP 6833) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  18  Thread 0x7f9555ffb700 (LWP 6883) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  17  Thread 0x7f95557fa700 (LWP 6884) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  16  Thread 0x7f9554ff9700 (LWP 6885) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  15  Thread 0x7f952bfff700 (LWP 9440) "dispatch" 0x0000000000419bb8in runtime.selectgoImpl (sel=0xc209419aa8, ~r1=833378949596, ~r2=32)

    at /root/go/src/runtime/select.go:289

  14  Thread 0x7f952b7fe700 (LWP 6985) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  13  Thread 0x7f952affd700 (LWP 27636) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  12  Thread 0x7f952a7fc700 (LWP 21103) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  11  Thread 0x7f9529ffb700 (LWP 21104) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  10  Thread 0x7f95297fa700 (LWP 21105) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  9   Thread 0x7f9528ff9700 (LWP 21106) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  8   Thread 0x7f9517fff700 (LWP 21219) "dispatch" runtime.duffzero() at /root/go/src/runtime/asm_amd64.s:1543

  7   Thread 0x7f95177fe700 (LWP 20300) "dispatch" runtime.futex () at/root/go/src/runtime/sys_linux_amd64.s:278

  6   Thread 0x7f9516ffd700 (LWP 20301) "dispatch" runtime.unlock(l=0xc2091d54f8) at /root/go/src/runtime/lock_futex.go:105

  5   Thread 0x7f95167fc700 (LWP 20302) "dispatch" runtime.epollwait() at /root/go/src/runtime/sys_linux_amd64.s:400

  4   Thread 0x7f9515ffb700 (LWP 20303) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  3   Thread 0x7f95157fa700 (LWP 20304) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

  2   Thread 0x7f9514ff9700 (LWP 20305) "dispatch" runtime.futex ()at /root/go/src/runtime/sys_linux_amd64.s:278

* 1    Thread 0x7f95688ff700 (LWP 6805)"dispatch" 0x00000000004ee614 in encoding/json.(*encodeState).string(e=0xc20805cc60, s=...,

    ~r1=833370089584, ~r2=...) at/usr/local/go/src/encoding/json/encode.go:791

(gdb)

发现20301线程的编号是6

使用thread命令切换线程

(gdb) thread6

[Switchingto thread 6 (Thread 0x7f9516ffd700 (LWP 20301))]

#0  runtime.unlock (l=0xc2091d54f8) at/root/go/src/runtime/lock_futex.go:105

105  /root/go/src/runtime/lock_futex.go: No suchfile or directory.

使用bt命令显示线程栈

(gdb) bt

#0  runtime.unlock (l=0xc2091d54f8) at/root/go/src/runtime/lock_futex.go:105

#1  0x000000000041956c in runtime.selunlock(sel=0xc20938daa8) at /root/go/src/runtime/select.go:166

#2  0x0000000000419df8 in runtime.selectgoImpl(sel=0xc20938daa8, ~r1=833378376156, ~r2=32) at/root/go/src/runtime/select.go:532

#3  0x0000000000419602 in runtime.selectgo(sel=0xc20938daa8) at /root/go/src/runtime/select.go:183

#4  0x00000000004c87ec in17ce.com/normalrize_dispatch.deal (task=..., flow_file=..., ~r2=96)

    at/home/jindonglan/17ce/go/src/17ce.com/normalrize_dispatch/NormalrizeDispatch.go:188

#5  0x00000000004c8a3f in17ce.com/normalrize_dispatch.dispatch (task=..., flow_file=...,res=0xc20942d7a0)

    at/home/jindonglan/17ce/go/src/17ce.com/normalrize_dispatch/NormalrizeDispatch.go:222

#6  0x0000000000440d91 in runtime.goexit () at/root/go/src/runtime/asm_amd64.s:2232

#7  0x000000c2086013d0 in ?? ()

#8  0x0000000000000004 in ?? ()

#9  0x000000000000018f in ?? ()

#100x0000000000000004 in ?? ()

#110x000000c2086013f0 in ?? ()

#120x000000000000000d in ?? ()

#130x000000c2087df518 in ?? ()

#140x0000000000000002 in ?? ()

#150x000000000000000c in ?? ()

#160x0000000000000000 in ?? ()

再去重点检查这几个地方就ok了