思科4506交换机通过show process cpu 显示CPU 占用98%
show proc cpu
CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 0 13 0 0.00% 0.00% 0.00% 0 Chunk Manager
2 552 21883 25 0.00% 0.00% 0.00% 0 Load Meter
3 160 131 1221 0.00% 0.00% 0.00% 0 SpanTree Helper
4 0 1 0 0.00% 0.00% 0.00% 0 Deferred Events
5 103924 14732 7054 0.00% 0.06% 0.05% 0 Check heaps
6 4 2 2000 0.00% 0.00% 0.00% 0 Pool Manager
7 0 2 0 0.00% 0.00% 0.00% 0 Timers
8 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun
9 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
10 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit
11 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager
12 16 1825 8 0.00% 0.00% 0.00% 0 IPC Dynamic Cach
13 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager
14 644 109135 5 0.00% 0.00% 0.00% 0 IPC Periodic Tim
15 412 109136 3 0.00% 0.00% 0.00% 0 IPC Deferred Por
16 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manager
17 0 1 0 0.00% 0.00% 0.00% 0 IFS Agent Manage
18 162484 514005 316 0.23% 0.21% 0.22% 0 ARP Input
19 68 11 6181 0.00% 0.00% 0.00% 0 Entity MIB API
20 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect
21 856 109136 7 0.00% 0.00% 0.00% 0 Dynamic ARP Insp
22 2060 27333 75 0.00% 0.00% 0.00% 0 HC Counter Timer
23 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
24 1716 91507 18 0.00% 0.00% 0.00% 0 Net Background
25 0 5 0 0.00% 0.00% 0.00% 0 Logger
26 716 109117 6 0.00% 0.00% 0.00% 0 TTY Background
27 2072 111013 18 0.00% 0.00% 0.00% 0 Per-Second Jobs
28 38292 1947 19667 0.00% 0.04% 0.00% 0 Per-minute Jobs
29 2858508 8807360 324 4.63% 3.81% 3.72% 0 Cat4k Mgmt HiPri
30 84371564 67467994 1250 91.69% 92.18% 92.23% 0 Cat4k Mgmt LoPri
31 260 49358 5 0.00% 0.00% 0.00% 0 Galios Reschedul
32 0 2 0 0.00% 0.00% 0.00% 0 IOS ACL Helper
33 0 2 0 0.00% 0.00% 0.00% 0 NAM Manager
34 152 2 76000 0.00% 0.00% 0.00% 0 rf task
35 12428 405853 30 0.00% 0.00% 0.00% 0 Net Input
36 11620 23022 504 0.07% 0.03% 0.00% 0 Compute load avg
37 0 84 0 0.00% 0.00% 0.00% 0 BACK CHECK
38 0 1 0 0.00% 0.00% 0.00% 0 chkpt message ha
39 0 2 0 0.00% 0.00% 0.00% 0 cpf_process_msg_
40 0 1 0 0.00% 0.00% 0.00% 0 cpf_process_ipcQ
41 0 14 0 0.00% 0.00% 0.00% 0 AggMgr Process
42 0 1 0 0.00% 0.00% 0.00% 0 SFF8472
43 0 15 0 0.00% 0.00% 0.00% 0 Collection proce
44 0 3 0 0.00% 0.00% 0.00% 0 CEF switching ba
45 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R
46 0 2 0 0.00% 0.00% 0.00% 0 AAA Server
47 0 1 0 0.00% 0.00% 0.00% 0 AAA ACCT Proc
48 0 1 0 0.00% 0.00% 0.00% 0 ACCT Periodic Pr
49 688824 1399707 492 0.71% 0.62% 0.62% 0 Spanning Tree
50 4396 55767 78 0.00% 0.01% 0.00% 0 DTP Protocol
51 40 10946 3 0.00% 0.00% 0.00% 0 Ethchnl
52 3744 109502 34 0.00% 0.01% 0.00% 0 UDLD
53 12 934 12 0.00% 0.00% 0.00% 0 DHCP Snooping
54 20 1825 10 0.00% 0.00% 0.00% 0 Port-Security
55 2521700 4927911 511 1.59% 1.75% 1.77% 0 IP Input
56 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl
57 91908 37910 2424 0.07% 0.14% 0.11% 0 CDP Protocol
58 16 221 72 0.00% 0.02% 0.00% 0 Exec
61 148 2534 58 0.00% 0.00% 0.00% 0 CEF background p
62 0 2 0 0.00% 0.00% 0.00% 0 XDR mcast
63 0 1 0 0.00% 0.00% 0.00% 0 IPC LC Message H
64 0 1 0 0.00% 0.00% 0.00% 0 XDR RP Ping Back
65 8 925 8 0.00% 0.00% 0.00% 0 XDR RP backgroun
66 0 1 0 0.00% 0.00% 0.00% 0 XDR RP Test Back
67 0 1 0 0.00% 0.00% 0.00% 0 IP IRDP
68 88064 159928 550 0.07% 0.07% 0.07% 0 CEF: IPv4 proces
69 184 36 5111 0.00% 0.00% 0.00% 0 ADJ background
70 48 367 130 0.00% 0.00% 0.00% 0 L2MM
71 120 1447 82 0.00% 0.00% 0.00% 0 MRD
72 1068 11946 89 0.00% 0.00% 0.00% 0 IGMPSN
73 0 1 0 0.00% 0.00% 0.00% 0 IGMPSN-HA
74 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers
75 0 2 0 0.00% 0.00% 0.00% 0 L2TRACE SERVER
76 288 16798 17 0.00% 0.00% 0.00% 0 TCP Timer
77 40 51 784 0.00% 0.00% 0.00% 0 TCP Protocols
78 1012 5641 179 0.00% 0.00% 0.00% 0 HTTP CORE
79 0 1 0 0.00% 0.00% 0.00% 0 CHKPT EXAMPLE
80 0 1 0 0.00% 0.00% 0.00% 0 CHKPT DevTest
81 0 2 0 0.00% 0.00% 0.00% 0 ATIP_UDP_TSK
82 0 1 0 0.00% 0.00% 0.00% 0 DHCP Snooping HA
83 0 1 0 0.00% 0.00% 0.00% 0 Probe Input
84 0 1 0 0.00% 0.00% 0.00% 0 RARP Input
85 5380 33808 159 0.00% 0.00% 0.00% 0 DHCPD Receive
86 1132 2079 544 0.00% 0.00% 0.00% 0 IP Background
87 300 1856 161 0.00% 0.00% 0.00% 0 IP RIB Update
88 0 1 0 0.00% 0.00% 0.00% 0 COPS
89 16 1371 11 0.00% 0.00% 0.00% 0 Cluster L2
90 76 10946 6 0.00% 0.00% 0.00% 0 Cluster RARP
91 0 2 0 0.00% 0.00% 0.00% 0 LOCAL AAA
92 0 2 0 0.00% 0.00% 0.00% 0 AAA Cached Serve
93 0 2 0 0.00% 0.00% 0.00% 0 TPLUS
94 0 3 0 0.00% 0.00% 0.00% 0 RADIUS TEST CMD
95 0 2 0 0.00% 0.00% 0.00% 0 AAA SEND STOP EV
96 11256 754707 14 0.15% 0.04% 0.01% 0 PM Callback
97 380 3039 125 0.00% 0.00% 0.00% 0 VLAN Manager
98 8 913 8 0.00% 0.00% 0.00% 0 DHCPD Timer
99 4 16 250 0.00% 0.00% 0.00% 0 VTP Trap Process
100 0 2 0 0.00% 0.00% 0.00% 0 DHCP Security He
101 0 1 0 0.00% 0.00% 0.00% 0 DiagCard1/-1
102 0 1 0 0.00% 0.00% 0.00% 0 DiagCard2/-1
103 0 1 0 0.00% 0.00% 0.00% 0 DiagCard3/-1
104 0 1 0 0.00% 0.00% 0.00% 0 DiagCard4/-1
105 412 476 865 0.00% 0.00% 0.00% 0 Syslog Traps
106 0 2 0 0.00% 0.00% 0.00% 0 VTPMIB EDIT BUFF
107 0 3 0 0.00% 0.00% 0.00% 0 SPAN switch
108 0 2 0 0.00% 0.00% 0.00% 0 SNMP Timers
109 0 2 0 0.00% 0.00% 0.00% 0 IP SNMP
110 0 1 0 0.00% 0.00% 0.00% 0 PDU DISPATCHER
111 0 1 0 0.00% 0.00% 0.00% 0 SNMP ENGINE
112 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
113 80 49 1632 0.00% 0.00% 0.00% 0 SNMP Traps
114 1344 111625 12 0.00% 0.00% 0.00% 0 NTP
115 324 30976 10 0.00% 0.00% 0.00% 0 DHCPD Database
116 356 54653 6 0.00% 0.00% 0.00% 0 System polling
CPU utilization for five seconds: 99%/0%; one minute: 99%; five minutes: 99%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 0 13 0 0.00% 0.00% 0.00% 0 Chunk Manager
2 552 21883 25 0.00% 0.00% 0.00% 0 Load Meter
3 160 131 1221 0.00% 0.00% 0.00% 0 SpanTree Helper
4 0 1 0 0.00% 0.00% 0.00% 0 Deferred Events
5 103924 14732 7054 0.00% 0.06% 0.05% 0 Check heaps
6 4 2 2000 0.00% 0.00% 0.00% 0 Pool Manager
7 0 2 0 0.00% 0.00% 0.00% 0 Timers
8 0 2 0 0.00% 0.00% 0.00% 0 Serial Backgroun
9 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
10 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit
11 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager
12 16 1825 8 0.00% 0.00% 0.00% 0 IPC Dynamic Cach
13 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager
14 644 109135 5 0.00% 0.00% 0.00% 0 IPC Periodic Tim
15 412 109136 3 0.00% 0.00% 0.00% 0 IPC Deferred Por
16 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat Manager
17 0 1 0 0.00% 0.00% 0.00% 0 IFS Agent Manage
18 162484 514005 316 0.23% 0.21% 0.22% 0 ARP Input
19 68 11 6181 0.00% 0.00% 0.00% 0 Entity MIB API
20 0 1 0 0.00% 0.00% 0.00% 0 SERIAL A'detect
21 856 109136 7 0.00% 0.00% 0.00% 0 Dynamic ARP Insp
22 2060 27333 75 0.00% 0.00% 0.00% 0 HC Counter Timer
23 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
24 1716 91507 18 0.00% 0.00% 0.00% 0 Net Background
25 0 5 0 0.00% 0.00% 0.00% 0 Logger
26 716 109117 6 0.00% 0.00% 0.00% 0 TTY Background
27 2072 111013 18 0.00% 0.00% 0.00% 0 Per-Second Jobs
28 38292 1947 19667 0.00% 0.04% 0.00% 0 Per-minute Jobs
29 2858508 8807360 324 4.63% 3.81% 3.72% 0 Cat4k Mgmt HiPri
30 84371564 67467994 1250 91.69% 92.18% 92.23% 0 Cat4k Mgmt LoPri
31 260 49358 5 0.00% 0.00% 0.00% 0 Galios Reschedul
32 0 2 0 0.00% 0.00% 0.00% 0 IOS ACL Helper
33 0 2 0 0.00% 0.00% 0.00% 0 NAM Manager
34 152 2 76000 0.00% 0.00% 0.00% 0 rf task
35 12428 405853 30 0.00% 0.00% 0.00% 0 Net Input
36 11620 23022 504 0.07% 0.03% 0.00% 0 Compute load avg
37 0 84 0 0.00% 0.00% 0.00% 0 BACK CHECK
38 0 1 0 0.00% 0.00% 0.00% 0 chkpt message ha
39 0 2 0 0.00% 0.00% 0.00% 0 cpf_process_msg_
40 0 1 0 0.00% 0.00% 0.00% 0 cpf_process_ipcQ
41 0 14 0 0.00% 0.00% 0.00% 0 AggMgr Process
42 0 1 0 0.00% 0.00% 0.00% 0 SFF8472
43 0 15 0 0.00% 0.00% 0.00% 0 Collection proce
44 0 3 0 0.00% 0.00% 0.00% 0 CEF switching ba
45 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R
46 0 2 0 0.00% 0.00% 0.00% 0 AAA Server
47 0 1 0 0.00% 0.00% 0.00% 0 AAA ACCT Proc
48 0 1 0 0.00% 0.00% 0.00% 0 ACCT Periodic Pr
49 688824 1399707 492 0.71% 0.62% 0.62% 0 Spanning Tree
50 4396 55767 78 0.00% 0.01% 0.00% 0 DTP Protocol
51 40 10946 3 0.00% 0.00% 0.00% 0 Ethchnl
52 3744 109502 34 0.00% 0.01% 0.00% 0 UDLD
53 12 934 12 0.00% 0.00% 0.00% 0 DHCP Snooping
54 20 1825 10 0.00% 0.00% 0.00% 0 Port-Security
55 2521700 4927911 511 1.59% 1.75% 1.77% 0 IP Input
56 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl
57 91908 37910 2424 0.07% 0.14% 0.11% 0 CDP Protocol
58 16 221 72 0.00% 0.02% 0.00% 0 Exec
61 148 2534 58 0.00% 0.00% 0.00% 0 CEF background p
62 0 2 0 0.00% 0.00% 0.00% 0 XDR mcast
63 0 1 0 0.00% 0.00% 0.00% 0 IPC LC Message H
64 0 1 0 0.00% 0.00% 0.00% 0 XDR RP Ping Back
65 8 925 8 0.00% 0.00% 0.00% 0 XDR RP backgroun
66 0 1 0 0.00% 0.00% 0.00% 0 XDR RP Test Back
67 0 1 0 0.00% 0.00% 0.00% 0 IP IRDP
68 88064 159928 550 0.07% 0.07% 0.07% 0 CEF: IPv4 proces
69 184 36 5111 0.00% 0.00% 0.00% 0 ADJ background
70 48 367 130 0.00% 0.00% 0.00% 0 L2MM
71 120 1447 82 0.00% 0.00% 0.00% 0 MRD
72 1068 11946 89 0.00% 0.00% 0.00% 0 IGMPSN
73 0 1 0 0.00% 0.00% 0.00% 0 IGMPSN-HA
74 0 1 0 0.00% 0.00% 0.00% 0 Socket Timers
75 0 2 0 0.00% 0.00% 0.00% 0 L2TRACE SERVER
76 288 16798 17 0.00% 0.00% 0.00% 0 TCP Timer
77 40 51 784 0.00% 0.00% 0.00% 0 TCP Protocols
78 1012 5641 179 0.00% 0.00% 0.00% 0 HTTP CORE
79 0 1 0 0.00% 0.00% 0.00% 0 CHKPT EXAMPLE
80 0 1 0 0.00% 0.00% 0.00% 0 CHKPT DevTest
81 0 2 0 0.00% 0.00% 0.00% 0 ATIP_UDP_TSK
82 0 1 0 0.00% 0.00% 0.00% 0 DHCP Snooping HA
83 0 1 0 0.00% 0.00% 0.00% 0 Probe Input
84 0 1 0 0.00% 0.00% 0.00% 0 RARP Input
85 5380 33808 159 0.00% 0.00% 0.00% 0 DHCPD Receive
86 1132 2079 544 0.00% 0.00% 0.00% 0 IP Background
87 300 1856 161 0.00% 0.00% 0.00% 0 IP RIB Update
88 0 1 0 0.00% 0.00% 0.00% 0 COPS
89 16 1371 11 0.00% 0.00% 0.00% 0 Cluster L2
90 76 10946 6 0.00% 0.00% 0.00% 0 Cluster RARP
91 0 2 0 0.00% 0.00% 0.00% 0 LOCAL AAA
92 0 2 0 0.00% 0.00% 0.00% 0 AAA Cached Serve
93 0 2 0 0.00% 0.00% 0.00% 0 TPLUS
94 0 3 0 0.00% 0.00% 0.00% 0 RADIUS TEST CMD
95 0 2 0 0.00% 0.00% 0.00% 0 AAA SEND STOP EV
96 11256 754707 14 0.15% 0.04% 0.01% 0 PM Callback
97 380 3039 125 0.00% 0.00% 0.00% 0 VLAN Manager
98 8 913 8 0.00% 0.00% 0.00% 0 DHCPD Timer
99 4 16 250 0.00% 0.00% 0.00% 0 VTP Trap Process
100 0 2 0 0.00% 0.00% 0.00% 0 DHCP Security He
101 0 1 0 0.00% 0.00% 0.00% 0 DiagCard1/-1
102 0 1 0 0.00% 0.00% 0.00% 0 DiagCard2/-1
103 0 1 0 0.00% 0.00% 0.00% 0 DiagCard3/-1
104 0 1 0 0.00% 0.00% 0.00% 0 DiagCard4/-1
105 412 476 865 0.00% 0.00% 0.00% 0 Syslog Traps
106 0 2 0 0.00% 0.00% 0.00% 0 VTPMIB EDIT BUFF
107 0 3 0 0.00% 0.00% 0.00% 0 SPAN switch
108 0 2 0 0.00% 0.00% 0.00% 0 SNMP Timers
109 0 2 0 0.00% 0.00% 0.00% 0 IP SNMP
110 0 1 0 0.00% 0.00% 0.00% 0 PDU DISPATCHER
111 0 1 0 0.00% 0.00% 0.00% 0 SNMP ENGINE
112 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
113 80 49 1632 0.00% 0.00% 0.00% 0 SNMP Traps
114 1344 111625 12 0.00% 0.00% 0.00% 0 NTP
115 324 30976 10 0.00% 0.00% 0.00% 0 DHCPD Database
116 356 54653 6 0.00% 0.00% 0.00% 0 System polling
通过交换机 show proc cpu的命令输出的信息可以看到:
在5秒、1分钟、5分钟内 Cat4k Mgmt LoPri进程的CPU占用率分别为 91.69% 92.18% 92.23%
显然是这个进程消耗了大量的CPU运行资源,从而导致交换机没有新的资源处理新的请求。
处理器运行两个进程,Cat4k Mgmt HiPri和Cat4k Mgmt LoPri,这两个进程同其他进程一样要占用cpu时间。当Cat4k平台上某项进程占用cpu超出了应分配的cpu时间,Cat4k Mgmt LoPri会接管这项进程,使其他进程能够得到cpu时间。同样当某个进程占用cpu时间没有超过规定的cpu利用率,Cat4k Mgmt HiPri会接管这个进程。
而Cat4k Mgmt LoPri进程CPU占用率超过90%,我们判断一定是某一个进程大大超过了CPU应分配的时间,而Cat4k Mgmt LoPri进程试图接管这一进程,而导致了Cat4k Mgmt LoPri 进程的CPU占用时间超过了90%。根据这一思路只要我们找出这个进程,将其关闭或找出这个进程的作用,就可以解决这个问题并能够找出CPU占用率过高的原因。
在5秒、1分钟、5分钟内 Cat4k Mgmt LoPri进程的CPU占用率分别为 91.69% 92.18% 92.23%
显然是这个进程消耗了大量的CPU运行资源,从而导致交换机没有新的资源处理新的请求。
处理器运行两个进程,Cat4k Mgmt HiPri和Cat4k Mgmt LoPri,这两个进程同其他进程一样要占用cpu时间。当Cat4k平台上某项进程占用cpu超出了应分配的cpu时间,Cat4k Mgmt LoPri会接管这项进程,使其他进程能够得到cpu时间。同样当某个进程占用cpu时间没有超过规定的cpu利用率,Cat4k Mgmt HiPri会接管这个进程。
而Cat4k Mgmt LoPri进程CPU占用率超过90%,我们判断一定是某一个进程大大超过了CPU应分配的时间,而Cat4k Mgmt LoPri进程试图接管这一进程,而导致了Cat4k Mgmt LoPri 进程的CPU占用时间超过了90%。根据这一思路只要我们找出这个进程,将其关闭或找出这个进程的作用,就可以解决这个问题并能够找出CPU占用率过高的原因。
通过show platform health 命令进一步更为详细的检查Catalyst 4506交换机的进程我们发现K2CpuMan Review进程有异常, 如下:
show pla hea
%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU
Lj-poll 1.00 0.01 2 144 100 500 0 0 0 0:26
GalChassisVp-review 3.00 0.17 10 28 100 500 0 0 0 2:50
S2w-JobEventSchedule 10.00 0.25 10 9 100 500 0 0 0 4:35
Stub-JobEventSchedul 10.00 2.36 10 34 100 500 2 2 1 35:16
StatValueMan Update 1.00 0.05 1 0 100 500 0 0 0 1:58
Pim-review 0.10 0.00 1 0 100 500 0 0 0 0:08
Ebm-host-review 1.00 0.31 8 4 100 500 0 0 0 0:27
Ebm-port-review 0.10 0.00 1 0 100 500 0 0 0 0:00
Protocol-aging-revie 0.20 0.00 2 0 100 500 0 0 0 0:00
Acl-Flattener 1.00 0.00 10 5 100 500 0 0 0 0:00
KxAclPathMan create/ 1.00 0.00 10 5 100 500 0 0 0 0:14
KxAclPathMan update 2.00 0.00 10 14 100 500 0 0 0 0:00
KxAclPathMan reprogr 1.00 0.00 2 1 100 500 0 0 0 0:00
TagMan-RecreateMtegR 1.00 0.00 10 5 100 500 0 0 0 0:00
K2CpuMan Review 30.00 69.84 30 26 100 500 112 106 78 1248:14
K2AccelPacketMan: Tx 10.00 2.74 20 0 100 500 12 12 10 254:45
K2AccelPacketMan: Au 0.10 0.00 0 0 100 500 0 0 0 0:00
K2AclMan-taggedFlatA 1.00 0.00 10 5 100 500 0 0 0 0:00
K2AclCamMan stale en 1.00 0.00 10 5 100 500 0 0 0 0:00
K2AclCamMan hw stats 3.00 0.50 10 5 100 500 0 0 0 7:24
K2AclCamMan kx stats 1.00 0.01 10 5 100 500 0 0 0 3:03
K2AclCamMan Audit re 1.00 7.76 10 5 100 500 5 0 0 4:07
K2AclPolicerTableMan 1.00 0.00 10 1 100 500 0 0 0 0:17
K2L2 Address Table R 2.00 3.25 12 5 100 500 0 0 0 7:25
K2L2 New Static Addr 2.00 0.00 10 5 100 500 0 0 0 0:00
K2L2 New Multicast A 2.00 0.00 10 5 100 500 0 0 0 0:00
K2L2 Dynamic Address 2.00 0.00 10 5 100 500 0 0 0 0:00
K2L2 Vlan Table Revi 2.00 0.00 12 8 100 500 0 0 0 0:02
K2 L2 Destination Ca 2.00 0.00 10 0 100 500 0 0 0 0:00
K2PortMan Review 2.00 1.53 15 11 100 500 2 1 1 26:42
Gigaport65535 Review 0.40 0.04 4 1 100 500 0 0 0 1:15
Gigaport65535 Review 0.40 0.07 4 1 100 500 0 0 0 1:15
Gigaport65535 Review 0.40 0.06 4 1 100 500 0 0 0 1:15
Gigaport65535 Review 0.40 0.05 4 1 100 500 0 0 0 1:17
Gigaport65535 Review 0.40 0.05 4 1 100 500 0 0 0 1:17
Gigaport65535 Review 0.40 0.06 4 1 100 500 0 0 0 1:16
Gigaport65535 Review 0.40 0.05 4 1 100 500 0 0 0 1:17
Gigaport65535 Review 0.40 0.05 4 1 100 500 0 0 0 1:17
Gigaport65535 Review 0.40 0.12 4 0 100 500 0 0 0 5:23
Gigaport65535 Review 0.40 0.09 4 0 100 500 0 0 0 5:20
Gigaport65535 Review 0.40 0.08 4 0 100 500 0 0 0 5:17
Gigaport65535 Review 0.40 0.09 4 0 100 500 0 0 0 5:19
Gigaport65535 Review 0.40 0.04 4 1 100 500 0 0 0 1:18
Gigaport65535 Review 0.40 0.08 4 1 100 500 0 0 0 1:17
Gigaport65535 Review 0.40 0.06 4 1 100 500 0 0 0 1:16
Gigaport65535 Review 0.40 0.06 4 1 100 500 0 0 0 1:16
Gigaport65535 Review 0.40 0.05 4 1 100 500 0 0 0 1:24
Gigaport65535 Review 0.40 0.05 4 1 100 500 0 0 0 1:22
Gigaport65535 Review 0.40 0.07 4 5 100 500 0 0 0 1:22
Gigaport65535 Review 0.40 0.07 4 1 100 500 0 0 0 1:22
Gigaport65535 Review 0.40 0.07 4 11 100 500 0 0 0 1:22
Gigaport65535 Review 0.40 0.06 4 1 100 500 0 0 0 1:23
Gigaport65535 Review 0.40 0.05 4 9 100 500 0 0 0 1:22
Gigaport65535 Review 0.40 0.05 4 1 100 500 0 0 0 1:21
Gigaport65535 Review 0.40 0.08 4 0 100 500 0 0 0 5:19
Gigaport65535 Review 0.40 0.11 4 0 100 500 0 0 0 5:22
Gigaport65535 Review 0.40 0.08 4 0 100 500 0 0 0 5:19
Gigaport65535 Review 0.40 0.10 4 0 100 500 0 0 0 5:19
Gigaport65535 Review 0.40 0.08 4 0 100 500 0 0 0 5:18
Gigaport65535 Review 0.40 0.09 4 0 100 500 0 0 0 5:18
Gigaport65535 Review 0.40 0.11 4 0 100 500 0 0 0 5:20
Gigaport65535 Review 0.40 0.09 4 0 100 500 0 0 0 5:19
K2Fib cam usage revi 2.00 0.00 15 0 100 500 0 0 0 0:00
K2Fib IrmFib Review 2.00 0.00 15 0 100 500 0 0 0 0:00
K2Fib Vrf Default Ro 2.00 0.00 15 0 100 500 0 0 0 0:00
K2Fib AdjRepop Revie 2.00 0.00 15 0 100 500 0 0 0 0:00
K2Fib Vrf Unpunt Rev 2.00 0.00 15 0 100 500 0 0 0 0:08
K2Fib Consistency Ch 1.00 12.04 5 2 100 500 2 1 1 27:08
K2FibAdjMan Stats Re 2.00 0.38 10 8 100 500 0 0 0 5:20
K2FibAdjMan Host Mov 2.00 0.00 10 7 100 500 0 0 0 0:14
K2FibAdjMan Adj Chan 2.00 0.00 10 0 100 500 0 0 0 0:00
K2FibMulticast Signa 2.00 0.03 10 2 100 500 0 0 0 0:46
K2FibMulticast Entry 2.00 0.00 10 6 100 500 0 0 0 0:00
K2FibMulticast Irm M 2.00 0.00 10 7 100 500 0 0 0 0:00
K2FibFastDropMan Rev 2.00 0.00 7 0 100 500 0 0 0 0:00
K2FibPbr route map r 2.00 0.41 20 5 100 500 0 0 0 4:02
K2FibPbr flat acl pr 2.00 0.08 20 1 100 500 0 0 0 0:45
K2FibPbr consolidati 2.00 0.01 10 0 100 500 0 0 0 0:12
K2FibPerVlanPuntMan 2.00 0.00 15 2 100 500 0 0 0 0:00
K2FibFlowCache flow 2.00 0.00 10 0 100 500 0 0 0 0:07
K2FibFlowCache flow 2.00 0.00 10 0 100 500 0 0 0 0:00
K2FibFlowCache adj r 2.00 0.00 10 0 100 500 0 0 0 0:07
K2FibFlowCache flow 2.00 0.00 10 0 100 500 0 0 0 0:02
K2MetStatsMan Review 2.00 0.15 5 2 100 500 0 0 0 6:51
K2FibMulticast MET S 2.00 0.00 10 0 100 500 0 0 0 0:00
K2QosDblMan Rate DBL 2.00 0.09 7 0 100 500 0 0 0 1:48
IrmFibThrottler Thro 2.00 0.07 7 3 100 500 0 0 0 0:43
K2 VlanStatsMan Revi 2.00 0.93 15 4 100 500 1 1 0 13:44
K2 Packet Memory Dia 2.00 0.31 15 8 100 500 2 1 0 16:06
K2 L2 Aging Table Re 2.00 0.07 20 3 100 500 0 0 0 2:42
RkiosPortMan Port Re 2.00 5.07 12 35 100 500 4 4 3 57:49
Rkios Module State R 4.00 0.02 40 1 100 500 0 0 0 0:30
Rkios Online Diag Re 4.00 0.01 40 0 100 500 0 0 0 0:26
RkiosIpPbr IrmPort R 2.00 0.01 10 1 100 500 0 0 0 0:40
RkiosAclMan Review 3.00 0.05 30 1 100 500 0 0 0 0:55
MatMan Review 0.50 0.00 4 0 100 500 0 0 0 0:00
Slot 2 ILC Manager R 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 2 ILC S2wMan Re 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 3 ILC Manager R 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 3 ILC S2wMan Re 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 4 ILC Manager R 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 4 ILC S2wMan Re 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 5 ILC Manager R 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 5 ILC S2wMan Re 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 6 ILC Manager R 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 6 ILC S2wMan Re 3.00 0.00 10 0 100 500 0 0 0 0:00
EthHoleLinecardMan(1 2.00 0.03 10 0 100 500 0 0 0 0:29
EthHoleLinecardMan(2 2.00 0.18 10 4 100 500 0 0 0 3:55
-------------
%CPU Totals 211.80 111.31
Allocation ceiling Current allocation
------------------ ------------------
kbytes % in use kbytes % in use
Linecard 1's Store 258.00 4% 12.04 100%
Linecard 2's Store 258.00 4% 12.72 100%
Linecard 3's Store 258.00 60% 155.24 100%
Linecard 4's Store 258.00 50% 131.53 100%
Linecard 5's Store 258.00 0% 0.00 0%
Linecard 6's Store 258.00 0% 0.00 0%
TSM objects ------------------ ------------------
RkiosSysPacketBuf 250.00 0% 0.56 0%
PacketBufRaw 20355.00 100% 20355.00 100%
PacketBufRawJumbo 732.81 25% 183.20 100%
Packet 1026.56 2% 30.17 99%
PacketInfoItem 390.62 0% 0.19 0%
VbufNodes2400 80.75 0% 0.00 0%
VbufNodes1600 55.75 0% 12.19 0%
VbufNodes400 73.00 8% 10.26 61%
VbufNodes64 62.00 0% 3.39 0%
GalGbicEntrys 11.34 0% 0.00 0%
PimPhyports 875.00 24% 215.25 100%
PimPorts 796.87 31% 247.03 100%
PimModules 150.00 1% 2.34 100%
PimSlots 5.00 2% 0.11 100%
PimChassis 33.50 6% 2.09 100%
EbmVlans 2688.00 1% 30.18 100%
EbmVlanGroupEntrys 1920.00 0% 4.80 100%
EbmPorts 184.00 30% 55.70 100%
EbmPortHostEntrys 384.00 0% 0.00 0%
EbmIeNodes 536.00 0% 4.18 100%
EbmPortVlanAclFeatur 896.00 0% 0.00 0%
EbmSortedHostTableIt 1.87 0% 0.00 0%
EbmSortedGroupTableI 1.75 0% 0.05 0%
IrmVrfs 6.85 1% 0.10 100%
IrmFibAdjs 768.00 1% 16.12 83%
IrmPortEtherAddrEntr 500.00 0% 0.00 0%
IrmFibEntries 10240.00 0% 39.76 86%
AclL4Op 384.00 0% 0.17 100%
AclL4OpTriplet 256.00 0% 0.15 100%
AclClassifier 768.00 0% 0.18 100%
AclFeature 2512.68 0% 0.42 100%
Acl 384.00 0% 0.10 100%
Ace 1280.00 0% 1.40 100%
AceActionDescStorage 256.00 0% 0.00 0%
AclListNode 256.00 0% 0.06 100%
AceListNode 25600.00 0% 0.55 61%
AclClassifierActionL 512.00 0% 0.09 100%
AclLayerFeatureListN 512.00 0% 0.06 50%
AclClassifierListNod 256.00 0% 0.00 0%
TableMapMan NameToTa 27.00 0% 0.00 0%
TableMapAllocator 59.00 0% 0.00 0%
FlatAcl 512.00 0% 0.09 33%
FlatAce 3840.00 0% 3.98 64%
FlatAceActionListNod 76800.00 0% 1.59 64%
FlatAclL4OpSetStorag 1024.00 0% 0.00 0%
FlatAclCacheNode 1024.00 0% 0.06 100%
QoS Policers 1672.00 0% 0.00 0%
KxAclPath 1024.00 0% 2.25 100%
KxAclPathListNode 256.00 0% 0.00 0%
ConfigToHwAfMap 418.78 0% 0.00 0%
HwToCfgAceMap 192.00 0% 0.00 0%
CommandTables 48.00 21% 10.26 100%
K2FibPbrFlatRouteMap 515.85 0% 0.00 0%
K2FibPbrExpandedFlat 2304.00 0% 0.00 0%
K2FibPbrFlatRouteMap 320.00 0% 0.00 0%
K2FibVrfs 38.59 1% 0.59 100%
K2 Netflow FPT Entri 11.97 0% 0.00 0%
K2TxPacket 384.00 0% 0.09 0%
K2TxPacketInfo 256.00 0% 0.17 0%
EbmVlanHostEntrys 2048.00 1% 35.50 92%
MatEntrys 7680.00 0% 8.90 100%
MatEntryTableIterato 1.12 0% 0.03 0%
RkiosAclMan NamedGal 43.75 0% 0.00 0%
Rkios Acl VlanMaps 272.00 0% 0.00 0%
Rkios Acl VlanMapEnt 1015.62 0% 0.00 0%
Rkios QoS PolicyMaps 1315.93 0% 0.00 0%
Rkios QoS ClassMaps 896.00 0% 0.02 100%
AclToIosFilterMapLis 384.00 0% 0.00 0%
Rkios QoS Policers 380.00 0% 0.00 0%
RkiosTableMap Galios 3.00 0% 0.00 0%
Event Nodes 84.00 0% 0.75 8%
Event Nodes 84.00 2% 17.71 13%
KxAclTagPairNode 2176.00 0% 0.13 100%
KxAclMappingTableEnt 96.02 0% 0.84 0%
KxAclMappingTableEnt 64.01 0% 0.00 0%
KxAclTaggedFlatAcl 1024.00 0% 0.06 100%
RkisoIpPbrRouteMaps 97.65 0% 0.00 0%
IrmFlows 256.00 0% 0.00 0%
------------------ ------------------
TSM totals 184649.73 11% 21299.07 99%
IOS软件设计的K2CpuMan Review进程CPU占用率最高为30% 而实际却为69.84,大大超过了软件当初设计得阀值,而其他的进程都在软件设计的设定的范围之内,因此我们可以初步判断是K2CpuMan Review 进程出现了意外,致使CPU的管理进程Cat4k Mgmt LoPri试图接管这一进程,而最终导致交换机CPU占用率过高,导致资源的最终耗尽。从而影响了整个网络运行的稳定性。
通过Cisco提供的文档来看,K2CpuMan Review进程主要处理“Hit”CPU的数据包,Catalyist 4506交换方式是CEF,并不是过程交换,在正常的情况下除设备管理、监测、路由计算外,数据包的转发主要由交换引擎来完成,并不需要CPU作过多的处理。K2CpuMan Review占用的CPU资源过多,但各接口的流量并没有明显增大,显然不是正常的负载过重导致,一定是很多异常的数据包Hit了CPU,为了查清楚 CPU受到的异常数据包性质。我们通过命令:
Monitor session 1 soure interface cpu
Monitor session 1 des interface gi 3/8?
把CPU收到的数据包镜像到了gi 3/8端口,然后在gi 3/8端口接上天元龙马的IDS和装有Sniffer和IRIS等数据分析软件的计算机。
此时天元龙马的IDS非常精确的监测到大量的SQL蠕虫病毒***和大量Syn flood***,(详见天元龙马公司IDS检测报告),此时我们已经可以初步判定是网内SQL蠕虫病毒的爆发最终导致集团公司Catalyst 4506交换机CPU资源的耗尽。(由于流量过大,sniffer和IRIS等软件死机)
为了验证我们的判断,我们把SQL蠕虫病毒***的 TCP 1433端口加入到了访问控制列表里面,在访问控制列表应用到接口之后,CPU的利用率立刻下降到15%左右,最后稳定在13%--18%, K2CpuMan Review 进程的CPU占用率也下降到10.4%,远低于IOS软件设定30%的目标,这进一步验证了我们的判断。同时根据天元龙马公司的提供的***源地址,我们找到了其中一台,并发现了该计算机已中蠕虫病毒,正在向外发送大量的***数据包。至此我们可以断定:是网内部分机器中SQL蠕虫病毒,导致了这次交换机工作的异常。
应用访问控制列表后的运行状态:
%CPU %CPU RunTimeMax Priority Average %CPU Total
Target Actual Target Actual Fg Bg 5Sec Min Hour CPU
Lj-poll 1.00 0.01 2 144 100 500 0 0 0 0:26
GalChassisVp-review 3.00 0.17 10 28 100 500 0 0 0 2:50
S2w-JobEventSchedule 10.00 0.25 10 9 100 500 0 0 0 4:35
Stub-JobEventSchedul 10.00 2.36 10 34 100 500 2 2 1 35:16
StatValueMan Update 1.00 0.05 1 0 100 500 0 0 0 1:58
Pim-review 0.10 0.00 1 0 100 500 0 0 0 0:08
Ebm-host-review 1.00 0.31 8 4 100 500 0 0 0 0:27
Ebm-port-review 0.10 0.00 1 0 100 500 0 0 0 0:00
Protocol-aging-revie 0.20 0.00 2 0 100 500 0 0 0 0:00
Acl-Flattener 1.00 0.00 10 5 100 500 0 0 0 0:00
KxAclPathMan create/ 1.00 0.00 10 5 100 500 0 0 0 0:14
KxAclPathMan update 2.00 0.00 10 14 100 500 0 0 0 0:00
KxAclPathMan reprogr 1.00 0.00 2 1 100 500 0 0 0 0:00
TagMan-RecreateMtegR 1.00 0.00 10 5 100 500 0 0 0 0:00
K2CpuMan Review 30.00 69.84 30 26 100 500 112 106 78 1248:14
K2AccelPacketMan: Tx 10.00 2.74 20 0 100 500 12 12 10 254:45
K2AccelPacketMan: Au 0.10 0.00 0 0 100 500 0 0 0 0:00
K2AclMan-taggedFlatA 1.00 0.00 10 5 100 500 0 0 0 0:00
K2AclCamMan stale en 1.00 0.00 10 5 100 500 0 0 0 0:00
K2AclCamMan hw stats 3.00 0.50 10 5 100 500 0 0 0 7:24
K2AclCamMan kx stats 1.00 0.01 10 5 100 500 0 0 0 3:03
K2AclCamMan Audit re 1.00 7.76 10 5 100 500 5 0 0 4:07
K2AclPolicerTableMan 1.00 0.00 10 1 100 500 0 0 0 0:17
K2L2 Address Table R 2.00 3.25 12 5 100 500 0 0 0 7:25
K2L2 New Static Addr 2.00 0.00 10 5 100 500 0 0 0 0:00
K2L2 New Multicast A 2.00 0.00 10 5 100 500 0 0 0 0:00
K2L2 Dynamic Address 2.00 0.00 10 5 100 500 0 0 0 0:00
K2L2 Vlan Table Revi 2.00 0.00 12 8 100 500 0 0 0 0:02
K2 L2 Destination Ca 2.00 0.00 10 0 100 500 0 0 0 0:00
K2PortMan Review 2.00 1.53 15 11 100 500 2 1 1 26:42
Gigaport65535 Review 0.40 0.04 4 1 100 500 0 0 0 1:15
Gigaport65535 Review 0.40 0.07 4 1 100 500 0 0 0 1:15
Gigaport65535 Review 0.40 0.06 4 1 100 500 0 0 0 1:15
Gigaport65535 Review 0.40 0.05 4 1 100 500 0 0 0 1:17
Gigaport65535 Review 0.40 0.05 4 1 100 500 0 0 0 1:17
Gigaport65535 Review 0.40 0.06 4 1 100 500 0 0 0 1:16
Gigaport65535 Review 0.40 0.05 4 1 100 500 0 0 0 1:17
Gigaport65535 Review 0.40 0.05 4 1 100 500 0 0 0 1:17
Gigaport65535 Review 0.40 0.12 4 0 100 500 0 0 0 5:23
Gigaport65535 Review 0.40 0.09 4 0 100 500 0 0 0 5:20
Gigaport65535 Review 0.40 0.08 4 0 100 500 0 0 0 5:17
Gigaport65535 Review 0.40 0.09 4 0 100 500 0 0 0 5:19
Gigaport65535 Review 0.40 0.04 4 1 100 500 0 0 0 1:18
Gigaport65535 Review 0.40 0.08 4 1 100 500 0 0 0 1:17
Gigaport65535 Review 0.40 0.06 4 1 100 500 0 0 0 1:16
Gigaport65535 Review 0.40 0.06 4 1 100 500 0 0 0 1:16
Gigaport65535 Review 0.40 0.05 4 1 100 500 0 0 0 1:24
Gigaport65535 Review 0.40 0.05 4 1 100 500 0 0 0 1:22
Gigaport65535 Review 0.40 0.07 4 5 100 500 0 0 0 1:22
Gigaport65535 Review 0.40 0.07 4 1 100 500 0 0 0 1:22
Gigaport65535 Review 0.40 0.07 4 11 100 500 0 0 0 1:22
Gigaport65535 Review 0.40 0.06 4 1 100 500 0 0 0 1:23
Gigaport65535 Review 0.40 0.05 4 9 100 500 0 0 0 1:22
Gigaport65535 Review 0.40 0.05 4 1 100 500 0 0 0 1:21
Gigaport65535 Review 0.40 0.08 4 0 100 500 0 0 0 5:19
Gigaport65535 Review 0.40 0.11 4 0 100 500 0 0 0 5:22
Gigaport65535 Review 0.40 0.08 4 0 100 500 0 0 0 5:19
Gigaport65535 Review 0.40 0.10 4 0 100 500 0 0 0 5:19
Gigaport65535 Review 0.40 0.08 4 0 100 500 0 0 0 5:18
Gigaport65535 Review 0.40 0.09 4 0 100 500 0 0 0 5:18
Gigaport65535 Review 0.40 0.11 4 0 100 500 0 0 0 5:20
Gigaport65535 Review 0.40 0.09 4 0 100 500 0 0 0 5:19
K2Fib cam usage revi 2.00 0.00 15 0 100 500 0 0 0 0:00
K2Fib IrmFib Review 2.00 0.00 15 0 100 500 0 0 0 0:00
K2Fib Vrf Default Ro 2.00 0.00 15 0 100 500 0 0 0 0:00
K2Fib AdjRepop Revie 2.00 0.00 15 0 100 500 0 0 0 0:00
K2Fib Vrf Unpunt Rev 2.00 0.00 15 0 100 500 0 0 0 0:08
K2Fib Consistency Ch 1.00 12.04 5 2 100 500 2 1 1 27:08
K2FibAdjMan Stats Re 2.00 0.38 10 8 100 500 0 0 0 5:20
K2FibAdjMan Host Mov 2.00 0.00 10 7 100 500 0 0 0 0:14
K2FibAdjMan Adj Chan 2.00 0.00 10 0 100 500 0 0 0 0:00
K2FibMulticast Signa 2.00 0.03 10 2 100 500 0 0 0 0:46
K2FibMulticast Entry 2.00 0.00 10 6 100 500 0 0 0 0:00
K2FibMulticast Irm M 2.00 0.00 10 7 100 500 0 0 0 0:00
K2FibFastDropMan Rev 2.00 0.00 7 0 100 500 0 0 0 0:00
K2FibPbr route map r 2.00 0.41 20 5 100 500 0 0 0 4:02
K2FibPbr flat acl pr 2.00 0.08 20 1 100 500 0 0 0 0:45
K2FibPbr consolidati 2.00 0.01 10 0 100 500 0 0 0 0:12
K2FibPerVlanPuntMan 2.00 0.00 15 2 100 500 0 0 0 0:00
K2FibFlowCache flow 2.00 0.00 10 0 100 500 0 0 0 0:07
K2FibFlowCache flow 2.00 0.00 10 0 100 500 0 0 0 0:00
K2FibFlowCache adj r 2.00 0.00 10 0 100 500 0 0 0 0:07
K2FibFlowCache flow 2.00 0.00 10 0 100 500 0 0 0 0:02
K2MetStatsMan Review 2.00 0.15 5 2 100 500 0 0 0 6:51
K2FibMulticast MET S 2.00 0.00 10 0 100 500 0 0 0 0:00
K2QosDblMan Rate DBL 2.00 0.09 7 0 100 500 0 0 0 1:48
IrmFibThrottler Thro 2.00 0.07 7 3 100 500 0 0 0 0:43
K2 VlanStatsMan Revi 2.00 0.93 15 4 100 500 1 1 0 13:44
K2 Packet Memory Dia 2.00 0.31 15 8 100 500 2 1 0 16:06
K2 L2 Aging Table Re 2.00 0.07 20 3 100 500 0 0 0 2:42
RkiosPortMan Port Re 2.00 5.07 12 35 100 500 4 4 3 57:49
Rkios Module State R 4.00 0.02 40 1 100 500 0 0 0 0:30
Rkios Online Diag Re 4.00 0.01 40 0 100 500 0 0 0 0:26
RkiosIpPbr IrmPort R 2.00 0.01 10 1 100 500 0 0 0 0:40
RkiosAclMan Review 3.00 0.05 30 1 100 500 0 0 0 0:55
MatMan Review 0.50 0.00 4 0 100 500 0 0 0 0:00
Slot 2 ILC Manager R 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 2 ILC S2wMan Re 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 3 ILC Manager R 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 3 ILC S2wMan Re 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 4 ILC Manager R 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 4 ILC S2wMan Re 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 5 ILC Manager R 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 5 ILC S2wMan Re 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 6 ILC Manager R 3.00 0.00 10 0 100 500 0 0 0 0:00
Slot 6 ILC S2wMan Re 3.00 0.00 10 0 100 500 0 0 0 0:00
EthHoleLinecardMan(1 2.00 0.03 10 0 100 500 0 0 0 0:29
EthHoleLinecardMan(2 2.00 0.18 10 4 100 500 0 0 0 3:55
-------------
%CPU Totals 211.80 111.31
Allocation ceiling Current allocation
------------------ ------------------
kbytes % in use kbytes % in use
Linecard 1's Store 258.00 4% 12.04 100%
Linecard 2's Store 258.00 4% 12.72 100%
Linecard 3's Store 258.00 60% 155.24 100%
Linecard 4's Store 258.00 50% 131.53 100%
Linecard 5's Store 258.00 0% 0.00 0%
Linecard 6's Store 258.00 0% 0.00 0%
TSM objects ------------------ ------------------
RkiosSysPacketBuf 250.00 0% 0.56 0%
PacketBufRaw 20355.00 100% 20355.00 100%
PacketBufRawJumbo 732.81 25% 183.20 100%
Packet 1026.56 2% 30.17 99%
PacketInfoItem 390.62 0% 0.19 0%
VbufNodes2400 80.75 0% 0.00 0%
VbufNodes1600 55.75 0% 12.19 0%
VbufNodes400 73.00 8% 10.26 61%
VbufNodes64 62.00 0% 3.39 0%
GalGbicEntrys 11.34 0% 0.00 0%
PimPhyports 875.00 24% 215.25 100%
PimPorts 796.87 31% 247.03 100%
PimModules 150.00 1% 2.34 100%
PimSlots 5.00 2% 0.11 100%
PimChassis 33.50 6% 2.09 100%
EbmVlans 2688.00 1% 30.18 100%
EbmVlanGroupEntrys 1920.00 0% 4.80 100%
EbmPorts 184.00 30% 55.70 100%
EbmPortHostEntrys 384.00 0% 0.00 0%
EbmIeNodes 536.00 0% 4.18 100%
EbmPortVlanAclFeatur 896.00 0% 0.00 0%
EbmSortedHostTableIt 1.87 0% 0.00 0%
EbmSortedGroupTableI 1.75 0% 0.05 0%
IrmVrfs 6.85 1% 0.10 100%
IrmFibAdjs 768.00 1% 16.12 83%
IrmPortEtherAddrEntr 500.00 0% 0.00 0%
IrmFibEntries 10240.00 0% 39.76 86%
AclL4Op 384.00 0% 0.17 100%
AclL4OpTriplet 256.00 0% 0.15 100%
AclClassifier 768.00 0% 0.18 100%
AclFeature 2512.68 0% 0.42 100%
Acl 384.00 0% 0.10 100%
Ace 1280.00 0% 1.40 100%
AceActionDescStorage 256.00 0% 0.00 0%
AclListNode 256.00 0% 0.06 100%
AceListNode 25600.00 0% 0.55 61%
AclClassifierActionL 512.00 0% 0.09 100%
AclLayerFeatureListN 512.00 0% 0.06 50%
AclClassifierListNod 256.00 0% 0.00 0%
TableMapMan NameToTa 27.00 0% 0.00 0%
TableMapAllocator 59.00 0% 0.00 0%
FlatAcl 512.00 0% 0.09 33%
FlatAce 3840.00 0% 3.98 64%
FlatAceActionListNod 76800.00 0% 1.59 64%
FlatAclL4OpSetStorag 1024.00 0% 0.00 0%
FlatAclCacheNode 1024.00 0% 0.06 100%
QoS Policers 1672.00 0% 0.00 0%
KxAclPath 1024.00 0% 2.25 100%
KxAclPathListNode 256.00 0% 0.00 0%
ConfigToHwAfMap 418.78 0% 0.00 0%
HwToCfgAceMap 192.00 0% 0.00 0%
CommandTables 48.00 21% 10.26 100%
K2FibPbrFlatRouteMap 515.85 0% 0.00 0%
K2FibPbrExpandedFlat 2304.00 0% 0.00 0%
K2FibPbrFlatRouteMap 320.00 0% 0.00 0%
K2FibVrfs 38.59 1% 0.59 100%
K2 Netflow FPT Entri 11.97 0% 0.00 0%
K2TxPacket 384.00 0% 0.09 0%
K2TxPacketInfo 256.00 0% 0.17 0%
EbmVlanHostEntrys 2048.00 1% 35.50 92%
MatEntrys 7680.00 0% 8.90 100%
MatEntryTableIterato 1.12 0% 0.03 0%
RkiosAclMan NamedGal 43.75 0% 0.00 0%
Rkios Acl VlanMaps 272.00 0% 0.00 0%
Rkios Acl VlanMapEnt 1015.62 0% 0.00 0%
Rkios QoS PolicyMaps 1315.93 0% 0.00 0%
Rkios QoS ClassMaps 896.00 0% 0.02 100%
AclToIosFilterMapLis 384.00 0% 0.00 0%
Rkios QoS Policers 380.00 0% 0.00 0%
RkiosTableMap Galios 3.00 0% 0.00 0%
Event Nodes 84.00 0% 0.75 8%
Event Nodes 84.00 2% 17.71 13%
KxAclTagPairNode 2176.00 0% 0.13 100%
KxAclMappingTableEnt 96.02 0% 0.84 0%
KxAclMappingTableEnt 64.01 0% 0.00 0%
KxAclTaggedFlatAcl 1024.00 0% 0.06 100%
RkisoIpPbrRouteMaps 97.65 0% 0.00 0%
IrmFlows 256.00 0% 0.00 0%
------------------ ------------------
TSM totals 184649.73 11% 21299.07 99%
IOS软件设计的K2CpuMan Review进程CPU占用率最高为30% 而实际却为69.84,大大超过了软件当初设计得阀值,而其他的进程都在软件设计的设定的范围之内,因此我们可以初步判断是K2CpuMan Review 进程出现了意外,致使CPU的管理进程Cat4k Mgmt LoPri试图接管这一进程,而最终导致交换机CPU占用率过高,导致资源的最终耗尽。从而影响了整个网络运行的稳定性。
通过Cisco提供的文档来看,K2CpuMan Review进程主要处理“Hit”CPU的数据包,Catalyist 4506交换方式是CEF,并不是过程交换,在正常的情况下除设备管理、监测、路由计算外,数据包的转发主要由交换引擎来完成,并不需要CPU作过多的处理。K2CpuMan Review占用的CPU资源过多,但各接口的流量并没有明显增大,显然不是正常的负载过重导致,一定是很多异常的数据包Hit了CPU,为了查清楚 CPU受到的异常数据包性质。我们通过命令:
Monitor session 1 soure interface cpu
Monitor session 1 des interface gi 3/8?
把CPU收到的数据包镜像到了gi 3/8端口,然后在gi 3/8端口接上天元龙马的IDS和装有Sniffer和IRIS等数据分析软件的计算机。
此时天元龙马的IDS非常精确的监测到大量的SQL蠕虫病毒***和大量Syn flood***,(详见天元龙马公司IDS检测报告),此时我们已经可以初步判定是网内SQL蠕虫病毒的爆发最终导致集团公司Catalyst 4506交换机CPU资源的耗尽。(由于流量过大,sniffer和IRIS等软件死机)
为了验证我们的判断,我们把SQL蠕虫病毒***的 TCP 1433端口加入到了访问控制列表里面,在访问控制列表应用到接口之后,CPU的利用率立刻下降到15%左右,最后稳定在13%--18%, K2CpuMan Review 进程的CPU占用率也下降到10.4%,远低于IOS软件设定30%的目标,这进一步验证了我们的判断。同时根据天元龙马公司的提供的***源地址,我们找到了其中一台,并发现了该计算机已中蠕虫病毒,正在向外发送大量的***数据包。至此我们可以断定:是网内部分机器中SQL蠕虫病毒,导致了这次交换机工作的异常。
应用访问控制列表后的运行状态: