ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT
os: centos 7.6
db: oracle 19.3
oracle rac 环境,其中一台实例报错如下.
alert 日志报错
$ tail -f /u01/app/oracle/diag/rdbms/orcls1/orcls12/trace/alert_orcls12.log
2019-06-25T08:24:42.755760+08:00
Errors in file /u01/app/oracle/diag/rdbms/orcls1/orcls12/trace/orcls12_ppa6_8473.trc (incident=89577) (PDBNAME=CDB$ROOT):
ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT
Errors in file /u01/app/oracle/diag/rdbms/orcls1/orcls12/trace/orcls12_ppa6_8473.trc (incident=89578) (PDBNAME=CDB$ROOT):
ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT
Errors in file /u01/app/oracle/diag/rdbms/orcls1/orcls12/trace/orcls12_ppa6_8473.trc (incident=89579) (PDBNAME=CDB$ROOT):
ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT
查看 pga 相关参数值
SQL> show parameter pga;
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
pga_aggregate_limit big integer 10G
pga_aggregate_target big integer 5G
pga_aggregate_limit 设置的已经够大了,而且是一个空库,没有任何负载.
查看 os 层面内存使用最大的几个进程
# ps aux|head -1;ps aux|grep -v PID|sort -rn -k +4|head -10
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
oracle 8285 1.2 17.6 43819240 11619632 ? Ss Jun13 216:34 ora_dia0_orcls12
root 11470 2.0 1.8 18679252 1189572 ? Sl Jun12 378:01 /u01/app/grid/product/19.0.0/grid_1/jdk/jre/bin/java -server -Xms128m -Xmx256m -Djava.awt.headless=true -Ddisable.checkForUpdate=true -XX:ParallelGCThreads=5 oracle.rat.tfa.TFAMain /u01/app/grid/product/19.0.0/grid_1/tfa/paynodea1/tfa_home
oracle 8660 0.0 0.5 32469668 334224 ? Ss Jun13 1:43 ora_p001_orcls12
grid 5920 0.5 0.5 5205416 354596 ? Sl Jun13 93:44 /u01/app/grid/product/19.0.0/grid_1/jdk/bin/java -server -Xms30M -Xmx512M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -DTRACING.ENABLED=false -XX:ParallelGCThreads=1 -cp /u01/app/grid/product/19.0.0/grid_1/jlib/cha.jar:/u01/app/grid/product/19.0.0/grid_1/jlib/chaconfig.jar:/u01/app/grid/product/19.0.0/grid_1/jlib/cha-diag-msg.jar:/u01/app/grid/product/19.0.0/grid_1/jlib/clsce.jar:/u01/app/grid/product/19.0.0/grid_1/jlib/srvm.jar:/u01/app/grid/product/19.0.0/grid_1/jlib/srvmhas.jar:/u01/app/grid/product/19.0.0/grid_1/jlib/srvmasm.jar:/u01/app/grid/product/19.0.0/grid_1/jlib/netcfg.jar:/u01/app/grid/product/19.0.0/grid_1/jdbc/lib/ojdbc8.jar:/u01/app/grid/product/19.0.0/grid_1/ucp/lib/ucp.jar:/u01/app/grid/product/19.0.0/grid_1/jlib/fte.jar:/u01/app/grid/product/19.0.0/grid_1/jlib/jai_core.jar:/u01/app/grid/product/19.0.0/grid_1/jlib/mlibwrapper_jai.jar:/u01/app/grid/product/19.0.0/grid_1/jlib/vecmath.jar:/u01/app/grid/product/19.0.0/grid_1/jlib/jai_codec.jar:/u01/app/grid/product/19.0.0/grid_1/jlib/jh.jar oracle.cha.server.CHADDriver
root 6497 139 0.3 5798204 238808 ? Rl Jun13 23562:22 /usr/bin/gnome-shell
root 11950 3.8 0.3 2481528 226064 ? Sl Jun13 644:51 /usr/lib64/firefox/firefox
grid 5359 0.5 0.3 2673844 230384 ? SLsl Jun13 95:04 /u01/app/grid/product/19.0.0/grid_1/bin/ocssd.bin
root 5342 0.1 0.2 1178080 151780 ? SLsl Jun13 24:13 /u01/app/grid/product/19.0.0/grid_1/bin/cssdagent
root 5309 1.5 0.2 1591256 177844 ? SLsl Jun13 254:39 /u01/app/grid/product/19.0.0/grid_1/bin/osysmond.bin
root 5307 0.1 0.2 1174704 149580 ? SLsl Jun13 23:59 /u01/app/grid/product/19.0.0/grid_1/bin/cssdmonitor
ora_dia0_orcls12 看起来占了 17.6% ,有点偏大.
查看 error 文件 orcls12_ppa6_8473.trc 如下,
$ more /u01/app/oracle/diag/rdbms/orcls1/orcls12/trace/orcls12_ppa6_8473.trc
Trace file /u01/app/oracle/diag/rdbms/orcls1/orcls12/trace/orcls12_ppa6_8473.trc
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0
Build label: RDBMS_19.3.0.0.0DBRU_LINUX.X64_190417
ORACLE_HOME: /u01/app/oracle/product/19.0.0/dbhome_1
System name: Linux
Node name: nodea1
Release: 3.10.0-957.5.1.el7.x86_64
Version: #1 SMP Fri Feb 1 14:54:57 UTC 2019
Machine: x86_64
Instance name: orcls12
Redo thread mounted by this instance: 1
Oracle process number: 191
Unix process pid: 8473, image: oracle@nodea1 (PPA6)
*** 2019-06-25T08:24:42.632795+08:00 (CDB$ROOT(1))
*** SESSION ID:(2949.60469) 2019-06-25T08:24:42.632870+08:00
*** CLIENT ID:() 2019-06-25T08:24:42.632882+08:00
*** SERVICE NAME:(SYS$BACKGROUND) 2019-06-25T08:24:42.632890+08:00
*** MODULE NAME:(MMON_SLAVE) 2019-06-25T08:24:42.632898+08:00
*** ACTION NAME:(Autotask Slave Action) 2019-06-25T08:24:42.632906+08:00
*** CLIENT DRIVER:() 2019-06-25T08:24:42.632914+08:00
*** CONTAINER ID:(1) 2019-06-25T08:24:42.632922+08:00
Process may have gone over pga_aggregate_limit
Just allocated 65536 bytes
Dumping short stack in preparation for potential ORA-4036
----- Abridged Call Stack Trace -----
ksedsts()+426<-ksm_pga_limit_short_stack()+1087<-ksm_check_over_limit()+697<-ksmarfg()+330<-kgh_invoke_alloc_cb()+162<-kghgex()+2713<-kghfnd()+386<-k
ghalo()+4853<-kghgex()+593<-kghfnd()+386<-kghalo()+4853<-kghgex()+593<-kghalf()+554<-kghalp()+798<-qcuAllocIdn()+41
<-qcucidn()+36<-qcpiid3()+98<-qcpicnm()+374<-qcpiapr()+1057<-qcpiafa()+97
----- End of Abridged Call Stack Trace -----
Partial short call stack signature: 0xaaeb7af68380038b
=======================================
PRIVATE MEMORY SUMMARY FOR THIS PROCESS
---------------------------------------
******************************************************
PRIVATE HEAP SUMMARY DUMP
5844 KB total:
5218 KB commented, 365 KB permanent
261 KB free (0 KB in empty extents),
2570 KB, 2 heaps: "callheap " 24 KB free held
820 KB, 1 heap: "kxs-heap-c " 216 KB free held
712 KB, 1 heap: "diag pga "
685 KB, 169 chunks: "ksipc pga chnk " 10 KB free held
222 KB, 1 heap: "Fixed UGA heap " 1 KB free held
------------------------------------------------------
Summary of subheaps at depth 1
4346 KB total:
3647 KB commented, 384 KB permanent
315 KB free (0 KB in empty extents),
2527 KB, 1 heap: "TCHK^6d9b8a7e " 16 KB free held
577 KB, 1 chunk: "dbgraim: pdb stage handle "
221 KB, 1 chunk: "Fixed Uga "
------------------------------------------------------
Summary of subheaps at depth 2
2619 KB total:
2602 KB commented, 10 KB permanent
7 KB free (0 KB in empty extents),
276 KB, 6596 chunks: "idndef : qcuAllocIdn "
256 KB, 1486 chunks: "coldef: qcopCreateCol "
242 KB, 6149 chunks: "chedef : qcuatc "
193 KB, 1643 chunks: "strdef: qcopCreateStr "
154 KB, 1117 chunks: "optdef: qcopCreateOptInter"
114 KB, 431 chunks: "kccdef : qcsvwsci "
114 KB, 429 chunks: "kccdef : qcdlgcd "
100 KB, 412 chunks: "qkshtqb : qkshtQBAtomicAll"
99 KB, 17 chunks: "297.kggms "
90 KB, 146 chunks: "ckydef : kkdlcky "
71 KB, 1140 chunks: "seldef: qcopCreateSel "
63 KB, 98 chunks: "frodef:qcpitnm "
58 KB, 1054 chunks: "jsoncol:qcpistoreparsetext"
55 KB, 1054 chunks: "prstxt:qcpiStoreParseText "
46 KB, 226 chunks: "qkshtfr : qkshtTabAtomicAl"
40 KB, 965 chunks: "num_buf : prsstr "
39 KB, 449 chunks: "qkshtAtomic : qkshtAtomicC"
35 KB, 46 chunks: "qbcdef:qcpiqbk "
34 KB, 1 heap: "7107.kgght " 1 KB free held
34 KB, 368 chunks: "idndef : qcpiexl "
30 KB, 15 chunks: "kkmevw: view_txt "
27 KB, 11 chunks: "qcpifqtqc : qcsidn "
27 KB, 377 chunks: "qkshtHReport:qkshtAtomHRep"
21 KB, 239 chunks: "oratext : qkshtHRepSetText"
18 KB, 453 chunks: "strdef_buf : prsstr "
17 KB, 146 chunks: "kkeis : kkdlcky "
17 KB, 113 chunks: "lazdef: qcopCreateLaz "
16 KB, 1 chunk: "alloc server hndl "
16 KB, 2 chunks: "ttc conv buf alloc "
15 KB, 189 chunks: "logdef: qcopCreateLog "
15 KB, 62 chunks: "kksgaAlloc elem "
13 KB, 58 chunks: "kkets:kkdlgtd "
signalling ORA-4036 interrupt
DDE: Problem Key 'ORA 4036' was flood controlled (0x6) (incident: 89577)
ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT
DDE: Problem Key 'ORA 4036' was flood controlled (0x6) (incident: 89578)
ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT
DDE: Problem Key 'ORA 4036' was flood controlled (0x6) (incident: 89579)
ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT
分析
SQL> set lines 200;
set pages 200;
col name format a50;
col value format 999999999999999;
col unit format a10;
col con_id format 99;
select * from v$pgastat;
NAME VALUE UNIT CON_ID
-------------------------------------------------- -------------------- ---------- ------
aggregate PGA target parameter 5,368,709,120 bytes 0
aggregate PGA auto target 335,544,320 bytes 0
global memory bound 536,862,720 bytes 0
total PGA inuse 13,589,449,728 bytes 0
total PGA allocated 14,577,980,416 bytes 0
maximum PGA allocated 14,589,383,680 bytes 0
total freeable PGA memory 8,716,288 bytes 0
MGA allocated (under PGA) 536,870,912 bytes 0
maximum MGA allocated 536,870,912 bytes 0
process count 218 0
max processes count 246 0
PGA memory freed back to OS 13,721,796,608 bytes 0
total PGA used for auto workareas 0 bytes 0
maximum PGA used for auto workareas 35,154,944 bytes 0
total PGA used for manual workareas 0 bytes 0
maximum PGA used for manual workareas 1,069,056 bytes 0
over allocation count 80,204 0
bytes processed 51,796,836,352 bytes 0
extra bytes read/written 0 bytes 0
cache hit percentage 100 percent 0
recompute count (total) 334,865 0
21 rows selected.
PGA 的分配明显偏大,一个空库能使用了这么多 pga,有点搞笑.
total PGA inuse 13,589,449,728 bytes 0
total PGA allocated 14,577,980,416 bytes 0
maximum PGA allocated 14,589,383,680 bytes 0
v$process 里有记录 pga 的分配使用大小
select *
from v$process
where 1=1
order by pga_used_mem desc
;
oracle@paynodea1 (DIA0) 的 pga 内存使用明显异常,对应操作系统 oracle 用户的 8285 进程,和前面查看内存使用排名相符.
# su - grid
$ sqlplus / as sysdba
SQL> set lines 200;
set pages 200;
column name format a25;
column pname format a12;
column "MegaBytes" format a10;
set numwidth 6;
select s.sid, s.serial#,p.pid, p.spid,p.pname, sn.name, round(ss.value/(1024 *1024))||'Mb' "MegaBytes"
from v$sesstat ss, v$statname sn, v$session s, v$process p
where s.paddr = p.addr
and sn.statistic# = ss.statistic#
and s.sid = ss.sid
and sn.name in ('session pga memory' , 'session pga memory max')
and p.pname like 'DIA%'
order by ss.value
/
SID SERIAL# PID SPID PNAME NAME MegaBytes
------ ------- ------ ------------------------ ------------ ------------------------- ----------
1141 3837 12 8268 DIAG session pga memory 8Mb
1141 3837 12 8268 DIAG session pga memory max 8Mb
1996 32053 21 8285 DIA0 session pga memory 12000Mb
1996 32053 21 8285 DIA0 session pga memory max 12000Mb
杀掉 DIA0 session 后,实例会重启一个进程
2019-06-25T10:07:15.967688+08:00
Process termination requested for pid 19193 [source = rdbms], [info = 2] [request issued by pid: 18183, uid: 10001]
2019-06-25T10:07:16.996820+08:00
KILL SESSION for sid=(1996, 47010):
Reason = alter system kill session
Mode = KILL HARD SAFE -/-/-
Requestor = USER (orapid = 195, ospid = 18183, inst = 2)
Owner = Process: DIA0 (orapid = 21, ospid = 19193)
Result = ORA-0
2019-06-25T10:07:19.171730+08:00
Restarting dead background process DIA0
Starting background process DIA0
2019-06-25T10:07:19.195689+08:00
DIA0 started with pid=21, OS id=18960
os内存也得到了释放
# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 558604 173596 10071120 0 0 81 48 8683 13364 0 0 99 0 0
0 0 0 557968 173596 10071120 0 0 145 49 8070 13185 0 0 99 0 0
3 0 0 557320 173596 10071120 0 0 81 160 8428 13220 0 0 99 0 0
1 0 0 480344 173596 10071224 0 0 97 173 10465 13905 2 1 97 0 0
2 0 0 541276 173596 10071176 0 0 81 252 21906 33637 5 1 94 0 0
4 0 0 450152 173596 10071240 0 0 81 49 16280 23034 4 1 95 0 0
2 0 0 5122416 173596 10069208 0 0 81 472 23653 26264 4 2 94 0 0
2 0 0 12127104 173596 10069512 0 0 93 493 35441 35639 5 3 92 0 0
3 0 0 12032968 173596 10069540 0 0 81 48 15058 22644 4 1 95 0 0
1 0 0 12082336 173596 10069508 0 0 97 3457 18977 27614 5 1 94 0 0
5 0 0 11993120 173596 10070012 0 0 589 48 21483 34140 5 1 93 0 0
1 0 0 12032256 173596 10070016 0 0 81 781 11349 15391 3 1 97 0 0
但是 DIA0 为什么会占用这么多内存
进一步分析
$ cd /u01/app/oracle/diag/rdbms/orcls1/orcls12/trace
$ egrep -i kjzcmsg orcls12_dia*
orcls12_dia0_19193_base_3.trc: 10 GB, 315919 chunks: "kjzcmsg " 2377 KB free held
orcls12_dia0_19193_base_3.trc: 10 GB, 316826 chunks: "kjzcmsg " 2377 KB free held
orcls12_dia0_19193_base_4.trc: 10 GB, 317735 chunks: "kjzcmsg " 2377 KB free held
orcls12_dia0_19193_base_4.trc: 10 GB, 318649 chunks: "kjzcmsg " 2377 KB free held
$ vi orcls12_dia0_19193_base_3.trc
=======================================
PRIVATE MEMORY SUMMARY FOR THIS PROCESS
---------------------------------------
******************************************************
PRIVATE HEAP SUMMARY DUMP
11 GB total:
9700 MB commented, 445 MB permanent
928 MB free (0 KB in empty extents),
9648 MB, 313187 chunks: "kjzcmsg " 2377 KB free held
2019-06-25 08:26:29.539 :kjzgrcfg_msghdlr(): Step 1 remote_inc:10 prop_inc:10 state:5 master:0
*** 2019-06-25T08:26:30.349957+08:00 (CDB$ROOT(1))
2019-06-25 08:26:30.349 :kjzgrcfg_finalize(): Reconfig finished inc:10 state:5
2019-06-25 08:26:30.358 :kjzgrcfg_msghdlr(): Step 1 remote_inc:10 prop_inc:10 state:5 master:0
*** 2019-06-25T08:26:31.353997+08:00 (CDB$ROOT(1))
2019-06-25 08:26:31.353 :kjzgrcfg_finalize(): Reconfig finished inc:10 state:5
2019-06-25 08:26:31.366 :kjzgrcfg_msghdlr(): Step 1 remote_inc:10 prop_inc:10 state:5 master:0
*** 2019-06-25T08:26:32.353036+08:00 (CDB$ROOT(1))
2019-06-25 08:26:32.353 :kjzgrcfg_finalize(): Reconfig finished inc:10 state:5
2019-06-25 08:26:32.361 :kjzgrcfg_msghdlr(): Step 1 remote_inc:10 prop_inc:10 state:5 master:0
*** 2019-06-25T08:26:33.360969+08:00 (CDB$ROOT(1))
2019-06-25 08:26:33.360 :kjzgrcfg_finalize(): Reconfig finished inc:10 state:5
2019-06-25 08:26:33.378 :kjzgrcfg_msghdlr(): Step 1 remote_inc:10 prop_inc:10 state:5 master:0
*** 2019-06-25T08:26:34.360883+08:00 (CDB$ROOT(1))
2019-06-25 08:26:34.360 :kjzgrcfg_finalize(): Reconfig finished inc:10 state:5
2019-06-25 08:26:34.369 :kjzgrcfg_msghdlr(): Step 1 remote_inc:10 prop_inc:10 state:5 master:0
*** 2019-06-25T08:26:35.365295+08:00 (CDB$ROOT(1))
2019-06-25 08:26:35.365 :kjzgrcfg_finalize(): Reconfig finished inc:10 state:5
2019-06-25 08:26:35.377 :kjzgrcfg_msghdlr(): Step 1 remote_inc:10 prop_inc:10 state:5 master:0
*** 2019-06-25T08:26:36.371975+08:00 (CDB$ROOT(1))
2019-06-25 08:26:36.371 :kjzgrcfg_finalize(): Reconfig finished inc:10 state:5
2019-06-25 08:26:36.381 :kjzgrcfg_msghdlr(): Step 1 remote_inc:10 prop_inc:10 state:5 master:0
*** 2019-06-25T08:26:37.372227+08:00 (CDB$ROOT(1))
2019-06-25 08:26:37.372 :kjzgrcfg_finalize(): Reconfig finished inc:10 state:5
2019-06-25 08:26:37.387 :kjzgrcfg_msghdlr(): Step 1 remote_inc:10 prop_inc:10 state:5 master:0
*** 2019-06-25T08:26:37.989766+08:00 (CDB$ROOT(1))
IPCLW:[0.126613030]{E}[WAIT]:PROTO: [1561422397989523]RETRANS DBG local acnh 0x7f99450fc930 dump:
IPCLW:[0.126613031]{-}[WAIT]:UTIL: [1561422397989523] ACNH 0x7f99450fc930 State: 1 SMSN: 1960512957 PKT(1960584397.241853529) # Pending: 4096
IPCLW:[0.126613032]{-}[WAIT]:UTIL: [1561422397989523] Peer: DIA0.KSXP_diag.18564 AckSeq: 241849433
IPCLW:[0.126613033]{-}[WAIT]:UTIL: [1561422397989523] Flags: 0x20000000 IHint: 0x21c543ed0000001f THint: 0x263119b40000001d
IPCLW:[0.126613034]{-}[WAIT]:UTIL: [1561422397989523] Local Address: 10.0.6.21:13604 Remote Address: 10.0.6.23:12712
IPCLW:[0.126613035]{-}[WAIT]:UTIL: [1561422397989523] Remote PID: ver 0 flags 1 trans 2 tos 0 opts 0 xdata3 6151 xdata2 ea30c360
IPCLW:[0.126613036]{-}[WAIT]:UTIL: [1561422397989523] : mmsz 32768 mmr 4096 mms 4096 xdata 8b1b8c61
IPCLW:[0.126613037]{-}[WAIT]:UTIL: [1561422397989523] IVPort: 24819 TVPort: 35937 IMPT: 46395 RMPT: 24913 Pending Sends: Yes Unacked Sends: Yes
IPCLW:[0.126613038]{-}[WAIT]:UTIL: [1561422397989523] Send Engine Queued: Yes sshdl -1 ssts 0 rtts 1561422397989750 snderrchk 3 creqcnt 1 credits 0/8
IPCLW:[0.126613039]{-}[WAIT]:UTIL: [1561422397989523] Unackd Messages 1960580301 -> 1960584396. SSEQ 241849433 Send Time: INVALID TIME SMSN # Xmits: 4797760 EMSN INVALID TIME
IPCLW:[0.126613040]{-}[WAIT]:UTIL: [1561422397989523] Pending send queue:
IPCLW:[0.126613041]{-}[WAIT]:UTIL: [1561422397989523] [0] mbuf 0x7f994513d7d0 MSN 1960580301 Seq 241849433 -> 241849434 # XMits: 4797760
IPCLW:[0.126613042]{-}[WAIT]:UTIL: [1561422397989523] [1] mbuf 0x7f994513dbf0 MSN 1960580302 Seq 241849434 -> 241849435 # XMits: 4797759
IPCLW:[0.126613043]{-}[WAIT]:UTIL: [1561422397989523] [2] mbuf 0x7f994513d3b0 MSN 1960580303 Seq 241849435 -> 241849436 # XMits: 4797758
IPCLW:[0.126613044]{-}[WAIT]:UTIL: [1561422397989523] [3] mbuf 0x7f994513cf90 MSN 1960580304 Seq 241849436 -> 241849437 # XMits: 4797757
IPCLW:[0.126613045]{-}[WAIT]:UTIL: [1561422397989523] [4] mbuf 0x7f994513cb70 MSN 1960580305 Seq 241849437 -> 241849438 # XMits: 4797680
IPCLW:[0.126613046]{-}[WAIT]:UTIL: [1561422397989523] [5] mbuf 0x7f994513c750 MSN 1960580306 Seq 241849438 -> 241849439 # XMits: 4797679
IPCLW:[0.126613047]{-}[WAIT]:UTIL: [1561422397989523] [6] mbuf 0x7f994513c330 MSN 1960580307 Seq 241849439 -> 241849440 # XMits: 4797679
IPCLW:[0.126613048]{-}[WAIT]:UTIL: [1561422397989523] [7] mbuf 0x7f994513bf10 MSN 1960580308 Seq 241849440 -> 241849441 # XMits: 4797678
IPCLW:[0.126613049]{-}[WAIT]:UTIL: [1561422397989523] [8] mbuf 0x7f994513baf0 MSN 1960580309 Seq 241849441 -> 241849442 # XMits: 0
IPCLW:[0.126613050]{-}[WAIT]:UTIL: [1561422397989523] [9] mbuf 0x7f994513b6d0 MSN 1960580310 Seq 241849442 -> 241849443 # XMits: 0
IPCLW:[0.126613051]{-}[WAIT]:UTIL: [1561422397989523] [10] mbuf 0x7f994513b2b0 MSN 1960580311 Seq 241849443 -> 241849444 # XMits: 0
IPCLW:[0.126613052]{-}[WAIT]:UTIL: [1561422397989523] [11] mbuf 0x7f994513ae90 MSN 1960580312 Seq 241849444 -> 241849445 # XMits: 0
IPCLW:[0.126613053]{-}[WAIT]:UTIL: [1561422397989523] [12] mbuf 0x7f994513aa70 MSN 1960580313 Seq 241849445 -> 241849446 # XMits: 0
IPCLW:[0.126613054]{-}[WAIT]:UTIL: [1561422397989523] [13] mbuf 0x7f994513a650 MSN 1960580314 Seq 241849446 -> 241849447 # XMits: 0
IPCLW:[0.126613055]{-}[WAIT]:UTIL: [1561422397989523] [14] mbuf 0x7f994513a230 MSN 1960580315 Seq 241849447 -> 241849448 # XMits: 0
IPCLW:[0.126613056]{-}[WAIT]:UTIL: [1561422397989523] [15] mbuf 0x7f9945139e10 MSN 1960580316 Seq 241849448 -> 241849449 # XMits: 0
IPCLW:[0.126613057]{-}[WAIT]:UTIL: [1561422397989523] [16] mbuf 0x7f99451399f0 MSN 1960580317 Seq 241849449 -> 241849450 # XMits: 0
出现了 Local Address: 10.0.6.21:13604 Remote Address: 10.0.6.23:12712 ,难道跟私网通讯多播有关.
但是在 mos 上又没有找到有用的信息,难道要定时杀掉 DIAG,DIA0 进程?
先记录一下,有解决方案了再更新.
参考:
<< High memory used for ASM diag/dia0 process. (文档 ID 2174882.1)>>
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=9ti1xt5c1_4&_afrLoop=246841723431860#SYMPTOM
<< dia0 eat too many memory>>
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=9ti1xt5c1_4&_afrLoop=247499448309782
<< Bug 28111583 - RAC Instance Crashes due to High Traffic (文档 ID 28111583.8)>>
https://support.oracle.com/epmos/faces/SearchDocDisplay?_adf.ctrl-state=9ti1xt5c1_4&_afrLoop=253072788203903