ASMB的BUG（ORA-04030 kfmditer）导致数据库宕机

ASMB的BUG（ORA-04030 kfmditer）导致数据库宕机
现象：
客户的一个重要生产系统RAC的一个实例宕机，查看alert日志：

Fri Jun 21 17:05:52 2013
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_asmb_11391.trc (incident=31397):
ORA-04030: out of process memory when trying to allocate 592 bytes (callheap,kfmditer)
Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31397/jyj1_asmb_11391_i31397.trc

Fri Jun 21 17:05:55 2013
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_rbal_11389.trc (incident=31389):
ORA-04030: out of process memory when trying to allocate bytes (,)
Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31389/jyj1_rbal_11389_i31389.trc
Fri Jun 21 17:06:14 2013
Instance terminated by ASMB, pid = 11391

查看asmb trace文件：
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_asmb_11391.trc (incident=31397):
ORA-04030: out of process memory when trying to allocate 592 bytes (callheap,kfmditer)
Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31397/jyj1_asmb_11391_i31397.trc
Fri Jun 21 17:05:52 2013
Trace dumping is performing id=[cdmp_20130621170552]
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_asmb_11391.trc:
ORA-04030: out of process memory when trying to allocate 592 bytes (callheap,kfmditer)
ASMB (ospid: 11391): terminating the instance due to error 4030
System state dump is made for local instance
System State dumped to trace file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_diag_11345.trc
Fri Jun 21 17:05:53 2013
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_lms0_11363.trc (incident=31301):
ORA-04030: out of process memory when trying to allocate bytes (,)
Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31301/jyj1_lms0_11363_i31301.trc
Fri Jun 21 17:05:53 2013
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_lmon_11359.trc (incident=31277):
ORA-04030: out of process memory when trying to allocate bytes (,)
Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31277/jyj1_lmon_11359_i31277.trc
Fri Jun 21 17:05:53 2013
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_lms1_11367.trc (incident=31309):
ORA-04030: out of process memory when trying to allocate bytes (,)
Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31309/jyj1_lms1_11367_i31309.trc
Fri Jun 21 17:05:54 2013
ORA-1092 : opitsk aborting process
Fri Jun 21 17:05:54 2013
License high water mark = 327
Fri Jun 21 17:05:55 2013
Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_rbal_11389.trc (incident=31389):
ORA-04030: out of process memory when trying to allocate bytes (,)
Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31389/jyj1_rbal_11389_i31389.trc
Fri Jun 21 17:06:14 2013
Instance terminated by ASMB, pid

jyj1_asmb_11391_i31397.trc：

Dump file /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31397/jyj1_asmb_11391_i31397.trc
Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /opt/app/ora11gR1db
System name: Linux
Node name: KSJYJ_DB01
Release: 2.6.18-164.el5
Version: #1 SMP Thu Sep 3 04:15:13 EDT 2009
Machine: x86_64
Instance name: jyj1
Redo thread mounted by this instance: 1
Oracle process number: 24
Unix process pid: 11391, image: oracle@KSJYJ_DB01 (ASMB)


*** 2013-06-21 17:05:52.045
*** SESSION ID:(532.1) 2013-06-21 17:05:52.046
*** CLIENT ID:() 2013-06-21 17:05:52.046
*** SERVICE NAME:(SYS$BACKGROUND) 2013-06-21 17:05:52.046
*** MODULE NAME:() 2013-06-21 17:05:52.046
*** ACTION NAME:() 2013-06-21 17:05:52.046
 
Dump continued from file: /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_asmb_11391.trc
ORA-04030: out of process memory when trying to allocate 592 bytes (callheap,kfmditer)

========= Dump for incident 31397 (ORA 4030) ========

*** 2013-06-21 17:05:52.046
----- SQL Statement (None) -----
Current SQL information unavailable - no cursor.

skdstdst <- ksedst1 <- ksedst <- dbkedDefDump <- ksedmp
 <- ksfdmp <- dbgexPhaseII <- dbgexProcessError <- dbgeExecuteForError <- dbgePostErrorKGE
 <- 1774 <- dbkePostKGE_kgsf <- kgesev <- kgesec3 <- kghnospc
 <- kghalf <- kfmdIterInit <- kfkIterInit <- kfnbIostatiterOp <- 110
 <- kfnbRun <- ksbrdp <- opirip <- opidrv <- sou2o

Process state
-----------------------

SO: 0x940dd1b98, type: 2, owner: (nil), flag: INIT/-/-/0x00 if: 0x3 c: 0x3
 proc=0x940dd1b98, name=process, file=ksu.h LINE:10286, pg=0
 (process) Oracle pid:24, ser:1, calls cur/top: 0x920f28eb8/0x920f28eb8
 flags: (0x6) SYSTEM
 int error: 0, call error: 0, sess error: 0, txn error 0
 (post info) last post received: 0 0 34
 last post received-location: ksr2.h LINE:594 ID:ksrpublish
 last process to post me: 950dfd540 47 2
 last post sent: 0 0 64
 last post sent-location: kso2.h LINE:316 ID:ksoreq_reply
 last process posted by me: 930e5c948 1 0
 (latch info) wait_event=0 bits=0
 Process Group: DEFAULT, pseudo proc: 0x950e4c060
 O/S info: user: oracle, term: UNKNOWN, ospid: 11391
 OSD pid info: Unix process pid: 11391, image: oracle@KSJYJ_DB01 (ASMB)
Dump of memory from 0x00000009D0DC0A10 to 0x00000009D0DC0C18


分析：
从报错信息（ORA-04030）看来，怀疑是Oracle的BUG导致的，因为以前碰到过类似的ASMB进程内存泄露的BUG，
于是搜索metalink关键词：asmb 04030
发现第一篇就跟客户的问题吻合。
ASMB process grows raising ora-4030 intermittently (Doc ID 735180.1)
ASMB process grows on memory, eventually leading to ora-4030 errors
which causes DB crash.

The reported error:
ORA-04030: out of process memory when trying to allocate 552 Bytes (callheap,kfmditer)
 
In the ASMB process heapdump we can see most of memory chunks are for 'kfmditer',
example:

 BreakDown
 ~~~~~~~~~
 Type     Count   Sum        Average
 ~~~~     ~~~~~   ~~~        ~~~~~~~
 Free     285684  142841492  500.00
 kfmditer 285685  157698132  552.00   <-- 在ASMB的HEAPDUMP中也看到了绝大多数都为kfmditer的内存片

 Total = 300539624 bytes 293495.73k 286.62MB
 
 这个BUG在11.1以后的大版本中都有出现，但是在以下的patchset中被修复：
 
 This issue is fixed in

11.2.0.1 (Base Release)
11.1.0.7.1 (Patch Set Update)
10.2.0.5 (Server Patch Set)
11.1.0.7 Patch 11 on Windows Platforms
11.1.0.7 RAC Recommended Patch Bundle #1
11.1.0.6 Patch 11 on Windows Platforms

如果不想做patchset升级的话，也可以直接打个小Patch 6851110可以解决这个问题。
You can check if Patch 6851110 is available for your patchset release and
O/S environment.:  Patch 6851110

解决方法：
在客户的数据库上打patch  6851110，经过持续观察一段时间，该问题未再现。