Oracle RAC内部错误:ORA-00600[kjbmprlst:shadow]一例
一套Linux x86-64上的11.2.0.1 4节点RAC系统中LMS GCS服务进程遭遇到内部错误ORA-00600[kjbmprlst:shadow],导致节点实例意外终止,具体日志如下:
Fri Jul 08 02:04:43 2011 Errors in file /u01/app/oracle/diag/rdbms/PROD/PROD1/trace/PROD1_lms1_536.trc (incident=1011732): ORA-00600: internal error code, arguments: [kjbmprlst:shadow], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/PROD/PROD1/incident/incdir_1011732/PROD1_lms1_536_i1011732.trc Fri Jul 08 02:04:44 2011 Trace dumping is performing id=[cdmp_20110708020444] Errors in file /u01/app/oracle/diag/rdbms/PROD/PROD1/trace/PROD1_lms1_536.trc: ORA-00600: internal error code, arguments: [kjbmprlst:shadow], [], [], [], [], [], [], [], [], [], [], [] Errors in file /u01/app/oracle/diag/rdbms/PROD/PROD1/trace/PROD1_lms1_536.trc: ORA-00600: internal error code, arguments: [kjbmprlst:shadow], [], [], [], [], [], [], [], [], [], [], [] LMS1 (ospid: 536): terminating the instance due to error 484 Fri Jul 08 02:04:45 2011 opiodr aborting process unknown ospid (27387) as a result of ORA-1092 System state dump is made for local instance System State dumped to trace file /u01/app/oracle/diag/rdbms/PROD/PROD1/trace/PROD1_diag_513.trc Fri Jul 08 02:04:54 2011 Termination issued to instance processes. Waiting for the processes to exit Fri Jul 08 02:04:58 2011 ORA-1092 : opitsk aborting process该ORA-00600[kjbmprlst:shadow]错误定位为11.2.0.1上的Bug 10121589或Bug 9458781：
Bug 10121589 ORA-600 [kjbmprlst:shadow] can occur in RAC
Affects:
Product (Component) Oracle Server (Rdbms)
Range of versions believed to be affected Versions BELOW 12.1
Versions confirmed as being affected
11.2.0.1
Platforms affected Generic (all / most platforms affected)
Fixed:
This issue is fixed in
12.1 (Future Release)
11.2.0.2 Bundle Patch 2 for Exadata Database
11.2.0.1 Bundle Patch 7 for Exadata Database
Symptoms:
Related To:
Internal Error May Occur (ORA-600)
ORA-600 [kjbmprlst:shadow]
RAC (Real Application Clusters) / OPS
Description
An ORA-600 [kjbmprlst:shadow] can occur if the fix for bug 9979039
is present.
Note:
One off patches for 10200390 should also include this fix.
Bug 9458781 Missing close message to master leaves closed lock dangling crashing the instance with assorted Internal error
Affects:
Product (Component) Oracle Server (Rdbms)
Range of versions believed to be affected Versions >= 11.2.0.1 but BELOW 11.2.0.2
Versions confirmed as being affected
11.2.0.1
Platforms affected Generic (all / most platforms affected)
Fixed:
This issue is fixed in
11.2.0.2 (Server Patch Set)
11.2.0.1 Bundle Patch 4 for Exadata Database
Symptoms:
Related To:
Instance May Crash
Internal Error May Occur (ORA-600)
ORA-600 [KJBMPRLST:SHADOW]
ORA-600 [KJBMOCVT:RID]
ORA-600 [KJBRREF:PKEY]
ORA-600 [KJBRASR:PKEY]
RAC (Real Application Clusters) / OPS
Description
A lock is closed without sending a message to the master.
This causes closed lock dangling at the master crashing the instance with different internal errors.
Reported internal errors so far are :
- KJBMPRLST:SHADOW
- KJBMOCVT:RID
- KJBRREF:PKEY
- KJBRASR:PKEY 该kjbmprlst:shadow内部函数用以管理kjbm shadow锁(/libserver10.a/kjbm.o )信息,存在某个已关闭的lock没有及时message给master node的代码漏洞,目前除了安装补丁外没有已验证的workaround办法(disable drm似乎是无效的):
oradebug lkdebug (track resources, take dumps)
KCL history
KJBL history
KJL history
PCM (GCS) and non-PCM (GES) resources are kept separate and use separate code paths.
GES:
Resource table: kjr and kjrt
Lock table: kjlt
Processes: kjpt
GCS:
Resource table: kjbr
Lock table: kjbl
DLM Structures (continued)
/* PCM resource structure */
typedef struct kjbr { /* 68 bytes on sun4u */
kjsolk hash_q_kjbr; /* hash list : hp */
ub4 resname_kjbr[2]; /* the resource name */
kjsolk scan_q_kjbr; /* chain to lmd scan q of grantable resources */
kjsolk grant_q_kjbr; /* list of granted resources */
kjsolk convert_q_kjbr; /* list of resources being converted */
ub4 diskscn_bas_kjbr; /* scn(base) known to be on disk */
ub2 diskscn_wrap_kjbr; /* scn(wrap) known to be on disk */
ub2 writereqscn_wrap_kjbr; /* scn(wrap) requested for write */
ub4 writereqscn_bas_kjbr; /* scn(base) requested for write */
struct kjbl *sender_kjbr; /* lock elected to send block */
ub2 senderver_kjbr; /* version# of above lock */
ub2 writerver_kjbr; /* version# of lock below */
struct kjbl *writer_kjbr; /* lock elected to write block */
ub1 mode_role_kjbr; /* one of 'n', 's', 'x' && one of 'l' or 'g' */
ub1 flags_kjbr; /* ignorewip, free etc. */
ub1 rfpcount_kjbr; /* refuse ping counter */
ub1 history_kjbr; /* resource operation history */
kxid xid_kjbr; /* split transaction ID */
} kjbr ;
/* kjbl - PCM lock structure
** Clients and most of the DLM will use the KJUSER* or KJ_* modes and kscns */
typedef struct kjbl { /* 52 bytes on sun4u */
union { /* discriminate lock@master and lock@client */
struct { /* for lock@master */
kgglk state_q_kjbl; /* link to chain to resource */
kjbopqi *rqinfo_kjbl; /* target bid */
struct kjbr *resp_kjbl; /* pointer to my resource */
} kjbllam; /* KJB Lock Lock At Master */
struct { /* for lock@client */
ub4 disk_base_kjbl; /* disk version(base) for replay */
ub2 disk_wrap_kjbl; /* disk version(wrap) for replay */
ub1 master_node_kjbl; /* master instance# */
ub1 client_flag_kjbl; /* flags specific to client locks */
ub2 update_seq_kjbl; /* last update to master */
} kjbllac; /* KJB Lock Lock At Client */
} kjblmcd; /* KJB Lock Master Client Discrimnant */
void *remote_lockp_kjbl; /* pointer to client lock or shadow */
ub2 remote_ver_kjbl; /* remote lock version# */
ub2 ver_kjbl; /* my version# */
ub2 msg_seq_kjbl; /* client->master seq# */
ub2 reqid_kjbl; /* requestid for convert */
ub2 creqid_kjbl; /* requestid for convert that has been cancelled */
ub2 pi_wrap_kjbl; /* scn(wrap) of highest pi */
ub4 pi_base_kjbl; /* scn(base) of highest pi */
ub1 mode_role_kjbl; /* one of 'n', 's', 'x' && one of 'l' or 'g' */
ub1 state_kjbl; /* _L|_R|_W|_S, notify, which q, lock type */
ub1 node_kjbl; /* instance lock belongs to */
ub1 flags_kjbl; /* lock flag bits */
ub2 rreqid_kjbl; /* save the reqid */
ub2 write_wrap_kjbl; /* last write request version(wrap) */
ub4 write_base_kjbl; /* last write request version(base) */
ub4 history_kjbl; /* lock operation history */
} kjbl;
PCM DLM locks that are owned by the local instance are allocated and embedded in an LE structure.
PCM DLM locks that are owned by remote instances and mastered by the local instance are allocated in SHARED_POOL.
PCM Locks and Resources
Fields of interest in the kclle structure: kcllerls or releasing; kcllelnm or name(id1,id2);
kcllemode or held-mode; kclleacq or acquiring; kcllelck or DLM lock.
Fields of interest in the kjbr structure: resname_kjbr[2] or resource name; grant_q_kjbr or grant queue;
convert_q_kjbr or convert queue; mode_role_kjbr, which is a bitwise merge of grant mode and
role-interpreted NULL(0x00), S(0x01), X(0x02), L0 Local (0x00), G0 Global without PI (0x08), G1 Global with PI (0x018).
The field mode_role_kjbl in kjbl is a bitwise merge of grant, request, and lock mode: 0x00 if grant NULL;
0x01 if grant S; 0x02 if grant X; 0x04 lock has been opened at master; 0x08 if global role (otherwise local);
0x10 has one or more PI; 0x20 if request CR; 0x40 if request S; 0x80 if request X.
Someone has to keep a list of all buffers and where they are mastered
This is called Global Resource Directory (GRD)
GRD is present on all the instances of the cluster
To find out the master:
select b.dbablk, r.kjblmaster master_node
from x$le l, x$kjbl r, x$bh b
where b.obj =
and b.le_addr = l.le_addr
and l.le_kjbl = r.kjbllockp Oracle Support宣称可以通过 11.2.0.2 (Server Patch Set)或 11.2.0.1 Bundle Patch 4 for Exadata Database修复该bug，但是有迹象表明在11.2.0.2上仍可能发生该ORA-00600[kjbmprlst:shadow]内部错误，同时该bug更多地发生在超过2个节点的RAC系统中。