Oracle_ORA-00600: internal error code, arguments: [729]

1）、错误现象

Errors in file /oracle/app/admin/orayz1/udump/oyz1a_ora_1130868.trc:
ORA-00600: internal error code, arguments: [729], [152], [space leak], [], [], [], [], []
ORA-02068: following severe error from DBLINK_YY4A
ORA-03113: end-of-file on communication channel
Sat Dec 26 08:35:08 2015
Trace dumping is performing id=[cdmp_20151226083508]

---ORA-00600: internal error code, arguments: [729], [152], [space leak], [], [], [], [], []  代码分析----

a. the first bracketed number [729] is the common argument for space leak problems.
b. the second number [1067976] is the number of bytes leaked by the error.
c. the third argument is always [space leak].

2）、分析Trace文件

/oracle/app/admin/orayz1/udump/oyz1a_ora_1130868.trc

******** ERROR: UGA memory leak detected 152 ********
******************************************************
HEAP DUMP heap name="session heap"  desc=1103761b0
 extent sz=0xff80 alt=32767 het=32767 rec=0 flg=3 opc=3
 parent=1100091f0 owner=70000001ce82ab0 nex=0 xsz=0xff80
EXTENT 0 addr=1108e5df0

a. the memory was leaked from the UGA area
b. the amount leaked is reported again in the text (152 bytes).
c. the above few lines describe this dump as SESSION HEAP with the descriptor 0x1103761b0.

3）、计算泄露内存大小

*** 2015-12-26 08:35:07.908
******** ERROR: UGA memory leak detected 152 ********
******************************************************
HEAP DUMP heap name="session heap"  desc=1103761b0
 extent sz=0xff80 alt=32767 het=32767 rec=0 flg=3 opc=3
 parent=1100091f0 owner=70000001ce82ab0 nex=0 xsz=0xff80
EXTENT 0 addr=1108e5df0
  Chunk        1108e5e00 sz=    65392    free      "               "
EXTENT 1 addr=1108d5df0
  Chunk        1108d5e00 sz=    47512    free      "               "
  Chunk        1108e1798 sz=      408    perm      "perm           "  alo=200
  Chunk        1108e1930 sz=    17472    free      "               "
EXTENT 2 addr=1108c5df0
  Chunk        1108c5e00 sz=    65392    free      "               "
EXTENT 3 addr=1108b5df0
  Chunk        1108b5e00 sz=    34640    free      "               "
  Chunk        1108be550 sz=       56    freeable  "koddcal dvoid  "
  Chunk        1108be588 sz=       96    freeable  "kodmcon kodmc  "
  Chunk        1108be5e8 sz=    30600    free      "               "
EXTENT 4 addr=1108a5df0
  Chunk        1108a5e00 sz=    65392    free      "               "
EXTENT 5 addr=110350080
  Chunk        110350090 sz=    65392    free      "               "
EXTENT 6 addr=110340098
  Chunk        1103400a8 sz=    16152    perm      "perm           "  alo=16152
  Chunk        110343fc0 sz=    49216    free      "               "
Total heap size    =   457720
FREE LISTS:
 Bucket 0 size=56
 Bucket 1 size=88
 Bucket 2 size=152
 Bucket 3 size=168
 Bucket 4 size=280
 Bucket 5 size=432
 Bucket 6 size=536
 Bucket 7 size=1048
 Bucket 8 size=2072
 Bucket 9 size=4120
 Bucket 10 size=8216
 Bucket 11 size=16408
  Chunk        1108be5e8 sz=    30600    free      "               "
  Chunk        1108e1930 sz=    17472    free      "               "
 Bucket 12 size=32792
  Chunk        110343fc0 sz=    49216    free      "               "
  Chunk        110350090 sz=    65392    free      "               "
  Chunk        1108a5e00 sz=    65392    free      "               "
  Chunk        1108b5e00 sz=    34640    free      "               "
  Chunk        1108c5e00 sz=    65392    free      "               "
  Chunk        1108d5e00 sz=    47512    free      "               "
  Chunk        1108e5e00 sz=    65392    free      "               "
 Bucket 13 size=65560
 Bucket 14 size=131096
 Bucket 15 size=262168
 Bucket 16 size=524312
 Bucket 17 size=2097176
Total free space   =   441008
UNPINNED RECREATABLE CHUNKS (lru first):
PERMANENT CHUNKS:
  Chunk        1108e1798 sz=      408    perm      "perm           "  alo=200
  Chunk        1103400a8 sz=    16152    perm      "perm           "  alo=16152
Permanent space    =    16560
******************************************************

FREEABLE and RECREATABLE chunks总和等于1067976 byte(leaked memory)

4)、会话状态分析

SO: 70000001ce82ab0, type: 4, owner: 70000001cce2718, flag: INIT/-/-/0x00
    (session) trans: 0, creator: 70000001cce2718, flag: (100041) USR/- BSY/-/-/DEL/-/-
              DID: 0000-0000-00000000, short-term DID: 0000-0000-00000000
              txn branch: 0
              oct: 0, prv: 0, sql: 700000047f8e230, psql: 700000024e58c68, user: 30/DCM
    O/S info: user: 4A_dunjinge, term: SD4ACITRIX13, ospid: 18616:39048, machine: SD4A\SD4ACITRIX13
              program: plsqldev.exe
    last wait for 'SQL*Net break/reset to dblink' blocking sess=0x0 seq=56492 wait_time=1
                driver id=28646573, break?=0, =0
    temporary object counter: 0
    ----------------------------------------
    SO: 70000001d5243f0, type: 13, owner: 70000001cce2718, flag: INIT/-/-/0x00
    (broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: 70000001cce2718,
                       event: 2282559, last message event: 2282559,
                       last message waited event: 2282559, messages read: 0
                       channel: (70000001d55f428) system events broadcast channel
                                scope: 0, event: 2286898, last mesage event: 0,
                                publishers/subscribers: 0/34,
                                messages published: 0

a.在logoff的时候发生UGA中的session heap发生内存泄露

b.该进程是一个用户异常退出数据库时产生的，并且该程序已经被释放。

5)、出现ORA-00600[729]原因

What is a space (memory) leak?

Memory leak problems generally occur when Oracle is trying to free memory allocated to a process. The memory leak dump is generally discovered during session logoff, when Oracle frees the heaps that are allocated for the user process. 

When a user connects to Oracle, a user process is created and at that time the heap is allocated. Every process will have its own memory heap.

The memory is organized in to heaps and every heap consists of one or more extents. Each extent contains a series of contiguous memory chunks, and these chunks can be either FREE or ALLOCATED. The Generic Heap Manager takes care of allocating and deallocating the memory chunks, with the help of FREE LISTS and LRU LISTS. 

Chunk types are as follows:

1. FREE 
2. FREEABLE 
3. RECREATABLE 
4. PERMANENT 
5. FREEABLE WITH MARK 

It is not mandatory that each extent only contains one type of chunk. Extents can contain various types of chunks. When processes require memory chunks, they are allocated as needed. Oracle keeps track of the amount of memory allocated for the process internally.

When the process terminates, all of the memory that has been allocated for the process is automatically released. When the memory is released the allocated heaps are freed. Generally, when the heap is freed the only chunks that the process should identify as allocated are the PERMANENT chunks and FREE chunks on the freelist. If the process finds there are still FREEABLE or RECREATABLE chunks remaining, then the process has not properly deallocated the memory. This situation is considered a space leak. 

Oracle generally performs space leak checks for the SGA Heap, UGA Heap, Large Pool Heap and PGA Heap. A space leak error will result in a trace file in the BACKGROUND_DUMP_DESTINATION or USER_DUMP_DESTINATION.

A space leak problem will generally capture the trace information and the heap dump.

In the alert.log, the error will be reported as:

Errors in file d:\oracle\admin\testdb\bdump\yourfilenamehere.trc:

ORA-00600: internal error code, arguments: [729], [560], [space leak], [], [], [], [], []

After the operating system and Oracle process header information you will see the following in the trace file:

*** 2006-10-03 18:43:11.598 
*** SESSION ID:(34.50354) 2006-10-03 18:43:11.597 
******** ERROR: UGA memory leak detected 560 ********

The above lines tells us that this memory leak is from the UGA and the number of bytes leaked is 560.

****************************************************** 
HEAP DUMP heap name="session heap" desc=0xaef81d0 
extent sz=0xffb8 alt=32767 het=32767 rec=0 flg=3 opc=3 
parent=0xaeb63e0 owner=0x7a4b7078 nex=(nil) xsz=0xffb8 

The above few lines describe this dump as SESSION HEAP with the descriptor 0xaef81d0. 

In the dump below, you can see the chunk not released is Chunk 4085a350. This is a recreatable chunk and the size is 560 bytes, which is exactly the number of bytes reported as leaked in the error ORA-00600 [729], [560], [space_leak]. 

When analyzing memory leak issues, identify the FREEABLE and RECREATABLE chunks. Ideally, they should equal the sum of the total amount of the memory bytes leaked in the error reported. 

If you look at the memory dump below, you see the total size of freeable and recreatable chunks equals 560, which is the amount of the memory bytes leaked. 

EXTENT 0 addr=0x407cf048 
Chunk 407cf050 sz= 65456 free " " 
EXTENT 1 addr=0x408a0048 
Chunk 408a0050 sz= 65456 free " " 
EXTENT 2 addr=0x40890048 
Chunk 40890050 sz= 65456 free " " 
EXTENT 3 addr=0x40850048 
Chunk 40850050 sz= 41728 free " " 
Chunk 4085a350 sz= 560 recreate "bind var heap " latch=(nil) 
EXTENT 4 addr=0x407df048 
Chunk 407df050 sz= 65456 free " " 
EXTENT 5 addr=0x40f91048 
Chunk 40f91050 sz= 65456 free " " 
EXTENT 6 addr=0x40880048 
Chunk 40880050 sz= 65456 free " " 
EXTENT 7 addr=0x40870048 
Chunk 40870050 sz= 65456 free " " 

6）、ORA-00600[729]处理方案

How do I handle a space leak?

Step 1. Review the alert.log to verify the error and obtain the trace file information. 

The alert.log will report an error similar to the following:

Sat Dec 02 21:52:17 2006 
Errors in file d:\oracle\admin\testdb\udump\testdb_ora_5928.trc: 
ORA-00600: internal error code, arguments: [729], [152], [space leak], [], [], [], [], [] 
a. the first bracketed number [729] is the common argument for space leak problems. 
b. the second number [152] is the number of bytes leaked by the error. 
c. the third argument is always [space leak].

Step 2. Open the associated trace file.

Below the operating system and Oracle process header information, you will see the following:

*** 2006-12-13 02:01:13.859 
*** SESSION ID:(54.11635) 2006-12-13 02:01:13.859 
******** ERROR: UGA memory leak detected 152 ******** 
****************************************************** 
The above error states: 
a. the memory was leaked from the UGA area 
b. the amount leaked is reported again in the text (152 bytes).

Step 3. Verify the leak was during session logoff.

a. Search the trace file for "Call Stack Trace". If "opilof" is referenced in the stack, the error is happening at session logoff. The section will look similar to the following: 

----- Call Stack Trace ----- 
calling call entry argument values in hex 
location type point (? means dubious value) 
-------------------- -------- -------------------- ---------------------------- 
ksedmp()+184 ? ksedst() 800000010000B938 ? 
ksfdmp()+32 ? ksedmp() 800003FFBFFF6418 ? 
kgeriv()+152 ? ksfdmp() 20000000B168 ? 
kgesiv()+132 ? kgeriv() 40000000000002D9 ? 
ksesic2()+124 ? kgesiv() 000000000 ? 
ksmuhe()+1040 ? ksesic2() 000000000 ? 
ksmugf()+400 ? ksmuhe() 000000000 ? 
ksuxds()+2692 ? ksmugf() 800003FFBFFF4020 ? 
ksudel()+104 ? ksuxds() 8000000100131B38 ? 
opilof()+876 ? ksudel() 800003FFBFFF5808 ? 
opiodr()+2416 ? opilof() 0650AB9D8 ? 
ttcpip()+1320 ? opiodr() 8000000100004790 ? 
opitsk()+1260 ? ttcpip() 000000100 ? 
opiino()+1484 ? opitsk() 8000000100138268 ? 
opiodr()+2416 ? opiino() 000001560 ? 
opidrv()+752 ? opiodr() 800003FFBFFF0870 ? 
sou2o()+40 ? opidrv() 000000000 ? 
main()+228 ? sou2o() 000000000 ?

b. The session state object has following information in flags that shows that the session has been deleted:
SO: 7000000abdd9290, type: 4, owner: 7000000ab97a2b8, flag: INIT/-/-/0x00
(session) sid: 617 trans: 0, creator: 7000000ab97a2b8, flag: (100041) USR/- BSY/-/-/DEL/-/-
DID: 0000-0000-00000000, short-term DID: 0000-0000-00000000

4. Are you using Dedicated Server or Multi Threaded Server?

a. If you are using Dedicated Server the impact of this error will end when the process exits. The result of this error is minimal and presents no real problem. The leak will be in the UGA. 
b. If you are using Multi Threaded Servers(MTS) and/or an XA transaction process manager/monitor, the leaked memory is in the SGA. It is important to review the alert.log for other errors such as ORA-4030 or ORA-4031 to ensure the SGA is not experiencing additional resource related problems.

5. Can I ignore the leak?

a. Are there any other errors? If there are no other errors reported at the same time, this may be a case where the error was a rare occurrence and can be safely ignored. As a rule of thumb, leaks less than 90,000 bytes in size are considered to be of low significance. The solution in this case is to set event 10262 (see below). 
b. Is the leak in the SGA? The alert.log should be reviewed for additional errors such as ORA-4030 and ORA-4031 to ensure there are no additional problems with the shared pool or operating system memory. 
c. Does the error reproduce with a given task? If so, this is a case that should be investigated further because the leak could be a known bug. 
See 
  Note 31056.1 - ORA-600 [729] UGA Space Leak 
for a list of known bugs and fixes.

Setting EVENT 10262:

After the amount of bytes leaked has been determined to be minimal, and no other errors have been noted, you can choose to safely ignore the errors. Leaks up to 90,000 bytes in size are considered to be within the allowable range.

In cases where the amount of bytes leaked is allowable and you do not want the errors to be reported in the alert.log file, set event 10262. This will stop reporting of the ORA-600 [729] error in the alert log file up to the specified number of bytes.

a. Set the following event in init.ora parameter file. This example disables reporting for space leaks less than 90000 bytes:

event = "10262 trace name context forever, level 90000"

b. Stop and restart the database.

If the level is set to 1, space leak checking is disabled. This is not advised because large memory leaks will be missed.

If the event is set to a value greater than 1, any space leak up to the number specified in the event is ignored.

  Are you encountering a bug?

There are a number of bugs which can cause this issue. These are the most prevalent at this time: 
1.
Search your incident trace file for the following:
 'XDB' and kgbt'

If you find them, this bug is likely:
Bug 7499301 - Memory leak in XDB / ORA-600 [729] Note 7499301.8

This issue is fixed in
    12.1.0.1 (Base Release)
    11.2.0.2 (Server Patch Set) 
2.
Search your incident trace file for the following:
kllcqgf:kllsltb
This bug is likely:
Bug 14385220 - Private memory leak of "kllcqgf:kllsltb" memory (ORA-600 [729] / ORA-600 [723] 
note 14385220.8
This is not likely to be a large leak. If the errors are small, you can mask them by setting event 10262 
as indicated above.
This issue is fixed in
    12.2 (Future Release) 
3.
Search your incident trace file for the following:
"kxs-role"
This bug is likely:
Bug 9474750 - ORA-600 [729] space leak of "kxs-krole" memory Note 9474750.8
This issue is fixed in    
    12.1.0.1 (Base Release)
    11.2.0.2 (Server Patch Set) 
4.
Search your incident trace file for the following:
'hssh' 
'horkmal' 
'ncoxdc'
If you find any or all of these this bug is possible:

Bug 9365381 - ORA-600 [729] having called an external procedure followed by PMON dump Note 9365381.8
This issue is fixed in
    12.1.0.1 (Base Release)
    11.2.0.3 (Server Patch Set) 
5.
For a complete list of known fixed bugs, please see the following note:
Note 31056.1 ORA-600 [729] "UGA Space Leak"

 

REFERENCES

NOTE:19837.1 - OERR: ORA 4031 "unable to allocate %s bytes of shared memory ("%s","%s","%s")"

NOTE:21235.1 - EVENT: 10262 "Do not check for memory leaks" - Reference Note
NOTE:31056.1 - ORA-600 [729] "UGA Space Leak"

参考:Understanding and Diagnosing ORA-600 [729] Space Leak Errors [ID 403584.1]