ORA-02409:超时:分布式事务处理等待锁定ORA-02063

ORA-02409:超时:分布式事务处理等待锁定ORA-02063

一、错误现象与环境
    前端应用程序运行时出现下面的错误提示：
         事件添加失败:ORA-02409;超时：分布式事务处理等待锁定
        ORA-02063:紧接着line(源于ITSPFDB.US.ORACLE.COM)  
    
    该应用程序后台对应的数据库为db01(oraclce 8)，使用了db link到数据库itspfdb(oracle 9)
    下面的SQL信息来自Oracle 8的数据库   

SQL> select * from v$version; BANNER ---------------------------------------------------------------- Oracle8i Enterprise Edition Release 8.1.7.4.1 - Production PL/SQL Release 8.1.7.4.0 - Production CORE 8.1.7.2.1 Production TNS for 32-bit Windows: Version 8.1.7.4.0 - Production NLSRTL Version 3.4.1.0.0 - Production SQL> select * from dba_db_links; OWNER DB_LINK USERNAME HOST CREATED --------------- ------------------------------ --------------- -------------------- ----------- PUBLIC ITSPFDB.US.ORACLE.COM HECH itspfdb 2008-7-3 18 SQL> select * from v$dblink; DB_LINK OWNER_ID LOGGED_ON HETEROGENEOUS PROTOCOL OPEN_CURSORS IN_TRANSACTION UPDATE_SENT COMMIT_POINT_STRENGTH --------------------- -------- --------- ------------- -------- ------------ -------------- ----------- --------------------- ITSPFDB.US.ORACLE.COM 0 YES YES UNKN 0 YES NO 1 创建了同义词 SQL> col TABLE_OWNER format a15 SQL> select * from dba_synonyms where db_link is not null; OWNER SYNONYM_NAME TABLE_OWNER TABLE_NAME DB_LINK --------------- ------------------------------ --------------- ------------------------------ ------------------------------ HECH T_EVENT_HECH511 ITSPF T_EVENT_HECH511 ITSPFDB.US.ORACLE.COM

二、alert log file与dump信息
    错误日志出现在Oracle 8中，如下：
    Errors in file D:\oracle\admin\DB01\udump\ORA03992.TRC:
    ORA-02049: 超时: 分布式事务处理等待锁定
    ORA-02063: 紧接着line（源于ITSPFDB.US.ORACLE.COM）
    
    Mon Jul 11 10:24:13 2011
    Errors in file D:\oracle\admin\DB01\udump\ORA01084.TRC:
    ORA-02049: 超时: 分布式事务处理等待锁定
    ORA-02063: 紧接着line（源于ITSPFDB.US.ORACLE.COM）

    dump文件信息：
    
    Dump file D:\oracle\admin\DB01\udump\ORA03992.TRC
    Mon Jul 11 10:23:10 2011
    ORACLE V8.1.7.4.1 - Production vsnsta=0
    vsnsql=f vsnxtr=3
    Windows 2000 Version 5.2 , CPU type 586
    Oracle8i Enterprise Edition Release 8.1.7.4.1 - Production
    With the Partitioning option
    JServer Release 8.1.7.4.1 - Production
    Windows 2000 Version 5.2 , CPU type 586
    Instance name: db01
    
    Redo thread mounted by this instance: 1
    
    Oracle process number: 21
    
    Windows thread id: 3992, image: ORACLE.EXE
    
    *** 2011-07-11 10:23:10.313
    *** SESSION ID83.24650) 2011-07-11 10:23:10.266
    FATAL ERROR IN TWO-TASK SERVER: error = 12571
    *** 2011-07-11 10:23:10.313
    ksedmp: internal or fatal error
    ORA-02049: 超时: 分布式事务处理等待锁定
    ORA-02063: 紧接着line（源于ITSPFDB.US.ORACLE.COM）
    Current SQL statement for this session:
    insert into T_EVENT_HECH511(id,event_id, start_unit_id, end_unit_id, event_level, state, event_type, start_date,
        end_date, acc_id, event_source, summary, remark, event_reason, region_no, modify_date, acc_level)
        values(73544,'110711100028',1354,1371,0,0,001,to_date('2011-7-11 10:22:01','yyyy-mm-dd hh24:mi:ss')
        ,null,null,2,'xxxx yyyy','','0','11',sysdate,null)
    ----- Call Stack Trace -----
    calling                          call     entry                            argument values in hex
    location                         type     point                            (? means dubious value)
    --------------------             -------- --------------------             ----------------------------
    _ksedmp+a8                       CALLrel  _ksedst+0                        
                                                                               8EE474
    _opitsk+f4a                      CALLrel  _ksedmp+0                        2
    _opiino+50c                      CALLrel  _opitsk+0                        0
    _opiodr+507                      CALLreg  00000000                         3C 4
                                                                               D8EFBFC
    _opidrv+384                      CALLrel  _opiodr+0                        3C 4
                                                                               D8EFBFC
                                                                               0
    _sou2o+19                        CALLrel  _opidrv+0                        
    _opimai+10c                      CALLrel  _sou2o+0                         
    _OracleThreadStart@4+4b8         CALLrel  _opimai+0                        2
                                                                               D8EFE78
    77E1A98D                         CALLreg  00000000                         
    ----- Argument/Register Address Dump -----
    Argument/Register addr=d8ee474.  
    Dump of memory from 0xD8EE434 to 0xD8EE574
    D8EE430          00000003 00000000 6552211C      [.........!Re]
    D8EE440 00000033 00000000 FFFFFFFD 00000048  [3...........H...]
    D8EE450 00000000 00000000 00000000 0D001758  [............X...]
    D8EE460 00000000 0000001D 0D8EE480 0041E6E4  [..............A.]
    D8EE470 00000001 00E3F1BC 00E3F1BC 00E3F1BC  [................]
    D8EE480 00E3F1C4 00E3F1BC 00E3F1BC 00E3F1BC  [................]
    D8EE490 00E3F1BC 00E3F1C4 00000000 00000000  [................]
    D8EE4A0 00000000 0D001758 0D8EE55C 0041F5F9  [....X...\.....A.]
    D8EE4B0 0D8EE4C0 0000001C 0D001758 00000002  [........X.......]
    D8EE4C0 2A2A2A0A 31303220 37302D31 2031312D  [.*** 2011-07-11 ]
    D8EE4D0 323A3031 30313A33 32332D25 2D252073  [10:23:10%-32s %-]
    D8EE4E0 25207338 7332332D 00000020 0D8EE504  [8s %-32s .......]
    D8EE4F0 60B15218 0D410030 45384420 38374546  [.R.`0.A. D8EFE78]
    D8EE500 25207300 7332332D 342D2520 0D000A73  [.s %-32s %-4s...]
    D8EE510 0000311B 655221B4 0D000033 00000000  [.1...!Re3.......]
    D8EE520 00000000 00000000 00000000 00000000  [................]
    D8EE530 00000000 0D8EFFEC 00000000 0D8EFE78  [............x...]
    D8EE540 00000000 00000001 00000001 00000000  [................]
    D8EE550 0D8EDCBC 00000003 0D8EE580 004844CD  [.............DH.]
    D8EE560 0D8EE498 00000000 0000311B 0D0043F0  [.........1...C..]
    D8EE570 00000000                             [....]            
    Argument/Register addr=d8efbfc.  
    Dump of memory from 0xD8EFBBC to 0xD8EFCFC
    D8EFBB0                            00000001              [....]
    D8EFBC0 77F32AC3 77E2AAA3 0D414AFC 60840538  [.*.w...w.JA.8..`]
    D8EFBD0 00000000 00000000 00000000 0D8EFBEC  [................]
    D8EFBE0 00000000 00000000 00000000 00000000  [................]
    D8EFBF0 7FF93C00 0D8EFC14 77E2AB09 0D8EFD83  [.<.........w....]
    D8EFC00 FFFFFFFF 00000000 00000001 00000000  [................]
    D8EFC10 77F3331F 77F32397 0D8EFFB8 0040318D  [.3.w.#.w.....1@.]
    D8EFC20 00000002 0D8EFE78 00000000 00000000  [....x...........]
    D8EFC30 00E79084 0D007188 77FB2340 77F4D0E7  [.....q..@#.w...w]
    D8EFC40 00000000 7FFDF000 00000000 00000024  [............$...]
    D8EFC50 00000001 00000000 44414853 00000000  [........SHAD....]
    D8EFC60 FFFF0000 00000002 01208F98 00000000  [.......... .....]
    D8EFC70 0000175C 00000001 00000000 00000000  [\...............]
    D8EFC80 00000070 FFFFFFFF FFFFFFFF 77F4D045  [p...........E..w]
    D8EFC90 77F4D10F 00246E68 71A81570 7FFDF000  [...whn$.p..q....]
    D8EFCA0 77FB23B4 0D8EFC40 00000000 0D8EFD0C  [.#.w@...........]
    D8EFCB0 77F35A34 77F7C7B0 FFFFFFFF 77F4D0E7  [4Z.w...w.......w]
    D8EFCC0 77F3363F 77F4CEE5 0D8EFD30 00000000  [?6.w...w0.......]
    D8EFCD0 00E79084 00000000 00000000 00000000  [................]
    D8EFCE0 00000000 00000000 00000000 00000000  [................]
    D8EFCF0 00000000 00000000 7FFDF000           [............]    
    Argument/Register addr=d8efe78.  
    Dump of memory from 0xD8EFE38 to 0xD8EFF78
    D8EFE30                   F6C0AB75 E16EACF0          [u.....n.]
    D8EFE40 82695620 00000000 F6C0AAF4 F6C0AB30  [ Vi.........0...]
    D8EFE50 80585240 805691F3 FF8D730C 81A31A08  [@RX...V..s......]
    D8EFE60 8132F238 C0015B3C 055CF000 00000000  [8.2.<[....\.....]
    D8EFE70 00000005 000001B3 0D8EFD78 0D8EFD83  [........x.......]
    D8EFE80 00000004 00000000 00000004 61726F2A  [............*ora]
    D8EFE90 77617073 65725F6E 5F796C70 31306264  [spawn_reply_db01]
    D8EFEA0 3939335F 00002A32 F873F120 804E8905  [_3992*.. .s...N.]
    D8EFEB0 00000246 C01FFE08 00000002 80500F2E  [F.............P.]
    D8EFEC0 00000002 80500EE0 00000000 7FF81000  [......P.........]
    D8EFED0 804F0CB0 F6C0AB9C 00BD7867 7FF81000  [..O.....gx......]
    D8EFEE0 02F81000 F6C0ABE8 804F7C80 00000000  [.........|O.....]
    D8EFEF0 FFDFF548 00BD7867 804FBDFC 81A31C20  [H...gx....O. ...]
    D8EFF00 81A31A08 81A31AD8 00000000 7FF81000  [................]
    D8EFF10 820521D0 055D0000 00000000 81907800  [.!....]......x..]
    D8EFF20 00000000 81A31AD8 FFFFFFFF 00000000  [................]
    D8EFF30 804E7E36 F6C0ABFC 8147B680 F873F120  [6~N.......G. .s.]
    D8EFF40 F873F900 804EF800 8147B720 8147B680  [..s...N. .G...G.]
    D8EFF50 8147B810 00000001 8147B680 00000001  [..G.......G.....]
    D8EFF60 00000000 00000000 F6C0AC30 804EFA20  [........0... .N.]
    D8EFF70 8147B680 8147B7E0                    [..G...G.]        
    ----- End of Call Stack Trace -----
    ===================================================
    Files currently opened by this process:
    ===================================================
    PROCESS STATE
    -------------
    Process global information:
         process: 23707e4, call: 0, xact: 268e070, curses: 0, usrses: 23b8c40
      ----------------------------------------
      SO: 23707e4, type: 1, owner: 0, pt: 0, flag: INIT/-/-/0x00
      (process) Oracle pid=21, calls cur/top: 0/3da179c, flag: (0) -
                int error: 0, call error: 0, sess error: 0, txn error 0
      (post info) last post received: 85 0 4
                  last post received-location: kslpsr
                  last process to post me: 236d23c 1 2
                  last post sent: 0 0 13
                  last post sent-location: ksasnd
                  last process posted by me: 236d23c 1 2
        (latch info) wait_event=0 bits=0
        Process Group: DEFAULT, pseudo proc: 2389f20
        O/S info: user: SYSTEM, term: TESTCOMPUTER, ospid: 3992
        OSD pid info: 3992
        ----------------------------------------
        SO: 23b8c40, type: 3, owner: 23707e4, pt: 0, flag: INIT/-/-/0x00
        (session) trans: 268e070, creator: 23707e4, flag: (8000041) USR/- -/-/-/-/-/-
                  DID: 0001-0015-00000025, short-term DID: 0000-0000-00000000
                  txn branch: 26afb2c
                  oct: 0, prv: 0, user: 43/HECH
        O/S info: user: Administrator, term: CHINA-175944CD0, ospid: 2100:3876, machine: WORKGROUP\CHINA-175944CD0
                  program: VB6.EXE
        last wait for 'SQL*Net break/reset to client' blocking sess=0x0 seq=827 wait_time=-2
                    driver id=54435000, break?=1, =0
          ----------------------------------------
          SO: 27efcfc, type: 36, owner: 23b8c40, flag: INIT/-/-/0x00
          LIBRARY OBJECT PIN: pin=27efcfc handle=0 lock=27e3dd4
          user=23b8c40 session=23b8c40 count=0 mask=0000 savepoint=105 flags=[00]
          ----------------------------------------
          SO: 27e3dd4, type: 35, owner: 23b8c40, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=27e3dd4 handle=2ee8d28 mode=N
          call pin=0 session pin=27efcfc
          user=23b8c40 session=23b8c40 count=1 flags=[00] savepoint=103
          LIBRARY OBJECT HANDLE: handle=2ee8d28
          namespace=CRSR flags=RON/KGHP/PN0/[10010000]
          kkkk-dddd-llll=0000-0041-0041 lock=N pin=0 latch=2
          lwt=2ee8d40[2ee8d40,2ee8d40] ltm=2ee8d48[2ee8d48,2ee8d48]
          pwt=2ee8d58[2ee8d58,2ee8d58] ptm=2ee8db0[2ee8db0,2ee8db0]
          ref=2ee8d30[4070294,4070294] lnd=2ee8dbc[2ee8dbc,2ee8dbc]
            LIBRARY OBJECT: object=2be4a38
            type=CRSR flags=EXS[0001] pflags= [00] status=VALD load=0
            DEPENDENCIES: count=1 size=16
            TRANSLATIONS: count=1 size=16
            DATA BLOCKS:
            data#     heap  pointer status pins change
            ----- -------- -------- ------ ---- ------
                0  4147cd0  2dfbdf4 I/P/A     0 NONE  
                6  2be4acc  290e0f4 I/-/A     0 NONE  
          ----------------------------------------
          SO: 27f11b4, type: 35, owner: 23b8c40, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=27f11b4 handle=2e72ac0 mode=N
          call pin=27e5a60 session pin=0
          user=23b8c40 session=23b8c40 count=1 flags=[00] savepoint=102
          LIBRARY OBJECT HANDLE: handle=2e72ac0
          name=
    insert into T_EVENT_HECH511(id,event_id, start_unit_id, end_unit_id, event_level, state, event_type, start_date,
        end_date, acc_id, event_source, summary, remark, event_reason, region_no, modify_date, acc_level)    
        values(73544,'110711100028',1354,1371,0,0,001,to_date('2011-7-11 10:22:01','yyyy-mm-dd hh24:mi:ss'),
        null,null,2,'xxxx yyyy','','0','11',sysdate,null)
          hash=3ee5c0de timestamp=07-11-2011 10:22:08
          namespace=CRSR flags=RON/KGHP/TIM/PN0/LRG/[10010001]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch=2
          lwt=2e72ad8[2e72ad8,2e72ad8] ltm=2e72ae0[2e72ae0,2e72ae0]
          pwt=2e72af0[2e72af0,2e72af0] ptm=2e72b48[2e72b48,2e72b48]
          ref=2e72ac8[2e72ac8,2e72ac8] lnd=2e72b54[2e72b54,2e72b54]
            LIBRARY OBJECT: object=407010c
            type=CRSR flags=EXS[0001] pflags= [00] status=VALD load=0
            CHILDREN: size=16
            child#    table reference   handle
            ------ -------- --------- --------
                 0  40702c8   4070294  2ee8d28
            DATA BLOCKS:
            data#     heap  pointer status pins change
            ----- -------- -------- ------ ---- ------
                0  3142c14  4070190 I/P/A     0 NONE  
          ----------------------------------------
          SO: 27e57d8, type: 36, owner: 23b8c40, flag: INIT/-/-/0x00
          LIBRARY OBJECT PIN: pin=27e57d8 handle=0 lock=27f0db8
          user=23b8c40 session=23b8c40 count=0 mask=0000 savepoint=52 flags=[00]
          ----------------------------------------
          SO: 27f0db8, type: 35, owner: 23b8c40, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=27f0db8 handle=2d9063c mode=N
          call pin=0 session pin=27e57d8
          user=23b8c40 session=23b8c40 count=1 flags=[00] savepoint=52
          LIBRARY OBJECT HANDLE: handle=2d9063c
          namespace=CRSR flags=RON/KGHP/PN0/[10010000]
          kkkk-dddd-llll=0000-0041-0041 lock=N pin=0 latch=2
          lwt=2d90654[2d90654,2d90654] ltm=2d9065c[2d9065c,2d9065c]
          pwt=2d9066c[2d9066c,2d9066c] ptm=2d906c4[2d906c4,2d906c4]
          ref=2d90644[34cde88,34cde88] lnd=2d906d0[2d906d0,2d906d0]
            LIBRARY OBJECT: object=30cdcd4
            type=CRSR flags=EXS[0001] pflags= [00] status=VALD load=0
            DEPENDENCIES: count=3 size=16
            AUTHORIZATIONS: count=1 size=16 minimum entrysize=16
            ACCESSES: count=1 size=16
            TRANSLATIONS: count=1 size=16
            DATA BLOCKS:
            data#     heap  pointer status pins change
            ----- -------- -------- ------ ---- ------
                0  3c48610  3be60e4 I/P/A     0 NONE  
                6  30cdd68  38f2bc8 I/-/A     0 NONE  
          ----------------------------------------
          SO: 27e4d4c, type: 35, owner: 23b8c40, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=27e4d4c handle=3c827f4 mode=N
          call pin=27efbdc session pin=0
          user=23b8c40 session=23b8c40 count=1 flags=[00] savepoint=51
          LIBRARY OBJECT HANDLE: handle=3c827f4
          name=select to_char(sysdate,'yyyy-mm-dd hh24:mi:ss') from dual
          hash=58512020 timestamp=07-05-2011 07:30:48
          namespace=CRSR flags=RON/KGHP/TIM/PN0/SML/[12010000]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch=2
          lwt=3c8280c[3c8280c,3c8280c] ltm=3c82814[3c82814,3c82814]
          pwt=3c82824[3c82824,3c82824] ptm=3c8287c[3c8287c,3c8287c]
          ref=3c827fc[3c827fc,3c827fc] lnd=3c82888[3c82888,3c82888]
            LIBRARY OBJECT: object=28cd708
            type=CRSR flags=EXS[0001] pflags= [00] status=VALD load=0
            CHILDREN: size=16
            child#    table reference   handle
            ------ -------- --------- --------
                 0  28cd798   34cde88  2d9063c
            DATA BLOCKS:
            data#     heap  pointer status pins change
            ----- -------- -------- ------ ---- ------
                0  432818c  3fa8550 I/P/A     0 NONE  
          ----------------------------------------
          SO: 27f29c0, type: 35, owner: 23b8c40, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=27f29c0 handle=2ff5230 mode=N
          call pin=27f8f3c session pin=0
          user=23b8c40 session=23b8c40 count=1 flags=[00] savepoint=26
          LIBRARY OBJECT HANDLE: handle=2ff5230
          name=HECH.SJ_T_HECH_EVENT_ID
          hash=3353c42f timestamp=07-16-2008 16:10:12
          namespace=TABL/PRCD/TYPE flags=KGHP/TIM/SML/[02000000]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch=1
          lwt=2ff5248[2ff5248,2ff5248] ltm=2ff5250[2ff5250,2ff5250]
          pwt=2ff5260[2ff5260,2ff5260] ptm=2ff52b8[2ff52b8,2ff52b8]
          ref=2ff5238[2ff5238,2ff5238] lnd=2ff52c4[2e101c8,3ca2508]
            LIBRARY OBJECT: object=3114000
            type=SQNC flags=EXS/LOC[0005] pflags= [00] status=VALD load=0
            DATA BLOCKS:
            data#     heap  pointer status pins change
            ----- -------- -------- ------ ---- ------
                0  31650d8  3114084 I/-/A     0 NONE  
          ----------------------------------------
          SO: 3500484, type: 35, owner: 23b8c40, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=3500484 handle=3a872e4 mode=N
          call pin=27f24bc session pin=0
          user=23b8c40 session=23b8c40 count=1 flags=[00] savepoint=16
          LIBRARY OBJECT HANDLE: handle=3a872e4
          name=HECH.SJ_T_HECH_EVENT_EVENTID
          hash=16d2ed55 timestamp=07-11-2011 00:00:03
          namespace=TABL/PRCD/TYPE flags=KGHP/TIM/SML/[02000000]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch=1
          lwt=3a872fc[3a872fc,3a872fc] ltm=3a87304[3a87304,3a87304]
          pwt=3a87314[3a87314,3a87314] ptm=3a8736c[3a8736c,3a8736c]
          ref=3a872ec[3a872ec,3a872ec] lnd=3a87378[3c807b8,40a7a50]
            LIBRARY OBJECT: object=339b2f8
            type=SQNC flags=EXS/LOC[0005] pflags= [00] status=VALD load=0
            DATA BLOCKS:
            data#     heap  pointer status pins change
            ----- -------- -------- ------ ---- ------
                0  3402d74  339b37c I/-/A     0 NONE  
          ----------------------------------------
          SO: 26afb2c, type: 26, owner: 23b8c40, pt: 0, flag: INIT/-/-/0x00
          (branch) trn = 0x268e070, flg = 0x02, state = 0x00 bno=1 ser=6
           creator = 0x23b8c40 uid = 0x2b serial# = 24650
           DB01.AIX.dfc38ba4.3.62.40474
            ----------------------------------------
            SO: 268e070, type: 25, owner: 26afb2c, pt: 0, flag: INIT/-/-/0x00
            (trans) bsn = 3, flg = 0x401e03, flg2 = 0x80, prx = 0x0, ros = 2147483647, spn = 105
            efd = 10
            parent xid: 0x0000.000.00000000
            env: (scn: 0x0b31.6691d413  xid: 0x0003.03e.00009e1a  uba: 0x00800407.9127.1b)
            cev: (spc = 4792  usi = 3  ubk tsn: 1 rdba: 0x00800407  useg tsn: 1 rdba: 0x00800402
                  hwm uba: 0x00800407.9127.1b  col uba: 0x00000000.0000.00
                  num bl: 1 bk list: 0x2673154)
            (gtrid 499) DB01.AIX.dfc38ba4.3.62.40474
              #branches = 1, type = 0x2
              branches     =  26afb2c, 0, 0, 0, 0, 0, 0, 0
              transactions =  268e070, 0, 0, 0, 0, 0, 0, 0
             branch#0 #refs = 1, #prep'd = 0, flags = 0x0, npi = N
            (enqueue) TX-0003003E-00009E1A        DID: 0001-0015-00000025
            lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
            res: 244a3e4, mode: X, prv: 244a3ec, sess: 23b8c40, proc: 23707e4
                    xga: 0x0, heap: UGA
              ----------------------------------------
              SO: 265d584, type: 22, owner: 268e070, pt: 0, flag: INIT/-/-/0x00
              DML LOCK: tab=24946 flg=01 chi=0
                        his[0]: mod=3 spn=92
              (enqueue) TM-00006172-00000000        DID: 0001-0015-00000025
              lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
              res: 244a15c, mode: SX, prv: 265da50, sess: 23b8c40, proc: 23707e4
              ----------------------------------------
              SO: 2673154, type: 24, owner: 268e070, pt: 0, flag: -/-/-/0x00
              (List of Blocks) next index = 7
              index   savepoint   itli   buffer hint
              --------------------------------------
                  0          71      3   0x 251cf50
                  1          74      2   0x 24b8ea8
                  2          76      2   0x 24d1eb0
                  3          78      2   0x 2500340
                  4          80      3   0x 257ea40
                  5          82      2   0x 2542690
                  6          95      3   0x 2562e20
              ----------------------------------------
              SO: 265d494, type: 22, owner: 268e070, pt: 0, flag: INIT/-/-/0x00
              DML LOCK: tab=25111 flg=01 chi=0
                        his[0]: mod=3 spn=68
              (enqueue) TM-00006217-00000000        DID: 0001-0015-00000025
              lv: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
              res: 244a42c, mode: SX, prv: 265d618, sess: 23b8c40, proc: 23707e4
          ----------------------------------------
          SO: 27eff84, type: 36, owner: 23b8c40, flag: INIT/-/-/0x00
          LIBRARY OBJECT PIN: pin=27eff84 handle=422c49c mode=S lock=27f0f98
          user=23b8c40 session=23b8c40 count=1 mask=0041 savepoint=2 flags=[00]
          ----------------------------------------
          SO: 27f0f98, type: 35, owner: 23b8c40, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=27f0f98 handle=422c49c mode=N
          call pin=0 session pin=27eff84
          user=23b8c40 session=23b8c40 count=1 flags=PNS/[08] savepoint=2
          LIBRARY OBJECT HANDLE: handle=422c49c
          namespace=CRSR flags=RON/KGHP/PN0/[10010000]
          kkkk-dddd-llll=0000-0041-0041 lock=N pin=S latch=2
          lwt=422c4b4[422c4b4,422c4b4] ltm=422c4bc[422c4bc,422c4bc]
          pwt=422c4cc[422c4cc,422c4cc] ptm=422c524[422c524,422c524]
          ref=422c4a4[3971fc4,3971fc4] lnd=422c530[422c530,422c530]
            LIBRARY OBJECT: object=38cb388
            type=CRSR flags=EXS[0001] pflags= [00] status=VALD load=0
            DEPENDENCIES: count=2 size=16
            AUTHORIZATIONS: count=1 size=16 minimum entrysize=16
            ACCESSES: count=1 size=16
            TRANSLATIONS: count=2 size=16
            DATA BLOCKS:
            data#     heap  pointer status pins change
            ----- -------- -------- ------ ---- ------
                0  2f96fe4  28895e8 I/P/A     0 NONE  
                6  38cb41c  4235720 I/P/A     1 NONE  
          ----------------------------------------
          SO: 27f131c, type: 35, owner: 23b8c40, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=27f131c handle=3c20354 mode=N
          call pin=27ef75c session pin=0
          user=23b8c40 session=23b8c40 count=1 flags=[00] savepoint=1
          LIBRARY OBJECT HANDLE: handle=3c20354
          name=
    select region_name from t_region_info where region_id =
        (select region_id from t_unit_info_branch where unit_id = '1354' and rownum=1)
          hash=44f1d7ab timestamp=07-11-2011 09:07:18
          namespace=CRSR flags=RON/KGHP/TIM/PN0/MED/[50010000]
          kkkk-dddd-llll=0000-0001-0001 lock=N pin=0 latch=2
          lwt=3c2036c[3c2036c,3c2036c] ltm=3c20374[3c20374,3c20374]
          pwt=3c20384[3c20384,3c20384] ptm=3c203dc[3c203dc,3c203dc]
          ref=3c2035c[3c2035c,3c2035c] lnd=3c203e8[3c203e8,3c203e8]
            LIBRARY OBJECT: object=2a71a8c
            type=CRSR flags=EXS[0001] pflags= [00] status=VALD load=0
            CHILDREN: size=16
            child#    table reference   handle
            ------ -------- --------- --------
                 0  3972190   3971fc4  422c49c
            DATA BLOCKS:
            data#     heap  pointer status pins change
            ----- -------- -------- ------ ---- ------
                0  38dee34  2a71b10 I/P/A     0 NONE  
          ----------------------------------------
          SO: 350022c, type: 35, owner: 23b8c40, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=350022c handle=42f5cf0 mode=N
          call pin=27f8d8c session pin=0
          user=23b8c40 session=23b8c40 count=1 flags=CBB/[02] savepoint=23
          LIBRARY OBJECT HANDLE: handle=42f5cf0
          name=SYS.DATABASE
          hash=aea96fa0 timestamp=12-31-4712 23:59:59
          namespace=EVNT flags=KGHP/TIM/SML/[02000000]
          kkkk-dddd-llll=0000-0009-0009 lock=N pin=0 latch=1
          lwt=42f5d08[42f5d08,42f5d08] ltm=42f5d10[42f5d10,42f5d10]
          pwt=42f5d20[42f5d20,42f5d20] ptm=42f5d78[42f5d78,42f5d78]
          ref=42f5cf8[42f5cf8,42f5cf8] lnd=42f5d84[42f5d84,42f5d84]
            LIBRARY OBJECT: object=3254378
            type=EVNT flags=EXS/LOC[0005] pflags= [00] status=VALD load=0
            DATA BLOCKS:
            data#     heap  pointer status pins change
            ----- -------- -------- ------ ---- ------
                0  310a4a0  325445c I/-/A     0 NONE  
                3  325440c  3d89b54 I/-/A     0 NONE  
          ----------------------------------------
          SO: 27f2df8, type: 35, owner: 23b8c40, flag: INIT/-/-/0x00
          LIBRARY OBJECT LOCK: lock=27f2df8 handle=40abcf0 mode=N
          call pin=27f14fc session pin=0
          user=23b8c40 session=23b8c40 count=1 flags=CBB/[02] savepoint=23
          LIBRARY OBJECT HANDLE: handle=40abcf0
          name=HECH.HECH
          hash=7677ed7 timestamp=12-31-4712 23:59:59
          namespace=EVNT flags=KGHP/TIM/SML/[02000000]
          kkkk-dddd-llll=0000-0009-0009 lock=N pin=0 latch=1
          lwt=40abd08[40abd08,40abd08] ltm=40abd10[40abd10,40abd10]
          pwt=40abd20[40abd20,40abd20] ptm=40abd78[40abd78,40abd78]
          ref=40abcf8[40abcf8,40abcf8] lnd=40abd84[40abd84,40abd84]
            LIBRARY OBJECT: object=2ab2cf4
            type=EVNT flags=EXS/LOC[0005] pflags= [00] status=VALD load=0
            DATA BLOCKS:
            data#     heap  pointer status pins change
            ----- -------- -------- ------ ---- ------
                0  41b89c8  2ab2dd8 I/-/A     0 NONE  
                3  2ab2d88  31ef814 I/-/A     0 NONE  
          ----------------------------------------
          SO: 326c1d8, type: 34, owner: 23b8c40, pt: 0, flag: INIT/-/-/0x00
          user lock: lock=326c1d8 mode=S
          user resource: user=27fc940 uid=43 mode=S
          ----------------------------------------
          SO: 3d415a0, type: 34, owner: 23b8c40, pt: 0, flag: INIT/-/-/0x00
          user lock: lock=3d415a0 mode=S
          user resource: user=27fc940 uid=43 mode=S
        ----------------------------------------
        SO: 2472308, type: 9, owner: 23707e4, pt: 0, flag: INIT/-/-/0x00
        (broadcast handle) flag: (2) ACTIVE SUBSCRIBER, owner: 23707e4,
                           event: 43885, last message event: 43885, messages read: 0
                           channel: (2473f60) system events broadcast channel
                                    scope: 101, event: 43889, last mesage event: 0,
                                    publishers/subscribers: 0/28,
                                    messages published: 0
        ----------------------------------------
        SO: 3da179c, type: 2, owner: 23707e4, pt: 0, flag: INIT/-/-/0x00
        (call) sess: cur 23b8c40, rec 0, usr 23b8c40; depth: 0
    END OF PROCESS STATE

三、分析与解决
    ORA-02049:
        timeout:distributed transaction waiting for lock
    cause:exceeded INIT.ORA distributed_lock_timeout seconds waiting for lock.
    action:treat as a deadlock.
    
    下面的ID 576929.1来自metalink
    Oracle Trace Error:The 'FATAL ERROR IN TWO-TASK SERVER: error =12571' [ID 576929.1]
    --------------------------------------------------------------------------------
       Modified 22-APR-2010     Type HOWTO     Status ARCHIVED  
    
    In this Document
      Goal
      Solution
    --------------------------------------------------------------------------------
    Applies to:
    Oracle Agile Application Administration
    Information in this document applies to any platform.
    ***Checked for relevance on 22-April-2010***
    
    Oracle 8.1.7
    
    Goal
    Oracle Trace Error:The 'FATAL ERROR IN TWO-TASK SERVER: error =12571'
    
    Solution
    Restart the eHub services
    
    The ora-12571 error usually occurs when the client abruptly terminates with a Ctrl-C or powers down.
    OR
    The ora-12571 error, usually, indicates an instability with Oracle with the network. Communication Gap between eHub
        and Database Server.

    从上面的情形可知，两种情况导致了ora-12571。一是意外终止或按下Ctrl-C，二是网络不稳定所致，而导致前面描述的ORA-02409
    
    更多有关ORA-02409的文章
        ID 164839.1
        ID 19332.1
        ID 789517.1

四、更多参考

有关性能优化请参考

    Oracle硬解析与软解析

    共享池的调整与优化(Sharedpool Tuning)

    Buffercache 的调整与优化(一)

    Oracle表缓存(cachingtable)的使用

 

有关ORACLE体系结构请参考

    Oracle表空间与数据文件

    Oracle密码文件

    Oracle参数文件

    Oracle联机重做日志文件(ONLINE LOG FILE)

    Oracle控制文件(CONTROLFILE)

    Oracle归档日志

    Oracle回滚(ROLLBACK)和撤销(UNDO)

    Oracle数据库实例启动关闭过程

    Oracle10g SGA 的自动化管理

    Oracle实例和Oracle数据库(Oracle体系结构)

 

有关闪回特性请参考

    Oracle闪回特性(FLASHBACK DATABASE)

    Oracle闪回特性(FLASHBACK DROP & RECYCLEBIN)

    Oracle闪回特性(Flashback Query、FlashbackTable)

    Oracle闪回特性(Flashback Version、Flashback Transaction)

 

有关基于用户管理的备份和备份恢复的概念请参考

    Oracle冷备份

    Oracle热备份

    Oracle备份恢复概念

    Oracle实例恢复

    Oracle基于用户管理恢复的处理(详细描述了介质恢复及其处理)

    SYSTEM表空间管理及备份恢复

    SYSAUX表空间管理及恢复

 

有关RMAN的备份恢复与管理请参考

    RMAN 概述及其体系结构

    RMAN 配置、监控与管理

    RMAN 备份详解

    RMAN 还原与恢复

    RMANcatalog 的创建和使用

    基于catalog 创建RMAN存储脚本

基于catalog 的RMAN 备份与恢复

使用RMAN迁移文件系统数据库到ASM

    RMAN 备份路径困惑(使用plus archivelog时)

 

有关ORACLE故障请参考

    ORA-32004的错误处理

    ORA-01658错误

    CRS-0215错误处理

    ORA-00119，ORA-00132 错误处理

    又一例SPFILE设置错误导致数据库无法启动

    对参数FAST_START_MTTR_TARGET= 0 的误解及设定

    SPFILE错误导致数据库无法启动(ORA-01565)

 

有关ASM请参考

    创建ASM实例及ASM数据库

    ASM 磁盘、目录的管理

    使用 ASMCMD 工具管理ASM目录及文件

 

有关SQL/PLSQL请参考

    SQLPlus常用命令

    替代变量与SQL*Plus环境设置

    使用Uniread实现SQLplus翻页功能

    SQL 基础-->SELECT 查询

    SQL 基础--> NEW_VALUE的使用

    SQL 基础--> 集合运算(UNION与UNION ALL)

    SQL 基础--> 常用函数

    SQL 基础--> 视图(CREATEVIEW)

    SQL 基础--> 创建和管理表

    SQL 基础--> 多表查询

    SQL 基础--> 过滤和排序

    SQL 基础--> 子查询

    SQL 基础--> 分组与分组函数

    SQL 基础--> 层次化查询(STARTBY ... CONNECT BY PRIOR)

    SQL 基础--> ROLLUP与CUBE运算符实现数据汇总

    PL/SQL--> 游标

    PL/SQL--> 异常处理(Exception)

    PL/SQL--> 语言基础

    PL/SQL--> 流程控制

    PL/SQL--> PL/SQL记录

    PL/SQL--> 包的创建与管理

    PL/SQL--> 隐式游标(SQL%FOUND)

    PL/SQL--> 包重载、初始化

    PL/SQL--> DBMS_DDL包的使用

    PL/SQL--> DML 触发器

    PL/SQL--> INSTEAD OF 触发器

    PL/SQL--> 存储过程

    PL/SQL--> 函数

    PL/SQL--> 动态SQL

    PL/SQL--> 动态SQL的常见错误

 

有关ORACLE其它特性

    Oracle常用目录结构(10g)

    使用OEM,SQL*Plus,iSQL*Plus 管理Oracle实例

    日志记录模式(LOGGING、FORCE LOGGING 、NOLOGGING)

    表段、索引段上的LOGGING与NOLOGGING

    OralceOMF 功能详解

    Oracle用户、对象权限、系统权限  

    Oracle角色、配置文件

    Oracle分区表

    Oracle外部表

    使用外部表管理Oracle 告警日志(ALAERT_$SID.LOG)

    簇表及簇表管理(Index clustered tables)

    数据泵 EXPDP 导出工具的使用

    数据泵 IMPDP 导入工具的使用

    导入导出 Oracle 分区表数据

    SQL*Loader使用方法

    启用用户进程跟踪

    配置非默认端口的动态服务注册

    配置ORACLE 客户端连接到数据库

    systemsys，sysoper sysdba 的区别

    ORACLE_SID、DB_NAME、INSTANCE_NAME、DB_DOMIAN、GLOBAL_NAME

    Oracle补丁全集 (Oracle 9i 10g 11g Path)

    Oracle10.2.0.1 升级到10.2.0.4

    Oracle彻底 kill session