ORA-15096: lost disk write detected的解决与修复
案例背景
4节点Extend RAC，存储RAID 5校验异常，修复之后SOLDATA磁盘组无法mount，报错ORA-15096: lost disk write detected。
mount SOLDATA磁盘组ASM日志：
Fri Sep 25 00:31:57 2020
NOTE: GMON heartbeating for grp 2 (SOLDATA)
GMON querying group 2 at 5 for pid 27, osid 187323
Fri Sep 25 00:31:57 2020
NOTE: cache is mounting group SOLDATA created on 2019/04/12 15:10:32
NOTE: cache opening disk 0 of grp 2: SOLDATA_0000 path:/dev/emcpowerb
NOTE: group 2 (SOLDATA) high disk header ckpt advanced to fcn 0.714
NOTE: 09/25/20 00:31:57 SOLDATA.F1X0 found on disk 0 au 10 fcn 0.714 datfmt 2
NOTE: cache opening disk 1 of grp 2: SOLDATA_0001 path:/dev/emcpowerc
NOTE: cache opening disk 2 of grp 2: SOLDATA_0002 path:/dev/emcpowerd
NOTE: cache opening disk 3 of grp 2: SOLDATA_0003 path:/dev/emcpowerh
Fri Sep 25 00:31:57 2020
NOTE: cache mounting (first) external redundancy group 2/0xB82BB917 (SOLDATA)
Fri Sep 25 00:31:57 2020
* allocate domain 2, invalid = TRUE
kjbdomatt send to inst 2
Fri Sep 25 00:31:57 2020
NOTE: attached to recovery domain 2
Fri Sep 25 00:31:57 2020
NOTE: crash recovery of group SOLDATA will recover thread=1 ckpt=28.3507 domain=2 inc#=2 instnum=2
NOTE: crash recovery of group SOLDATA will recover thread=2 ckpt=39.8576 domain=2 inc#=4 instnum=1
NOTE: crash recovery of group SOLDATA will recover thread=3 ckpt=21.9043 domain=2 inc#=6 instnum=4
NOTE: crash recovery of group SOLDATA will recover thread=4 ckpt=22.6878 domain=2 inc#=12 instnum=3
* validated domain 2, flags = 0x0
NOTE: BWR validation signaled ORA-15096
Fri Sep 25 00:31:57 2020
Errors in file /u01/product/grid/crs/diag/asm/+asm/+ASM1/trace/+ASM1_ora_187323.trc:
ORA-15096: lost disk write detected
NOTE: crash recovery signalled OER-15096
ERROR: ORA-15096 signalled during mount of diskgroup SOLDATA查看ORA-15096的描述，官方提供的action还是比较悲观的。
[grid@lx1 ~]$ oerr ora 15096
15096, 0000, "lost disk write detected"
// *Cause: A failure either by disk hardware or disk software caused a disk
// write to to be lost, even though ASM received acknowledgement that
// the write completed. Alternatively, a clustering hardware failure
// or a clustering software failure resulted in an ASM instance
// believing that another ASM instance had crashed, when in fact it
// was still active.
// *Action: The disk group is corrupt and cannot be recovered. The disk group
// must be recreated, and its contents restored from backups.KFED读取4个thread的acd checkpoint分别为：
thread 1(inst_id 2) acdc:
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 7 ; 0x002: KFBTYP_ACDC
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 3 ; 0x008: file=3
kfbh.check: 2236936757 ; 0x00c: 0x8554f235
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfracdc.eyec[0]: 65 ; 0x000: 0x41
kfracdc.eyec[1]: 67 ; 0x001: 0x43
kfracdc.eyec[2]: 68 ; 0x002: 0x44
kfracdc.eyec[3]: 67 ; 0x003: 0x43
kfracdc.thread: 1 ; 0x004: 0x00000001
kfracdc.lastAba.seq: 4294967295 ; 0x008: 0xffffffff
kfracdc.lastAba.blk: 4294967295 ; 0x00c: 0xffffffff
kfracdc.blk0: 1 ; 0x010: 0x00000001
kfracdc.blks: 10751 ; 0x014: 0x000029ff
kfracdc.ckpt.seq: 28 ; 0x018: 0x0000001c
kfracdc.ckpt.blk: 3507 ; 0x01c: 0x00000db3
kfracdc.fcn.base: 5091911 ; 0x020: 0x004db247
kfracdc.fcn.wrap: 0 ; 0x024: 0x00000000
kfracdc.bufBlks: 256 ; 0x028: 0x00000100
kfracdc.strt112.seq: 2 ; 0x02c: 0x00000002
kfracdc.strt112.blk: 0 ; 0x030: 0x00000000thread 2(inst_id 1) acdc:
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 7 ; 0x002: KFBTYP_ACDC
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 10752 ; 0x004: blk=10752
kfbh.block.obj: 3 ; 0x008: file=3
kfbh.check: 3866731362 ; 0x00c: 0xe679a362
kfbh.fcn.base: 943 ; 0x010: 0x000003af
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfracdc.eyec[0]: 65 ; 0x000: 0x41
kfracdc.eyec[1]: 67 ; 0x001: 0x43
kfracdc.eyec[2]: 68 ; 0x002: 0x44
kfracdc.eyec[3]: 67 ; 0x003: 0x43
kfracdc.thread: 2 ; 0x004: 0x00000002
kfracdc.lastAba.seq: 4294967295 ; 0x008: 0xffffffff
kfracdc.lastAba.blk: 4294967295 ; 0x00c: 0xffffffff
kfracdc.blk0: 10753 ; 0x010: 0x00002a01
kfracdc.blks: 10751 ; 0x014: 0x000029ff
kfracdc.ckpt.seq: 39 ; 0x018: 0x00000027
kfracdc.ckpt.blk: 8576 ; 0x01c: 0x00002180
kfracdc.fcn.base: 5092052 ; 0x020: 0x004db2d4
kfracdc.fcn.wrap: 0 ; 0x024: 0x00000000
kfracdc.bufBlks: 256 ; 0x028: 0x00000100
kfracdc.strt112.seq: 2 ; 0x02c: 0x00000002
kfracdc.strt112.blk: 0 ; 0x030: 0x0000000thread 3(inst_id 4) acdc:
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 7 ; 0x002: KFBTYP_ACDC
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 21504 ; 0x004: blk=21504
kfbh.block.obj: 3 ; 0x008: file=3
kfbh.check: 1613963016 ; 0x00c: 0x60331f08
kfbh.fcn.base: 995 ; 0x010: 0x000003e3
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfracdc.eyec[0]: 65 ; 0x000: 0x41
kfracdc.eyec[1]: 67 ; 0x001: 0x43
kfracdc.eyec[2]: 68 ; 0x002: 0x44
kfracdc.eyec[3]: 67 ; 0x003: 0x43
kfracdc.thread: 3 ; 0x004: 0x00000003
kfracdc.lastAba.seq: 4294967295 ; 0x008: 0xffffffff
kfracdc.lastAba.blk: 4294967295 ; 0x00c: 0xffffffff
kfracdc.blk0: 21505 ; 0x010: 0x00005401
kfracdc.blks: 10751 ; 0x014: 0x000029ff
kfracdc.ckpt.seq: 21 ; 0x018: 0x00000015
kfracdc.ckpt.blk: 9043 ; 0x01c: 0x00002353
kfracdc.fcn.base: 5092244 ; 0x020: 0x004db394
kfracdc.fcn.wrap: 0 ; 0x024: 0x00000000
kfracdc.bufBlks: 256 ; 0x028: 0x00000100
kfracdc.strt112.seq: 2 ; 0x02c: 0x00000002
kfracdc.strt112.blk: 0 ; 0x030: 0x00000000thread 4(inst_id 3) acdc:
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 7 ; 0x002: KFBTYP_ACDC
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 32256 ; 0x004: blk=32256
kfbh.block.obj: 3 ; 0x008: file=3
kfbh.check: 587082185 ; 0x00c: 0x22fe29c9
kfbh.fcn.base: 1039 ; 0x010: 0x0000040f
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfracdc.eyec[0]: 65 ; 0x000: 0x41
kfracdc.eyec[1]: 67 ; 0x001: 0x43
kfracdc.eyec[2]: 68 ; 0x002: 0x44
kfracdc.eyec[3]: 67 ; 0x003: 0x43
kfracdc.thread: 4 ; 0x004: 0x00000004
kfracdc.lastAba.seq: 4294967295 ; 0x008: 0xffffffff
kfracdc.lastAba.blk: 4294967295 ; 0x00c: 0xffffffff
kfracdc.blk0: 32257 ; 0x010: 0x00007e01
kfracdc.blks: 10751 ; 0x014: 0x000029ff
kfracdc.ckpt.seq: 22 ; 0x018: 0x00000016
kfracdc.ckpt.blk: 6878 ; 0x01c: 0x00001ade
kfracdc.fcn.base: 5091930 ; 0x020: 0x004db25a
kfracdc.fcn.wrap: 0 ; 0x024: 0x00000000
kfracdc.bufBlks: 256 ; 0x028: 0x00000100
kfracdc.strt112.seq: 2 ; 0x02c: 0x00000002
kfracdc.strt112.blk: 0 ; 0x030: 0x00000000报错的trace可以看到是在做acd前滚recover的时候出现了异常，其实ORA-15096错误的原因就是在前滚过程中，ACD block描述的元数据块变更与实际元数据块不一致导致的。
*** 2020-09-25 03:33:39.996
kfdp_query: callcnt 23 grp 2 (SOLDATA)
NOTE: group 2 (SOLDATA) high disk header ckpt advanced to fcn 0.714
*** 2020-09-25 03:33:40.201
2020-09-25 03:33:40.201684 : Start recovery for domain=2, valid=0, flags=0x4
NOTE: crash recovery of group SOLDATA will recover thread=1 ckpt=28.3507 domain=2 inc#=2 instnum=2
NOTE: crash recovery of group SOLDATA will recover thread=2 ckpt=39.8576 domain=2 inc#=4 instnum=1
NOTE: crash recovery of group SOLDATA will recover thread=3 ckpt=21.9043 domain=2 inc#=6 instnum=4
NOTE: crash recovery of group SOLDATA will recover thread=4 ckpt=22.6878 domain=2 inc#=12 instnum=3
2020-09-25 03:33:40.232217 : Validate domain 2
2020-09-25 03:33:40.235370 : kjbvalidate: bcasted validate msg for domain=2
* kjbvalidate: validated domain 2, flags = 0x0
lost disk write detected during recovery:
fn=1 blk=303 last written kfcn: 0.5092245 BWR in thd=3 ABA 21.9044 --recover ACD thread 3 seq 21 block 9044时报错
mirror side: 0
OSM metadata block dump:
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 4 ; 0x002: KFBTYP_FILEDIR --报错的是303号文件的filedir
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 303 ; 0x004: blk=303
kfbh.block.obj: 1 ; 0x008: file=1
kfbh.check: 3027249708 ; 0x00c: 0xb4702a2c
kfbh.fcn.base: 5091095 ; 0x010: 0x004daf17
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfffdb.node.incarn: 1005593101 ; 0x000: A=1 NUMM=0x1df81106
kfffdb.node.frlist.number: 4294967295 ; 0x004: 0xffffffff
kfffdb.node.frlist.incarn: 0 ; 0x008: A=0 NUMM=0x0
kfffdb.hibytes: 0 ; 0x00c: 0x00000000
kfffdb.lobytes: 11776 ; 0x010: 0x00002e00
kfffdb.xtntcnt: 1 ; 0x014: 0x00000001
kfffdb.xtnteof: 1 ; 0x018: 0x00000001
kfffdb.blkSize: 512 ; 0x01c: 0x00000200
kfffdb.flags: 17 ; 0x020: O=1 S=0 S=0 D=0 C=1 I=0 R=0 A=0
kfffdb.fileType: 13 ; 0x021: 0x0d
....
recovery of group SOLDATA failed due to the following error(s):
ORA-15096: lost disk write detected
NOTE: messaging CKPT to quiesce pins Unix process pid: 175408, image: oracle@s189066 (TNS V1-V3)
kfdp_dismount(): callcnt 24 grp 2
----- Abridged Call Stack Trace -----
ksedsts()+244<-kfdp_dismountInt()+388<-kfdp_dismount()+21<-kfgTermCache()+452<-kfgRecoverDismount()+437<-kfgRecoverMount()+289<-kfgscDelete()+2514<-kss_del_cb()+257<-kssdel()+239<-kfgscFinalize()+1184<-kfgForEachKfgsc()+310<-kfgsoFinalize()+163<-kfgFinalize()+433
<-kfxdrvMount()+5067<-kfxdrvEntry()+2227<-opiexe()+22673<-opiosq0()+4534<-kpoal8()+1268<-opiodr()+1165<-ttcpip()+2699<-opitsk()+1740<-opiino()+945<-opiodr()+1165<-opidrv()+587<-sou2o()+145<-opimai_real()+154<-ssthrdmain()+412<-main()+236<-__libc_start_main()+245
----- End of Abridged Call Stack Trace -----
ASM name of disk 0x158e1d868 (2:0:SOLDATA_0000:/dev/emcpowerb) is being cleared
ASM name of disk 0x158e1d428 (2:1:SOLDATA_0001:/dev/emcpowerc) is being cleared
ASM name of disk 0x158e1d000 (2:2:SOLDATA_0002:/dev/emcpowerd) is being cleared
ASM name of disk 0x158e1c798 (2:3:SOLDATA_0003:/dev/emcpowerh) is being cleared
2020-09-25 03:33:40.534: [ CSSCLNT]clssgsgrppubdata: group (ocr_s-cluster) not found
kfx procr_get_online_conf [ocrret:0][configured:0][ebuf:]KFED读取Recover报错的ACD BLOCK：
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 8 ; 0x002: KFBTYP_CHNGDIR
kfbh.datfmt: 2 ; 0x003: 0x02
kfbh.block.blk: 30548 ; 0x004: blk=30548
kfbh.block.obj: 3 ; 0x008: file=3
kfbh.check: 3288383010 ; 0x00c: 0xc400be22
kfbh.fcn.base: 5092244 ; 0x010: 0x004db394
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfracdb2.aba.seq: 21 ; 0x000: 0x00000015
kfracdb2.aba.blk: 9043 ; 0x004: 0x00002353
kfracdb2.ents: 1 ; 0x008: 0x0001
kfracdb2.ub2spare: 0 ; 0x00a: 0x0000
kfracdb2.instNum: 4 ; 0x00c: 0x00000004
kfracdb2.timestamp: 1051980058 ; 0x010: 2020-9-24 16:40:58
kfracdb2.lge[0].valid: 1 ; 0x014: V=1 B=0 M=0
kfracdb2.lge[0].chgCount: 1 ; 0x015: 0x01
kfracdb2.lge[0].len: 56 ; 0x016: 0x0038
kfracdb2.lge[0].kfcn.base: 5092245 ; 0x018: 0x004db395
kfracdb2.lge[0].kfcn.wrap: 0 ; 0x01c: 0x00000000
kfracdb2.lge[0].bcd[0].kfbl.blk: 303 ; 0x020: blk=303
kfracdb2.lge[0].bcd[0].kfbl.obj: 1 ; 0x024: file=1
kfracdb2.lge[0].bcd[0].kfcn.base:5091095 ; 0x028: 0x004daf17
kfracdb2.lge[0].bcd[0].kfcn.wrap: 0 ; 0x02c: 0x00000000
kfracdb2.lge[0].bcd[0].oplen: 8 ; 0x030: 0x0008
kfracdb2.lge[0].bcd[0].blkIndex: 47 ; 0x032: 0x002f
kfracdb2.lge[0].bcd[0].flags: 28 ; 0x034: F=0 N=0 F=1 L=1 V=1 A=0 C=0
kfracdb2.lge[0].bcd[0].opcode: 135 ; 0x036: 0x0087
kfracdb2.lge[0].bcd[0].kfbtyp: 4 ; 0x038: KFBTYP_FILEDIR
kfracdb2.lge[0].bcd[0].redund: 17 ; 0x039: SCHE=0x1 NUMB=0x1
kfracdb2.lge[0].bcd[0].pad: 63903 ; 0x03a: 0xf99f
kfracdb2.lge[0].bcd[0].KFFFD_COMMIT.modts.hi:33105680 ; 0x03c: HOUR=0x10 DAYS=0x18 MNTH=0x9 YEAR=0x7e4
kfracdb2.lge[0].bcd[0].KFFFD_COMMIT.modts.lo:0 ; 0x040: USEC=0x0 MSEC=0x0 SECS=0x0 MINS=0x0
kfracdb2.lge[0].bcd[0].au[0]: 21 ; 0x044: 0x00000015
kfracdb2.lge[0].bcd[0].disks[0]: 1 ; 0x048: 0x0001
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 8 ; 0x002: KFBTYP_CHNGDIR
kfbh.datfmt: 2 ; 0x003: 0x02
kfbh.block.blk: 30549 ; 0x004: blk=30549
kfbh.block.obj: 3 ; 0x008: file=3
kfbh.check: 3308033219 ; 0x00c: 0xc52c94c3
kfbh.fcn.base: 5092245 ; 0x010: 0x004db395
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfracdb2.aba.seq: 21 ; 0x000: 0x00000015
kfracdb2.aba.blk: 9044 ; 0x004: 0x00002354
kfracdb2.ents: 1 ; 0x008: 0x0001
kfracdb2.ub2spare: 0 ; 0x00a: 0x0000
kfracdb2.instNum: 4 ; 0x00c: 0x00000004
kfracdb2.timestamp: 1051980105 ; 0x010: 2020-9-24 16:41:45
kfracdb2.lge[0].valid: 3 ; 0x014: V=1 B=1 M=0
kfracdb2.lge[0].chgCount: 1 ; 0x015: 0x01
kfracdb2.lge[0].len: 64 ; 0x016: 0x0040
kfracdb2.lge[0].kfcn.base: 5092246 ; 0x018: 0x004db396
kfracdb2.lge[0].kfcn.wrap: 0 ; 0x01c: 0x00000000
kfracdb2.lge[0].bcd[0].kfbl.blk: 0 ; 0x020: blk=0
kfracdb2.lge[0].bcd[0].kfbl.obj: 0 ; 0x024: file=0
kfracdb2.lge[0].bcd[0].kfcn.base: 0 ; 0x028: 0x00000000
kfracdb2.lge[0].bcd[0].kfcn.wrap: 0 ; 0x02c: 0x00000000
kfracdb2.lge[0].bcd[0].oplen: 24 ; 0x030: 0x0018
kfracdb2.lge[0].bcd[0].blkIndex: 0 ; 0x032: 0x0000
kfracdb2.lge[0].bcd[0].flags: 0 ; 0x034: F=0 N=0 F=0 L=0 V=0 A=0 C=0
kfracdb2.lge[0].bcd[0].opcode: 4 ; 0x036: 0x0004
kfracdb2.lge[0].bcd[0].kfbtyp: 0 ; 0x038: KFBTYP_INVALID --这里明显错了
kfracdb2.lge[0].bcd[0].redund: 0 ; 0x039: SCHE=0x0 NUMB=0x0
kfracdb2.lge[0].bcd[0].pad: 63903 ; 0x03a: 0xf99f
kfracdb2.lge[0].bcd[0].KFR_BWR1.bwr.kfbl.blk:303 ; 0x03c: blk=303
kfracdb2.lge[0].bcd[0].KFR_BWR1.bwr.kfbl.obj:1 ; 0x040: file=1
kfracdb2.lge[0].bcd[0].KFR_BWR1.bwr.kfcn.base:5092245 ; 0x044: 0x004db395
kfracdb2.lge[0].bcd[0].KFR_BWR1.bwr.kfcn.wrap:0 ; 0x048: 0x00000000
kfracdb2.lge[0].bcd[0].KFR_BWR1.blkIdx:47 ; 0x04c: 0x002f
kfracdb2.lge[0].bcd[0].KFR_BWR1.disk1:1 ; 0x04e: 0x0001
kfracdb2.lge[0].bcd[0].KFR_BWR1.au1: 21 ; 0x050: 0x00000015KFED读取Recover报错的元数据Block：
kfbh.endian: 1 ; 0x000: 0x01
kfbh.hard: 130 ; 0x001: 0x82
kfbh.type: 4 ; 0x002: KFBTYP_FILEDIR
kfbh.datfmt: 1 ; 0x003: 0x01
kfbh.block.blk: 303 ; 0x004: blk=303
kfbh.block.obj: 1 ; 0x008: file=1
kfbh.check: 3027249708 ; 0x00c: 0xb4702a2c
kfbh.fcn.base: 5091095 ; 0x010: 0x004daf17
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
kfffdb.node.incarn: 1005593101 ; 0x000: A=1 NUMM=0x1df81106
kfffdb.node.frlist.number: 4294967295 ; 0x004: 0xffffffff
kfffdb.node.frlist.incarn: 0 ; 0x008: A=0 NUMM=0x0
kfffdb.hibytes: 0 ; 0x00c: 0x00000000
kfffdb.lobytes: 11776 ; 0x010: 0x00002e00
kfffdb.xtntcnt: 1 ; 0x014: 0x00000001
kfffdb.xtnteof: 1 ; 0x018: 0x00000001
kfffdb.blkSize: 512 ; 0x01c: 0x00000200
kfffdb.flags: 17 ; 0x020: O=1 S=0 S=0 D=0 C=1 I=0 R=0 A=0
kfffdb.fileType: 13 ; 0x021: 0x0d
kfffdb.dXrs: 17 ; 0x022: SCHE=0x1 NUMB=0x1
kfffdb.iXrs: 17 ; 0x023: SCHE=0x1 NUMB=0x1
kfffdb.dXsiz[0]: 4294967295 ; 0x024: 0xffffffff
kfffdb.dXsiz[1]: 0 ; 0x028: 0x00000000
kfffdb.dXsiz[2]: 0 ; 0x02c: 0x00000000
kfffdb.iXsiz[0]: 4294967295 ; 0x030: 0xffffffff
kfffdb.iXsiz[1]: 0 ; 0x034: 0x00000000
kfffdb.iXsiz[2]: 0 ; 0x038: 0x00000000
kfffdb.xtntblk: 1 ; 0x03c: 0x0001
kfffdb.break: 60 ; 0x03e: 0x003c
kfffdb.priZn: 0 ; 0x040: KFDZN_COLD
kfffdb.secZn: 0 ; 0x041: KFDZN_COLD
kfffdb.ub2spare: 0 ; 0x042: 0x0000
kfffdb.alias[0]: 1378 ; 0x044: 0x00000562
kfffdb.alias[1]: 4294967295 ; 0x048: 0xffffffff
kfffdb.strpwdth: 1 ; 0x04c: 0x01
kfffdb.strpsz: 20 ; 0x04d: 0x14
kfffdb.usmsz: 0 ; 0x04e: 0x0000
kfffdb.crets.hi: 33083859 ; 0x050: HOUR=0x13 DAYS=0xe MNTH=0x4 YEAR=0x7e3
kfffdb.crets.lo: 1679484928 ; 0x054: USEC=0x0 MSEC=0x2ba SECS=0x1 MINS=0x19
kfffdb.modts.hi: 33105590 ; 0x058: HOUR=0x16 DAYS=0x15 MNTH=0x9 YEAR=0x7e4
kfffdb.modts.lo: 0 ; 0x05c: USEC=0x0 MSEC=0x0 SECS=0x0 MINS=0x0
kfffdb.dasz[0]: 0 ; 0x060: 0x00
kfffdb.dasz[1]: 0 ; 0x061: 0x00
kfffdb.dasz[2]: 0 ; 0x062: 0x00
kfffdb.dasz[3]: 0 ; 0x063: 0x00
kfffdb.permissn: 0 ; 0x064: 0x00
kfffdb.ub1spar1: 0 ; 0x065: 0x00
kfffdb.ub2spar2: 0 ; 0x066: 0x0000
kfffdb.user.entnum: 0 ; 0x068: 0x0000
kfffdb.user.entinc: 0 ; 0x06a: 0x0000
kfffdb.group.entnum: 0 ; 0x06c: 0x0000
kfffdb.group.entinc: 0 ; 0x06e: 0x0000
kfffdb.paswdblk: 0 ; 0x070: 0x00000000
kfffdb.mdbid: 0 ; 0x074: 0x00000000
kfffdb.pdbid: 0 ; 0x078: 0x00000000
kfffdb.pfno: 0 ; 0x07c: 0x00000000
kfffdb.ndeps: 0 ; 0x080: 0x0000
kfffdb.pgnam: ; 0x082: length=0
kfffdb.usm: ; 0x0a0: length=0
kfffdb.fmtBlks: 0 ; 0x49c: 0x00000000
kfffde[0].xptr.au: 2829 ; 0x4a0: 0x00000b0d
...明白了来龙去脉之后修复方法有两种：
修改thread 3 acd checkpoint；
修改303号文件filedir的fscn。
关于作者
李翔宇，云和恩墨西区交付技术顾问，长期服务移动运营商行业客户，熟悉Oracle性能优化，故障诊断，特殊恢复。
今年的数据技术嘉年华大会上，李翔宇老师将带来题为《在通过案例深入解析Oracle内部原理》的演讲，与大家一起探索CBO和ASM rebalance的一些内部机制，精彩不容错过！
更多数据库行业相关内容，欢迎光临 2021 数据技术嘉年华 ：https://www.modb.pro/dtc2021（扫描下方二维码免费领取大会门票）
END
推荐阅读：267页！2020年度数据库技术年刊
推荐下载：2020数据技术嘉年华PPT下载
2020数据技术嘉年华近50个PPT下载、视频回放已上传墨天轮平台，可在“数据和云”公众号回复关键词“2020DTC”获得！
你知道吗？我们的视频号里已经发布了很多精彩的内容，快去看看吧！↓↓↓
点击下图查看更多 ↓
云和恩墨大讲堂 | 一个分享交流的地方
长按，识别二维码，加入万人交流社群
请备注：云和恩墨大讲堂
点个“在看”
你的喜欢会被看到❤