FAILED TO READ MORROR SIDE 1

FAILED TO READ MORROR SIDE 1

问题描述：

环境：
RAC p740 p550
-bash-3.2$ srvctl status database -d kmis

Instance kmis1 is running on node p550
Instance kmis2 is not running on node p740


AIX
-bash-3.2$ oslevel -r
6100-08

数据库 oracle 11.2.2

$ srvctl start database -d kmis

PRCR-1079 : Failed to start resource ora.kmis.db

CRS-5017: The resource action "ora.ARCHDG.dg start" encountered the following error:

ORA-15032: not all alterations performed

ORA-15017: diskgroup "ARCHDG" cannot be mounted

ORA-15063: ASM discovered an insufficient number of disks for diskgroup "ARCHDG"

ORA-15080: synchronous I/O operation to a disk failed

CRS-2674: Start of 'ora.ARCHDG.dg' on 'p740' failed

CRS-2632: There are no more servers to try to place resource 'ora.kmis.db' on that would satisfy its placement policy

$

$ srvctl start instance -d kmis -i kmis2 -o open;

PRCR-1013 : Failed to start resource ora.kmis.db

PRCR-1064 : Failed to start resource ora.kmis.db on node p740

CRS-5017: The resource action "ora.ARCHDG.dg start" encountered the following error:

ORA-15032: not all alterations performed

ORA-15017: diskgroup "ARCHDG" cannot be mounted

ORA-15063: ASM discovered an insufficient number of disks for diskgroup "ARCHDG"

ORA-15080: synchronous I/O operation to a disk failed

1

p550

ASMCMD> ls -lState    Type    Rebal  NameMOUNTED  EXTERN  N      ARCHDG/MOUNTED  NORMAL  N      CRS/MOUNTED  EXTERN  N      DATADG/ASMCMD> host

p740上

MOUNTED  NORMAL  N      CRS/MOUNTED  EXTERN  N      DATADG/挂载不上 ARCDG

一挂载就报错

odm：

Bug 10422126 : FAILED TO READ MORROR SIDE 1	转到底部

Bug 属性

类型	B - Defect	已在产品版本中修复	12.1
严重性	2 - Severe Loss of Service	产品版本	11.2.0.1
状态	90 - Closed, Verified by Filer	平台	212 - IBM AIX on POWER Systems (64-bit)
创建时间	2010-12-22	平台版本	5.3
更新时间	2014-3-19	基本 Bug	N/A
数据库版本	11.2.0.1	影响平台	Generic
产品源	Oracle

相关产品

产品线	Oracle Database Products	系列	Oracle Database Suite
区域	Oracle Database	产品	5 - Oracle Database - Enterprise Edition

Hdr: 10422126 11.2.0.1 RDBMS 11.2.0.1 ASM PRODID-5 PORTID-212
Abstract: FAILED TO READ MORROR SIDE 1
 
*** 12/22/10 03:46 am ***
----
 
PROBLEM:
--------
Alert log spooling this error continously and filling up available space.
Here is the message.
-----------------------
WARNING: failed to read mirror side 1 of virtual extent 665 logical extent 0 
of file 280 in group [1.445758315] from disk ASM_ESBQA001_DATA1_0005 
allocation unit 75811 reason error; if possible,will try another mirror side
-----------------------
 
When the problem starts, errors spin in the alert log until disk space is 
exausted. We have found that the process that is refrenced in the trace is 
usually an orphaned ebiz form.
 
Recent Changes - Upgrade to 11.2.0.1
 
Over the weekend we have upgraded Production Database to 11.2.0.1.  We've had 
2 occurrence of the issue on 2 of the RAC nodes.
 
The issue had happened in QA and we have tried to drop the disk and add new 
disk again, but it did not solve the problem.
 
One more thing to note is, when this happens, there was a "run-away" process 
that also generates the same error message in a trace file. As long as that 
session/process is killed, the error messages stops.  Please investigate.
 
DIAGNOSTIC ANALYSIS:
--------------------
This looks closely related to bug.10051315 and bug.10055063
But customer has applied these patches already, even then we still see the 
same warning messages.
 
We are still trying to get information from customer to understand how that 
session is being killed(not sure could be from database end --or application 
end).
 
Customer confirmed the following :
---------------
1.) Are they in-house scripts scheduled in database/os which are going to 
kill the inactive sessions at database/os level ?
-- There are not any scripts scheduled to kill sessions.  When this error 
starts happening, we kill the process manually.
 
2.) Is this happening only during specific intervals.
-- I cannot see a pattern as to when it is happening.  It seems random.
 
3.) Please confirm - the ASM instance is functional (one can connect to it, 
run queries on the various v$asm% views, like: v$asm_diskgroup, v$asm_disk)
-- Yes, ASM is functional during the error.  All other sessions continue to 
run without errors.
---------------
 
WORKAROUND:
-----------
As long as that session/process is killed, the error messages stops.
 
RELATED BUGS:
-------------
bug.10051315 and bug.10055063
 
REPRODUCIBILITY:
----------------
Its random.
 
TEST CASE:
----------
NA
 
STACK TRACE:
------------
Function List   (to Full stack)   (to Summary stack)
 
     ksedsts <- ksdxfstk <- ksdxcb <- sspuser <- f0
        <- sdbgrfucf_close_file <- sdbgrfcf_close_file <- dbgrlWriteTextOnly 
<- dbgrlWriteAlertDetail_int <- dbgrlWriteAlertDetail
         <- dbgrlWriteAlertText <- dbkrlWriteAlertText <- dbkrlPril <- ksdfmw 
<- ksdwrt
          <- kfioCompleteIO <- kfioWaitIO <- kfioRequestPriv <- kfioRequest 
<- ksfd_kfioRequest
           <- ksfd_osmio <- ksfd_io <- ksfdread <- kcfrbd1 <- kcbzib
            <- kcbgcur <- kturRollbackToSavepoint <- ktcrsp1 <- ksures <- 
ksuxds
             <- kssdel <- kssdch <- ksudlp <- opidcl <- opidrv
              <- sou2o <- opimai_real <- ssthrdmain <- main
 
SUPPORTING INFORMATION:
-----------------------
Customer’s Management name:      Al Fischer
Customer's Management 24x7 contact number:      +1-616-648-6244
Customer’s Management email:      [email protected]
Primary Customer Email address:      eric.lai
Secondary Contact Name:      George Zhang
Secondary Contact Email:      (EMAIL ADDRESS)  [email protected]
 
24 HOUR CONTACT INFORMATION FOR P1 BUGS:
----------------------------------------
 
DIAL-IN INFORMATION:
--------------------
 
IMPACT DATE:
------------
" system at risk of crashing as alert logging is filling up; causing forms to 
hang at Call Center"
 
*** 12/22/10 03:57 am ***
*** 12/22/10 04:20 am ***
*** 12/22/10 04:21 am ***
*** 12/22/10 04:21 am *** (CHG: Sta->16)
*** 12/22/10 04:33 am ***
*** 12/22/10 05:35 am ***
*** 12/22/10 06:08 am ***
*** 12/22/10 06:09 am ***
*** 12/22/10 06:10 am ***
*** 12/22/10 08:06 am ***
*** 12/22/10 11:25 am *** 
*** 12/22/10 11:25 am ***
*** 12/22/10 01:47 pm *** (CHG: Sta->10)
*** 12/22/10 01:47 pm ***
*** 12/22/10 07:46 pm ***
*** 12/22/10 07:58 pm ***
File versions form GI & RDBMS homes.
 
Below is the info from GI_HOME.
 
[usax02w] $ ar -t -v -X32_64 $GI_HOME/lib/libserver11.a kfd.o
rw-rw-rw- 10000/1026  393236 Nov 15 13:29 2010 kfd.o
[usax02w] $ ar -t -v -X32_64 $GI_HOME/lib/libserver11.a kfdp.o
rw-rw-rw- 10000/1026  271741 Nov 15 13:29 2010 kfdp.o
[usax02w] $ ar -t -v -X32_64 $GI_HOME/lib/libserver11.a kfkio.o
rw-rw-rw- 10000/1026  103575 Nov 15 13:29 2010 kfkio.o
[usax02w] $ ar -t -v -X32_64 $GI_HOME/lib/libserver11.a kfksubs.o
rw-rw-rw- 10000/1026  166215 Nov 15 13:29 2010 kfksubs.o
 
 
 Below is the info from RDBMS_HOME
 
[usax02w] $ echo $ORACLE_HOME
/u01/apps/oracle/product/11.2.0/db_1
[usax02w] $ ar -t -v -X32_64 $ORACLE_HOME/lib/libserver11.a kfd.o
rw-rw-rw- 10000/1026  393236 Nov 15 13:29 2010 kfd.o
[usax02w] $ ar -t -v -X32_64 $ORACLE_HOME/lib/libserver11.a kfdp.o
rw-rw-rw- 10000/1026  271741 Nov 15 13:29 2010 kfdp.o
[usax02w] $ ar -t -v -X32_64 $ORACLE_HOME/lib/libserver11.a kfkio.o
rw-rw-rw- 10000/1026  103575 Nov 15 13:29 2010 kfkio.o
[usax02w] $ ar -t -v -X32_64 $ORACLE_HOME/lib/libserver11.a kfksubs.o
rw-rw-rw- 10000/1026  166215 Nov 15 13:29 2010 kfksubs.o
 
 
 
Regards,
Prathap
*** 12/22/10 08:01 pm *** (CHG: Sta->16)
*** 12/22/10 08:25 pm ***
*** 12/23/10 01:48 am ***
*** 12/23/10 02:46 am ***
*** 12/23/10 06:25 am *** (CHG: Sta->11)
*** 12/23/10 06:25 am ***
*** 12/23/10 06:26 am *** 
*** 12/23/10 06:26 am ***
*** 12/24/10 01:43 am *** 
*** 12/24/10 01:43 am ***
*** 12/27/10 11:33 am ***
*** 12/27/10 11:45 am ***
*** 12/27/10 11:51 am ***
*** 12/27/10 12:22 pm *** ESCALATED
*** 12/27/10 12:22 pm ***
*** 12/27/10 12:29 pm ***
*** 12/28/10 12:42 am *** (CHG: Sta->30)
*** 12/28/10 12:42 am ***
*** 12/28/10 05:00 am *** (CHG: Sta->16)
*** 12/28/10 05:00 am ***
*** 12/28/10 05:08 am *** (CHG: Sta->11)
*** 12/28/10 09:37 am ***
*** 12/29/10 02:51 am ***
*** 12/31/10 03:05 am ***
*** 01/02/11 08:28 pm ***
*** 01/04/11 01:43 am *** (CHG: Sta->30)
*** 01/04/11 01:43 am ***
*** 01/04/11 01:54 am ***
*** 01/04/11 04:50 am *** (CHG: Sta->11)
*** 01/04/11 04:50 am ***
*** 01/05/11 01:26 am *** (CHG: Sta->30)
*** 01/05/11 01:26 am ***
*** 01/10/11 10:46 am ***
*** 01/10/11 10:46 am *** (CHG: Sta->11)
*** 01/11/11 03:14 am ***
*** 01/13/11 11:32 pm ***
RELEASE NOTES:
]]  kfk SO is deleted, while there are still allocated sessions.
 
 
REDISCOVERY INFORMATION:
 The following error, even with fix 10051315:
 
 WARNING: IO Failed. group:1 disk(number.incarnation):5.0xe0d14bb7 
 disk_path:/dev/ASM_1457_4 
          AU:75811 disk_offset(bytes):79494299648 io_size:8192 operation:Read 
 type:synchronous 
          result:I/O error process_id:2338940 
          subsys:System iop:0x110939310 bufp:0x70000016f2ec000 
 osderr:0x434c5344 osderr1:0x0
WORKAROUND:
None
*** 01/13/11 11:44 pm *** 
*** 01/13/11 11:44 pm *** 
*** 01/13/11 11:44 pm *** (CHG: Sta->80)
*** 01/16/11 11:05 pm ***
*** 01/16/11 11:17 pm ***
*** 01/16/11 11:36 pm ***
*** 01/17/11 12:01 am ***
*** 01/17/11 07:09 am ***
*** 01/17/11 07:11 am ***
*** 01/17/11 09:25 am ***
*** 01/17/11 10:16 am ***
*** 01/17/11 11:14 pm ***
*** 01/18/11 05:06 am ***
*** 01/18/11 11:47 am ***
*** 01/18/11 11:52 pm ***
*** 01/19/11 02:05 am ***
*** 01/19/11 05:39 am ***
*** 01/19/11 05:45 am ***
*** 01/19/11 07:43 am ***
*** 01/19/11 08:44 am ***
*** 01/19/11 06:12 pm ***
*** 01/19/11 06:14 pm ***
*** 01/19/11 10:53 pm ***
*** 01/20/11 12:11 am ***
*** 01/20/11 04:04 am ***
*** 01/20/11 06:11 am ***
*** 01/20/11 06:59 am ***
*** 01/20/11 09:13 am ***
*** 01/20/11 10:44 am ***
*** 01/20/11 10:44 am ***
*** 01/20/11 11:00 am ***
*** 01/20/11 11:38 am ***
*** 01/20/11 04:58 pm ***
*** 01/21/11 04:05 am ***
*** 01/24/11 11:33 am ***
*** 01/25/11 02:17 am ***
*** 01/27/11 02:58 am ***
*** 01/27/11 03:16 am ***
*** 01/27/11 03:37 am ***
*** 01/27/11 03:49 am ***
*** 02/02/11 04:37 am ***
*** 02/04/11 08:25 pm ***
*** 02/05/11 02:46 am ***
*** 02/05/11 02:58 am ***
*** 02/05/11 02:59 am ***
*** 02/05/11 02:29 pm ***
*** 02/05/11 07:19 pm ***
*** 02/05/11 09:37 pm ***
*** 02/07/11 01:37 am ***
*** 02/07/11 06:14 am ***
*** 02/07/11 12:35 pm ***
*** 02/07/11 06:46 pm ***
*** 02/07/11 08:33 pm ***
*** 02/08/11 01:52 pm ***
*** 02/08/11 02:42 pm ***
*** 02/08/11 10:32 pm ***
*** 02/09/11 11:51 pm ***
*** 02/10/11 08:56 am ***
*** 02/10/11 08:59 am ***
*** 02/10/11 09:30 am ***
*** 02/11/11 11:52 pm ***
*** 02/12/11 05:10 pm ***
*** 02/15/11 02:55 am ***
*** 02/15/11 10:13 am ***
*** 02/16/11 01:07 am ***
*** 02/16/11 06:34 am ***
*** 02/18/11 01:39 am ***
*** 02/18/11 01:40 am ***
*** 02/24/11 07:20 am ***
*** 02/24/11 07:35 am ***
*** 02/25/11 12:34 am ***
*** 03/02/11 07:59 am ***
*** 03/02/11 08:25 am ***
*** 03/03/11 01:22 am ***
*** 03/03/11 03:34 pm ***
*** 03/07/11 11:54 am ***
*** 03/08/11 08:53 pm ***
*** 03/09/11 10:44 pm ***
*** 04/20/11 01:17 pm ***
*** 04/21/11 09:34 am ***
*** 04/21/11 01:46 pm ***
*** 04/22/11 03:47 pm ***
*** 04/29/11 12:33 pm ***
*** 04/29/11 01:21 pm ***
*** 05/02/11 07:53 am ***
*** 05/03/11 01:12 am ***
*** 05/03/11 01:37 am ***
*** 05/03/11 02:11 am ***
*** 05/11/11 03:00 am ***
*** 05/11/11 03:44 am ***
*** 05/16/11 08:07 am ***
*** 05/17/11 11:31 am ***
*** 05/17/11 11:34 am ***
*** 05/17/11 03:14 pm ***
*** 05/17/11 03:16 pm ***
*** 05/18/11 06:00 am ***
*** 05/18/11 06:11 am ***
*** 05/18/11 08:46 am ***
*** 06/09/11 12:44 pm ***
*** 06/10/11 10:21 am ***
*** 06/10/11 10:21 am ***
*** 06/11/11 04:44 am ***
*** 06/14/11 10:14 pm ***
*** 06/17/11 12:33 pm ***
*** 07/04/11 01:09 pm ***
*** 07/06/11 04:03 pm ***
*** 07/06/11 04:56 pm ***
*** 07/08/11 11:47 am ***
*** 07/08/11 12:16 pm ***
*** 07/12/11 07:28 am ***
*** 07/14/11 07:58 pm *** 
*** 07/17/11 05:29 am ***
*** 07/17/11 05:30 am ***
*** 07/18/11 03:23 pm ***
*** 07/26/11 03:20 pm ***
*** 07/26/11 03:23 pm ***
*** 07/27/11 03:13 am ***
*** 07/27/11 03:15 am ***
*** 07/27/11 07:33 am ***
*** 08/02/11 01:34 pm ***
*** 08/03/11 01:03 am *** 
*** 08/03/11 01:03 am ***
*** 08/03/11 01:15 am *** 
*** 08/03/11 01:15 am ***
*** 08/05/11 08:00 am ***
*** 08/05/11 08:11 am ***
*** 08/05/11 10:42 am ***
*** 08/11/11 01:22 am ***
*** 08/12/11 04:00 am ***
*** 08/12/11 10:03 am ***
*** 08/18/11 12:46 pm ***
*** 08/22/11 12:14 pm ***
*** 09/07/11 04:49 am ***
*** 09/09/11 02:55 pm ***
*** 09/14/11 02:40 pm ***
*** 10/14/11 02:15 am ***
*** 10/14/11 02:15 am *** ESCALATION -> CLOSED
*** 10/14/11 02:15 am *** (CHG: Sta->90)
*** 10/14/11 02:21 am *** 
*** 10/14/11 02:21 am ***
*** 10/14/11 09:56 am ***
*** 11/01/11 01:55 am ***
*** 11/01/11 10:49 am ***
*** 11/01/11 10:49 am ***
*** 11/27/11 01:15 am ***
*** 12/01/11 09:11 am ***
*** 01/17/12 03:29 pm ***
*** 01/23/12 08:26 pm ***
*** 01/23/12 08:26 pm ***
*** 03/14/12 01:07 am ***
*** 03/18/12 11:22 pm ***
*** 03/22/12 12:44 am ***
*** 04/02/12 06:10 am ***
*** 04/24/12 10:45 am ***
*** 01/26/13 08:03 am ***
*** 01/31/13 07:35 am ***
*** 02/24/13 11:27 pm ***
*** 04/16/13 02:14 am ***
*** 04/19/13 07:23 am ***
*** 05/12/13 09:12 pm ***
*** 05/20/13 06:28 am ***
*** 06/25/13 01:16 am ***
*** 07/08/13 09:41 pm ***
*** 07/09/13 09:44 pm ***
*** 07/09/13 10:24 pm ***
*** 07/17/13 03:02 am ***
*** 07/18/13 02:35 am ***
*** 08/12/13 03:54 am ***
*** 08/20/13 12:21 am ***
*** 09/30/13 12:26 pm ***
*** 10/23/13 05:08 pm ***
*** 10/29/13 08:22 am ***
*** 10/29/13 09:56 am ***
*** 10/29/13 09:56 am ***
*** 12/18/13 03:38 am ***
*** 01/23/14 09:24 pm ***
*** 02/11/14 09:58 am ***
*** 02/24/14 03:11 am ***
*** 02/24/14 06:02 am ***
*** 02/24/14 12:34 pm ***
*** 02/24/14 03:58 pm ***
*** 02/25/14 02:28 am ***
*** 03/18/14 04:34 am ***
*** 03/18/14 04:55 am ***
*** 03/18/14 05:09 am ***
*** 03/18/14 10:37 pm ***
*** 03/19/14 12:39 am ***