WARNING: failed to read mirror side 1 of virtual extent 229 logical extent
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent
-bash-3.2$ cat /etc/release
Oracle Solaris 10 8/11 s10s_u10wos_17b SPARC
Copyright (c) 1983, 2011, Oracle and/or its affiliates. All rights reserved.
Assembled 23 August 2011
-bash-3.2$
SQL> select * from V$version;
BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
PL/SQL Release 11.2.0.2.0 - Production
CORE 11.2.0.2.0 Production
TNS for Solaris: Version 11.2.0.2.0 - Production
NLSRTL Version 11.2.0.2.0 - Production
solaris10 系统，oracle 11g RAC 警告日志里不停的报下面的错误：
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
WARNING: failed to read mirror side 1 of virtual extent 229 logical extent 0 of file 271 in group [5.973205957] from disk SYS_0000 allocation unit 155234 reason error; if possible,will try another mirror side
Errors in file /oracle/PROD/diag/rdbms/prod9/PROD91/trace/PROD91_ora_6910.trc:
WARNING: Read Failed. group:5 disk:0 AU:155234 offset:811008 size:8192
查看 PROD92_ora_12246.trc文件内容如下：
-bash-3.2$ tail -200 PROD92_ora_12246.trc
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 2020 logical extent 0 of file 266 in group [5.973115434] from disk SYS_0000 allocation unit 146775 reason error; if possible,will try another mirror side
WARNING: Read Failed. group:5 disk:0 AU:146775 offset:57344 size:8192
path:/dev/rdsk/c5t600144F08176764B00004FEABD4E0003d0s6
incarnation:0xf0f060f0 synchronous result:'I/O error'
subsys:System iop:0xffffffff7b4f4000 bufp:0x549584000 osderr:0x434c5344 osderr1:0x0
*** DUMP FILE SIZE IS LIMITED TO 10485760 BYTES ***
查看solaris系统日志如下：
Jul 21 12:46:52 rac2 last message repeated 1 time
Jul 21 12:57:49 rac2 CLSD: [ID 770310 daemon.notice] The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous wi
th the mean cluster time.
Jul 21 13:08:46 rac2 last message repeated 1 time
Jul 21 13:19:51 rac2 CLSD: [ID 770310 daemon.notice] The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous wi
th the mean cluster time.
Jul 21 13:31:04 rac2 last message repeated 1 time
Jul 21 13:37:44 rac2 fctl: [ID 517869 kern.warning] WARNING: fp(6)::GPN_ID for D_ID=10000 failed
Jul 21 13:37:44 rac2 fctl: [ID 517869 kern.warning] WARNING: fp(6)::N_x Port with D_ID=10000, PWWN=21000024ff04c87f disappeared from fabric
Jul 21 13:39:54 rac2 fctl: [ID 517869 kern.warning] WARNING: fp(6)::N_x Port with D_ID=10000, PWWN=21000024ff04c87f reappeared in fabric
Jul 21 13:41:52 rac2 CLSD: [ID 770310 daemon.notice] The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous wi
th the mean cluster time.
Jul 21 13:52:41 rac2 last message repeated 1 time
Jul 21 14:03:22 rac2 CLSD: [ID 770310 daemon.notice] The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous wi
th the mean cluster time.
Jul 21 14:14:19 rac2 last message repeated 1 time
Jul 21 14:25:08 rac2 CLSD: [ID 770310 daemon.notice] The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous wi
th the mean cluster time.
Jul 21 14:35:57 rac2 last message repeated 1 time
导致在22日下午5点多，系统被占满：
Jul 22 17:13:02 rac2 CLSD: [ID 770310 daemon.notice] The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous wi
th the mean cluster time.
Jul 22 17:23:59 rac2 last message repeated 1 time
Jul 22 17:30:32 rac2 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Jul 22 17:34:45 rac2 last message repeated 63 times
Jul 22 17:34:48 rac2 CLSD: [ID 770310 daemon.notice] The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous wi
th the mean cluster time.
Jul 22 17:34:49 rac2 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Jul 22 17:35:17 rac2 last message repeated 7 times
Jul 22 17:35:21 rac2 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Jul 22 17:41:58 rac2 last message repeated 99 times
Jul 22 17:42:02 rac2 ufs: [ID 845546 kern.notice] NOTICE: alloc: /: file system full
Huge number of alert log entries: 'WARNING: IO Failed...' 'WARNING: failed to read mirror side 1 of virtual extent ...' [ID 1274852.1]
Database instance throwing large number of entries in the alert log:
WARNING: IO Failed. group:1 disk(number.incarnation):118.0x2ce5a485
disk_path:/dev/rhdiskasm175
AU:40573 disk_offset(bytes):42543947776 io_size:8192 operation:Read
type:synchronous
result:I/O error process_id:2658412
subsys:System iop:0x110ac00a0 bufp:0x700000296c4e000
osderr:0x434c5344 osderr1:0x0
WARNING: failed to read mirror side 1 of virtual extent 7044 logical extent 0
of file 259 in group [1.3607450813] from disk D
SKGRP1_0118 allocation unit 40573 reason error; if possible,will try another
mirror side
WARNING: IO Failed. group:1 disk(number.incarnation):118.0x2ce5a485
disk_path:/dev/rhdiskasm175
AU:40573 disk_offset(bytes):42543947776 io_size:8192 operation:Read
type:synchronous
result:I/O error process_id:2658412
subsys:System iop:0x110ac00a0 bufp:0x700000296c4e000
osderr:0x434c5344 osderr1:0x0
Cause
Check:
1) ASM side:
* Make sure the ASM instance remains up and running, despite of these entries/messages.
* Verify that all ASM disk groups remain mounted and have no offlined ASM disks.
* Overall, the ASM instance is functional (one can connect to it, run queries on the various v$asm% views, like: v$asm_diskgroup, v$asm_disk)
2) The database remains open and operational (no outage).
3) Storage level:
* The sysadmin, storage admin, and storage vendor has verified no errors in the OS system logs nor any other storage related logs, in regards to any of the disks or devices utilized by the ASM disk groups.
Furthermore, they have ran all due IO and hardware diagnostics on these devices/disks and they found no real IO or hardware related issue.
4) After all above checked okay, then this is related toBug:10051315,Bug:10055063andBug:10422126which fix is included in 11.2.0.3
Solution
Either use the work-around or apply the merge patch.
1) Work-around:
Prevent the file system from filling up by a) saving these files to another location with larger storage space or some type of offline storage (i.e., tape, etc.) AND b) either truncate or remove the logs / trc files periodically and systematically (i.e, cron job, etc)
2) Merge patch:
Select proper Oracle Release/version and platform, and apply patch:https://updates.oracle.com/download/10422126.html
For 11.2.0.2.0 and if these errors persists after patch 10440054 (merge 10051315, 10055063) was installed, then install the following merge patch (10410054, 10422126)): https://updates.oracle.com/download/11808931.html
For 11.2.0.2.1, the patch 11830021 can be applied (merge of 10051315 + 10055063 + 10422126)
For 11.2.0.2.3, the patch 12746685 can be applied (merge of 10051315 + 10055063 + 10422126)
Notes:
* If you find a different patch for the same Oracle Release/Version and platform, use the latest one supplied (see column "Last Updated" from the
respective patch's url - example: https://updates.oracle.com/download/11808931.html)
* The patch MUST be applied to the db_home AND could be also installed into the asm_home (although not required)
* If the patch for your release/version or/and platform is not available, please file an Service Request and Oracle Support will assist.
更多参考：