Oracle不能对部分表进行读写Error:ORA-15080/ORA-15025/ORA-27041

   有一套Oracle RAC,突然反映说不能对部分表格进行读写操作了,但是数据库正常,大部分数据读写也正常,先将服务切至另一台Server使Oracle对外正常服务,然后开始troubleshooting。


   HP-UX系统层面没有问题,存储3par没有预警,确认链路正常,以上步骤是为了确认不是个别共享存储出问题导致的数据无法访问。然后查看trace log进行分析(/oracle/app/diag/rdbms/mesdb/mesdb2/trace/alter_mesdb2.log),发现以下报错且不断循环:


ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 713 logical extent 0 of file 532 in group 9 on disk 0 allocation unit 595268
Fri Apr 11 12:13:00 2014
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 66 logical extent 0 of file 310 in group 9 on disk 1 allocation unit 171803
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 66 logical extent 0 of file 310 in group 9 on disk 1 allocation unit 171803
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_10102.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
WARNING: failed to read mirror side 1 of virtual extent 12 logical extent 0 of file 281 in group [9.1884348661] from disk MESDBDATA_0000  allocation unit 5885 reason error; if possible, will try another mirror side
WARNING: failed to read mirror side 1 of virtual extent 12 logical extent 0 of file 281 in group [9.1884348661] from disk MESDBDATA_0000  allocation unit 5885 reason error; if possible, will try another mirror side
WARNING: failed to read mirror side 1 of virtual extent 12 logical extent 0 of file 281 in group [9.1884348661] from disk MESDBDATA_0000  allocation unit 5885 reason error; if possible, will try another mirror side
WARNING: failed to read mirror side 1 of virtual extent 12 logical extent 0 of file 281 in group [9.1884348661] from disk MESDBDATA_0000  allocation unit 5885 reason error; if possible, will try another mirror side
Fri Apr 11 12:13:00 2014
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_12188.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_12188.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_12188.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 47 logical extent 0 of file 289 in group 9 on disk 1 allocation unit 95318
Fri Apr 11 12:13:01 2014
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_6690.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_6690.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
WARNING: failed to read mirror side 1 of virtual extent 683 logical extent 0 of file 312 in group [9.1884348661] from disk MESDBDATA_0001  allocation unit 179306 reason error; if possible, will try another mirror side
Fri Apr 11 12:13:03 2014
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_5920.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_5920.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
WARNING: failed to read mirror side 1 of virtual extent 685 logical extent 0 of file 312 in group [9.1884348661] from disk MESDBDATA_0000  allocation unit 134472 reason error; if possible, will try another mirror side
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_5920.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_5920.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
WARNING: failed to read mirror side 1 of virtual extent 685 logical extent 0 of file 312 in group [9.1884348661] from disk MESDBDATA_0000  allocation unit 134472 reason error; if possible, will try another mirror side
Fri Apr 11 12:13:09 2014
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_4434.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_4434.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_4434.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 188 logical extent 0 of file 260 in group 9 on disk 1 allocation unit 1631380
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_4434.trc:
ORA-01114: IO error writing block to file  (block # )
Fri Apr 11 12:13:14 2014
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_12098.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata3"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_12098.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata3"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_12098.trc:
ORA-15080: synchronous I/O operation to a disk failed
WARNING: failed to write mirror side 1 of virtual extent 6293 logical extent 0 of file 710 in group 9 on disk 2 allocation unit 1302838
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_12098.trc:
ORA-01114: IO error writing block to file  (block # )
Fri Apr 11 12:13:15 2014
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_6690.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_6690.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
WARNING: failed to read mirror side 1 of virtual extent 683 logical extent 0 of file 312 in group [9.1884348661] from disk MESDBDATA_0001  allocation unit 179306 reason error; if possible, will try another mirror side
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_6690.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_6690.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
WARNING: failed to read mirror side 1 of virtual extent 683 logical extent 0 of file 312 in group [9.1884348661] from disk MESDBDATA_0001  allocation unit 179306 reason error; if possible, will try another mirror side
Fri Apr 11 12:13:32 2014
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_12016.trc:
ORA-15025: "/dev/asmdisk/mesdbdata2" .
ORA-27041:
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_12016.trc:
ORA-15025: "/dev/asmdisk/mesdbdata2" .
ORA-27041:
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
WARNING: failed to read mirror side 1 of virtual extent 38 logical extent 0 of file 257 in group [9.1884348661] from disk MESDBDATA_0001  allocation unit 305 reason error; if possible, will try another mirror side
Fri Apr 11 12:13:33 2014
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_5920.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_5920.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
WARNING: failed to read mirror side 1 of virtual extent 685 logical extent 0 of file 312 in group [9.1884348661] from disk MESDBDATA_0000  allocation unit 134472 reason error; if possible, will try another mirror side
Fri Apr 11 12:13:38 2014
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_6690.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_6690.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
WARNING: failed to read mirror side 1 of virtual extent 683 logical extent 0 of file 312 in group [9.1884348661] from disk MESDBDATA_0001  allocation unit 179306 reason error; if possible, will try another mirror side
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_5920.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_5920.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata1"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3
WARNING: failed to read mirror side 1 of virtual extent 685 logical extent 0 of file 312 in group [9.1884348661] from disk MESDBDATA_0000  allocation unit 134472 reason error; if possible, will try another mirror side
Errors in file /oracle/app/diag/rdbms/mesdb/mesdb2/trace/mesdb2_ora_6690.trc:
ORA-15025: could not open disk "/dev/asmdisk/mesdbdata2"
ORA-27041: unable to open file
HPUX-ia64 Error: 13: Permission denied
Additional information: 3


报错从10:55开始,询问相关有权限的人员,均说无人进行相关操作,查看log报错,很多人说是需要调整内核参数,但是说法都不太统一,不敢下手。后来看到could not open disk "/dev/asmdisk/mesdbdata2"和Permission denied,在猜想是不是有人更改了数据盘权限,当然这是最后想到的,因为改权限是最不可能的事情,这毕竟是生产DB,进到/dev/asmdisk下:


fabdb2#[/dev/rdisk]ll /dev/asmdisk
total 0
lrwxr-xr-x   1 root       sys             18 Aug  2  2013 fgmsarch -> /dev/rdisk/disk192
lrwxr-xr-x   1 root       sys             18 Aug  2  2013 fgmsctl1 -> /dev/rdisk/disk158
lrwxr-xr-x   1 root       sys             18 Aug  2  2013 fgmsctl2 -> /dev/rdisk/disk175
lrwxr-xr-x   1 root       sys             18 Aug  2  2013 fgmsdata -> /dev/rdisk/disk226
lrwxr-xr-x   1 root       sys             18 Aug  2  2013 fgmsidx -> /dev/rdisk/disk209
lrwxr-xr-x   1 root       sys             17 Aug  2  2013 mesdbarch -> /dev/rdisk/disk73
lrwxr-xr-x   1 root       sys             17 Aug  2  2013 mesdbctl1 -> /dev/rdisk/disk39
lrwxr-xr-x   1 root       sys             17 Aug  2  2013 mesdbctl2 -> /dev/rdisk/disk56
lrwxr-xr-x   1 root       sys             18 Aug  2  2013 mesdbdata1 -> /dev/rdisk/disk107
lrwxr-xr-x   1 root       sys             18 Aug  2  2013 mesdbdata2 -> /dev/rdisk/disk124
lrwxr-xr-x   1 root       sys             18 Aug  2  2013 mesdbdata3 -> /dev/rdisk/disk141
lrwxr-xr-x   1 root       sys             17 Aug  2  2013 mesdbidx -> /dev/rdisk/disk90
lrwxr-xr-x   1 root       sys             17 Aug  2  2013 voting -> /dev/rdisk/disk22


发现权限正常,然后通过映射关系mesdbdata2 -> /dev/rdisk/disk124到/dev/rdisk目录现查看disk124的权限,发现异常:


fabdb2#[/dev/rdisk]ll /dev/rdisk
total 0
crw-r-----   1 bin        sys         13 0x00001f Apr 11 03:56 disk107
crw-r-----   1 bin        sys         13 0x000020 Apr 11 03:29 disk124
crw-r-----   1 bin        sys         13 0x000021 Apr 11 02:45 disk141
crw-r-----   1 bin        sys         13 0x000022 Dec 25 12:35 disk158
crw-r-----   1 bin        sys         13 0x000023 Dec 25 12:35 disk175
crw-r-----   1 bin        sys         13 0x000024 Apr 11 04:28 disk192
crw-r-----   1 bin        sys         13 0x000001 Aug  1  2013 disk2
crw-r-----   1 bin        sys         13 0x000025 Aug  2  2013 disk209
crw-r-----   1 bin        sys         13 0x00001a Apr 11 14:35 disk22
crw-r-----   1 bin        sys         13 0x000026 Apr 11 04:29 disk226
crw-r-----   1 bin        sys         13 0x000037 Apr 11 10:12 disk242
crw-r-----   1 bin        sys         13 0x000040 Mar 27 14:19 disk251
crw-r-----   1 bin        sys         13 0x000042 Mar 28 15:51 disk277
crw-r-----   1 bin        sys         13 0x000017 Aug  1  2013 disk2_p1
crw-r-----   1 bin        sys         13 0x000018 Aug  1  2013 disk2_p2
crw-r-----   1 bin        sys         13 0x000019 Aug  1  2013 disk2_p3
crw-r-----   1 bin        sys         13 0x000003 Aug  1  2013 disk3
crw-r-----   1 bin        sys         13 0x00001b Mar 11 00:27 disk39
crw-r-----   1 bin        sys         13 0x00001c Mar 11 00:27 disk56
crw-r-----   1 bin        sys         13 0x00001d Apr 11 11:00 disk73
crw-r-----   1 bin        sys         13 0x00001e Apr 11 03:29 disk90


我们看到这里都是属于bin用户和sys组,其实正常应该是如下才对:


fabdb1#[/dev/asmdisk]ll /dev/rdisk
total 0
crwxrwxr-x   1 grid       asmadmin    13 0x000022 Apr 10 03:01 disk107
crwxrwxr-x   1 grid       asmadmin    13 0x000023 Mar 20 13:47 disk124
crwxrwxr-x   1 grid       asmadmin    13 0x000024 Apr 10 03:03 disk141
crwxrwxr-x   1 grid       asmadmin    13 0x000025 Apr 11 11:25 disk158
crwxrwxr-x   1 grid       asmadmin    13 0x000026 Dec 25 12:35 disk175
crw-r-----   1 bin        sys         13 0x000001 Aug  1  2013 disk18
crw-r-----   1 bin        sys         13 0x000018 Aug  1  2013 disk18_p1
crw-r-----   1 bin        sys         13 0x00001c Aug  1  2013 disk18_p2
crw-r-----   1 bin        sys         13 0x00001d Aug  1  2013 disk18_p3
crw-r-----   1 bin        sys         13 0x000003 Aug  1  2013 disk19
crwxrwxr-x   1 grid       asmadmin    13 0x000027 Dec 25 12:35 disk192
crw-r-----   1 bin        sys         13 0x000019 Aug  1  2013 disk19_p1
crw-r-----   1 bin        sys         13 0x00001a Aug  1  2013 disk19_p2
crw-r-----   1 bin        sys         13 0x00001b Aug  1  2013 disk19_p3
crw-r-----   1 bin        sys         13 0x000005 Aug  1  2013 disk20
crwxrwxr-x   1 grid       asmadmin    13 0x000028 Apr 11 00:30 disk209
crwxrwxr-x   1 grid       asmadmin    13 0x000029 Aug  3  2013 disk226
crwxrwxr-x   1 grid       asmadmin    13 0x00002a Apr  9 00:31 disk243
crwxrwxr-x   1 grid       asmadmin    13 0x00001e Sep 26  2013 disk39
crwxrwxr-x   1 grid       asmadmin    13 0x00001f Feb 22 06:00 disk56
crwxrwxr-x   1 grid       asmadmin    13 0x000020 Dec 10 00:29 disk73
crwxrwxr-x   1 grid       asmadmin    13 0x000021 Apr 11 12:30 disk90


因为Oracle 11gR2 RAC下磁盘都是交给ASM管理了,我们会把共享磁盘都划给grid用户和asmadmin组以便于ASM拥有权限管理,所以找到此原因后,按照对于的磁盘将ASM管理的磁盘重新付与grid用户和asmadmin组,权限是660,之后恢复正常。


由于没有开启审计功能虽然可以查看何人何时登录OS,但是没办法查知究竟是何人执行了什么操作才引起的。

你可能感兴趣的:(oracle,RAC,11gr2,ORA-27041,ORA-15080,ORA-15025)