HP-UX+11.2.0.3RAC因裸设备权限不一致导致RAC一个节点报错的解决

故障情况：
HP-UX+11.2.0.3RAC；未使用ASM使用了裸设备；--这里就不讨论11gR2使用裸设备的坑了，历史问题了。。
在存储划盘过来后，A节点的裸设备权限不正确，B节点权限正确；
然后，将裸设备做为数据文件加入到表空间后，很快，遇到日志切换后开始报错(因为检查点)；
此时，已经排查到磁盘权限问题并修改为正确的权限；但是A节点ALERT日志中仍然在报错。
报错信息：

Mon Jan 11 09:41:10 2016
Thread 1 advanced to log sequence 169313 (LGWR switch)
  Current log# 1 seq# 169313 mem# 0: /dev/vgpAAAdb11/rrredo_11_2048
Mon Jan 11 09:41:20 2016
Archived Log entry 240885 added for thread 1 sequence 169312 ID 0x360aa448 dest 1:
Mon Jan 11 09:42:08 2016
Errors in file /app/oracle/diag/rdbms/p1AAAdb/P1AAADB1/trace/P1AAADB1_m002_8386.trc
ORA-01157: cannot identify/lock data file 743 - see DBWR trace file
ORA-01110: data file 743: '/dev/vgpAAAdb17/rLV_AAA_L_D01'
Errors in file /app/oracle/diag/rdbms/p1AAAdb/P1AAADB1/trace/P1AAADB1_m002_8386.trc:
ORA-01157: cannot identify/lock data file 744 - see DBWR trace file
ORA-01110: data file 744: '/dev/vgpAAAdb18/rLV_AAA_M_D01'
Mon Jan 11 09:42:28 2016
Thread 1 advanced to log sequence 169314 (LGWR switch)
  Current log# 2 seq# 169314 mem# 0: /dev/vgpAAAdb11/rrredo_12_2048
Mon Jan 11 09:42:46 2016

解决方法：
此时需要在问题节点上将报错的数据文件OFFLINE，然后再对数据文件进行RECOVER；然后重新ONLINE数据文件即可；
未报错的节点上可以不操作；
---------
解决过程：

SQL> select file_name,status from dba_data_files where file_id=743;
select file_name,status from dba_data_files where file_id=743
                             *
ERROR at line 1:
ORA-01157: cannot identify/lock data file 743 - see DBWR trace file
ORA-01110: data file 743: '/dev/vgpAAAdb17/rLV_AAA_L_D01'

----offline操作

SQL> alter database datafile '/dev/vgpAAAdb17/rLV_AAA_L_D01' offline;
Database altered.
SQL> alter database datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01' offline;
Database altered.
----online操作
SQL> alter database datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01' online;
alter database datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01' online
*
ERROR at line 1:
ORA-01113: file 744 needs media recovery
ORA-01110: data file 744: '/dev/vgpAAAdb18/rLV_AAA_M_D01'

----recover数据文件操作

SQL> recover datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01';
Media recovery complete.
SQL> alter database datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01' online;
Database altered.

SQL> alter database datafile  '/dev/vgpAAAdb17/rLV_AAA_L_D01' online;
alter database datafile  '/dev/vgpAAAdb17/rLV_AAA_L_D01' online
*
ERROR at line 1:
ORA-01113: file 743 needs media recovery
ORA-01110: data file 743: '/dev/vgpAAAdb17/rLV_AAA_L_D01'

SQL> recover datafile '/dev/vgpAAAdb17/rLV_AAA_L_D01';
Media recovery complete.
SQL> alter database datafile  '/dev/vgpAAAdb17/rLV_AAA_L_D01' online;
Database altered.

----进行检查点及日志切换操作，均可以正常进行。--稍后业务部门对表等数据查询验证均正常。

SQL> alter system checkpoint;
System altered.
SQL> alter system switch logfile;
System altered.
SQL> select file_name,status from dba_data_files where file_id=743;
FILE_NAME
--------------------------------------------------------------------------------
STATUS
---------
/dev/vgpAAAdb17/rLV_AAA_L_D01
AVAILABLE
SQL> select file_name,status from dba_data_files where file_id=744;
FILE_NAME
--------------------------------------------------------------------------------
STATUS
---------
/dev/vgpAAAdb18/rLV_AAA_M_D01
AVAILABLE

------------------
对应的ALERT日志：

Mon Jan 11 10:01:56 2016
Recovery of Online Redo Log: Thread 2 Group 15 Seq 71551 Reading mem 0
  Mem# 0: /dev/vgpAAAdb11/rrredo_27_2048
Mon Jan 11 10:02:08 2016
Recovery of Online Redo Log: Thread 1 Group 6 Seq 169318 Reading mem 0
  Mem# 0: /dev/vgpAAAdb11/rrredo_16_2048
Mon Jan 11 10:02:09 2016
Errors in file /app/oracle/diag/rdbms/p1AAAdb/P1AAADB1/trace/P1AAADB1_m005_11643.trc:
ORA-01135: file 743 accessed for DML/query is offline
ORA-01110: data file 743: '/dev/vgpAAAdb17/rLV_AAA_L_D01'
Errors in file /app/oracle/diag/rdbms/p1AAAdb/P1AAADB1/trace/P1AAADB1_m005_11643.trc:
ORA-01135: file 744 accessed for DML/query is offline
ORA-01110: data file 744: '/dev/vgpAAAdb18/rLV_AAA_M_D01'
Mon Jan 11 10:02:22 2016
Recovery of Online Redo Log: Thread 1 Group 7 Seq 169319 Reading mem 0
  Mem# 0: /dev/vgpAAAdb11/rrredo_17_2048
Media Recovery Complete (P1AAADB1)
Completed: ALTER DATABASE RECOVER  datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01'  
Mon Jan 11 10:02:37 2016
alter database datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01' online
Completed: alter database datafile '/dev/vgpAAAdb18/rLV_AAA_M_D01' online
Mon Jan 11 10:03:00 2016
alter database datafile  '/dev/vgpAAAdb17/rLV_AAA_L_D01' online
ORA-1113 signalled during: alter database datafile  '/dev/vgpAAAdb17/rLV_AAA_L_D01' online...
Mon Jan 11 10:03:14 2016
ALTER DATABASE RECOVER  datafile '/dev/vgpAAAdb17/rLV_AAA_L_D01'  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 4 Seq 169316 Reading mem 0
  Mem# 0: /dev/vgpAAAdb11/rrredo_14_2048
Recovery of Online Redo Log: Thread 2 Group 14 Seq 71550 Reading mem 0
  Mem# 0: /dev/vgpAAAdb11/rrredo_26_2048
Recovery of Online Redo Log: Thread 1 Group 5 Seq 169317 Reading mem 0
  Mem# 0: /dev/vgpAAAdb11/rrredo_15_2048
Recovery of Online Redo Log: Thread 2 Group 15 Seq 71551 Reading mem 0
  Mem# 0: /dev/vgpAAAdb11/rrredo_27_2048
Mon Jan 11 10:03:32 2016
Recovery of Online Redo Log: Thread 1 Group 6 Seq 169318 Reading mem 0
  Mem# 0: /dev/vgpAAAdb11/rrredo_16_2048
Mon Jan 11 10:03:45 2016
Recovery of Online Redo Log: Thread 1 Group 7 Seq 169319 Reading mem 0
  Mem# 0: /dev/vgpAAAdb11/rrredo_17_2048
Media Recovery Complete (P1AAADB1)
Completed: ALTER DATABASE RECOVER  datafile '/dev/vgpAAAdb17/rLV_AAA_L_D01'  
Mon Jan 11 10:03:58 2016
alter database datafile  '/dev/vgpAAAdb17/rLV_AAA_L_D01' online
Completed: alter database datafile  '/dev/vgpAAAdb17/rLV_AAA_L_D01' online
Mon Jan 11 10:04:42 2016
Thread 1 advanced to log sequence 169320 (LGWR switch)
  Current log# 8 seq# 169320 mem# 0: /dev/vgpAAAdb11/rrredo_18_2048