RAC 实例不能启动 手工启动实例报错 ORA-03113 end-of-file on communication channel
前几天弄好的一个Oracle 10gR2的 RAC. 刚才启动的时候实例无法启动，开始以为是反应慢，就等了一会。 过去了N长时间还没有启动。 觉得不太对劲。
[root@rac1 bin]# ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE OFFLINE
ora....oltp.cs application ONLINE OFFLINE
ora....cl1.srv application ONLINE OFFLINE
ora....cl2.srv application ONLINE OFFLINE
ora....l1.inst application ONLINE OFFLINE
ora....l2.inst application ONLINE OFFLINE
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
手动用srvctl 命令启动也报错：
[root@rac1 bin]# ./srvctl start instance -d orcl -i orcl1
PRKP-1001 : Error starting instance orcl1 on node rac1
CRS-0215: Could not start resource 'ora.orcl.orcl1.inst'.
因为下午在这个实例上重建了OCR和Voting disk。 所以开始以为是这个过程导致的。 在那用ocrdump命令将OCR dump出来，然后查看。 相关resource的属性没有问题。
想起来以前看到的一个错误，srvctl 不能启动， 但是sqlplus能启动。 参考：
ORA-16032 Can not Start Instance via srvctl but via sqlplus is fine [ID 1062071.1]
http://blog.csdn.net/tianlesoftware/archive/2010/11/19/6020432.aspx
就尝试用sqlplus 启动了：
[oracle@rac1 dbs]$ export ORACLE_SID=orcl1
[oracle@rac1 dbs]$ sqlplus /nolog
SQL*Plus: Release 10.2.0.1.0 - Production on Thu Dec 2 19:18:22 2010
Copyright (c) 1982, 2005, Oracle. All rights reserved.
SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 222298112 bytes
Fixed Size 1218628 bytes
Variable Size 125831100 bytes
Database Buffers 92274688 bytes
Redo Buffers 2973696 bytes
Database mounted.
ORA-03113: end-of-file on communication channel
mount 之后直接挂掉，郁闷。 所以去看了一下alert log：
Errors in file /u01/app/oracle/admin/orcl/udump/orcl1_ora_14889.trc:
ORA-16038: log 1 sequence# 2 cannot be archived
ORA-00254: error in archive control string ''
ORA-00312: online log 1 thread 1: '+DATA/orcl/onlinelog/group_1.261.736599263'
ORA-00312: online log 1 thread 1: '+FRA/orcl/onlinelog/group_1.257.736599275'
ORA-15173: entry 'archivelog' does not exist in directory '/'
USER: terminating instance due to error 16038
看到以上信息，小兴奋了一下。 因为找到原因了。 将数据库启动到mount 状态。 确认下归档目录：
SQL> startup mount;
ORACLE instance started.
Total System Global Area 222298112 bytes
Fixed Size 1218628 bytes
Variable Size 96470972 bytes
Database Buffers 121634816 bytes
Redo Buffers 2973696 bytes
Database mounted.
SQL> show parameter log_
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_1 string LOCATION=+FRA/archivelog/orcl/
...
将RAC的归档放在了ASM上。 所以到ASM 上看了一下，确实没有后面的几个目录。 用asmcmd连上ASM. 在DATA和FRA目录下分别才创建了archivelog/orcl 这个目录：
[oracle@rac1 bdump]$ asmcmd
ASMCMD> ls
DATA/
FRA/
ASMCMD> cd DATA/
ASMCMD> ls
ORCL/
ASMCMD> mkdir archivelog
ASMCMD> cd archivelog
ASMCMD> ls
ASMCMD> mkdir orcl
ASMCMD>cd ../..
ASMCMD> cd FRA
ASMCMD> ls
ORCL/
ASMCMD> mkdir archivelog
ASMCMD> ls
ORCL/
archivelog/
ASMCMD> cd archivelog
ASMCMD> ls
ASMCMD> mkdir orcl
在将mount的实例open:
SQL> alter database open;
Database altered.
成功open. 用srvctl 启动了另一个实例：
[oracle@rac1 bin]$ ./srvctl start instance -d orcl -i orcl1
查看进程：
[oracle@rac1 bin]$ ./crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.orcl.db application ONLINE ONLINE rac1
ora....oltp.cs application ONLINE ONLINE rac1
ora....cl1.srv application ONLINE ONLINE rac1
ora....cl2.srv application ONLINE ONLINE rac2
ora....l1.inst application ONLINE ONLINE rac1
ora....l2.inst application ONLINE ONLINE rac2
ora....SM1.asm application ONLINE ONLINE rac1
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application ONLINE ONLINE rac1
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip application ONLINE ONLINE rac1
ora....SM2.asm application ONLINE ONLINE rac2
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application ONLINE ONLINE rac2
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip application ONLINE ONLINE rac2
成功启动。
小结：
官网的那个是因为使用了环境变量导致的。 我的这个是因为归档目录不存在导致的。 归档设置虽然是个小问题，影响确是很大的。
------------------------------------------------------------------------------
Blog： http://blog.csdn.net/tianlesoftware
网上资源： http://tianlesoftware.download.csdn.net
相关视频：http://blog.csdn.net/tianlesoftware/archive/2009/11/27/4886500.aspx
DBA1 群：62697716(满); DBA2 群：62697977(满)
DBA3 群：62697850 DBA 超级群：63306533;
聊天 群：40132017
--加群需要在备注说明Oracle表空间和数据文件的关系，否则拒绝申请