AIX系统下RAC故障之一--CRS服务启动失败（coredump）
AIX系统下RAC故障之一--CRS服务启动失败（coredump）
系统环境：
操作系统： AIX 5300-08
集群软件： CRS 10.2.0.1
数据库： Oracle 10.2.0.1
故障现象：
在安装完CRS后，运行root.sh脚本出现故障，CRS服务启动失败：
[root@aix224@ install]#/u01/crs_1/root.sh
chown: /u01/crs_1/crs/auth: No such file or directory
chmod: /u01/crs_1/crs/auth: No such file or directory
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
chown: /u01/crs_1/crs/auth: No such file or directory
chmod: /u01/crs_1/crs/auth: No such file or directory
WARNING: directory '/u01' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: aix223 aix223-priv aix223
node 2: aix224 aix224-priv aix224
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
aix223
aix224
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
/u01/crs_1/install/rootconfig[965]: 401480 Abort(coredump)
Waiting for the Oracle CRSD and EVMD to start
/u01/crs_1/install/rootconfig[965]: 401486 Abort(coredump)
Waiting for the Oracle CRSD and EVMD to start
/u01/crs_1/install/rootconfig[965]: 401492 Abort(coredump)
Waiting for the Oracle CRSD and EVMD to start
/u01/crs_1/install/rootconfig[965]: 401498 Abort(coredump)
Waiting for the Oracle CRSD and EVMD to start
/u01/crs_1/install/rootconfig[965]: 401504 Abort(coredump)
Waiting for the Oracle CRSD and EVMD to start
/u01/crs_1/install/rootconfig[965]: 401510 Abort(coredump)
Waiting for the Oracle CRSD and EVMD to start
/u01/crs_1/install/rootconfig[965]: 401516 Abort(coredump)
Waiting for the Oracle CRSD and EVMD to start
/u01/crs_1/install/rootconfig[965]: 401522 Abort(coredump)
Waiting for the Oracle CRSD and EVMD to start
/u01/crs_1/install/rootconfig[965]: 401528 Abort(coredump)
Waiting for the Oracle CRSD and EVMD to start
/u01/crs_1/install/rootconfig[965]: 401534 Abort(coredump)
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
/u01/crs_1/install/rootconfig[965]: 401600 Abort(coredump)
Waiting for the Oracle CRSD and EVMD to start
/u01/crs_1/install/rootconfig[965]: 401606 Abort(coredump)
Waiting for the Oracle CRSD and EVMD to start
/u01/crs_1/install/rootconfig[965]: 401612 Abort(coredump)
Waiting for the Oracle CRSD and EVMD to start
Timed out waiting for the CRS stack to start.
查看日志信息：
[root@aix224@ install]#cd /u01/crs_1/log/aix224/
[root@aix224@ aix224]#ls
admin client cssd racg
alertaix224.log crsd evmd
[root@aix224@ aix224]#cd crsd
[root@aix224@ crsd]#ls
core core.2015-09-15-18:20:53 core.2015-09-15-18:22:52
core.2015-09-15-18:20:05 core.2015-09-15-18:20:59 core.2015-09-15-18:22:58
core.2015-09-15-18:20:31 core.2015-09-15-18:21:05 core.2015-09-15-18:23:04
core.2015-09-15-18:20:36 core.2015-09-15-18:21:10 crsd.log
core.2015-09-15-18:20:42 core.2015-09-15-18:21:16
core.2015-09-15-18:20:48 core.2015-09-15-18:22:47
解决方法：
[root@aix224@ install]#cd /u01/crs_1/log/aix224/
[root@aix224@ aix224]#ls
admin client cssd racg
alertaix224.log crsd evmd
[root@aix224@ aix224]#cd crsd
[root@aix224@ crsd]#ls
core core.2015-09-15-18:20:53 core.2015-09-15-18:22:52
core.2015-09-15-18:20:05 core.2015-09-15-18:20:59 core.2015-09-15-18:22:58
core.2015-09-15-18:20:31 core.2015-09-15-18:21:05 core.2015-09-15-18:23:04
core.2015-09-15-18:20:36 core.2015-09-15-18:21:10 crsd.log
core.2015-09-15-18:20:42 core.2015-09-15-18:21:16
core.2015-09-15-18:20:48 core.2015-09-15-18:22:47
解决方法：
重新清除OCR配置信息:
[root@aix224@ /]# dd if=/dev/zero of=/dev/rrac_ocr bs=8192 count=2560
建立以下目录：
[root@aix224@ /]#mkdir -p /u01/crs_1/crs/auth
[root@aix224@ /]#chgrp dba /u01/crs_1/crs/auth
[root@aix224@ /]#chmod 755 /u01/crs_1/crs/auth
在两个节点重新运行root.sh（aix223、aix224）：
[root@aix224@ /]#mkdir -p /u01/crs_1/crs/auth
[root@aix224@ /]#chgrp dba /u01/crs_1/crs/auth
[root@aix224@ /]#chmod 755 /u01/crs_1/crs/auth
在两个节点重新运行root.sh（aix223、aix224）：
[root@aix224@ install]#./rootdelete.sh
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources.
Error while stopping resources. Possible cause: CRSD is down.
Stopping CSSD.
Unable to communicate with the CSS daemon.
Shutdown has begun. The daemons should exit soon.
Checking to see if Oracle CRS stack is down...
Oracle CRS stack is not running.
Oracle CRS stack is down now.
Removing script for Oracle Cluster Ready services
Updating ocr file for downgrade
Cleaning up SCR settings in '/etc/oracle/scls_scr'
Shutting down Oracle Cluster Ready Services (CRS):
Stopping resources.
Error while stopping resources. Possible cause: CRSD is down.
Stopping CSSD.
Unable to communicate with the CSS daemon.
Shutdown has begun. The daemons should exit soon.
Checking to see if Oracle CRS stack is down...
Oracle CRS stack is not running.
Oracle CRS stack is down now.
Removing script for Oracle Cluster Ready services
Updating ocr file for downgrade
Cleaning up SCR settings in '/etc/oracle/scls_scr'
[root@aix224@ /]#/u01/crs_1/root.sh
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: aix223 aix223-priv aix223
node 2: aix224 aix224-priv aix224
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
aix223
aix224
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
The given interface(s), "en0, en1" is not public. Public interfaces should be used to configure virtual IPs.
---脚本运行成功，然后以root用户执行vipca脚本
查看CRS 服务：
[root@aix224@ bin]#crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy
CSS appears healthy
CRS appears healthy
EVM appears healthy
[root@aix224@ bin]#crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.aix223.gsd application ONLINE ONLINE aix223
ora.aix223.ons application ONLINE ONLINE aix223
ora.aix223.vip application ONLINE ONLINE aix223
ora.aix224.gsd application ONLINE ONLINE aix224
ora.aix224.ons application ONLINE ONLINE aix224
ora.aix224.vip application ONLINE ONLINE aix224
---CRS服务启动正常！！