CRS-0184: Cannot communicate with the CRS daemon.之节点脱离集群问题解决
1、系统环境：
OEL6.5企业版
Oracle11.2.0.2 RAC + ASM
2、报错信息：
$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
$ crsctl check has
CRS-4639: Could not contact Oracle High Availability Services
3、报错原因查找：
在节点1节点查看IP地址，VIP已经不存在
$ ifconfig -a
en0: flags=1e080863,480
inet 192.168.0.117 netmask 0xffffff00 broadcast 192.168.0.255
tcp_sendspace 262144 tcp_recvspace 262144 rfc1323 1
en1: flags=1e080863,480
inet 10.10.10.1 netmask 0xff000000 broadcast 10.255.255.255
tcp_sendspace 65536 tcp_recvspace 65536 rfc1323 1
在节点2上运行“crs_stat -t”发现，节点1好像已经退出集群环境
# su - grid
$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE node2
ora....ER.lsnr ora....er.type ONLINE ONLINE node2
ora....N1.lsnr ora....er.type ONLINE OFFLINE
ora.OCR.dg ora....up.type ONLINE ONLINE node2
ora.asm ora.asm.type ONLINE ONLINE node2
ora.cvu ora.cvu.type ONLINE OFFLINE
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE node2
ora.node1.vip ora....t1.type ONLINE OFFLINE
ora....SM2.asm application ONLINE ONLINE node2
ora....E2.lsnr application ONLINE ONLINE node2
ora.node2.gsd application OFFLINE OFFLINE
ora.node2.ons application ONLINE ONLINE node2
ora.node2.vip ora....t1.type ONLINE ONLINE node2
ora.oc4j ora.oc4j.type ONLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE node2
ora.prod.db ora....se.type ONLINE ONLINE node2
ora....ry.acfs ora....fs.type ONLINE ONLINE node2
ora.scan1.vip ora....ip.type ONLINE OFFLINE
从输出中看到节点1的VIP已经飘逸到节点2，从而判断节点1已经脱离集群。
通过"ifconfig"查看节点2上的ip信息。
4、问题解决：
清理节点1的配置信息，打算重新运行root.sh
# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -verbose -deconfig -force
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Stop failed, or completed with errors.
CRS-4544: Unable to connect to OHAS
CRS-4000: Command Stop failed, or completed with errors.
ACFS-9313: No ADVM/ACFS installation detected.
Successfully deconfigured Oracle clusterware stack on this node
#
#
# /u01/app/11.2.0/grid/crs/install/roothas.pl -verbose -deconfig -force
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Delete failed, or completed with errors.
CRS-4047: No Oracle Clusterware components configured.
CRS-4000: Command Stop failed, or completed with errors.
You must kill ohasd processes or reboot the system to properly
cleanup the processes started by Oracle clusterware
ACFS-9313: No ADVM/ACFS installation detected.
Either /etc/oracle/olr.loc does not exist or is not readable
Make sure the file exists and it has read and execute access
Failure in execution (rc=-1, 512, A file or directory in the path name does not exist.) for command /etc/ohasd deinstall
Successfully deconfigured Oracle Restart stack
环境清理后，运行root.sh
# ./u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
User ignored Prerequisites during installation
User grid has the required capabilities to run CSSD in realtime mode
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node node2, number 2, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
PRKO-2190 : VIP exists for node node1, VIP name node1-vip
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
查看结果：
$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE node1
ora....ER.lsnr ora....er.type ONLINE ONLINE node1
ora....N1.lsnr ora....er.type ONLINE ONLINE node1
ora.OCR.dg ora....up.type ONLINE ONLINE node1
ora.asm ora.asm.type ONLINE ONLINE node1
ora.cvu ora.cvu.type ONLINE ONLINE node2
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE node1
ora....SM1.asm application ONLINE ONLINE node1
ora....E1.lsnr application ONLINE ONLINE node1
ora.node1.gsd application OFFLINE OFFLINE
ora.node1.ons application ONLINE ONLINE node1
ora.node1.vip ora....t1.type ONLINE ONLINE node1
ora....SM2.asm application ONLINE ONLINE node2
ora....E2.lsnr application ONLINE ONLINE node2
ora.node2.gsd application OFFLINE OFFLINE
ora.node2.ons application ONLINE ONLINE node2
ora.node2.vip ora....t1.type ONLINE ONLINE node2
ora.oc4j ora.oc4j.type ONLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE node1
ora.prod.db ora....se.type ONLINE ONLINE node1
ora....ry.acfs ora....fs.type ONLINE ONLINE node1
ora.scan1.vip ora....ip.type ONLINE ONLINE node1