oracle rac 重启报错 The Cluster Time Synchronization Service aborted on host rac1.

今天没事重启了一下rac ，原本以为会没啥问题的，没想到rac2，rac3 节点都正常起来了，但rac1 始终起不来，查看日志，如下:

[ctssd(3088)]CRS-2402:The Cluster Time Synchronization Service aborted on host rac1. Details at  in /u01/app/11.2.0/grid_1/log/rac1/ctssd/octssd.log.
2017-03-09 16:06:40.950: 
[ctssd(3106)]CRS-2405:The Cluster Time Synchronization Service on host rac1 is shutdown by user
2017-03-09 16:06:45.255: 
[ctssd(3114)]CRS-2401:The Cluster Time Synchronization Service started on host rac1.
2017-03-09 16:06:45.256: 
[ctssd(3114)]CRS-2407:The new Cluster Time Synchronization Service reference node is host rac2.
2017-03-09 16:06:48.405: 
[ctssd(3114)]CRS-2404:The Cluster Time Synchronization Service detects that the local time is significantly different from the mean cluster time. Details in /u01/app/11.2.0/grid_1/log/rac1/ctssd/octssd.log.
2017-03-09 16:06:48.980: 
[ctssd(3114)]CRS-2402:The Cluster Time Synchronization Service aborted on host rac1. Details at  in /u01/app/11.2.0/grid_1/log/rac1/ctssd/octssd.log.
2017-03-09 16:06:49.995: 
[ohasd(2308)]CRS-2807:Resource 'ora.asm' failed to start automatically.
2017-03-09 16:06:49.995: 
[ohasd(2308)]CRS-2807:Resource 'ora.crsd' failed to start automatically.
2017-03-09 16:06:49.995: 
[ohasd(2308)]CRS-2807:Resource 'ora.ctssd' failed to start automatically.
2017-03-09 16:06:49.995: 
[ohasd(2308)]CRS-2807:Resource 'ora.evmd' failed to start automatically.

尝试手动重启crs，发现最后还是起不来，最好在看一下报错信息，发现The Cluster Time Synchronization Service aborted on host rac1。

于是根据报错信息，发现rac1的时间和rac2，rac3 已经不同了，于是赶紧把rac1的时间改回来，

[root@rac1 ~]# date -s '2017-03-05 13:50:12'

[root@rac1 ~]# hwclock -w

然后手动重启

[root@rac1 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'rac1'
CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@rac1 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac1 bin]# ps -ef |grep pmon
grid      4130     1  0 13:53 ?        00:00:00 asm_pmon_+ASM1
root      4323  2970  0 13:53 pts/0    00:00:00 grep pmon
[root@rac1 bin]#