Oracle 19C集群管理- 主机时间，ctss异常导致ora.asm资源启动异常分析

1 故障描述

今日数据库主机1节点进行重启，重启完成后发现，节点的时间比平常时间快了将近1小时。

而数据库也无法正常启动。

2 故障分析

查看数据库启动到哪个阶段

[grid@orcl1~]$ crsctl status res -t -init

--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  OFFLINE      ORCL1          STABLE
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       gzrac01                  STABLE
ora.crf
      1        ONLINE  ONLINE       gzrac01                  STABLE
ora.crsd
      1        ONLINE  ONLINE       gzrac01                  STABLE
ora.cssd
      1        ONLINE  ONLINE       gzrac01                  STABLE
ora.cssdmonitor
      1        ONLINE  ONLINE       gzrac01                  STABLE
ora.ctssd
      1        ONLINE  OFFLINE      gzrac01                  OBSERVER,STABLE
ora.diskmon
      1        OFFLINE OFFLINE                               STABLE
ora.drivers.acfs
      1        ONLINE  ONLINE       gzrac01                  STABLE
ora.evmd
      1        ONLINE  ONLINE       gzrac01                  STABLE
ora.gipcd
      1        ONLINE  ONLINE       gzrac01                  STABLE
ora.gpnpd
      1        ONLINE  ONLINE       gzrac01                  STABLE
ora.mdnsd
      1        ONLINE  ONLINE       gzrac01                  STABLE
ora.storage
      1        ONLINE  ONLINE       gzrac01                  STABLE
--------------------------------------------------------------------------------

查看ctss日志发现异常，ctss无法判断是否启动异常，导致ora.asm无法启动。

[grid@gzrac01 ~]$ crsctl status  res "ora.asm" -init

NAME=ora.asm
TYPE=ora.asm.type
TARGET=ONLINE
STATE=ONLINE on gzrac01

[grid@gzrac01 ~]$ crsctl status  res "ora.asm" -init -p

NAME=ora.asm
TYPE=ora.asm.type
ACL=owner:grid:rw-,pgrp:oinstall:rw-,other::r--,user:grid:rwx
ACTIONS=CRSDASM_START,user:grid relocate_clients,user:grid
ACTION_SCRIPT=
ACTION_TIMEOUT=600
ACTIVE_PLACEMENT=0
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
AUTO_START=restore
CARDINALITY=1
CHECK_ARGS=
CHECK_COMMAND=
CHECK_INTERVAL=1
CHECK_TIMEOUT=600
CLEAN_ARGS=
CLEAN_COMMAND=
CLEAN_TIMEOUT=60
CSS_CRITICAL=no
DAEMON_LOGGING_LEVELS=GIPC=2,GIPCBASE=2,GIPCCLSA=2,GIPCCLSC=2,GIPCEXMP=2,GIPCGEN=2,GIPCGMOD=2,GIPCHALO=2,GIPCHAUP=2
,GIPCHDEM=2,GIPCHEAD=2,GIPCHGEN=2,GIPCHGNS=2,GIPCHLCK=2,GIPCHTHR=2,GIPCHWRK=2,GIPCLIB=2,GIPCMUX=2,GIPCNET=2,GIPCNUL
L=2,GIPCOSD=2,GIPCPKT=2,GIPCSMEM=2,GIPCTLS=2,GIPCTRAC=2,GIPCWAIT=2,GIPCXCPT=2
DAEMON_TRACE_FILE_OPTIONS=filesize=26214400,numsegments=10
DELETE_TIMEOUT=60
DESCRIPTION="ASM instance"
DETACHED=true
ENABLED=1
FAILOVER_DELAY=0
FAILURE_INTERVAL=3
FAILURE_THRESHOLD=5
GEN_RESTART=StartCompleted
GEN_USR_ORA_INST_NAME=+ASM1
HOSTING_MEMBERS=
IGNORE_TARGET_ON_FAILURE=no
INSTANCE_FAILOVER=1
INTERMEDIATE_TIMEOUT=600
LOAD=1
LOGGING_LEVEL=1
MODIFY_TIMEOUT=60
OFFLINE_CHECK_INTERVAL=0
OS_CRASH_THRESHOLD=0
OS_CRASH_UPTIME=0
PID_FILE=
PLACEMENT=restricted
PROCESS_TO_MONITOR=
RELOCATE_BY_DEPENDENCY=1
RELOCATE_KIND=offline
RESOURCE_GROUP=
RESTART_ATTEMPTS=5
RESTART_DELAY=0
SCRIPT_TIMEOUT=600
SERVER_CATEGORY=ora.hub.category
SERVER_POOLS=*
SPFILE=
START_ARGS=
START_COMMAND=
START_CONCURRENCY=0
START_DEPENDENCIES=hard(ora.cssd,ora.cluster_interconnect.haip,ora.ctssd)pullup
(ora.cssd,ora.cluster_interconnect.haip,ora.ctssd)weak(ora.drivers.acfs)
START_TIMEOUT=600
STOP_ARGS=
STOP_COMMAND=
STOP_CONCURRENCY=0
STOP_DEPENDENCIES=hard(intermediate:ora.cssd,shutdown:ora.cluster_interconnect.haip)
STOP_TIMEOUT=600
TARGET_DEFAULT=default
UNRESPONSIVE_TIMEOUT=180
UPTIME_THRESHOLD=1h
USER_WORKLOAD=no
USE_STICKINESS=0
USR_ORA_ENV=
USR_ORA_INST_NAME=
USR_ORA_OPEN_MODE=mount
USR_ORA_OPI=false
USR_ORA_STOP_MODE=immediate
WORKLOAD_CPU=0
WORKLOAD_CPU_CAP=0
WORKLOAD_MEMORY_MAX=0
WORKLOAD_MEMORY_TARGET=0

[grid@gzrac01 ~]$ 

3 故障处理

将系统时间修改正确，并将NTP启动修改为自启动

systemctl start ntpd

systemctl enable ntpd

ntpupdate -u 10.228.11.2

hwclock -w

crsctl start  res "ora.asm" -init