ORA-39511|静默安装 Oracle 19c RAC 时遇到的头疼事儿
作者 | JiekeXu
来源 | JiekeXu DBA之路（ID: JiekeXu_IT）
大家好，我是 JiekeXu,很高兴又和大家见面了,今天和大家一起来看看 ORA-39511 问题处理，欢迎点击上方蓝字关注我，标星或置顶，更多干货第一时间到达！
事情是这么个事，看官慢慢听我道来，从上周开始，便陆陆续续的安装操作系统准备安装 RAC，上周五安装好系统，设置好一些参数及基础配置，便解压 GI 安装软件准备安装。
这里也暂时不说详细的配置过程，只说明与本次问题相关的配置，以后再说详细安装过程。本次 IP 及主机名规划如下 hosts 文件所示，公网使用 192.168.75 网段，私网使用 10.10.10 网段。VMware 16 虚拟机使用 VMnet8 和 VMnet1 本地虚拟网卡。
[grid@jiekexu-r1:/home/grid]$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
#public ip
192.168.75.128 jiekexu-r1
192.168.75.129 jiekexu-r2
#private ip
10.10.10.128 jiekexu-r1-priv
10.10.10.129 jiekexu-r2-priv
#vip
192.168.75.130 jiekexu-r1-vip
192.168.75.131 jiekexu-r2-vip
#scanip
192.168.75.132 jiekexu-racscan
而 Linux 虚拟机的两块网卡 ens32 ens33 分别配置如下
[root@jiekexu-r1 network-scripts]# more ifcfg-ens32
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="dhcp"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="yes"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens32"
UUID="e8332396-7724-410f-9887-1fc0e27fc818"
DEVICE="ens32"
ONBOOT="yes"
GATEWAY="192.168.75.2"
IPADDR="192.168.75.128"
PREFIX="24"
IPV6_PRIVACY="no"
[root@jiekexu-r1 network-scripts]# more ifcfg-ens33
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
DEVICE=ens33
HWADDR="00:0C:29:2C:79:D8"
uuid="c96bc909-188e-ec64-3a96-6a90982b08ad"
ONBOOT=yes
GATEWAY="10.10.10.1"
IPADDR="10.10.10.128"
PREFIX="8"
然后解压 GI 软件,解压 RU 补丁 32900083 ，替换 OPatch，配置互信。
分别使用grid和oracle用户调用sshUserSetup.sh脚本配置SSH connectivity，根据提示输入密码。
$ cd $ORACLE_HOME/oui/prov/resources/scripts
$ ./sshUserSetup.sh -user grid -hosts "jiekexu-r1 jiekexu-r2" -advanced -confirm -noPromptPassphrase
cd /u01/app/19.0.0/grid/oui/prov/resources/scripts
$ ./sshUserSetup.sh -user oracle -hosts "jiekexu-r1 jiekexu-r2" -advanced -confirm -noPromptPassphrase
--分别使用grid和oracle用户验证 SSH connectivity：
$ for h in jiekexu-r1 jiekexu-r1-priv jiekexu-r2 jiekexu-r2-priv;do
ssh -l grid -o StrictHostKeyChecking=no $h date;
done
$ for h in jiekexu-r1 jiekexu-r1-priv jiekexu-r2 jiekexu-r2-priv;do
ssh -l oracle -o StrictHostKeyChecking=no $h date;
done
GI 安装配置预检查
$/u01/app/19.0.0/grid/runcluvfy.sh stage -pre crsinst -n jiekexu-r1,jiekexu-r2 -method root
执行补丁和 GI 安装
#### 执行补丁和 GI 安装
$ORACLE_HOME/gridSetup.sh -silent -applyRU /home/soft/32900083/32895426 -force -noconfig -ignorePrereq -waitforcompletion \
-responseFile ${ORACLE_HOME}/install/response/gridsetup.rsp \
INVENTORY_LOCATION=/u01/app/oraInventory \
ORACLE_BASE=/u01/app/grid \
ORACLE_HOME=/u01/app/19.0.0/grid \
oracle.install.asm.OSDBA=asmdba \
oracle.install.asm.OSOPER=asmoper \
oracle.install.asm.OSASM=asmadmin \
oracle.install.crs.config.storageOption=ASM \
oracle.install.crs.config.useIPMI=false \
oracle.install.asm.diskGroup.name=OCR \
oracle.install.asm.diskGroup.AUSize=4 \
oracle.install.asm.diskGroup.redundancy=NORMAL \
oracle.install.crs.config.ClusterConfiguration=STANDALONE \
oracle.install.crs.config.configureAsExtendedCluster=false \
oracle.install.crs.config.gpnp.scanPort=1521 \
oracle.install.crs.config.gpnp.configureGNS=false \
oracle.install.option=CRS_CONFIG \
oracle.install.asm.SYSASMPassword=Oracle19C \
oracle.install.asm.monitorPassword=Oracle19C \
oracle.install.asm.diskGroup.diskDiscoveryString=/dev/asm_* \
oracle.install.asm.diskGroup.disks=/dev/asm_ocr01,/dev/asm_ocr02,/dev/asm_ocr03 \
oracle.install.crs.config.gpnp.scanName=jiekexu-racscan \
oracle.install.crs.config.clusterName=jiekexu-racscan \
oracle.install.crs.config.autoConfigureClusterNodeVIP=false \
oracle.install.crs.config.clusterNodes=jiekexu-r1:jiekexu-r1-vip,jiekexu-r2:jiekexu-r2-vip \
oracle.install.crs.config.networkInterfaceList=ens32:192.168.75.0:1,ens33:10.10.10.0:5 \
oracle.install.asm.configureGIMRDataDG=false \
oracle.install.crs.configureRHPS=false \
oracle.install.crs.config.ignoreDownNodes=false \
oracle.install.config.managementOption=NONE \
oracle.install.config.omsPort=0 \
oracle.install.crs.rootconfig.executeRootScript=false
一、出现问题
完成此步骤后，则需要执行 root.sh ,当在节点一执行 root.sh 时便出现了报错。
查看上图中 asmca 日志 ORA-39511 haip 无法启动，查看 mos 说与网卡有问题，则将网卡删除重建，甚至怀疑节点一主机的问题，弃用节点一主机，使用节点二主机复制新生成主机一，然后重新安装 GI。
ORA-39511: Start of CRS resource for instance ‘215’ failed with error:[CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘jiekexu-r1’
即使这样，当到这一步执行 root.sh 时报错还是一样，删网卡，加网卡，来来回回折腾了不下十次，花费了几天的时间，直到今天早上又重新配置了网卡删除了残余的 GI ，使用 ssh-keygen 配置互信，又来了一遍可还是同样的错误，真让人头疼，欲哭无泪了。。。
这里贴一下删除过程，以备不时之需。
**如果脚本执行错误，重复执行也无法修复时，则需要使用 deinstall 卸载重新安装。**
cd $ORACLE_HOME/deinstall
./deinstall
--根据提示输入和执行脚本
n n y 另开会话执行脚本 ./rootcrs.sh -force -deconfig -paramfile ……，等待十几分钟
rm -rf /opt/ORCLfmap
rm -rf /etc/oraInst.loc
--检查所有节点的配置文件/etc/oraInst.loc、/etc/oratab是否存在相关配置，若有，则删除之。
--使用 root 用户登录所有节点，执行：
rm -rf /etc/oracle /tmp/.oracle /var/tmp/.oracle /u01/app/oraInventory /u01/app/grid/* /tmp/GridSetupActions*
--卸载后新建 mkdir -p /u01/app/19.0.0/grid
chown -R grid:oinstall /u01
chmod -R 775 /u01
chown -R oracle:oinstall /u01/app/oracle
另一节点如果还没执行 root.sh 则直接 /u01/app/19.0.0/grid ：
cd $ORACLE_HOME
rm -rf * rm -rf .opatchauto_storage rm -rf .patch_storage
cd /u01/app/grid
rm -rf *
rm -rf /u01/app/oraInventory
在这期间，修改了 VMware 虚拟网卡 VMnet1 的配置，这是关键的一步.由于之前虚拟机没有使用过 10.10.10 网段的地址，VMnet1 还是原来的 192.168 网段地址。这里将其修改为 10.10.10.0 网段，注意子网掩码改为 255.255.255.0
然后下午排查问题时将 ens32 ens33 两块网卡配置为如下：
cat > /etc/sysconfig/network-scripts/ifcfg-ens32 <<"EOF"
DEVICE=ens32
NAME=ens32
IPADDR=192.168.75.128
NETMASK=255.255.255.0
GATEWAY=192.168.75.2
ONBOOT=yes
USERCTL=no
BOOTPROTO=static
HWADDR=00:0c:29:e4:ce:69
TYPE=Ethernet
IPV6INIT=no
DNS1=114.114.114.114
DNS2=8.8.8.8
NM_CONTROLLED=no
EOF
--注意第二块网卡不配置网关，否则系统默认网关就变成了第二块网卡的网关，从而导致系统不能上外网。
cat > /etc/sysconfig/network-scripts/ifcfg-ens33 <<"EOF"
DEVICE=ens33
NAME=ens33
IPADDR=10.10.10.128
NETMASK=255.0.0.0
ONBOOT=yes
USERCTL=no
BOOTPROTO=static
HWADDR=00:0c:29:e4:ce:73
TYPE=Ethernet
IPV6INIT=no
DNS1=114.114.114.114
DNS2=8.8.8.8
NM_CONTROLLED=no
EOF
顺便使用 kfod 排查了共享磁盘的配置是否正确的问题
[grid@jiekexu-r1:/home/grid]$ kfod disks=all status=true
Disk Size Header Path User Group
1: 10240 MB CANDIDATE /dev/asm_arch01 grid asmadmin
2: 10240 MB CANDIDATE /dev/asm_data01 grid asmadmin
3: 10240 MB CANDIDATE /dev/asm_data02 grid asmadmin
4: 3072 MB CANDIDATE /dev/asm_ocr01 grid asmadmin
5: 3072 MB CANDIDATE /dev/asm_ocr02 grid asmadmin
6: 3072 MB CANDIDATE /dev/asm_ocr03 grid asmadmin
KFOD-00313: No ASM instances available. CSS group services were successfully initialized by kgxgncin.
--直接使用 asmca 静默创建 OCR 也是同样的错误
[grid@jiekexu-r1:/home/grid]$ /u01/app/19.0.0/grid/bin/asmca -silent -diskGroupName OCR -diskList '/dev/asm_ocr03,/dev/asm_ocr01,/dev/asm_ocr02' -redundancy NORMAL -diskString '/dev/asm_*' -configureLocalASM -passwordFileLocation +OCR/orapwASM -au_size 4
ASM failed to start. Check /u01/app/grid/cfgtoollogs/asmca/asmca-210831PM030816.log for details.
[grid@jiekexu-r1:/home/grid]$ more /u01/app/grid/cfgtoollogs/asmca/asmca-210831PM030816.log
[USMInstance.configureLocalASM:3413] ORA-39511: Start of CRS resource for instance '215' failed with error:[CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'jiekexu-r1'
二、下午定位问题
既然是 HAIP 问题，那么能不能将其禁用掉，然后再执行 root.sh 呢？死马当活马医吧！！！
cd /u01/app/19.0.0/grid/bin
[root@jiekexu-r1 bin]# ./crsctl modify res ora.cluster_interconnect.haip -attr "ENABLED=0" -init
[root@jiekexu-r1 bin]# ./crsctl modify res ora.asm -attr "START_DEPENDENCIES='hard(ora.cssd,ora.ctssd)pullup(ora.cssd,ora.ctssd)weak(ora.drivers.acfs)',STOP_DEPENDENCIES='hard(intermediate:ora.cssd)'" -init
[root@jiekexu-r1 bin]#
[root@jiekexu-r1 bin]# /u01/app/19.0.0/grid/root.sh
禁用掉后执行 root.sh 如下报错： ORA-27504
[main] [ 2021-08-31 17:57:27.543 CST ] [UsmcaLogger.logException:187] ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:if_not_found failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgxpvaddr9
ORA-27303: additional information: requested interface 48.0.0.0 not found. Check output from ifconfig command
看情况还是网卡相关的问题，按照 MOS 上的文章(Doc ID 2356980.1)尝试了一下还是一样的报错：
Error ORA-27504: IPC error creating OSD context - ASM Failed to Start After OS Kernel Patching (Doc ID 2356980.1)
[root@jiekexu-r1 grid]# cd /u01/app/19.0.0/grid/bin
[root@jiekexu-r1 bin]# cd ..
[root@jiekexu-r1 grid]# cd crs/install/
[root@jiekexu-r1 install]# ./rootcrs.sh -unlock
[grid@jiekexu-r1:/home/grid]$ make -f $ORACLE_HOME/rdbms/lib/ins_rdbms.mk ipc_g ioracle
[root@jiekexu-r1 install]# ./rootcrs.sh -lock
报错一样，则先将禁用的 HAIP 回退掉。
[root@jiekexu-r1 bin]# ./crsctl modify res ora.cluster_interconnect.haip -attr "ENABLED=1" -init
[root@jiekexu-r1 bin]# ./crsctl modify res ora.asm -attr "START_DEPENDENCIES='hard(ora.cssd,ora.ctssd)pullup(ora.cssd,ora.ctssd)weak(ora.cluster_interconnect.haip,ora.drivers.acfs)',STOP_DEPENDENCIES='hard(intermediate:ora.cssd,shutdown:ora.cluster_interconnect.haip)'" -init
思来想去，还是在看看日志吧，除了安装时看的 rootcrs 日志和 asmca-210831PM061542.log 日志外，需要再看看 gipcd.trc日志，由于 19c 日志发生变化，则需要到如下路径下查看。日志发生变化的新特新之前也写过一篇文章，感兴趣的小伙伴可戳此查看。
/u01/app/grid/diag/crs/jiekexu-r1/crs/trace
tail -30f gipcd.trc
2021-08-31 18:06:58.778 :GIPCDCLT:3717416704: gipcdClientInterfaceQueryToMonitor: enqueue local interface query (2) to worklist
2021-08-31 18:06:58.778 :GIPCDCLT:3717416704: gipcdClientInterfaceRequest: sent local interface query
2021-08-31 18:06:58.827 : CLSINET:3717416704: (:CLSINE0018:)WARNING: failed to find interface available for interface definition ens33(:.*)?:10.10.10.0
2021-08-31 18:06:58.829 :GIPCDCLT:3717416704: gipcdUpdateLocalClient: successfully sent the inf update
2021-08-31 18:06:58.829 :GIPCDCLT:3717416704: gipcdClientInterfaceRequest: sent local interface list back to client
2021-08-31 18:06:58.830 : CLSINET:3713214208: (:CLSINE0018:)WARNING: failed to find interface available for interface definition ens33(:.*)?:10.10.10.0
2021-08-31 18:06:59.267 :GIPCDCLT:3717416704: gipcdClientThread: req from local client of type gipcdmsgtypeInterfaceRequest, endp 0000000000000395
2021-08-31 18:06:59.267 :GIPCDCLT:3717416704: gipcdClientInterfaceRequest: Received type(gipcdmsgtypeInterfaceRequest), endp(0000000000000395), len(1024), buf(0x7fd2c432c4a8):[hostname(jiekexu-r1), retStatus(gipcretSuccess)]
2021-08-31 18:06:59.267 :GIPCDCLT:3717416704: gipcdClientInterfaceQueryToMonitor: enqueue local interface query (2) to worklist
2021-08-31 18:06:59.267 :GIPCDCLT:3717416704: gipcdClientInterfaceRequest: sent local interface query
2021-08-31 18:06:59.315 : CLSINET:3717416704: (:CLSINE0018:)WARNING: failed to find interface available for interface definition ens33(:.*)?:10.10.10.0
2021-08-31 18:06:59.317 :GIPCDCLT:3717416704: gipcdUpdateLocalClient: successfully sent the inf update
2021-08-31 18:06:59.317 :GIPCDCLT:3717416704: gipcdClientInterfaceRequest: sent local interface list back to client
2021-08-31 18:06:59.831 :GIPCDMON:3713214208: gipcdMonitorCheckXfer: set new infQuery
2021-08-31 18:06:59.831 :GIPCDMON:3713214208: gipcdMonitorCheckXfer: set new infQuery
2021-08-31 18:06:59.932 : CLSINET:3713214208: (:CLSINE0018:)WARNING: failed to find interface available for interface definition ens33(:.*)?:10.10.10.
failed to find interface available for interface definition ens33(:.*)?:10.10.10.0
果然报错 ens33 网卡接口的问题，那么到底什么问题呢？网卡都重建过，还有哪里的问题？？？抱着尝试的心态想到好心的网友说的将子网掩码改为 24 试试，那就试试吧。
cd /etc/sysconfig/network-scripts/
--重启网卡后查看 gipcd.trc 文件则已经没有报错了
ifdown ens33
ifup ens33
--执行 root.sh 脚本成功，历时八天终于成功了
[root@jiekexu-r1 network-scripts]# cat /u01/app/19.0.0/grid/install/root_jiekexu-r1_2021-08-31_18-14-46-714402688.log
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/19.0.0/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/jiekexu-r1/crsconfig/rootcrs_jiekexu-r1_2021-08-31_06-14-47PM.log
2021/08/31 18:14:50 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2021/08/31 18:14:50 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2021/08/31 18:14:50 CLSRSC-363: User ignored prerequisites during installation
2021/08/31 18:14:50 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2021/08/31 18:14:52 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2021/08/31 18:14:53 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
2021/08/31 18:14:53 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2021/08/31 18:14:53 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2021/08/31 18:14:54 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2021/08/31 18:14:57 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2021/08/31 18:14:57 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2021/08/31 18:14:57 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2021/08/31 18:14:58 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2021/08/31 18:15:00 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2021/08/31 18:15:00 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2021/08/31 18:15:01 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2021/08/31 18:15:04 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
2021/08/31 18:15:22 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
ASM has been created and started successfully.
[DBT-30001] Disk groups created successfully. Check /u01/app/grid/cfgtoollogs/asmca/asmca-210831PM061542.log for details.
2021/08/31 18:16:34 CLSRSC-482: Running command: '/u01/app/19.0.0/grid/bin/ocrconfig -upgrade grid oinstall'
CRS-4256: Updating the profile
Successful addition of voting disk 14ed0aa13ffd4f89bfe2d79061f96fbc.
Successful addition of voting disk 27cc8fbc135f4fd3bf574b4d2e62531e.
Successful addition of voting disk c5c806e3a2414f74bf1c70f2add4a821.
Successfully replaced voting disk group with +OCR.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 14ed0aa13ffd4f89bfe2d79061f96fbc (/dev/asm_ocr03) [OCR]
2. ONLINE 27cc8fbc135f4fd3bf574b4d2e62531e (/dev/asm_ocr01) [OCR]
3. ONLINE c5c806e3a2414f74bf1c70f2add4a821 (/dev/asm_ocr02) [OCR]
Located 3 voting disk(s).
2021/08/31 18:17:29 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2021/08/31 18:18:28 CLSRSC-343: Successfully started Oracle Clusterware stack
2021/08/31 18:18:28 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2021/08/31 18:20:11 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2021/08/31 18:20:38 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
节点 2 执行 root.sh 也成功了，兴奋中……
[grid@jiekexu-r2:/home/grid]$ tail -30f /u01/app/19.0.0/grid/install/root_jiekexu-r2_2021-08-31_18-25-18-796337893.log
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/19.0.0/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19.0.0/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/grid/crsdata/jiekexu-r2/crsconfig/rootcrs_jiekexu-r2_2021-08-31_06-25-49PM.log
2021/08/31 18:25:53 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2021/08/31 18:25:53 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2021/08/31 18:25:53 CLSRSC-363: User ignored prerequisites during installation
2021/08/31 18:25:53 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2021/08/31 18:25:55 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2021/08/31 18:25:55 CLSRSC-594: Executing installation step 5 of 19: 'SetupOSD'.
Redirecting to /bin/systemctl restart rsyslog.service
2021/08/31 18:25:56 CLSRSC-594: Executing installation step 6 of 19: 'CheckCRSConfig'.
2021/08/31 18:25:58 CLSRSC-594: Executing installation step 7 of 19: 'SetupLocalGPNP'.
2021/08/31 18:26:02 CLSRSC-594: Executing installation step 8 of 19: 'CreateRootCert'.
2021/08/31 18:26:02 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2021/08/31 18:26:16 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2021/08/31 18:26:16 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2021/08/31 18:26:20 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2021/08/31 18:26:20 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2021/08/31 18:26:43 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2021/08/31 18:26:43 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2021/08/31 18:27:30 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2021/08/31 18:27:33 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
2021/08/31 18:27:42 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
2021/08/31 18:27:56 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2021/08/31 18:28:28 CLSRSC-343: Successfully started Oracle Clusterware stack
2021/08/31 18:28:29 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2021/08/31 18:29:13 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2021/08/31 18:29:19 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
执行完毕后查看集群状态如下：
配置集群
根据静默安装后的提示内容，执行完 root.sh 脚本后，需要执行 executeConfigTools 配置，更新节点信息。
/u01/app/19.0.0/grid/gridSetup.sh -silent -executeConfigTools -waitforcompletion \
-responseFile /u01/app/19.0.0/grid/install/response/gridsetup.rsp \
INVENTORY_LOCATION=/u01/app/oraInventory \
ORACLE_BASE=/u01/app/grid \
ORACLE_HOME=/u01/app/19.0.0/grid \
oracle.install.asm.OSDBA=asmdba \
oracle.install.asm.OSOPER=asmoper \
oracle.install.asm.OSASM=asmadmin \
oracle.install.crs.config.storageOption=ASM \
oracle.install.crs.config.useIPMI=false \
oracle.install.asm.diskGroup.name=OCR \
oracle.install.asm.diskGroup.AUSize=4 \
oracle.install.asm.diskGroup.redundancy=NORMAL \
oracle.install.crs.config.ClusterConfiguration=STANDALONE \
oracle.install.crs.config.configureAsExtendedCluster=false \
oracle.install.crs.config.gpnp.scanPort=1521 \
oracle.install.crs.config.gpnp.configureGNS=false \
oracle.install.option=CRS_CONFIG \
oracle.install.asm.SYSASMPassword=Oracle19C \
oracle.install.asm.monitorPassword=Oracle19C \
oracle.install.asm.diskGroup.diskDiscoveryString=/dev/asm_* \
oracle.install.asm.diskGroup.disks=/dev/asm_ocr01,/dev/asm_ocr02,/dev/asm_ocr03 \
oracle.install.crs.config.gpnp.scanName=jiekexu-racscan \
oracle.install.crs.config.clusterName=jiekexu-racscan \
oracle.install.crs.config.autoConfigureClusterNodeVIP=false \
oracle.install.crs.config.clusterNodes=jiekexu-r1:jiekexu-r1-vip,jiekexu-r2:jiekexu-r2-vip \
oracle.install.crs.config.networkInterfaceList=ens32:192.168.75.0:1,ens33:10.10.10.0:5 \
oracle.install.asm.configureGIMRDataDG=false \
oracle.install.crs.configureRHPS=false \
oracle.install.crs.config.ignoreDownNodes=false \
oracle.install.config.managementOption=NONE \
oracle.install.config.omsPort=0 \
oracle.install.crs.rootconfig.executeRootScript=false
Launching Oracle Grid Infrastructure Setup Wizard...
You can find the logs of this session at:
/u01/app/oraInventory/logs/GridSetupActions2021-08-31_06-44-55PM
You can find the log of this install session at:
/u01/app/oraInventory/logs/UpdateNodeList2021-08-31_06-44-55PM.log
Configuration failed.
[WARNING] [INS-43080] Some of the configuration assistants failed, were cancelled or skipped.
ACTION: Refer to the logs or contact Oracle Support Services.
注意：在 /u01/app/19.0.0/grid/install/response 目录下，新生成了 grid_2021-08-31_02-31-29PM.rsp 响应文件，可以使用新生成的，我这里沿用了前面的配置项也可以。
三、总结归纳
此次问题主要是出在 ens33 私网网卡上，由于在虚拟机外部的网络 VMnet1 中设置了子网掩码为 24，但在内部虚拟机中配置 ens33 时却将子网掩码写成了 8 导致 ens33 网卡不可用，不管新建网卡，还是新建虚拟机都没有想到要改 IP 这块，导致使用了原来的错误配置进行安装，故每次 root.sh 都是同样的问题。
在 RAC 安装配置过程中，每一步都很重要，都需要仔细认真，不可马虎，要做到一丝不苟。利用各种日志排查分析问题，如安装过程中产生的日志，集群启动过程中的各个进程日志以及操作系统日志等等，这些均是分析问题的关键因素，本次问题主要就是忽略了各个进程日志才导致了很多重复工作，以后需要引以为戒。
最后说一句子网掩码 (subnet mask) 又叫网络掩码、地址掩码、子网络遮罩，是一个应用于 TCP/IP 网络的 32 位二进制值。它可以屏蔽掉 IP 地址中的一部分，从而分离出 IP 地址中的网络部分与主机部分，基于子网掩码，管理员可以将网络进一步划分为若干子网。它必须结合 IP 地址一起使用。 子网掩码设置为 24 则只有 254 个可用地址，设置为 8 则有16777214 个地址可用。关于网络问题不是很懂，这里有一篇知乎上的帖子说明子网掩码是个啥？
https://www.zhihu.com/question/56895036/answer/150953183
最大踩坑，真心不易，耐心阅读，此文一定对您有帮助，欢迎点赞、在看与转发，写作不易，坚持写作更不易，您的点赞、转发，举手之劳，便是对我最大的支持，也能让更多的人看到更多的人受益，万分感谢！
———————————————————————————
公众号：JiekeXu DBA之路
墨天轮：https://www.modb.pro/u/4347
CSDN ：https://blog.csdn.net/JiekeXu
腾讯云：https://cloud.tencent.com/developer/user/5645107
———————————————————————————
【安装篇】- 基于 VMWARE Oracle Linux7.9 安装 Oracle19c RAC 详细配置方案
2021年7月国产数据库排行榜:openGauss高歌猛进，GBase持续下跌
Oracle 12c 及以上版本补丁更新说明及下载方法（收藏版）
Oracle 19c 19.10DBRU 最新补丁升级看这一篇就够了
Redhat 7.7 安装最新版 MongoDB 5.0.1 手册
ASM 管理的内部工具：KFED、KFOD、AMDU
性能优化|关于数据库历史性能问题的一道面试题
一线运维 DBA 五年经验常用 SQL 大全（二）
ORA-00349|激活 ADG 备库时遇到的问题
Oracle 轻量级实时监控工具 oratop
MySQL OCP 认证考试你知道吗？
Oracle 19C RAC 安装遇到的坑
国产数据库|TiDB 5.0 快速体验
Oracle 19C MAA 搭建指南
2021 年 8 月数据库英雄榜
Oracle 每日一题系列合集
百花齐放的国产数据库
