RedHat 5.6_x86_64 + ASM + RAW+ Oracle 10g RAC （八） 添加节点―安装clusterware
1、Extending the Oracle Clusterware Home
以oracle用户身份登录节点jason1（非新添加节点执行）中，进入$ORA_CRS_HOME/oui/bin目录执行./addNode.sh脚本Extending the Oracle Clusterware Home目录.如下：
点击下一步
填写第三节点信息
点击install，开始安装。
根据上图提示依次执行脚本，如下：
节点jason3
[root@jason3 ~]# cd /home/oracle/orabase/oraInventory/
[root@jason3 oraInventory]# ./orainstRoot.sh
Changing permissions of /home/oracle/orabase/oraInventory to 770.
Changing groupname of /home/oracle/orabase/oraInventory to oinstall.
The execution of the script is complete
[root@jason3 oraInventory]#
节点jason1
[root@jason1 mnt]# cd /home/oracle/orahome/product/10.2.0/crs/
[root@jason1 crs]# cd install
[root@jason1 install]# ./rootaddnode.sh
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Attempting to add 1 new nodes to the configuration
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 3: jason3 jason3-priv jason3
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
/home/oracle/orahome/product/10.2.0/crs/bin/srvctl add nodeapps -n jason3 -A jason3-vip/255.255.255.0/eth0 -o /home/oracle/orahome/product/10.2.0/crs
[root@jason1 install]#
节点jason3
[root@jason3 oraInventory]# cd /home/oracle/orahome/product/10.2.0/crs/
[root@jason3 crs]# pwd
/home/oracle/orahome/product/10.2.0/crs
[root@jason3 crs]# ./root.sh
WARNING: directory '/home/oracle/orahome/product/10.2.0' is not owned by root
WARNING: directory '/home/oracle/orahome/product' is not owned by root
WARNING: directory '/home/oracle/orahome' is not owned by root
WARNING: directory '/home/oracle' is not owned by root
No value set for the CRS parameter CRS_OCR_LOCATIONS. Using Values in paramfile.crs
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
OCR LOCATIONS = /dev/raw/raw1,/dev/raw/raw2
OCR backup directory '/home/oracle/orahome/product/10.2.0/crs/cdata/crs' does not exist. Creating now
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/home/oracle/orahome/product/10.2.0' is not owned by root
WARNING: directory '/home/oracle/orahome/product' is not owned by root
WARNING: directory '/home/oracle/orahome' is not owned by root
WARNING: directory '/home/oracle' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: jason1 jason1-priv jason1
node 2: jason2 jason2-priv jason2
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
jason1
jason2
jason3
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
[root@jason3 crs]# su - oracle
[oracle@jason3 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....N1.inst application ONLINE ONLINE jason1
ora....N2.inst application ONLINE ONLINE jason2
ora.JASON.db application ONLINE ONLINE jason2
ora....SM1.asm application ONLINE ONLINE jason1
ora....N1.lsnr application ONLINE ONLINE jason1
ora.jason1.gsd application ONLINE ONLINE jason1
ora.jason1.ons application ONLINE ONLINE jason1
ora.jason1.vip application ONLINE ONLINE jason1
ora....SM2.asm application ONLINE ONLINE jason2
ora....N2.lsnr application ONLINE ONLINE jason2
ora.jason2.gsd application ONLINE ONLINE jason2
ora.jason2.ons application ONLINE ONLINE jason2
ora.jason2.vip application ONLINE ONLINE jason2
ora.jason3.gsd application ONLINE ONLINE jason3
ora.jason3.ons application ONLINE ONLINE jason3
ora.jason3.vip application ONLINE ONLINE jason3
[oracle@jason3 ~]$
执行完成脚本后回到图像界面点击ok并退出安装界面。
2、添加新节点jason3 Oracle Notification Services (ONS)配信息到Oracle Cluster Registry (OCR)
在jason1节点中获取ons相关信息。
[oracle@jason1 ~]$ cat $ORA_CRS_HOME/opmn/conf/ons.config
localport=6113
remoteport=6200
loglevel=3
useocr=on
添加配置信息到OCR，使用命令为./racgons add_config docrac3:remote_port
docrac3：为需要添加的节点名称。
remote_port：上面查出的remoteport。
[oracle@jason1 ~]$ cd /home/oracle/orahome/product/10.2.0/crs/bin/
[oracle@jason1 bin]$ ./racgons add_config jason3:6200
[oracle@jason1 bin]$
添加完成之后再节点三jason3查看状态如下：
[oracle@jason3 ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....N1.inst application ONLINE ONLINE jason1
ora....N2.inst application ONLINE ONLINE jason2
ora.JASON.db application ONLINE ONLINE jason2
ora....SM1.asm application ONLINE ONLINE jason1
ora....N1.lsnr application ONLINE ONLINE jason1
ora.jason1.gsd application ONLINE ONLINE jason1
ora.jason1.ons application ONLINE ONLINE jason1
ora.jason1.vip application ONLINE ONLINE jason1
ora....SM2.asm application ONLINE ONLINE jason2
ora....N2.lsnr application ONLINE ONLINE jason2
ora.jason2.gsd application ONLINE ONLINE jason2
ora.jason2.ons application ONLINE ONLINE jason2
ora.jason2.vip application ONLINE ONLINE jason2
ora.jason3.gsd application ONLINE ONLINE jason3
ora.jason3.ons application ONLINE ONLINE jason3
ora.jason3.vip application ONLINE ONLINE jason3
[oracle@jason3 ~]$
在新节点上已安装集群软件，可以通过如下命令进行验证
$ORA_CRS_HOME/bin/cluvfy stage -post crsinst -n docrac3 -verbose
docrac3为新添加的节点名称。
官方文档要求在新添加节点执行验证，但是自己实验时ssh等效性经测试无任何问题，在新添加节点执行报错
[oracle@jason3 ~]$ cluvfy stage -post crsinst -n jason3 -verbose
Performing post-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "jason3"
Destination Node Reachable?
------------------------------------ ------------------------
jason3 yes
Result: Node reachability check passed from node "jason3".
Checking user equivalence...
Check: User equivalence for user "oracle"
Node Name Comment
------------------------------------ ------------------------
jason3 failed
Result: User equivalence check failed for user "oracle".
ERROR:
User equivalence unavailable on all the nodes.
Verification cannot proceed.
Post-check for cluster services setup was unsuccessful on all the nodes.
[oracle@jason3 ~]$
在节点1执行却正常如下：
[oracle@jason1 bin]$ ./cluvfy stage -post crsinst -n jason3 -verbose
Performing post-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "jason1"
Destination Node Reachable?
------------------------------------ ------------------------
jason3 yes
Result: Node reachability check passed from node "jason1".
Checking user equivalence...
Check: User equivalence for user "oracle"
Node Name Comment
------------------------------------ ------------------------
jason3 passed
Result: User equivalence check passed for user "oracle".
Checking Cluster manager integrity...
Checking CSS daemon...
Node Name Status
------------------------------------ ------------------------
jason3 running
Result: Daemon status check passed for "CSS daemon".
Cluster manager integrity check passed.
Checking cluster integrity...
Node Name
------------------------------------
jason1
jason2
jason3
Cluster integrity check passed
Checking OCR integrity...
Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations.
Uniqueness check for OCR device passed.
Checking the version of OCR...
OCR of correct Version "2" exists.
Checking data integrity of OCR...
Data integrity check for OCR passed.
OCR integrity check passed.
Checking CRS integrity...
Checking daemon liveness...
Check: Liveness for "CRS daemon"
Node Name Running
------------------------------------ ------------------------
jason3 yes
Result: Liveness check passed for "CRS daemon".
Checking daemon liveness...
Check: Liveness for "CSS daemon"
Node Name Running
------------------------------------ ------------------------
jason3 yes
Result: Liveness check passed for "CSS daemon".
Checking daemon liveness...
Check: Liveness for "EVM daemon"
Node Name Running
------------------------------------ ------------------------
jason3 yes
Result: Liveness check passed for "EVM daemon".
Liveness of all the daemons
Node Name CRS daemon CSS daemon EVM daemon
------------ ------------------------ ------------------------ ----------
jason3 yes yes yes
Checking CRS health...
Check: Health of CRS
Node Name CRS OK?
------------------------------------ ------------------------
jason3 yes
Result: CRS health check passed.
CRS integrity check passed.
Checking node application existence...
Checking existence of VIP node application
Node Name Required Status Comment
------------ ------------------------ ------------------------ ----------
jason3 yes exists passed
Result: Check passed.
Checking existence of ONS node application
Node Name Required Status Comment
------------ ------------------------ ------------------------ ----------
jason3 no exists passed
Result: Check passed.
Checking existence of GSD node application
Node Name Required Status Comment
------------ ------------------------ ------------------------ ----------
jason3 no exists passed
Result: Check passed.
Post-check for cluster services setup was successful.
[oracle@jason1 bin]$