RedHat 5.6_x86_64 + ASM + RAW+ Oracle 10g RAC （八） 添加节点―安装clusterware

1、Extending the Oracle Clusterware Home  

   以oracle用户身份登录节点jason1（非新添加节点执行）中，进入$ORA_CRS_HOME/oui/bin目录执行./addNode.sh脚本Extending the Oracle Clusterware Home目录.如下：

点击下一步

填写第三节点信息

点击install，开始安装。

根据上图提示依次执行脚本，如下：

节点jason3

[root@jason3 ~]# cd /home/oracle/orabase/oraInventory/

[root@jason3 oraInventory]# ./orainstRoot.sh

Changing permissions of /home/oracle/orabase/oraInventory to 770.

Changing groupname of /home/oracle/orabase/oraInventory to oinstall.

The execution of the script is complete

[root@jason3 oraInventory]#

节点jason1

[root@jason1 mnt]# cd /home/oracle/orahome/product/10.2.0/crs/

[root@jason1 crs]# cd install

[root@jason1 install]# ./rootaddnode.sh

clscfg: EXISTING configuration version 3 detected.

clscfg: version 3 is 10G Release 2.

Attempting to add 1 new nodes to the configuration

Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.

node <nodenumber>: <nodename> <private interconnect name> <hostname>

node 3: jason3 jason3-priv jason3

Creating OCR keys for user 'root', privgrp 'root'..

Operation successful.

/home/oracle/orahome/product/10.2.0/crs/bin/srvctl add nodeapps -n jason3 -A jason3-vip/255.255.255.0/eth0 -o /home/oracle/orahome/product/10.2.0/crs

[root@jason1 install]#

节点jason3

[root@jason3 oraInventory]# cd /home/oracle/orahome/product/10.2.0/crs/

[root@jason3 crs]# pwd

/home/oracle/orahome/product/10.2.0/crs

[root@jason3 crs]# ./root.sh

WARNING: directory '/home/oracle/orahome/product/10.2.0' is not owned by root

WARNING: directory '/home/oracle/orahome/product' is not owned by root

WARNING: directory '/home/oracle/orahome' is not owned by root

WARNING: directory '/home/oracle' is not owned by root

No value set for the CRS parameter CRS_OCR_LOCATIONS. Using Values in paramfile.crs

Checking to see if Oracle CRS stack is already configured

/etc/oracle does not exist. Creating it now.

OCR LOCATIONS =  /dev/raw/raw1,/dev/raw/raw2

OCR backup directory '/home/oracle/orahome/product/10.2.0/crs/cdata/crs' does not exist. Creating now

Setting the permissions on OCR backup directory

Setting up NS directories

Oracle Cluster Registry configuration upgraded successfully

WARNING: directory '/home/oracle/orahome/product/10.2.0' is not owned by root

WARNING: directory '/home/oracle/orahome/product' is not owned by root

WARNING: directory '/home/oracle/orahome' is not owned by root

WARNING: directory '/home/oracle' is not owned by root

clscfg: EXISTING configuration version 3 detected.

clscfg: version 3 is 10G Release 2.

Successfully accumulated necessary OCR keys.

Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.

node <nodenumber>: <nodename> <private interconnect name> <hostname>

node 1: jason1 jason1-priv jason1

node 2: jason2 jason2-priv jason2

clscfg: Arguments check out successfully.

NO KEYS WERE WRITTEN. Supply -force parameter to override.

-force is destructive and will destroy any previous cluster

configuration.

Oracle Cluster Registry for cluster has already been initialized

Startup will be queued to init within 30 seconds.

Adding daemons to inittab

Expecting the CRS daemons to be up within 600 seconds.

CSS is active on these nodes.

       jason1

       jason2

       jason3

CSS is active on all nodes.

Waiting for the Oracle CRSD and EVMD to start

Oracle CRS stack installed and running under init(1M)

[root@jason3 crs]# su - oracle

[oracle@jason3 ~]$ crs_stat -t

Name           Type           Target    State     Host        

------------------------------------------------------------

ora....N1.inst application    ONLINE    ONLINE    jason1      

ora....N2.inst application    ONLINE    ONLINE    jason2      

ora.JASON.db   application    ONLINE    ONLINE    jason2      

ora....SM1.asm application    ONLINE    ONLINE    jason1      

ora....N1.lsnr application    ONLINE    ONLINE    jason1      

ora.jason1.gsd application    ONLINE    ONLINE    jason1      

ora.jason1.ons application    ONLINE    ONLINE    jason1      

ora.jason1.vip application    ONLINE    ONLINE    jason1      

ora....SM2.asm application    ONLINE    ONLINE    jason2      

ora....N2.lsnr application    ONLINE    ONLINE    jason2      

ora.jason2.gsd application    ONLINE    ONLINE    jason2      

ora.jason2.ons application    ONLINE    ONLINE    jason2      

ora.jason2.vip application    ONLINE    ONLINE    jason2      

ora.jason3.gsd application    ONLINE    ONLINE    jason3      

ora.jason3.ons application    ONLINE    ONLINE    jason3      

ora.jason3.vip application    ONLINE    ONLINE    jason3

[oracle@jason3 ~]$

执行完成脚本后回到图像界面点击ok并退出安装界面。

2、添加新节点jason3 Oracle Notification Services (ONS)配信息到Oracle Cluster Registry (OCR)

   在jason1节点中获取ons相关信息。

[oracle@jason1 ~]$ cat $ORA_CRS_HOME/opmn/conf/ons.config

localport=6113

remoteport=6200

loglevel=3

useocr=on

添加配置信息到OCR，使用命令为./racgons add_config docrac3:remote_port

docrac3：为需要添加的节点名称。

remote_port：上面查出的remoteport。

[oracle@jason1 ~]$ cd /home/oracle/orahome/product/10.2.0/crs/bin/

[oracle@jason1 bin]$ ./racgons add_config jason3:6200

[oracle@jason1 bin]$

添加完成之后再节点三jason3查看状态如下：

[oracle@jason3 ~]$ crs_stat -t

Name           Type           Target    State     Host        

------------------------------------------------------------

ora....N1.inst application    ONLINE    ONLINE    jason1      

ora....N2.inst application    ONLINE    ONLINE    jason2      

ora.JASON.db   application    ONLINE    ONLINE    jason2      

ora....SM1.asm application    ONLINE    ONLINE    jason1      

ora....N1.lsnr application    ONLINE    ONLINE    jason1      

ora.jason1.gsd application    ONLINE    ONLINE    jason1      

ora.jason1.ons application    ONLINE    ONLINE    jason1      

ora.jason1.vip application    ONLINE    ONLINE    jason1      

ora....SM2.asm application    ONLINE    ONLINE    jason2      

ora....N2.lsnr application    ONLINE    ONLINE    jason2      

ora.jason2.gsd application    ONLINE    ONLINE    jason2      

ora.jason2.ons application    ONLINE    ONLINE    jason2      

ora.jason2.vip application    ONLINE    ONLINE    jason2      

ora.jason3.gsd application    ONLINE    ONLINE    jason3      

ora.jason3.ons application    ONLINE    ONLINE    jason3      

ora.jason3.vip application    ONLINE    ONLINE    jason3

[oracle@jason3 ~]$

在新节点上已安装集群软件，可以通过如下命令进行验证

$ORA_CRS_HOME/bin/cluvfy stage -post crsinst -n docrac3 -verbose

docrac3为新添加的节点名称。

   官方文档要求在新添加节点执行验证，但是自己实验时ssh等效性经测试无任何问题，在新添加节点执行报错

[oracle@jason3 ~]$ cluvfy stage -post crsinst -n jason3 -verbose

Performing post-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "jason3"

 Destination Node                      Reachable?              

 ------------------------------------  ------------------------

 jason3                                yes                    

Result: Node reachability check passed from node "jason3".

Checking user equivalence...

Check: User equivalence for user "oracle"

 Node Name                             Comment                

 ------------------------------------  ------------------------

 jason3                                failed                  

Result: User equivalence check failed for user "oracle".

ERROR:

User equivalence unavailable on all the nodes.

Verification cannot proceed.

Post-check for cluster services setup was unsuccessful on all the nodes.

[oracle@jason3 ~]$

在节点1执行却正常如下：

[oracle@jason1 bin]$  ./cluvfy stage -post crsinst -n jason3 -verbose

Performing post-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "jason1"

 Destination Node                      Reachable?              

 ------------------------------------  ------------------------

 jason3                                yes                    

Result: Node reachability check passed from node "jason1".

Checking user equivalence...

Check: User equivalence for user "oracle"

 Node Name                             Comment                

 ------------------------------------  ------------------------

 jason3                                passed                  

Result: User equivalence check passed for user "oracle".

Checking Cluster manager integrity...

Checking CSS daemon...

 Node Name                             Status                  

 ------------------------------------  ------------------------

 jason3                                running                

Result: Daemon status check passed for "CSS daemon".

Cluster manager integrity check passed.

Checking cluster integrity...

 Node Name                          

 ------------------------------------

 jason1                              

 jason2                              

 jason3                              

Cluster integrity check passed

Checking OCR integrity...

Checking the absence of a non-clustered configuration...

All nodes free of non-clustered, local-only configurations.

Uniqueness check for OCR device passed.

Checking the version of OCR...

OCR of correct Version "2" exists.

Checking data integrity of OCR...

Data integrity check for OCR passed.

OCR integrity check passed.

Checking CRS integrity...

Checking daemon liveness...

Check: Liveness for "CRS daemon"

 Node Name                             Running                

 ------------------------------------  ------------------------

 jason3                                yes                    

Result: Liveness check passed for "CRS daemon".

Checking daemon liveness...

Check: Liveness for "CSS daemon"

 Node Name                             Running                

 ------------------------------------  ------------------------

 jason3                                yes                    

Result: Liveness check passed for "CSS daemon".

Checking daemon liveness...

Check: Liveness for "EVM daemon"

 Node Name                             Running                

 ------------------------------------  ------------------------

 jason3                                yes                    

Result: Liveness check passed for "EVM daemon".

Liveness of all the daemons

 Node Name     CRS daemon                CSS daemon                EVM daemon

 ------------  ------------------------  ------------------------  ----------

 jason3        yes                       yes                       yes      

Checking CRS health...

Check: Health of CRS

 Node Name                             CRS OK?                

 ------------------------------------  ------------------------

 jason3                                yes                    

Result: CRS health check passed.

CRS integrity check passed.

Checking node application existence...

Checking existence of VIP node application

 Node Name     Required                  Status                    Comment  

 ------------  ------------------------  ------------------------  ----------

 jason3        yes                       exists                    passed    

Result: Check passed.

Checking existence of ONS node application

 Node Name     Required                  Status                    Comment  

 ------------  ------------------------  ------------------------  ----------

 jason3        no                        exists                    passed    

Result: Check passed.

Checking existence of GSD node application

 Node Name     Required                  Status                    Comment  

 ------------  ------------------------  ------------------------  ----------

 jason3        no                        exists                    passed    

Result: Check passed.

Post-check for cluster services setup was successful.

[oracle@jason1 bin]$