vmware server 安装oracle 10G RAC 实践（三）

NO1的配置算完成了。接下来配置NOD2。

关闭NOD1电源，拷贝NOD1目录下的文件到NOD2

在VMWARE SERVER控制台按ctrl+o，打开NOD2下的Red Hat Enterprise Linux 4.vmx

右击setting，打开options,改成NOD2

 

确定，点击start this virtul machine

选择 create a new identifier

以root用户登录修改网卡配置

出现下面错误，原因是我在公司配置143这个IP已经被使用了，该个未使用的IP地址，待会再来修改NOD1的hosts。修改主机名魏NOD2

查看主机名，添加/etc/hosts

[root@NOD1 ~]# uname -n

NOD2

[root@NOD1 ~]# vim /etc/hosts

127.0.0.1 localhost

128.1.100.144   nod1

128.1.100.145   nod2

128.1.100.201   nod1-vip

128.1.100.202   nod2-vip

10.10.10.100    nod1-priv

10.10.10.101    nod2-priv

 

修改[root@NOD1 ~]# vim /home/oracle/.bash_profile

ORACLE_SID=ORCL2

 

增加ssh用户等效性，在集群就需服务CRS和RAC安装过程中，oui必须以ORACLE用户身份将软件复制到所有RAC节点，而不提示输入口令。在ORACLE 10G中可以用ssh来完成这项工作。

重启NOD2，开启NOD1电源，我还是现在NOD1上配置

NOD2在重启过程中启动了硬件勘察工具，原因是刚才网卡配置探测了新的MAC地址，旧的没有删掉，删除掉就行了。

登录NOD1，修改/etc/hosts文件

128.1.100.145 nod2

 

以ORACLE用户生成公钥和私钥

[root@NOD1 ~]# su - oracle

[oracle@NOD1 ~]$ mkdir ~/.ssh

[oracle@NOD1 ~]$ chmod 700 ~/.ssh

[oracle@NOD1 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/oracle/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/oracle/.ssh/id_rsa.

Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.

The key fingerprint is:

60:96:10:80:22:b9:82:7b:1e:db:ab:10:78:a4:5e:fb oracle@NOD1

[oracle@NOD1 ~]$ ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/home/oracle/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/oracle/.ssh/id_dsa.

Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.

The key fingerprint is:

6c:c4:cb:94:9e:db:7d:38:02:e7:ae:f2:3a:46:88:dd oracle@NOD1

[oracle@NOD1 ~]$

 

在NOD2上执行

Last login: Thu Dec 31 10:30:10 2009 from nod1

[root@NOD2 ~]# su - oracle

[oracle@NOD2 ~]$ mkdir ~/.ssh

[oracle@NOD2 ~]$ chmod 700 ~/.ssh

[oracle@NOD2 ~]$ ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/home/oracle/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/oracle/.ssh/id_rsa.

Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.

The key fingerprint is:

6b:66:fe:ed:f1:bc:73:6e:e7:34:84:7a:43:ba:4e:03 oracle@NOD2

[oracle@NOD2 ~]$ ssh-keygen -t dsa

Generating public/private dsa key pair.

Enter file in which to save the key (/home/oracle/.ssh/id_dsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/oracle/.ssh/id_dsa.

Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.

The key fingerprint is:

84:5a:2a:6b:50:70:6f:4b:40:0b:3a:c0:64:6c:0f:f7 oracle@NOD2

[oracle@NOD2 ~]$

 

回到NOD1执行

[root@NOD1 ~]# su - oracle

[oracle@NOD1 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

[oracle@NOD1 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

[oracle@NOD1 ~]$ ssh nod2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

The authenticity of host 'nod2 (128.1.100.145)' can't be established.

RSA key fingerprint is e8:f1:5a:88:be:6e:ef:ad:5c:a1:2e:36:9c:74:4d:a0.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'nod2,128.1.100.145' (RSA) to the list of known hosts.

oracle@nod2's password:

[oracle@NOD1 ~]$ ssh nod2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

oracle@nod2's password:

[oracle@NOD1 ~]$ scp ~/.ssh/authorized_keys nod2:~/.ssh/authorized_keys

oracle@nod2's password:

authorized_keys                               100% 1644     1.6KB/s   00:00   

[oracle@NOD1 ~]$

 

验证登录是否需要密码

[oracle@NOD1 ~]$ ssh nod1 date

The authenticity of host 'nod1 (127.0.0.1)' can't be established.

RSA key fingerprint is e8:f1:5a:88:be:6e:ef:ad:5c:a1:2e:36:9c:74:4d:a0.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'nod1' (RSA) to the list of known hosts.

四 12月 31 10:57:52 CST 2009

[oracle@NOD1 ~]$ ssh nod1-priv date

The authenticity of host 'nod1-priv (10.10.10.100)' can't be established.

RSA key fingerprint is e8:f1:5a:88:be:6e:ef:ad:5c:a1:2e:36:9c:74:4d:a0.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'nod1-priv,10.10.10.100' (RSA) to the list of known hosts.

四 12月 31 10:58:00 CST 2009

[oracle@NOD1 ~]$ ssh nod2-priv date

The authenticity of host 'nod2-priv (10.10.10.101)' can't be established.

RSA key fingerprint is e8:f1:5a:88:be:6e:ef:ad:5c:a1:2e:36:9c:74:4d:a0.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'nod2-priv,10.10.10.101' (RSA) to the list of known hosts.

四 12月 31 10:50:07 CST 2009

[oracle@NOD1 ~]$

 

配置ASMLIB（两个节点上）

必须以ROOT用户执行

[root@NOD1 oracle]# /etc/init.d/oracleasm config

Usage: /etc/init.d/oracleasm {start|stop|restart|enable|disable|configure|createdisk|deletedisk|querydisk|listdisks|scandisks|status}

[root@NOD1 oracle]# /etc/init.d/oracleasm configure

Configuring the Oracle ASM library driver.

 

This will configure the on-boot properties of the Oracle ASM library

driver. The following questions will determine whether the driver is

loaded on boot and what permissions it will have. The current values

will be shown in brackets ('[]'). Hitting <ENTER> without typing an

answer will keep that current value. Ctrl-C will abort.

 

Default user to own the driver interface []: oracle

Default group to own the driver interface []: dba

Start Oracle ASM library driver on boot (y/n) [n]: y

Scan for Oracle ASM disks on boot (y/n) [y]: y

Writing Oracle ASM library driver configuration: done

Initializing the Oracle ASMLib driver: [ OK ]

Scanning the system for Oracle ASMLib disks: [ OK ]

[root@NOD1 oracle]#

 

创建ASM磁盘

[root@NOD1 oracle]# /etc/init.d/oracleasm createdisk VOL1 /dev/sdc

Marking disk "VOL1" as an ASM disk: [FAILED]

[root@NOD1 oracle]# /etc/init.d/oracleasm createdisk VOL1 /dev/sdc1

Marking disk "VOL1" as an ASM disk: [ OK ]

[root@NOD1 oracle]# /etc/init.d/oracleasm createdisk VOL2 /dev/sde1

Marking disk "VOL2" as an ASM disk: [ OK ]

[root@NOD1 oracle]# /etc/init.d/oracleasm createdisk VOL3 /dev/sdd1

Marking disk "VOL3" as an ASM disk: [ OK ]

[root@NOD1 oracle]# /etc/init.d/oracleasm scandisks

Scanning the system for Oracle ASMLib disks: [ OK ]

[root@NOD1 oracle]# /etc/init.d/oracleasm listdisks

VOL1

VOL2

VOL3

 

配置ORACLE集群文件系统OCFS2

下载ocfs2 软件包

http://oss.oracle.com/projects/ocfs2/files/RedHat/RHEL4/i386/1.2.9-1/

http://oss.oracle.com/projects/ocfs2-tools/files/RedHat/RHEL4/i386/1.2.7-1/

 

下载三个RPM包

ocfs2-2.6.9-78.EL-1.2.9-1.el4.i686.rpm

ocfs2console-1.2.7-1.el4.i386.rpm

ocfs2-tools-1.2.7-1.el4.i386.rpm

在两个节点上安装

[root@NOD1 ~]# ls

anaconda-ks.cfg     ocfs2-2.6.9-78.EL-1.2.9-1.el4.i686.rpm

Desktop             ocfs2console-1.2.7-1.el4.i386.rpm

install.log         ocfs2-tools-1.2.7-1.el4.i386.rpm

install.log.syslog

[root@NOD1 ~]# rpm -ivh ocfs2-2.6.9-78.EL-1.2.9-1.el4.i686.rpm

錯誤：相依性失敗：

        ocfs2-tools >= 1.2.6 是 ocfs2-2.6.9-78.EL-1.2.9-1.el4.i686 所需要的

[root@NOD1 ~]# rpm -ivh ocfs2-tools-1.2.7-1.el4.i386.rpm

準備中...                ########################################### [100%]

   1:ocfs2-tools            ########################################### [100%]

[root@NOD1 ~]# rpm -ivh ocfs2console-1.2.7-1.el4.i386.rpm

準備中...                ########################################### [100%]

   1:ocfs2console           ########################################### [100%]

[root@NOD1 ~]# rpm -ivh ocfs2-2.6.9-78.EL-1.2.9-1.el4.i686.rpm

準備中...                ########################################### [100%]

   1:ocfs2-2.6.9-78.EL      ########################################### [100%]

[root@NOD1 ~]# rpm -qa | grep ocfs

ocfs2-2.6.9-78.EL-1.2.9-1.el4

ocfs2-tools-1.2.7-1.el4

ocfs2console-1.2.7-1.el4

[root@NOD1 ~]#

 

打开Xmanager连接NOD1，启动终端输入

Export DISPLAY=128.1.100.204:0.0

遭遇错误

 

检查了一下内核和OCFS2包

发现时内核版本不对，还真不容易看出来

因该是

ocfs2-2.6.9-78.ELsmp-1.2.9-1.el4.i686.rpm

我下了

ocfs2-2.6.9-78.EL-1.2.9-1.el4.i686.rpm

少了一个smp。

卸载ocfs2-2.6.9-78.EL-1.2.9-1.el4

Rpm –e ocfs2-2.6.9-78.EL-1.2.9-1.el4

重新下载安装

安装完之后在两个节点上配置/etc/init.d/o2cb

[root@NOD1 ~]# /etc/init.d/o2cb configure

Configuring the O2CB driver.

 

This will configure the on-boot properties of the O2CB driver.

The following questions will determine whether the driver is loaded on

boot. The current values will be shown in brackets ('[]'). Hitting

<ENTER> without typing an answer will keep that current value. Ctrl-C

will abort.

 

Load O2CB driver on boot (y/n) [y]: y

Cluster to start on boot (Enter "none" to clear) [ocfs2]:

Specify heartbeat dead threshold (>=7) [61]: 61

Specify network idle timeout in ms (>=5000) [30000]: 30000

Specify network keepalive delay in ms (>=1000) [2000]: 2000

Specify network reconnect delay in ms (>=2000) [2000]: 2000

Writing O2CB configuration: OK

Starting O2CB cluster ocfs2: Failed

Cluster ocfs2 created

o2cb_ctl: Configuration error discovered while populating cluster ocfs2. None of its nodes were considered local. A node is considered local when its node name in the configuration matches this machine's host name.

Stopping O2CB cluster ocfs2: OK

[root@NOD1 ~]# /etc/init.d/o2cb status

Module "configfs": Loaded

Filesystem "configfs": Mounted

Module "ocfs2_nodemanager": Loaded

Module "ocfs2_dlm": Loaded

Module "ocfs2_dlmfs": Loaded

Filesystem "ocfs2_dlmfs": Mounted

Checking O2CB cluster ocfs2: Offline

[root@NOD1 ~]#

 

启动ocfs2console

添加nod1,nod2节点。

选择Propagate Configuration 复制到nod2

在NOD2上查看

[root@NOD2 ~]# cat /etc/ocfs2/cluster.conf

node:

        ip_port = 7777

        ip_address = 128.1.100.144

        number = 0

        name = nod1

        cluster = ocfs2

 

node:

        ip_port = 7777

        ip_address = 128.1.100.145

        number = 1

        name = nod2

        cluster = ocfs2

 

cluster:

        node_count = 2

        name = ocfs2

 

[root@NOD2 ~]#

 

 

重新在两个节点上配置O2CB驱动

[root@NOD1 ~]# /etc/init.d/o2cb unload

Stopping O2CB cluster ocfs2: OK

Unmounting ocfs2_dlmfs filesystem: OK

Unloading module "ocfs2_dlmfs": OK

Unmounting configfs filesystem: OK

Unloading module "configfs": OK

[root@NOD1 ~]# /etc/init.d/o2cb configure

Configuring the O2CB driver.

 

This will configure the on-boot properties of the O2CB driver.

The following questions will determine whether the driver is loaded on

boot. The current values will be shown in brackets ('[]'). Hitting

<ENTER> without typing an answer will keep that current value. Ctrl-C

will abort.

 

Load O2CB driver on boot (y/n) [y]: y

Cluster to start on boot (Enter "none" to clear) [ocfs2]:

Specify heartbeat dead threshold (>=7) [61]: 60

Specify network idle timeout in ms (>=5000) [30000]: 30000

Specify network keepalive delay in ms (>=1000) [2000]:

Specify network reconnect delay in ms (>=2000) [2000]:

Writing O2CB configuration: OK

Loading module "configfs": OK

Mounting configfs filesystem at /config: OK

Loading module "ocfs2_nodemanager": OK

Loading module "ocfs2_dlm": OK

Loading module "ocfs2_dlmfs": OK

Mounting ocfs2_dlmfs filesystem at /dlm: OK

Starting O2CB cluster ocfs2: OK

[root@NOD1 ~]#

 

 

格式化文件系统

打开ocfs2console---tasks----format

在两个节点上挂载文件系统

[root@NOD1 /]# su - oracle

[oracle@NOD1 ~]$ mkdir ocfs

[oracle@NOD1 ~]$ ls

ln ocfs p

[oracle@NOD1 ~]$ su

Password:

[root@NOD1 oracle]#

[oracle@NOD1 ~]$ su

Password:

[root@NOD1 oracle]# mount /dev/sdb

sdb   sdb1 

[root@NOD1 oracle]# mount /dev/sdb1 ocfs/

[root@NOD1 oracle]#

 

 

启动时挂载文件系统

[root@NOD2 ~]# vim /etc/fstab

/dev/sdb1 /home/oracle/ocfs ocfs2 _netdev,datavolume,nointr 0 0

 

创建 Oracle 集群件目录。在 OCR 和表决磁盘将驻留的 OCFS2 文件系统中创建目录

[root@NOD1 oracle]# mkdir /home/oracle/ocfs/clusterware

[root@NOD1 oracle]# chown -R oracle:dba /home/oracle/ocfs/

[root@NOD1 oracle]#

 

到目前为止，已经完成了再两个节点上的OCFS设置

 

安装VMWARE TOOLS，用来同步虚拟机和主机的时间

关闭NOD1,NOD2，载入linux.iso镜像F:\Program Files\VMware\VMware Server\linux.iso

启动，安装VMWARE TOOLS。

 

执行“vmware-toolbox”以显示 VMware Tools Properties 窗口。/* Wilson注：vmware-toolbox是一个可执行文件，直接在Terminal窗口中以root身份敲vmware-toolbox就可以执行它了 */ 在Options 选项卡下，选择Time synchronization between the virtual machine and the host operating system。您应该发现 tools.syncTime = "TRUE" 参数已经追加到虚拟机配置文件d:\vm\rac\rac1\Red Hat Enterprise Linux 4.vmx 中。

 

 

编辑 /boot/grub/grub.conf，并将选项“clock=pit nosmp noapic nolapic”添加到读取内核 /boot/ 的那一行。您已经将选项添加到两个内核，现在只需对特定内核进行更改。

3. #boot=/dev/sda

4. default=0

5. timeout=5

6. splashimage=(hd0,0)/boot/grub/splash.xpm.gz

7. hiddenmenu

8. title Enterprise (2.6.9-42.0.0.0.1.ELsmp)

9. root (hd0,0)

10. kernel /boot/vmlinuz-2.6.9-42.0.0.0.1.ELsmp ro

11. root=LABEL=/ rhgb quiet clock=pit nosmp noapic nolapic

12. initrd /boot/initrd-2.6.9-42.0.0.0.1.ELsmp.img

13. title Enterprise-up (2.6.9-42.0.0.0.1.EL)

14. root (hd0,0)

15. kernel /boot/vmlinuz-2.6.9-42.0.0.0.1.EL ro root=LABEL=/

16. rhgb quiet clock=pit nosmp noapic nolapic

17. initrd /boot/initrd-2.6.9-42.0.0.0.1.EL.img

18. 重新引导 rac1。

# reboot

 

 

 

 

使用SSH secure FILE TRANSFER把clusterware 和10G database安装程序上传到NOD1上

Unzip 到/home/目录下

Su 到oracle用户

进入到clusterware目录执行

[oracle@NOD1 clusterware]$ ./runInstaller

选择ORACrs10g_HOME安装目录

忽视内存要求警告

添加NOD2节点

NOD2

NOD2-PRIV

NOD2-VIP

指定网卡，一般默认就可以了

指定 Oracle 集群注册表 (OCR) 位置：选择External Redundancy。为了简单起见，这里将不镜像 OCR。在生产环境中，您可能会考虑复用 OCR 以实现更高的冗余。

指定 OCR 位置：/home/oracle/ocfs/clusterware/ocr

指定表决磁盘位置：选择 External Redundancy。同样，为了简单起见，我们选择不镜像表决磁盘。

表决磁盘位置：/home/oracle/ocfs/clusterware/votingdisk

安装

 

依次在NOD1,NOD2上以ROOT用户运行脚本

[root@NOD1 oracle]# /home/ora10g/oraInventory/orainstRoot.sh

[root@NOD2 ~]# /home/ora10g/oraInventory/orainstRoot.sh

[root@NOD1 oracle]# /home/ora10g/product/10.2.0/crs_1/root.sh

[root@NOD1 oracle]# /home/ora10g/product/10.2.0/crs_1/root.sh

 

 

在NOD2上执行root.sh报错

[root@NOD2 home]# /home/ora10g/product/10.2.0/crs_1/root.sh

WARNING: directory '/home/ora10g/product/10.2.0' is not owned by root

WARNING: directory '/home/ora10g/product' is not owned by root

WARNING: directory '/home/ora10g' is not owned by root

Checking to see if Oracle CRS stack is already configured

/etc/oracle does not exist. Creating it now.

 

Setting the permissions on OCR backup directory

Setting up NS directories

Oracle Cluster Registry configuration upgraded successfully

WARNING: directory '/home/ora10g/product/10.2.0' is not owned by root

WARNING: directory '/home/ora10g/product' is not owned by root

WARNING: directory '/home/ora10g' is not owned by root

clscfg: EXISTING configuration version 3 detected.

clscfg: version 3 is 10G Release 2.

assigning default hostname nod1 for node 1.

assigning default hostname nod2 for node 2.

Successfully accumulated necessary OCR keys.

Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.

node <nodenumber>: <nodename> <private interconnect name> <hostname>

node 1: nod1 nod1-priv nod1

node 2: nod2 nod2-priv nod2

clscfg: Arguments check out successfully.

 

NO KEYS WERE WRITTEN. Supply -force parameter to override.

-force is destructive and will destroy any previous cluster

configuration.

Oracle Cluster Registry for cluster has already been initialized

Startup will be queued to init within 90 seconds.

Adding daemons to inittab

Expecting the CRS daemons to be up within 600 seconds.

CSS is active on these nodes.

        nod1

        nod2

CSS is active on all nodes.

Waiting for the Oracle CRSD and EVMD to start

Oracle CRS stack installed and running under init(1M)

Running vipca(silent) for configuring nodeapps

在输入参数中指定的节点名 "NOD1" 无效。

在输入参数中指定的节点名 "NOD1" 无效。

 

在OUI界面点确定OCA检查失败。点确定退出

在XMANAGER窗口重新打开一个窗口，以ROOT身份运行

/home/ora10g/product/10.2.0/crs_1/bin/vipca弹出图形窗口，点击下一步

选择eth0，下一步，输入IP别名Nod1-vip，其他的信息自动弹出来。下一步，安装VIP，

GSD,ONS

确定后退出。

 

在NOD1,NOD2上验证集群服务设置

[root@NOD1 bin]# su - oracle

[oracle@NOD1 ~]$ cd /home/ora10g/product/10.2.0/crs_1/bin/

[oracle@NOD1 bin]$ ./cluvfy stage -post crsinst -n nod1,nod2

 

执行 群集服务设置 的后期检查

 

正在检查节点的可访问性...

节点 "NOD1" 的节点可访问性检查已通过。

 

 

正在检查等同用户...

用户 "oracle" 的等同用户检查已通过。

 

正在检查集群管理器完整性...

 

 

正在检查 CSS 守护程序...

"CSS daemon" 的 守护程序状态 检查已通过。

 

集群管理器完整性检查已通过。

 

正在检查集群完整性...

 

 

集群完整性检查已通过

 

 

正在检查 OCR 完整性...

 

正在检查是否缺少非集群配置...

所有节点都没有非集群的, 仅限本地的配置。

 

OCR 设备的唯一性检查已通过。

 

正在检查 OCR 的版本...

正确版本 "2" 的 OCR 存在。

 

正在检查 OCR 数据完整性...

OCR 数据完整性检查已通过。

 

OCR 完整性检查已通过。

 

正在检查 CRS 完整性...

 

正在检查守护程序的活动性...

"CRS daemon" 的活动性检查已通过。

 

正在检查守护程序的活动性...

"CSS daemon" 的活动性检查已通过。

 

正在检查守护程序的活动性...

"EVM daemon" 的活动性检查已通过。

 

正在检查 CRS 健康状况...

CRS 健康状况检查已通过。

 

CRS 完整性检查已通过。

 

正在检查节点应用程序是否存在...

 

 

正在检查 VIP 节点应用程序是否存在(必需)

检查已通过。

 

正在检查 ONS 节点应用程序是否存在(可选)

检查已通过。

 

正在检查 GSD 节点应用程序是否存在(可选)

检查已通过。

 

 

群集服务设置 的后期检查成功。

[oracle@NOD1 bin]$

检查通过

这里还是不知道那个节点名无效错误时怎么产生的。网上搜了一下，很多人遇到的是eth0 is not public。。。。是因为公共接口是不可路由的IP导致。哪位兄弟遇到过把解决方法告诉我一下。。。