Oracle RAC Study之--10gR2 RAC环境中在线添加、删除Votedisk和在线替换OCR的探讨
这里我们不讨论11gR2 RAC在线添加、删除Votedisk和OCR，在11gR2平台可以非常容易的实现。这里也不讨论Votedisk和OCR的还原，还原操作必须在停止Clusterware的情况下完成。下面以实际的例子讨论10gR2 RAC for Linux环境下，Votedisk和OCR的在线添加和删除。
一.Votedisk的在线添加和删除。
1).查看集群资源的状态：
[root@rhel1 bin]# ./crs_stat -t
Name Type Target State Host ------------------------------------------------------------ ora....L1.lsnr application ONLINE ONLINE rhel1 ora.rhel1.gsd application ONLINE ONLINE rhel1 ora.rhel1.ons application ONLINE ONLINE rhel1 ora.rhel1.vip application ONLINE ONLINE rhel1 ora....L2.lsnr application ONLINE ONLINE rhel2 ora.rhel2.gsd application ONLINE ONLINE rhel2 ora.rhel2.ons application ONLINE ONLINE rhel2 ora.rhel2.vip application ONLINE ONLINE rhel2
2).查看当前votedisk位置及文件名：
[root@rhel1 bin]# ./crsctl query css votedisk
0. 0 /u02/vdisk/vdisk1 1. 0 /u02/vdisk/vdisk2 2. 0 /u02/vdisk/vdisk3 located 3 votedisk(s).
3).查看ocssd进程号：
[root@rhel1 bin]# ps -e | grep -i ocssd
5942 ? 00:00:21 ocssd.bin
4).查看ocssd进程定位的votedisk文件：
[root@rhel1 bin]# cd /proc/5942/fd/
[root@rhel1 fd]# ll total 0 lr-x------ 1 oracle oinstall 64 Jun 12 18:18 0 -> /dev/null lrwx------ 1 oracle oinstall 64 Jun 12 18:18 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 18:18 10 -> socket:[13927] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 11 -> /u02/vdisk/vdisk1 lrwx------ 1 oracle oinstall 64 Jun 12 18:18 12 -> /u02/vdisk/vdisk1 lrwx------ 1 oracle oinstall 64 Jun 12 18:18 13 -> /u02/vdisk/vdisk2 lrwx------ 1 oracle oinstall 64 Jun 12 18:18 14 -> /u02/vdisk/vdisk2 lrwx------ 1 oracle oinstall 64 Jun 12 18:18 15 -> /u02/vdisk/vdisk3 lrwx------ 1 oracle oinstall 64 Jun 12 18:18 16 -> /u02/vdisk/vdisk3 l-wx------ 1 oracle oinstall 64 Jun 12 18:18 17 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log lr-x------ 1 oracle oinstall 64 Jun 12 18:18 18 -> /u01/app/oracle/crs/has/mesg/clsdus.msb lrwx------ 1 oracle oinstall 64 Jun 12 18:18 19 -> socket:[13974] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 18:18 20 -> socket:[13977] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 21 -> socket:[14446] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 22 -> socket:[13984] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 23 -> socket:[13987] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 24 -> socket:[13989] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 25 -> socket:[13993] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 26 -> socket:[13995] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 27 -> socket:[13996] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 28 -> socket:[14447] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 29 -> socket:[14733] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 18:18 30 -> socket:[14449] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 31 -> socket:[14739] lr-x------ 1 oracle oinstall 64 Jun 12 18:18 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb lr-x------ 1 oracle oinstall 64 Jun 12 18:18 5 -> /u02/ocr/ocr1 lr-x------ 1 oracle oinstall 64 Jun 12 18:18 6 -> /u02/ocr/ocr2 lr-x------ 1 oracle oinstall 64 Jun 12 18:18 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb l-wx------ 1 oracle oinstall 64 Jun 12 18:18 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log lrwx------ 1 oracle oinstall 64 Jun 12 18:18 9 -> socket:[13926]
文件说明当前ocssd定位的votedisk文件。分别是vdisk1、vdisk2和vdisk3。
5).在线添加Votedisk文件：
[root@rhel1 fd]# cd -
/u01/app/oracle/crs/bin [root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk4 -force Now formatting voting disk: /u03/vdisk/vdisk4 successful addition of votedisk /u03/vdisk/vdisk4. [root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk5 -force Now formatting voting disk: /u03/vdisk/vdisk5 successful addition of votedisk /u03/vdisk/vdisk5. [root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk6 -force Now formatting voting disk: /u03/vdisk/vdisk6 successful addition of votedisk /u03/vdisk/vdisk6.
执行上面的命令在线添加了3个votedisk文件到不同的ocfs2文件系统上，这样做的目的是为了实现存储设备的在线迁移。
6).查看当前的votedisk路径及文件名：
[root@rhel1 bin]# ./crsctl query css votedisk
0. 0 /u02/vdisk/vdisk1 1. 0 /u02/vdisk/vdisk2 2. 0 /u02/vdisk/vdisk3 3. 0 /u03/vdisk/vdisk4 4. 0 /u03/vdisk/vdisk5 5. 0 /u03/vdisk/vdisk6
located 6 votedisk(s).
从上面的输出结果可以看出，3个votedisk文件添加是成功的。
7).查看ocssd进程定位的votedisk文件：
[root@rhel1 bin]# cd /proc/5942/fd/
[root@rhel1 fd]# ll total 0 lr-x------ 1 oracle oinstall 64 Jun 12 18:18 0 -> /dev/null lrwx------ 1 oracle oinstall 64 Jun 12 18:18 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 18:18 10 -> socket:[13927] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 11 -> /u02/vdisk/vdisk1 lrwx------ 1 oracle oinstall 64 Jun 12 18:18 12 -> /u02/vdisk/vdisk1 lrwx------ 1 oracle oinstall 64 Jun 12 18:18 13 -> /u02/vdisk/vdisk2 lrwx------ 1 oracle oinstall 64 Jun 12 18:18 14 -> /u02/vdisk/vdisk2 lrwx------ 1 oracle oinstall 64 Jun 12 18:18 15 -> /u02/vdisk/vdisk3 lrwx------ 1 oracle oinstall 64 Jun 12 18:18 16 -> /u02/vdisk/vdisk3 l-wx------ 1 oracle oinstall 64 Jun 12 18:18 17 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log lr-x------ 1 oracle oinstall 64 Jun 12 18:18 18 -> /u01/app/oracle/crs/has/mesg/clsdus.msb lrwx------ 1 oracle oinstall 64 Jun 12 18:18 19 -> socket:[13974] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 18:18 20 -> socket:[13977] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 21 -> socket:[14446] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 22 -> socket:[13984] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 23 -> socket:[13987] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 24 -> socket:[13989] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 25 -> socket:[13993] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 26 -> socket:[13995] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 27 -> socket:[13996] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 28 -> socket:[14447] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 29 -> socket:[14733] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 18:18 30 -> socket:[14449] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 31 -> socket:[14739] lr-x------ 1 oracle oinstall 64 Jun 12 18:18 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb lr-x------ 1 oracle oinstall 64 Jun 12 18:18 5 -> /u02/ocr/ocr1 lr-x------ 1 oracle oinstall 64 Jun 12 18:18 6 -> /u02/ocr/ocr2 lr-x------ 1 oracle oinstall 64 Jun 12 18:18 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb l-wx------ 1 oracle oinstall 64 Jun 12 18:18 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log lrwx------ 1 oracle oinstall 64 Jun 12 18:18 9 -> socket:[13926]
从上面的输出可以看出，ocssd进程定位的votedisk文件并未发生变化，可以猜测，ocssd对votedisk的更新也不会同步到新 添加的3个votedisk文件中，那么在长时间的运行之后，重启服务器时新添加的votedisk过旧。
8).在线删除votedisk文件：
[root@rhel1 fd]# cd -
/u01/app/oracle/crs/bin [root@rhel1 bin]# ./crsctl delete css votedisk /u02/vdisk/vdisk1 -force successful deletion of votedisk /u02/vdisk/vdisk1. [root@rhel1 bin]# ./crsctl delete css votedisk /u02/vdisk/vdisk2 -force successful deletion of votedisk /u02/vdisk/vdisk2. [root@rhel1 bin]# ./crsctl delete css votedisk /u02/vdisk/vdisk3 -force successful deletion of votedisk /u02/vdisk/vdisk3.
9).查看删除votedisk后的votedisk路径及文件名：
[root@rhel1 bin]# ./crsctl query css votedisk
0. 0 /u03/vdisk/vdisk4 1. 0 /u03/vdisk/vdisk5 2. 0 /u03/vdisk/vdisk6 located 3 votedisk(s).
[root@rhel1 bin]# cd /proc/5942/fd/
[root@rhel1 fd]# ll total 0 lr-x------ 1 oracle oinstall 64 Jun 12 18:18 0 -> /dev/null lrwx------ 1 oracle oinstall 64 Jun 12 18:18 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 18:18 10 -> socket:[13927] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 11 -> /u02/vdisk/vdisk1 lrwx------ 1 oracle oinstall 64 Jun 12 18:18 12 -> /u02/vdisk/vdisk1 lrwx------ 1 oracle oinstall 64 Jun 12 18:18 13 -> /u02/vdisk/vdisk2 lrwx------ 1 oracle oinstall 64 Jun 12 18:18 14 -> /u02/vdisk/vdisk2 lrwx------ 1 oracle oinstall 64 Jun 12 18:18 15 -> /u02/vdisk/vdisk3 lrwx------ 1 oracle oinstall 64 Jun 12 18:18 16 -> /u02/vdisk/vdisk3 l-wx------ 1 oracle oinstall 64 Jun 12 18:18 17 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log lr-x------ 1 oracle oinstall 64 Jun 12 18:18 18 -> /u01/app/oracle/crs/has/mesg/clsdus.msb lrwx------ 1 oracle oinstall 64 Jun 12 18:18 19 -> socket:[13974] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 18:18 20 -> socket:[13977] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 21 -> socket:[14446] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 22 -> socket:[13984] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 23 -> socket:[13987] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 24 -> socket:[13989] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 25 -> socket:[13993] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 26 -> socket:[13995] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 27 -> socket:[13996] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 28 -> socket:[14447] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 29 -> socket:[14733] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 18:18 30 -> socket:[14449] lrwx------ 1 oracle oinstall 64 Jun 12 18:18 31 -> socket:[14739] lr-x------ 1 oracle oinstall 64 Jun 12 18:18 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb lr-x------ 1 oracle oinstall 64 Jun 12 18:18 5 -> /u02/ocr/ocr1 lr-x------ 1 oracle oinstall 64 Jun 12 18:18 6 -> /u02/ocr/ocr2 lr-x------ 1 oracle oinstall 64 Jun 12 18:18 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb l-wx------ 1 oracle oinstall 64 Jun 12 18:18 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log lrwx------ 1 oracle oinstall 64 Jun 12 18:18 9 -> socket:[13926]
定位未发生任何变化。
下面我们来尝试重启Clusterware，看会发生什么情况。
11).停止Clustereware：
[root@rhel1 fd]# cd -
/u01/app/oracle/crs/bin
[root@rhel1 bin]# ./crsctl stop crs
Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued.
在所有RAC节点停止Clusterware。
12).启动Clusterware：
[root@rhel1 bin]# ./crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly
只在一个节点启动Clusterware。
13).查看CRS组件的状态及votedisk文件：
[root@rhel1 bin]# ./crsctl check crs
CSS appears healthy CRS appears healthy EVM appears healthy
[root@rhel1 bin]# ./crsctl query css votedisk
0. 0 0 1. 0 0 2. 0 /u03/vdisk/vdisk6 located 3 votedisk(s).
可以发现，votedisk只有vdisk6显示正常，至少可以说明这种情况是不正常的，可以大胆猜想是由于在线增加votedisk引起的问题。
14).查看ocssd进程定位的votedisk文件：
[root@rhel1 bin]# ps -e | grep -i ocssd
9386 ? 00:00:04 ocssd.bin
[root@rhel1 bin]# cd /proc/9386/fd/
[root@rhel1 fd]# ll total 0 lr-x------ 1 oracle oinstall 64 Jun 12 18:30 0 -> /dev/null lrwx------ 1 oracle oinstall 64 Jun 12 18:30 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 18:30 10 -> socket:[44767] lrwx------ 1 oracle oinstall 64 Jun 12 18:30 11 -> /u03/vdisk/vdisk6 lrwx------ 1 oracle oinstall 64 Jun 12 18:30 12 -> /u03/vdisk/vdisk6 l-wx------ 1 oracle oinstall 64 Jun 12 18:30 13 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log lr-x------ 1 oracle oinstall 64 Jun 12 18:30 14 -> /u01/app/oracle/crs/has/mesg/clsdus.msb lrwx------ 1 oracle oinstall 64 Jun 12 18:30 15 -> socket:[44950] lrwx------ 1 oracle oinstall 64 Jun 12 18:30 16 -> socket:[44953] lrwx------ 1 oracle oinstall 64 Jun 12 18:30 17 -> socket:[45584] lrwx------ 1 oracle oinstall 64 Jun 12 18:30 18 -> socket:[44980] lrwx------ 1 oracle oinstall 64 Jun 12 18:30 19 -> socket:[44983] lrwx------ 1 oracle oinstall 64 Jun 12 18:30 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 18:30 20 -> socket:[44986] lrwx------ 1 oracle oinstall 64 Jun 12 18:30 21 -> socket:[44988] lrwx------ 1 oracle oinstall 64 Jun 12 18:30 22 -> socket:[44992] lrwx------ 1 oracle oinstall 64 Jun 12 18:30 23 -> socket:[44993] lrwx------ 1 oracle oinstall 64 Jun 12 18:30 24 -> socket:[45585] lrwx------ 1 oracle oinstall 64 Jun 12 18:30 25 -> socket:[45586] lrwx------ 1 oracle oinstall 64 Jun 12 18:30 26 -> socket:[45587] lrwx------ 1 oracle oinstall 64 Jun 12 18:30 27 -> socket:[45588] lrwx------ 1 oracle oinstall 64 Jun 12 18:30 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lr-x------ 1 oracle oinstall 64 Jun 12 18:30 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb lr-x------ 1 oracle oinstall 64 Jun 12 18:30 5 -> /u02/ocr/ocr1 lr-x------ 1 oracle oinstall 64 Jun 12 18:30 6 -> /u02/ocr/ocr2 lr-x------ 1 oracle oinstall 64 Jun 12 18:30 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb l-wx------ 1 oracle oinstall 64 Jun 12 18:30 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log lrwx------ 1 oracle oinstall 64 Jun 12 18:30 9 -> socket:[44766]
从上面的结果可以看出，ocssd进程只定位了vdisk6这一个文件。
15).停止所有节点的Clusterware：
[root@rhel1 fd]# cd -
/u01/app/oracle/crs/bin [root@rhel1 bin]# ./crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@rhel1 bin]# ps -ef | grep ora root 2265 2243 0 16:45 ? 00:00:01 hald-addon-storage: polling /dev/hdc root 8197 8174 0 18:23 pts/3 00:00:00 su - oracle oracle 8198 8197 0 18:23 pts/3 00:00:00 -bash oracle 10433 8198 0 18:29 pts/3 00:00:00 tail -f alertrhel1.log root 13299 3340 0 18:34 pts/0 00:00:00 grep ora root 30740 9263 0 17:59 pts/2 00:00:00 su - oracle oracle 30741 30740 0 17:59 pts/2 00:00:00 -bash
16).手动删除没有使用的vdisk4和vdisk5：
[root@rhel1 bin]# cd /u03/vdisk/
[root@rhel1 vdisk]# rm -rf vdisk4 vdisk5
17).在Clusterware脱机的情况下添加vdisk4和vdisk5：
[root@rhel1 vdisk]# cd -
/u01/app/oracle/crs/bin [root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk4 -force Now formatting voting disk: /u03/vdisk/vdisk4 successful addition of votedisk /u03/vdisk/vdisk4. [root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk5 -force Now formatting voting disk: /u03/vdisk/vdisk5 successful addition of votedisk /u03/vdisk/vdisk5.
17).修改votedisk正确的权限：
[root@rhel1 bin]# cd -
/u03/vdisk [root@rhel1 vdisk]# ll total 30000 -rw-r--r-- 1 root root 10240000 Jun 12 18:37 vdisk4 -rw-r--r-- 1 root root 10240000 Jun 12 18:37 vdisk5 -rw-r--r-- 1 oracle oinstall 10240000 Jun 12 18:19 vdisk6 [root@rhel1 vdisk]# chown oracle:oinstall *
18).启动Clusterware：
[root@rhel1 vdisk]# cd -
/u01/app/oracle/crs/bin [root@rhel1 bin]# ./crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly [root@rhel1 bin]# ./crsctl check css CSS appears healthy [root@rhel1 bin]# ./crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy
19).查看votedisk路径及文件名：
[root@rhel1 bin]# ./crsctl query css votedisk
0. 0 /u03/vdisk/vdisk4 1. 0 /u03/vdisk/vdisk5 2. 0 0 3. 0 0 4. 0 /u03/vdisk/vdisk4 located 5 votedisk(s).
可以看出vdisk4和vdisk5被正确添加，vdisk6不见了踪迹，说明10g votedisk在Clusterware的情况下是可以添加的，并且在脱机的情况下添加之后启动ocssd进程会立即定位到这些votedisk文件，所以这并不会导致前面新添加的votedisk未被定位的情况发生。
20).添加vdisk6和清理有问题的votedisk：
[root@rhel1 bin]# ./crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@rhel1 bin]# ps -ef | grep ora root 2265 2243 0 16:45 ? 00:00:02 hald-addon-storage: polling /dev/hdc root 8197 8174 0 18:22 pts/3 00:00:00 su - oracle oracle 8198 8197 0 18:22 pts/3 00:00:00 -bash oracle 20770 8198 0 18:50 pts/3 00:00:00 tail -f alertrhel1.log root 21487 3340 0 18:51 pts/0 00:00:00 grep ora root 30740 9263 0 17:59 pts/2 00:00:00 su - oracle oracle 30741 30740 0 17:59 pts/2 00:00:00 -bash [root@rhel1 bin]# ./crsctl add css votedisk /u03/vdisk/vdisk6 -force Now formatting voting disk: /u03/vdisk/vdisk6 successful addition of votedisk /u03/vdisk/vdisk6. [root@rhel1 bin]# ./crsctl query css votedisk 0. 0 /u03/vdisk/vdisk4 1. 0 /u03/vdisk/vdisk5 2. 0 /u03/vdisk/vdisk6 3. 0 0 4. 0 0 5. 0 /u03/vdisk/vdisk4 located 6 votedisk(s). [root@rhel1 bin]# ./crsctl delete css votedisk 0 -force successful deletion of votedisk 0. [root@rhel1 bin]# ./crsctl query css votedisk 0. 0 /u03/vdisk/vdisk4 1. 0 /u03/vdisk/vdisk5 2. 0 /u03/vdisk/vdisk6 3. 0 0 4. 0 /u03/vdisk/vdisk4 located 5 votedisk(s).
从上面返回的结果可以看出，虽然votedisk的路径和文件都是0，在删除0的时候也不会被全部删除，只会删除其中的一条。
[root@rhel1 bin]# ./crsctl delete css votedisk 0 -force successful deletion of votedisk 0. [root@rhel1 bin]# ./crsctl query css votedisk 0. 0 /u03/vdisk/vdisk4 1. 0 /u03/vdisk/vdisk5 2. 0 /u03/vdisk/vdisk6 3. 0 /u03/vdisk/vdisk4 located 4 votedisk(s). [root@rhel1 bin]# ./crsctl delete css votedisk /u03/vdisk/vdisk4 -force successful deletion of votedisk /u03/vdisk/vdisk4. [root@rhel1 bin]# ./crsctl query css votedisk 0. 0 /u03/vdisk/vdisk5 1. 0 /u03/vdisk/vdisk6 2. 0 /u03/vdisk/vdisk4 located 3 votedisk(s).
删除 /u03/vdisk/vdisk4的votedisk文件也只删除了其中一条记录。
21).启动Clusterware：
[root@rhel1 bin]# ./crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly [root@rhel1 bin]# ./crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy [root@rhel1 bin]# ./crsctl query css votedisk 0. 0 /u03/vdisk/vdisk5 1. 0 /u03/vdisk/vdisk6 2. 0 /u03/vdisk/vdisk4 located 3 votedisk(s).
从上面的例子可以说明一点，在10gR2 RAC 的环境中最好不要在Clusterware在线的情况下添加votedisk，虽然可以成功添加，但在重启Clusterware之后可能出现意想不到的问题。
二.OCR的在线替换（添加和删除）。
1).检查OCR的完整性：
[root@rhel1 bin]# ./ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 262120 Used space (kbytes) : 2512 Available space (kbytes) : 259608 ID : 2111044529 Device/File Name : /u02/ocr/ocr1 Device/File integrity check succeeded Device/File Name : /u02/ocr/ocr2 Device/File integrity check succeeded Cluster registry integrity check succeeded
2).提前将需要创建的ocr文件touch：
[root@rhel1 bin]# su - oracle [oracle@rhel1 ~]$ [oracle@rhel1 ~]$ cd /u03/ocr/ [oracle@rhel1 ocr]$ ls [oracle@rhel1 ocr]$ touch ocr3 [oracle@rhel1 ocr]$ touch ocr4
上面添加votedisk的时候并未提前touch votedisk文件，但某些情况下votedisk文件也需要提前touch。
3).替换ocrmirror文件，实际是一个迁移ocr的过程：
[root@rhel1 bin]# ./ocrconfig -replace ocrmirror /u03/ocr/ocr4 [root@rhel1 bin]# ./ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 262120 Used space (kbytes) : 2512 Available space (kbytes) : 259608 ID : 2111044529 Device/File Name : /u02/ocr/ocr1 Device/File integrity check succeeded Device/File Name : /u03/ocr/ocr4 Device/File integrity check succeeded Cluster registry integrity check succeeded
迁移后检查ocr的完整性，没问题。
4).查看ocssd进程定位的ocr文件：
ocr文件也是由ocssd进程来维护。 [root@rhel1 bin]# ps -e | grep -i ocssd 23379 ? 00:00:06 ocssd.bin [root@rhel1 bin]# cd /proc/23379/fd [root@rhel1 fd]# ll total 0 lr-x------ 1 oracle oinstall 64 Jun 12 19:00 0 -> /dev/null lrwx------ 1 oracle oinstall 64 Jun 12 19:00 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 19:00 10 -> socket:[62386] lrwx------ 1 oracle oinstall 64 Jun 12 19:00 11 -> /u03/vdisk/vdisk5 lrwx------ 1 oracle oinstall 64 Jun 12 19:00 12 -> /u03/vdisk/vdisk5 lrwx------ 1 oracle oinstall 64 Jun 12 19:00 13 -> /u03/vdisk/vdisk6 lrwx------ 1 oracle oinstall 64 Jun 12 19:00 14 -> /u03/vdisk/vdisk6 lrwx------ 1 oracle oinstall 64 Jun 12 19:00 15 -> /u03/vdisk/vdisk4 lrwx------ 1 oracle oinstall 64 Jun 12 19:00 16 -> /u03/vdisk/vdisk4 l-wx------ 1 oracle oinstall 64 Jun 12 19:00 17 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log lr-x------ 1 oracle oinstall 64 Jun 12 19:00 18 -> /u01/app/oracle/crs/has/mesg/clsdus.msb lrwx------ 1 oracle oinstall 64 Jun 12 19:00 19 -> socket:[62607] lrwx------ 1 oracle oinstall 64 Jun 12 19:00 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 19:00 20 -> socket:[62610] lrwx------ 1 oracle oinstall 64 Jun 12 19:00 21 -> socket:[62611] lrwx------ 1 oracle oinstall 64 Jun 12 19:00 22 -> socket:[62627] lrwx------ 1 oracle oinstall 64 Jun 12 19:00 23 -> socket:[62632] lrwx------ 1 oracle oinstall 64 Jun 12 19:00 24 -> socket:[62637] lrwx------ 1 oracle oinstall 64 Jun 12 19:00 25 -> socket:[62640] lrwx------ 1 oracle oinstall 64 Jun 12 19:00 26 -> socket:[62645] lrwx------ 1 oracle oinstall 64 Jun 12 19:00 27 -> socket:[62646] lrwx------ 1 oracle oinstall 64 Jun 12 19:00 28 -> socket:[62650] lrwx------ 1 oracle oinstall 64 Jun 12 19:00 29 -> socket:[62651] lrwx------ 1 oracle oinstall 64 Jun 12 19:00 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 19:00 30 -> socket:[62652] lrwx------ 1 oracle oinstall 64 Jun 12 19:00 31 -> socket:[62653] lr-x------ 1 oracle oinstall 64 Jun 12 19:00 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb lr-x------ 1 oracle oinstall 64 Jun 12 19:00 5 -> /u02/ocr/ocr1 lr-x------ 1 oracle oinstall 64 Jun 12 19:00 6 -> /u02/ocr/ocr2 lr-x------ 1 oracle oinstall 64 Jun 12 19:00 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb l-wx------ 1 oracle oinstall 64 Jun 12 19:00 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log lrwx------ 1 oracle oinstall 64 Jun 12 19:00 9 -> socket:[62385]
从上面的结果可以看出，ocssd进程定位的ocr文件还是oc1和ocr2，并未随着ocr的迁移发生改变，和votedisk的效果相同。
5).停止Clusterware：
[root@rhel1 fd]# cd -
/u01/app/oracle/crs/bin [root@rhel1 bin]# ./crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued.停止所有节点的Clusterware。
[root@rhel1 bin]# ps -ef | grep ora root 2265 2243 0 16:45 ? 00:00:02 hald-addon-storage: polling /dev/hdc root 8197 8174 0 18:22 pts/3 00:00:00 su - oracle oracle 8198 8197 0 18:22 pts/3 00:00:00 -bash root 26683 3340 0 19:01 pts/0 00:00:00 grep ora root 30740 9263 0 17:58 pts/2 00:00:00 su - oracle oracle 30741 30740 0 17:58 pts/2 00:00:00 -bash
6).在Clusterware脱机的情况下迁移ocr文件：
[root@rhel1 bin]# ./ocrconfig -replace ocr /u03/ocr/ocr3 PROT-1: Failed to initialize ocrconfig
由此可以说明OCR文件不能在脱机的情况下迁移。
7).启动Clusterware：
[root@rhel1 bin]# ./crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly [root@rhel1 bin]# ./crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy [root@rhel1 bin]# ./ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 262120 Used space (kbytes) : 2512 Available space (kbytes) : 259608 ID : 2111044529 Device/File Name : /u02/ocr/ocr1 Device/File integrity check succeeded Device/File Name : /u03/ocr/ocr4 Device/File integrity check succeeded Cluster registry integrity check succeeded
8).在线替换ocr文件：
[root@rhel1 bin]# ./ocrconfig -replace ocr /u03/ocr/ocr3 [root@rhel1 bin]# ./ocrcheck Status of Oracle Cluster Registry is as follows : Version : 2 Total space (kbytes) : 262120 Used space (kbytes) : 2512 Available space (kbytes) : 259608 ID : 2111044529 Device/File Name : /u03/ocr/ocr3 Device/File integrity check succeeded Device/File Name : /u03/ocr/ocr4 Device/File integrity check succeeded Cluster registry integrity check succeeded
替换成功，OCR文件一致性检查成功。
9).检查ocssd进程定位的ocr文件：
[root@rhel1 bin]# ps -e | grep ocssd 28226 ? 00:00:03 ocssd.bin [root@rhel1 bin]# cd /proc/28226/fd [root@rhel1 fd]# ll total 0 lr-x------ 1 oracle oinstall 64 Jun 12 19:04 0 -> /dev/null lrwx------ 1 oracle oinstall 64 Jun 12 19:04 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log l-wx------ 1 oracle oinstall 64 Jun 12 19:04 10 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log lrwx------ 1 oracle oinstall 64 Jun 12 19:04 11 -> /u03/vdisk/vdisk5 lrwx------ 1 oracle oinstall 64 Jun 12 19:04 12 -> /u03/vdisk/vdisk5 lrwx------ 1 oracle oinstall 64 Jun 12 19:04 13 -> /u03/vdisk/vdisk6 lrwx------ 1 oracle oinstall 64 Jun 12 19:04 14 -> /u03/vdisk/vdisk6 l-wx------ 1 oracle oinstall 64 Jun 12 19:04 15 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log lr-x------ 1 oracle oinstall 64 Jun 12 19:04 16 -> /u01/app/oracle/crs/has/mesg/clsdus.msb lrwx------ 1 oracle oinstall 64 Jun 12 19:04 17 -> /u03/vdisk/vdisk4 lrwx------ 1 oracle oinstall 64 Jun 12 19:04 18 -> /u03/vdisk/vdisk4 lrwx------ 1 oracle oinstall 64 Jun 12 19:04 19 -> socket:[69253] lrwx------ 1 oracle oinstall 64 Jun 12 19:04 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 19:04 20 -> socket:[69256] lrwx------ 1 oracle oinstall 64 Jun 12 19:04 21 -> socket:[69257] lrwx------ 1 oracle oinstall 64 Jun 12 19:04 22 -> socket:[69285] lrwx------ 1 oracle oinstall 64 Jun 12 19:04 23 -> socket:[69303] lrwx------ 1 oracle oinstall 64 Jun 12 19:04 24 -> socket:[69346] lrwx------ 1 oracle oinstall 64 Jun 12 19:04 25 -> socket:[69353] lrwx------ 1 oracle oinstall 64 Jun 12 19:04 26 -> socket:[69370] lrwx------ 1 oracle oinstall 64 Jun 12 19:04 27 -> socket:[69371] lrwx------ 1 oracle oinstall 64 Jun 12 19:04 28 -> socket:[69379] lrwx------ 1 oracle oinstall 64 Jun 12 19:04 29 -> socket:[69380] lrwx------ 1 oracle oinstall 64 Jun 12 19:04 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 19:04 30 -> socket:[69381] lrwx------ 1 oracle oinstall 64 Jun 12 19:04 31 -> socket:[69382] lr-x------ 1 oracle oinstall 64 Jun 12 19:04 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb lr-x------ 1 oracle oinstall 64 Jun 12 19:04 5 -> /u02/ocr/ocr1 lr-x------ 1 oracle oinstall 64 Jun 12 19:04 6 -> /u03/ocr/ocr4 lr-x------ 1 oracle oinstall 64 Jun 12 19:04 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb lrwx------ 1 oracle oinstall 64 Jun 12 19:04 8 -> socket:[68925] lrwx------ 1 oracle oinstall 64 Jun 12 19:04 9 -> socket:[68926]
定位文件未发生变化，需要重启才能定位新替换的OCR文件。
10).重启Clusterware：
[root@rhel1 fd]# cd - /u01/app/oracle/crs/bin [root@rhel1 bin]# ./crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources. Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@rhel1 bin]# ./crsctl start crs Attempting to start CRS stack The CRS stack will be started shortly [root@rhel1 bin]# ./crsctl check crs CSS appears healthy CRS appears healthy EVM appears healthy
11).查看ocssd进程定位的ocr文件：
[root@rhel1 bin]# ps -e | grep ocssd 30773 ? 00:00:02 ocssd.bin [root@rhel1 bin]# cd /proc/30773/fd [root@rhel1 fd]# ll total 0 lr-x------ 1 oracle oinstall 64 Jun 12 19:05 0 -> /dev/null lrwx------ 1 oracle oinstall 64 Jun 12 19:05 1 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 19:05 10 -> socket:[73581] lrwx------ 1 oracle oinstall 64 Jun 12 19:05 11 -> /u03/vdisk/vdisk5 lrwx------ 1 oracle oinstall 64 Jun 12 19:05 12 -> /u03/vdisk/vdisk5 lrwx------ 1 oracle oinstall 64 Jun 12 19:05 13 -> /u03/vdisk/vdisk6 lrwx------ 1 oracle oinstall 64 Jun 12 19:05 14 -> /u03/vdisk/vdisk6 lrwx------ 1 oracle oinstall 64 Jun 12 19:05 15 -> /u03/vdisk/vdisk4 lrwx------ 1 oracle oinstall 64 Jun 12 19:05 16 -> /u03/vdisk/vdisk4 l-wx------ 1 oracle oinstall 64 Jun 12 19:05 17 -> /u01/app/oracle/crs/log/rhel1/alertrhel1.log lr-x------ 1 oracle oinstall 64 Jun 12 19:05 18 -> /u01/app/oracle/crs/has/mesg/clsdus.msb lrwx------ 1 oracle oinstall 64 Jun 12 19:05 19 -> socket:[73764] lrwx------ 1 oracle oinstall 64 Jun 12 19:05 2 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 19:05 20 -> socket:[73767] lrwx------ 1 oracle oinstall 64 Jun 12 19:05 22 -> socket:[73769] lrwx------ 1 oracle oinstall 64 Jun 12 19:05 23 -> socket:[73772] lrwx------ 1 oracle oinstall 64 Jun 12 19:05 24 -> socket:[73774] lrwx------ 1 oracle oinstall 64 Jun 12 19:05 25 -> socket:[73778] lrwx------ 1 oracle oinstall 64 Jun 12 19:05 26 -> socket:[73780] lrwx------ 1 oracle oinstall 64 Jun 12 19:05 27 -> socket:[73781] lrwx------ 1 oracle oinstall 64 Jun 12 19:05 28 -> socket:[73787] lrwx------ 1 oracle oinstall 64 Jun 12 19:05 29 -> socket:[73788] lrwx------ 1 oracle oinstall 64 Jun 12 19:05 3 -> /u01/app/oracle/crs/log/rhel1/cssd/cssdOUT.log lrwx------ 1 oracle oinstall 64 Jun 12 19:05 30 -> socket:[73789] lrwx------ 1 oracle oinstall 64 Jun 12 19:05 31 -> socket:[73790] lr-x------ 1 oracle oinstall 64 Jun 12 19:05 4 -> /u01/app/oracle/crs/srvm/mesg/procus.msb lr-x------ 1 oracle oinstall 64 Jun 12 19:05 5 -> /u03/ocr/ocr3 lr-x------ 1 oracle oinstall 64 Jun 12 19:05 6 -> /u03/ocr/ocr4 lr-x------ 1 oracle oinstall 64 Jun 12 19:05 7 -> /u01/app/oracle/crs/css/mesg/clssus.msb l-wx------ 1 oracle oinstall 64 Jun 12 19:05 8 -> /u01/app/oracle/crs/log/rhel1/cssd/ocssd.log lrwx------ 1 oracle oinstall 64 Jun 12 19:05 9 -> socket:[73580] 重启后ocssd进程定位ocr文件成功。
从上面的OCR替换的例子可以看出，OCR的替换（添加和删除）只能在Clusterware在线的情况下才能执行，最好在替换成功，检查OCR一致性正确之后立即重启Clusterware，使得替换的OCR文件生效。
文章出自：http://ms.itpub.net/23135684/viewspace-732734/ ，感谢作者！