RAC表决磁盘管理和维护
一，查询磁盘状况
1，如何查看表决磁盘的相关信息
[root@rac-two bin]# ./crsctl query css votedisk
2，如何备份表决磁盘：
dd if=/dev/raw/raw3 of=/tmp/votedisk.bak
3，恢复表决磁盘：
dd if=/tmp/votedisk.bak of=/dev/raw/raw3
4，删除votedisk
/app/crs/product/11.0.6/crs/bin/crsctl delete crss votedisk /dev/raw/raw5
5，[grid@rac1 ~]$ crsctl delete css votedisk ORCL:DISK3
CRS-4258: Addition and deletion of voting files are not allowed because the voting files are on ASM
[grid@rac1 ~]$
6，[oracle@rac1 ~]$ /u01/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2680
Available space (kbytes) : 259440
ID : 2063842061
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user
[oracle@rac1 ~]$
7，oracle@rac1 init.d]$ su - grid
Password:
[grid@rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.LISTENER.lsnr
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.asm
ONLINE ONLINE rac1
ONLINE ONLINE rac2 Started
ora.eons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.gsd
OFFLINE OFFLINE rac1
OFFLINE OFFLINE rac2
ora.net1.network
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.ons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.registry.acfs
ONLINE ONLINE rac1
ONLINE ONLINE rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac1
ora.oc4j
1 OFFLINE OFFLINE
ora.rac1.vip
1 ONLINE ONLINE rac1
ora.rac2.vip
1 ONLINE ONLINE rac2
ora.radb.db
1 ONLINE ONLINE rac1
2 ONLINE ONLINE rac2 Open
ora.scan1.vip
1 ONLINE ONLINE rac1
[grid@rac1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.LISTENER.lsnr
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.asm
ONLINE ONLINE rac1
ONLINE ONLINE rac2 Started
ora.eons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.gsd
OFFLINE OFFLINE rac1
OFFLINE OFFLINE rac2
ora.net1.network
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.ons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.registry.acfs
ONLINE ONLINE rac1
ONLINE ONLINE rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac1
ora.oc4j
1 OFFLINE OFFLINE
ora.rac1.vip
1 ONLINE ONLINE rac1
ora.rac2.vip
1 ONLINE ONLINE rac2
ora.radb.db
1 ONLINE ONLINE rac1
2 ONLINE ONLINE rac2 Open
ora.scan1.vip
1 ONLINE ONLINE rac1
[grid@rac1 ~]$
二，备份与恢复
1，查看备份磁盘
[grid@rac1 ~]$ ocrconfig -showbackup
PROT-24: Auto backups for the Oracle Cluster Registry are not available
PROT-25: Manual backups for the Oracle Cluster Registry are not available
[grid@rac1 ~]$
2，恢复
ocrconfig -restore /app/crs/product/11.0.6/crs/cdata/racluster/backup01.ocr
3，查看配置
[grid@rac1 ~]$ more /etc/oracle/ocr.loc
ocrconfig_loc=+DATA
local_only=FALSE
[grid@rac1 ~]$
4，查看进程
[grid@rac1 ~]$ ps -ef | grep d.bin
root 4694 1 0 10:00 ? 00:00:13 /u01/grid/bin/ohasd.bin reboot
grid 4821 1 0 10:00 ? 00:00:28 /u01/grid/bin/oraagent.bin
root 4823 1 0 10:00 ? 00:00:04 /u01/grid/bin/orarootagent.bin
grid 4846 1 0 10:00 ? 00:00:00 /u01/grid/bin/gipcd.bin
grid 4859 1 0 10:00 ? 00:00:00 /u01/grid/bin/mdnsd.bin
grid 4874 1 0 10:00 ? 00:00:01 /u01/grid/bin/gpnpd.bin
root 8645 1 0 10:47 ? 00:00:04 /u01/grid/bin/cssdmonitor
root 8662 1 0 10:48 ? 00:00:05 /u01/grid/bin/cssdagent
grid 8664 1 0 10:48 ? 00:00:01 /u01/grid/bin/diskmon.bin -d -f
grid 8688 1 0 10:48 ? 00:00:40 /u01/grid/bin/ocssd.bin
root 8754 1 0 10:50 ? 00:00:01 /u01/grid/bin/octssd.bin
grid 8770 1 0 10:50 ? 00:00:02 /u01/grid/bin/evmd.bin
grid 8888 1 0 10:51 ? 00:00:00 /u01/grid/bin/oclskd.bin
root 8920 1 0 10:51 ? 00:00:07 /u01/grid/bin/crsd.bin reboot
root 8966 1 0 10:51 ? 00:00:00 /u01/grid/bin/oclskd.bin
grid 9013 8770 0 10:51 ? 00:00:00 /u01/grid/bin/evmlogger.bin -o /u01/grid/evm/log/evmlogger.info -l /u01/grid/evm/log/evmlogger.log
grid 9055 1 0 10:51 ? 00:00:06 /u01/grid/bin/oraagent.bin
root 9059 1 0 10:51 ? 00:00:42 /u01/grid/bin/orarootagent.bin
grid 9283 1 0 10:52 ? 00:00:00 /u01/grid/bin/tnslsnr LISTENER -inherit
oracle 9549 1 0 10:58 ? 00:00:26 /u01/grid/bin/oraagent.bin
oracle 9773 1 0 11:00 ? 00:00:00 /u01/grid/bin/oclskd.bin
grid 18618 1 0 13:46 ? 00:00:00 /u01/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid 22527 21370 0 14:58 pts/2 00:00:00 grep d.bin
[grid@rac1 ~]$
注：
occsd:用于管理与协调集群中各个节点的关系，并用于节点通信。该进程非常的重要，如果这个进程异常中止，会导致系统自动重启。在某些极端的情况下，如果ocssd无法正常启动，就会导致操作系统循环重启。
crsd：监控节点各个资源，当某个资源发生异常是，自动重启或者切换该资源。
evmd：是一个基于后台的事件检测程序。
oclskd：该守护进程是Oracle 11g(11.10.6)新增的一个后台进程，主要是用于监控RAC数据库节点实例，当某个实例挂起时，就重启该节点。
三，管理表决磁盘
voting disk （表决磁盘）主要用于记录节点成员信息，例如包包含哪些节点成员、节点添加删除信息的记录等 。
1，查看voting disk
要查看voting disk信息，可以执行如下命令：
crsctl query css votedisk
2，备份voting disk
voting disk 的信息很重要，需要定期备份votedisk，例如：
dd if=/dev/raw/raw3 of=/tmp/votedisk.bak
3，恢复voting disk
dd if=/tmp/votedisk.bak of=/dev/raw/rawe
（ 注：表决磁盘（votedisk）是为因网络导致脑裂的情况而创建的。表决磁盘在11G存储在ASM中，且如果放入ASM将有几点要求。
第一：表决磁盘文件必须全部放入ASM中；
第二：表决磁盘存在asm中的个数不能删除和添加，而是通过asm的normal，high、external冗余级别决定的。
如：在normal中必须有3个故障组存在3个votedisk，在high中必须要有5个故障组存在5个votedisk，那么在external只有1个votedisk
第三：表决磁盘文件在11G不在支持dd命令对其的备份和还原，而是支持crsctl相关命令
第四：表决磁盘文件的个数要是奇数，便于投票选举，且表决磁盘文件的个数最多为15个，但一般没必要超过5个。）
4，删除voting disk镜像盘
如果删除一块表决磁盘，可以执行如下命令：
crsctl delete css votedisk /dev/raw/raw5 -force
这个操作 以root用户身份。
5，添加一块表决磁盘
添加一块表决磁盘时，执行如下命令：
crsctl add css votedisk /dev/raw/raw5 -force
这个操作以root用户身份
OCR磁盘主要用户记录节点成员的配置信息，如数据库、服务、实例、vip地址、监听、应用进程等CRS资源配置信息，OCR可以存储在裸设备或者集群文件系统上。
(1)，检查ocr设置
可以通过如下命令查看当前的OCR设置ocrcheck
(2)，OCR的备份与恢复
默认情况下，Oracle每四个小时自动备份一个OCR，并保留三个有效版本，但是仅保存在集群的某一个节点上。
查看自动备份OCR文件的路径，执行如下命令
四，导入导出
1，ocrconfig -showbackup
2，ocrconfig -restore \>/app/oracrs/product/11.0.6/cdata/rac-cluster/backup01.ocr
3，其实OCR也可以通过手动的方式导出、导入、方法如下：
ocrconfig -export /tmp/ocr_bak
ocrconfig -import /tmp/ocr_bak
五，添加OCR镜像盘
向磁盘组中添加OCR磁盘的基本步骤如下：
1，执行命令crsctl stop crs 停掉CRS服务
2，创建用于镜像OCR的RAW设备，例如：/dev/raw/raw11
3，执行"ocrconfig -export"命令导出OCR的信息
4，修改/etc/oracle/ocr.loc 文件，添加新增的raw设备，
5，执行命令“orcconfig -import”将OCR备份信息导入磁盘
6，检查ocr设置信息，看是否增加成功
7，最后用"crsctl start crs" 启动crs服务
补充：
11g中可移动表决磁盘
[grid@rac-one peer]$ crsctl get css disktimeout
CRS-4678: Successful get disktimeout 200 for Cluster Synchronization Services.
[grid@rac-one peer]$
通过ocssd进程来管理
如何移动表决磁盘文件到另一asm磁盘组呢？
eg：
[grid@rac-two ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Tue Mar 25 23:31:24 2014
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> col name for a20
SQL> col path for a40
SQL> set linesize 200
SQL> select a.name,b.name,b.path,b.voting_file from v$asm_disk b,v$asm_diskgroup a where a.group_number=b.group_number;
NAME NAME PATH V
-------------------- -------------------- ---------------------------------------- -
DATADG DATADG_0000 /dev/asm-diskg N
TESTDG TESTDG_0004 /dev/asm-diskh N
TESTDG TESTDG_0000 /dev/asm-diski N
TESTDG TESTDG_0001 /dev/asm-diskj N
TESTDG TESTDG_0003 /dev/asm-diskf N
GIDG GIDG_0001 /dev/asm-diskc Y
TESTDG TESTDG_0002 /dev/asm-diskk N
GIDG GIDG_0000 /dev/asm-diskd Y
GIDG GIDG_0002 /dev/asm-diske Y
DATADG DATADG_0001 /dev/asm-diskb N
10 rows selected.
SQL> exit
另外还可以使用crsctl query css votedisk 获得信息
[grid@rac-two ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 3da4160ba0334f66bf4b2e6e9b38cceb (/dev/asm-diskd) [GIDG]
2. ONLINE 2797b9ed25e84f34bf8fa5948c8fedb6 (/dev/asm-diske) [GIDG]
3. ONLINE 69ade332aa214f3abfd8b77a410019b0 (/dev/asm-diskc) [GIDG]
Located 3 voting disk(s).
当votedisk属于asm组从而移动votedisk的时候将不能使用crsctl add css votedisk命令，否则会报错。但是如果原先votedisk属于集群文件系统文件，那么可以使用该命令进而replace
完成操作。
[grid@rac-two ~]$ crsctl add css votedisk +testdg
CRS-4671: This command is not supported for ASM diskgroups.
CRS-4000: Command Add failed, or completed with errors.
将votedisk从磁盘组GIDG 移动到TESTDG
[grid@rac-two ~]$ crsctl replace votedisk +TESTDG
Successful addition of voting disk 98f8244772c74fc0bf8c5ddc699775f1.
Successful addition of voting disk aa123f79eef94f9ebf12ed0953a3caa6.
Successful addition of voting disk 42e16333baa44f78bf00840265bff13d.
Successful deletion of voting disk 3da4160ba0334f66bf4b2e6e9b38cceb.
Successful deletion of voting disk 2797b9ed25e84f34bf8fa5948c8fedb6.
Successful deletion of voting disk 69ade332aa214f3abfd8b77a410019b0.
Successfully replaced voting disk group with +TESTDG.
CRS-4266: Voting file(s) successfully replaced
[grid@rac-two ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 98f8244772c74fc0bf8c5ddc699775f1 (/dev/asm-diski) [TESTDG]
2. ONLINE aa123f79eef94f9ebf12ed0953a3caa6 (/dev/asm-diskk) [TESTDG]
3. ONLINE 42e16333baa44f78bf00840265bff13d (/dev/asm-diskf) [TESTDG]
Located 3 voting disk(s).
[grid@rac-two ~]$
备份表决磁盘文件。
刚刚介绍了无需备份
如果磁盘文件损坏了那么如何修复呢？
首先要独占模式启动crs资源
crsctl start crs -excl
如果是存在asm中，使用crsctl replace votedisk +diskgroup
完成修复
如果是存在集群文件系统，那么需要查看votedisk信息，crsctl query css votedisk然后删除原先的失效的votedisk，进而在添加votedisk
命令为：crsctl add css votedisk ‘path'，
然后再使用root停止crs，crsctl stop crs
然后再使用root启动crs ，crsctl start crs