oracle11.2.0.4 rac搭建中的crs-4000错误解析
在系统环境为rhel6.5的服务器上，搭建数据库版本为oracle11.2.0.4的两节点的RAC，安装GRID时遭遇如下错误：
（注：vote、data和闪回盘都在存储设备上）
ASM created and started successfully.
Disk Group VOTE mounted successfully.
clscfg: -install mode specified
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Failed to create voting files on disk group VOTE.
Change to configuration failed, but was successfully rolled back.
CRS-4000: Command Replace failed, or completed with errors.
Voting file add failed
Failed to add voting disks at /g01/11ggrid/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 6930.
/g01/11ggrid/app/11.2.0/grid/perl/bin/perl -I/g01/11ggrid/app/11.2.0/grid/perl/lib -I/g01/11ggrid/app/11.2.0/grid/crs/install /g01/11ggrid/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
这个错误是说无法对磁盘组VOTE创建投票文件。改变配置失败，但成功回滚。
投票文件添加失败，未能在/g01/11ggrid/app/11.2.0/grid/crs/install添加表决磁盘。
碰到这个错误还是头一遭，很是诡异。因为之前使用ASMLIB挂盘都没有出现任何问题，但安装却出现这个问题，没法继续进行下去。
检查ASM的 alert.log日志发现：
ERROR: Could not create voting files. It spans across 161 AUs (max supported is 64 AUs)
ERROR: Voting file allocation failed for group VOTE
Errors in file /g01/11ggrid/app/11.2.0/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_7321.trc:
ORA-15303: Voting files could not be created in diskgroup VOTE due to small Allocation Unit size
查看对应GRID的仲裁盘对应的DM
#multipath -ll
sc_vote (360a980003830302d712b46586276736b) dm-11 NETAPP,LUN
size=5.0G features='4 queue_if_no_path pg_init_retries 50 retain_attached_hw_handle' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=2 status=active
`- 17:0:0:0 sdax 67:16 active ready running
继续检查一下DM-11块的大小问题，发现一些信息：
# cd /sys/block/dm-11/queue
# more *block_size
::::::::::::::
logical_block_size
::::::::::::::
512
::::::::::::::
physical_block_size
::::::::::::::
4096
究其原因是它表现出不同的逻辑和物理块大小。对于大多数的情况下，物理和逻辑块的大小是相同的“512”。
那么如何解决这个问题呢？查询ORACLE的官方网站，给出的解决方案是：
1. Oracle bug 11780656 is already fixed in 11.2.0.3 with compatible.asm = 11.2.0.3 but OUI does not allow to specify "compatible.asm" attribute to create OCRVOTE diskgroup.
- Bug 11780656 - ASM MANAGED VOTING FILES CANNOT BE MORE THAN 64X AU SIZE
2. Oracle bug 13999609 indicates that ASMLib driver only works with the expectation that logical block size and physical block size are 512/512 bytes.
- Bug 13999609 - PHYSICAL BLOCK SIZE REPORTED CAN CAUSE ISSUES WITH 10G DATABASES
SOLUTION
1] Possible workaround is to use '/dev/oracleasm/disks/*' path instead of "ORCL:*" when creating OCRVOTE diskgroup in OUI.
OR
2] Install the new “oracleasm-support-2.1.8-1” ASMLIB RPM package (which contains the permanent fix) as described in note 1500460.1
根据以上提示，我们可以在安装oracleasm-support-2.1.8-1的包，或者也可以在选择ASM盘时，改用
/dev/oracleasm/disks/*的路径来解决此问题。根据以上提示进行解决，发现还是无法成功装上GRID.
反复几次，没有找到解决的办法。由于后端使用的是NETAPP存储，所以就去NETAPP的官网查找此问题的CASE案例。发现其给出如下解决方案：
For details and caveats regarding this workaround, see the Oracle Alert.
Additionally, Oracle has provided a patch and configuration parameter to enable ASMlib to continue to function using the correct logical block size.
# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size
# reported by the underlying disk instead of the physical. The default
# is 'false'
(oracle 设置)
NetApp has also provided a workaround in versions 8.0.5, 8.1.3 and 8.2 of Data ONTAP 7-Mode. The workaround allows specified LUNs to continue to not report the logical blocks per physical block value. This work around should only be applied to LUNs used by Oracle ASMlib with the symptoms described in this article.
From the Data ONTAP 7-Mode CLI, enter the following commands:
> lun set report-physical-size <path> disable（netapp存储设置）
根据此提示，然后进入/etc/sysconfig目录，修改其下的oracleasm文件，将其中的
[oracle@rac1 ~]$ cat /etc/sysconfig/oracleasm
#
# This is a configuration file for automatic loading of the Oracle
# Automatic Storage Management library kernel driver. It is generated
# By running /etc/init.d/oracleasm configure. Please use that method
# to modify this file
#
# ORACLEASM_ENABLED: 'true' means to load the driver on boot.
ORACLEASM_ENABLED=true
# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.
ORACLEASM_UID=grid
# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.
ORACLEASM_GID=asmadmin
# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.
ORACLEASM_SCANBOOT=true
# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER=""
# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE=""
# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size
# reported by the underlying disk instead of the physical. The default
# is 'false'
ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false
将ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false改为true
并对应修改NETAPP上的存储设置参数
（注：Oracle提供了配置参数来启用ASMLib继续使用正确的逻辑块的大小的功能来避免这个问题。）
通过如上的处理，该问题得以解决，GRID最后安装成功。