谈谈Oracle ASM的限制

今天在客户现场安装Oracle rac时，创建ASM DISKGROUP时出现问题，以下为解决过程。记录之，以备忘。
客户操作系统版本为AIX 6105，Oracle crs版本为10.2.0.5,rdbms和asm版本为10.2.0.1。
当用dbca工具建立磁盘组时，检索不出备选磁盘。在确定rdisk权限等没问题之后，在启动ASM实例之后，尝试在1号节点手动建立磁盘组，不料建磁盘组时出现以下错误。

引用

$ sqlplus "/as sysdba"

SQL*Plus: Release 10.2.0.1.0 - Production on Thu Mar 31 13:44:37 2011

Copyright (c) 1982, 2005, Oracle.  All rights reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining options

SQL> create diskgroup data1 external redundancy disk '/dev/rhdisk7' size 2047G;
create diskgroup data1 external redundancy disk '/dev/rhdisk7' size 2047G
*
ERROR at line 1:
ORA-15018: diskgroup cannot be created
IBM AIX RISC System/6000 Error: 6: No such device or address
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error: 6: No such device or address
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error: 6: No such device or address
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error: 6: No such device or address
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error: 6: No such device or address
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error: 6: No such device or address
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error: 6: No such device or address
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error: 6: No such device or address
Additional information: -1
Additional information: 655360
IBM AIX RISC System/6000 Error:

红色标注可以看到，Oracle识别不了磁盘rhdisk7，但实际上rhdisk7确实存在。由于时间紧张，上google搜索，在一份otn的帖子中，一老外同行也碰到过类似问题，后来在解决问题描述时指出：

引用

Hi All,
I have resolved the issue now.
Since we are using SAN disks with raid 5. The disks size visible to ASm was not correct one.
So what i did is that I added SIZE clause during asm diskgroup creation.
create diskgroup oradata external redundany disk '/dev/rhdisk4' size 1000G;
by issuing this statement now I can create diskgroups.
Remember max size of a SAN disk for ASM is 2TB. We can't go beyond 2TB

询问客户之后，发现本案例中也是raid5 而且rdisk也超过了2T，于是按照网友指出的方法进行创磁盘组尝试。幸好创建成功！

引用

SQL> create diskgroup data external redundancy disk '/dev/rhdisk7' size 1990G;

Diskgroup created.

创建成功之后，碰到了ASM重启之后，识别不了磁盘组错误：

引用

SQL> startup
ASM instance started

Total System Global Area  130023424 bytes
Fixed Size                  2019136 bytes
Variable Size             102838464 bytes
ASM Cache                  25165824 bytes
ORA-15110: no diskgroups mounted

或者

引用

SQL> startup
ASM instance started

Total System Global Area  130023424 bytes
Fixed Size                  2019136 bytes
Variable Size             102838464 bytes
ASM Cache                  25165824 bytes
ORA-15032: not all alterations performed
ORA-15063: ASM discovered an insufficient number of disks for diskgroup
"DATA,DATA1"

解决办法为：
在asm参数文件中显示指明该asm需要mount的磁盘组

引用

asm_diskgroups='DATA','ARCH'
ASM_DISKSTRING ='/dev/rhdisk7','/dev/rhdisk8'

为什么ASM不能创建超过2T的硬盘呢？上metalink一查，Oracle针对这一现象，解释为bug：Bug 6453944 - ORA-15196 with ASM disks larger than 2TB [ID 6453944.8]

引用

ORA-15196 (ASM block corruption) can occur, if LUNs larger than 2TB are
presented to an ASM diskgroup.

As a result of the fix, ORA-15099 will be raised if larger than 2TB size
disk is specified. This is irrespective of the presence of asmlib.

In the future, this limitation will be removed.

Workaround:
  Do not add more than 2 TB size disk to a diskgroup.

此外metalink 736891.1对此bug引起的后果有较为详细的描述，其中最直接的后果就是损坏ASM元数据（metadata）：

引用

There are no clear indications of what can go wrong with ASM LUNs larger than 2TB – but one example is wrapping: when Oracle gets to the end of 2TB, it may wrap back to the start of the LUN and overwrite the ASM metadata

，并导致如下错误发生：

引用

ORA-15196 : invalid ASM block header [kfc.c:7997] [hard_kfbh] [4] [4] [34 != 130]

当然这一bug已经在以下版本修复：

引用

    11.2
    10.2.0.4 (Server Patch Set)
    11.1.0.7 (Server Patch Set)

那么除此之外，ASM还有什么限制呢?(metalink:ID 370921.1).我想除了2TB限制，其他限制达到的可能性还是比较低的。

引用

63 disk groups in a storage system
10,000 ASM disks in a storage system
2 terabyte maximum storage for each ASM disk (the Bug 6453944 allowed larger sizes, but that led to problems, see Note 736891.1 "ORA-15196 WITH ASM DISKS LARGER THAN 2TB")
40 exabyte maximum storage for each storage system
1 million files for each disk group
2.4 terabyte maximum storage for each file