quorum failgroup disk (3)
需求：
共两个diskgroup，+DATA和+SYS。其中+DATA有n个盘，external redundancy 类型；+SYS有3块盘，normal redundancy类型。
需要将OCR和VOTE存放到+SYS diskgroup，并将OCR冗余到+DATA diskgroup。
另外还需要将SYSTEM表空间存放到+SYS diskgroup，保证同一个extent，分布到不同磁盘（机柜）。
实施办法：
下列操作最好在安装CRS时完成，也可以在RAC安装完成后进行。
1、安装CRS时，创建+DATA diskgroup，external redundancy 类型。
2、CRS安装完成后，创建+SYS diskgroup，normal redundancy 类型。创建SQL如下：
create diskgroup sys
normal redundancy
quorum failgroup sysfg1 disk '/dev/rhdisk1'
regular failgroup sysfg2 disk '/dev/rhdisk2'
regular failgroup sysfg3 disk '/dev/rhdisk3'
attribute 'compatible.asm' = '11.2.0.0.0';
3、在集群的其他节点mount +SYS。
alter diskgroup mount sys;
4、迁移votedisk到+SYS。
crsctl replace votedisk +SYS
5、冗余ocr到+SYS。
ocrconfig -add +SYS
6、检查验证。
crsctl query css votedisk
ocrcheck
SQL> select group_number,name,type from v$asm_diskgroup;
GROUP_NUMBER NAME TYPE
------------ ------------------------------ ------
1 DATA EXTERN
SQL> col path for a30
SQL> select name,path from v$asm_disk;
NAME PATH
------------------------------ ------------------------------
/dev/asm_high3
/dev/asm_high2
/dev/asm_high1
/dev/asm_normal2
/dev/asm_normal1
DATA_0001 /dev/asm2
DATA_0000 /dev/asm1
7 rows selected.
SQL> create diskgroup sys normal redundancy
2 quorum failgroup sysfg1 disk '/dev/asm_high1'
3 regular failgroup sysfg2 disk '/dev/asm_high2'
4 regular failgroup sysfg3 disk '/dev/asm_high3'
5 attribute 'compatible.asm' = '11.2.0.0.0'
6 ;
Diskgroup created.
SQL>
<<< 创建sys diskgroup，这里三个盘各自为一个failgroup，其中盘1增加了quorum属性。
SQL> select name,path,redundancy from v$asm_disk;
NAME PATH REDUNDA
------------------------------ ------------------------------ -------
/dev/asm_normal2 UNKNOWN
/dev/asm_normal1 UNKNOWN
SYS_0002 /dev/asm_high3 UNKNOWN
SYS_0001 /dev/asm_high2 UNKNOWN
SYS_0000 /dev/asm_high1 UNKNOWN
DATA_0001 /dev/asm2 UNKNOWN
DATA_0000 /dev/asm1 UNKNOWN
7 rows selected.
SQL> select name,type from v$asm_diskgroup;
NAME TYPE
------------------------------ ------
DATA EXTERN
SYS NORMAL
<<< 其中DATA为external redundancy，SYS为normal redundancy。
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[grid@node1 ~]$ exit
logout
[root@node1 rules.d]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 846d42046b044fabbfefbbd4e6dc08dd (/dev/asm1) [DATA]
Located 1 voting disk(s).
<<< 当前votedisk在DATA diskgroup中。
[root@node1 rules.d]#
[root@node1 rules.d]# crsctl replace votedisk +SYS
Successful addition of voting disk a76c9cca4f884ffebf07adecbbaae129.
Successful addition of voting disk 8f58d3b4b2774f3ebfc37dca90ce1a69.
Successful addition of voting disk 924e7dfb80274febbf76789783bccff9.
Successful deletion of voting disk 846d42046b044fabbfefbbd4e6dc08dd.
Successfully replaced voting disk group with +SYS.
CRS-4266: Voting file(s) successfully replaced
[root@node1 rules.d]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE a76c9cca4f884ffebf07adecbbaae129 (/dev/asm_high1) [SYS]
2. ONLINE 8f58d3b4b2774f3ebfc37dca90ce1a69 (/dev/asm_high2) [SYS]
3. ONLINE 924e7dfb80274febbf76789783bccff9 (/dev/asm_high3) [SYS]
Located 3 voting disk(s).
[root@node1 rules.d]# ocrconfig -add +SYS
[root@node1 rules.d]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3244
Available space (kbytes) : 258876
ID : 179398566
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File Name : +SYS
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@node1 rules.d]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE a76c9cca4f884ffebf07adecbbaae129 (/dev/asm_high1) [SYS]
2. ONLINE 8f58d3b4b2774f3ebfc37dca90ce1a69 (/dev/asm_high2) [SYS]
3. ONLINE 924e7dfb80274febbf76789783bccff9 (/dev/asm_high3) [SYS]
Located 3 voting disk(s).
[root@node1 rules.d]#
<<< 迁移votedisk到SYS中，并冗余OCR到sys中。
SQL> select group_number,name,type from v$asm_diskgroup;
GROUP_NUMBER NAME TYPE
------------ ------------------------------ ------
1 DATA EXTERN
2 SYS NORMAL
SQL> select path from v$asm_disk where group_number = 2;
PATH
--------------------------------------------------------------------------------
/dev/asm_high3
/dev/asm_high2
/dev/asm_high1
SQL> col path for a30
SQL> c/path/path,disk_number
1* select path,disk_number from v$asm_disk where group_number = 2
SQL> /
PATH DISK_NUMBER
------------------------------ -----------
/dev/asm_high3 2
/dev/asm_high2 1
/dev/asm_high1 0
SQL> l
1* select path,disk_number,failgroup_type from v$asm_disk where group_number = 2
SQL> /
PATH DISK_NUMBER FAILGRO
------------------------------ ----------- -------
/dev/asm_high3 2 REGULAR
/dev/asm_high2 1 REGULAR
/dev/asm_high1 0 QUORUM
SQL>
<<< /dev/asm_high1为quorum，数据将无法进入这个盘。下面过程验证数据将无法进入这个盘。
SQL> create tablespace test datafile '+SYS' size 10m;
Tablespace created.
SQL>
<<< 在sys上创建test表空间。
<<< 通过v$asm_alias.file_number找到test的file_number为256，又知道diskgroup_number为2，代入下面的SQL将显示extent的分布。
SQL> select group_kffxp group#,disk_kffxp disk#,au_kffxp au#,xnum_kffxp extent# from x$kffxp
2 where
3 group_kffxp=&diskgroup_number
4 and number_kffxp=&file_number;
Enter value for diskgroup_number: 2
old 3: group_kffxp=&diskgroup_number
new 3: group_kffxp=2
Enter value for file_number: 256
old 4: and number_kffxp=&file_number
new 4: and number_kffxp=256
GROUP# DISK# AU# EXTENT#
---------- ---------- ---------- ----------
2 2 396 0
2 1 396 0
2 1 397 1
2 2 397 1
2 2 398 2
2 1 398 2
2 1 399 3
2 2 399 3
2 2 400 4
2 1 400 4
2 1 401 5
GROUP# DISK# AU# EXTENT#
---------- ---------- ---------- ----------
2 2 401 5
2 2 402 6
2 1 402 6
2 1 403 7
2 2 403 7
2 2 404 8
2 1 404 8
2 1 405 9
2 2 405 9
2 2 406 10
2 1 406 10
22 rows selected.
SQL> select group_kffxp group#,disk_kffxp disk#,
2 xnum_kffxp extent#,
3 case lxn_kffxp
4 when 0 then 'Primary Copy'
5 when 1 then 'Mirrored Copy'
6 when 2 then '2nd Mirrored Copy or metadata'
7 else 'Unknown' end type
8 from x$kffxp
9 where number_kffxp=256
10 and group_kffxp = 2
11 and xnum_kffxp!=65534
12 order by 3;
GROUP# DISK# EXTENT# TYPE
---------- ---------- ---------- -----------------------------
2 2 0 Primary Copy
2 1 0 Mirrored Copy
2 1 1 Primary Copy
2 2 1 Mirrored Copy
2 2 2 Primary Copy
2 1 2 Mirrored Copy
2 1 3 Primary Copy
2 2 3 Mirrored Copy
2 2 4 Primary Copy
2 1 4 Mirrored Copy
2 1 5 Primary Copy
GROUP# DISK# EXTENT# TYPE
---------- ---------- ---------- -----------------------------
2 2 5 Mirrored Copy
2 2 6 Primary Copy
2 1 6 Mirrored Copy
2 1 7 Primary Copy
2 2 7 Mirrored Copy
2 2 8 Primary Copy
2 1 8 Mirrored Copy
2 1 9 Primary Copy
2 2 9 Mirrored Copy
2 2 10 Primary Copy
2 1 10 Mirrored Copy
22 rows selected.
SQL>
<<< 可以看到每个区被冗余了一次，并且所有区只分布在disk# 1或2上，该表空间文件的区没有进入到disk# 0磁盘。