在11gR2 RAC中修改ASM DISK Path磁盘路径

有同学在T.askmaclean.com上提问关于修改11gR2中ASM DISK的路径问题，具体问题如下：  

aix 6.1，grid 11.2.0.3+asm11.2.0.3+rac 建数据库的时候使用的是aix自带的多路径软件mpio，建了diskgroup 现在改造成veritas dmp多路径，已经修改了asm的disk_strings＝/dev/vx/rdmp/*，crs/asm启动的时候已经可以识别到磁盘/dev/vx/rdmp/开头的磁盘，但是读取不回原来的diskgroup信息。 crs启动的报错日志： 2012-07-13 15:07:29.748: [ GPNP][1286]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2108 clsgpnp_profileCallUrlInt] get-profile call to url "ipc://GPNPD_ggtest1" disco "" [f=0 claimed- host: cname: seq: auth:] 2012-07-13 15:07:29.762: [ GPNP][1286]clsgpnp_profileCallUrlInt: [at clsgpnp.c:2236 clsgpnp_profileCallUrlInt] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote "ipc://GPNPD_ggtest1" disco "" 2012-07-13 15:07:29.762: [ CSSD][1286]clssnmReadDiscoveryProfile: voting file discovery string(/dev/vx/rdmp/*) 2012-07-13 15:07:29.762: [ CSSD][1286]clssnmvDDiscThread: using discovery string /dev/vx/rdmp/* for initial discovery 2012-07-13 15:07:29.762: [ SKGFD][1286]Discovery with str:/dev/vx/rdmp/*: 2012-07-13 15:07:29.762: [ SKGFD][1286]UFS discovery with :/dev/vx/rdmp/*: 2012-07-13 15:07:29.769: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_919: 2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_212: 2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_211: 2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_210: 2012-07-13 15:07:29.770: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_209: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_181: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/v_df8000_180: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_3: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_2: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_1: 2012-07-13 15:07:29.771: [ SKGFD][1286]Fetching UFS disk :/dev/vx/rdmp/disk_0: 2012-07-13 15:07:29.771: [ SKGFD][1286]OSS discovery with :/dev/vx/rdmp/*: 2012-07-13 15:07:29.771: [ SKGFD][1286]Handle 1115e7510 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_916: 2012-07-13 15:07:29.772: [ SKGFD][1286]Handle 1118758b0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_912: 2012-07-13 15:07:29.773: [ SKGFD][1286]Handle 1118d9cf0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_908: 2012-07-13 15:07:29.773: [ SKGFD][1286]Handle 1118da450 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_904: 2012-07-13 15:07:29.773: [ SKGFD][1286]Handle 1118dad70 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_903: 2012-07-13 15:07:29.802: [ CLSF][1286]checksum failed for disk:/dev/vx/rdmp/v_df8000_916: 2012-07-13 15:07:29.803: [ SKGFD][1286]Lib :UFS:: closing handle 1115e7510 for disk :/dev/vx/rdmp/v_df8000_916: 2012-07-13 15:07:29.803: [ SKGFD][1286]Lib :UFS:: closing handle 1118758b0 for disk :/dev/vx/rdmp/v_df8000_912: 2012-07-13 15:07:29.804: [ SKGFD][1286]Handle 1115e6710 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_202: 2012-07-13 15:07:29.808: [ SKGFD][1286]Handle 1115e7030 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_201: 2012-07-13 15:07:29.809: [ SKGFD][1286]Handle 1115e7ad0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_200: 2012-07-13 15:07:29.809: [ SKGFD][1286]Handle 1118733f0 from lib :UFS:: for disk :/dev/vx/rdmp/v_df8000_199: 2012-07-13 15:07:29.816: [ CLSF][1286]checksum failed for disk:/dev/vx/rdmp/v_df8000_186: 2012-07-13 15:07:29.816: [ SKGFD][1286]Lib :UFS:: closing handle 1118de5d0 for disk :/dev/vx/rdmp/v_df8000_186: 2012-07-13 15:07:29.816: [ CSSD][1286]clssnmvDiskVerify: Successful discovery of 0 disks 2012-07-13 15:07:29.816: [ CSSD][1286]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery 2012-07-13 15:07:29.816: [ CSSD][1286]clssnmvFindInitialConfigs: No voting files found 2012-07-13 15:07:29.816: [ CSSD][1286](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds 2012-07-13 15:07:30.169: [ CSSD][1029]clssgmExecuteClientRequest(): type(37) size(80) only connect and exit messages are allowed before lease acquisition proc(1115e4870) client(0)

    其需求为修改ASM DISK PATH的磁盘设备路径，但是由于11gR2 RAC+ASM的特殊性，导致CRS无法正常启动，虽然使用crsctl start crs -excl -nocrs的方式可以启动CSS服务和ASM实例，   但是最后还是报(clssnmCompleteInitVFDiscovery: Voting file not found)，这是由于Voteing file投票磁盘的位置没有合理更新导致的。     这里我们来学习一下，如何正确的修改11gR2 RAC+ASM下的ASM DISK路径：     1.我们来营造一个修改ASM DISK路径的环境，这里我们使用UDEV的设备名绑定服务，并利用UDEV将原本的ASM DISK从/dev/asm-disk* 修改为 /dev/rasm-disk*的形式， 这只需要修改udev rule文件即可实现：    

[grid@maclean1 ~]$ export  ORACLE_HOME=/g01/grid/app/11.2.0/grid

[grid@maclean1 ~]$ /g01/grid/app/11.2.0/grid/bin/sqlplus  / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Sun Jul 15 04:09:28 2012

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> show parameter diskstri

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskstring                       string      /dev/asm*

  可以看到当前ASM实例使用的asm_diskstring 为/dev/asm*， 切换到root用户修改UDEV RULE文件 ：    

[root@maclean1 rules.d]# cp 99-oracle-asmdevices.rules  99-oracle-asmdevices.rules.bak
[root@maclean1 rules.d]# vi 99-oracle-asmdevices.rules

[root@maclean1 rules.d]# cat 99-oracle-asmdevices.rules

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB09cadb31-cfbea255_", NAME="rasm-diskb", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB5f097069-59efb82f_", NAME="rasm-diskc", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB4e1a81c0-20478bc4_", NAME="rasm-diskd", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VBdcce9285-b13c5a27_", NAME="rasm-diske", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB82effe1a-dbca7dff_", NAME="rasm-diskf", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB950d279f-c581cb51_", NAME="rasm-diskg", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB14400d81-651672d7_", NAME="rasm-diskh", OWNER="grid", GROUP="asmadmin", MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="SATA_VBOX_HARDDISK_VB31b1237b-78aa22bb_", NAME="rasm-diski", OWNER="grid", GROUP="asmadmin", MODE="0660"

    以上修改了原始99-oracle-asmdevices.rules的UDEV RULE规则文件，生成的设备名被修改为/dev/rasm-disk*的形式，不同于之前的ASM DISK设备名， 这要求我们后续的一系列操作来保证RAC CRS可以正常启动。   当前运行时的votedisk和ocr 存放位置:  

[root@maclean1 rules.d]# /g01/grid/app/11.2.0/grid/bin/crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   6896bfc3d1464f9fbf0ea9df87e023ad (/dev/asm-diskb) [SYSTEMDG]
 2. ONLINE   58eb81b656084ff2bfd315d9badd08b7 (/dev/asm-diskc) [SYSTEMDG]
 3. ONLINE   6bf7324625c54f3abf2c942b1e7f70d9 (/dev/asm-diskd) [SYSTEMDG]
 4. ONLINE   43ad8ae20c354f5ebf7083bc30bf94cc (/dev/asm-diske) [SYSTEMDG]
 5. ONLINE   4c225359d51b4f93bfba01080664b3d7 (/dev/asm-diskf) [SYSTEMDG]
Located 5 voting disk(s).

[root@maclean1 rules.d]# /g01/grid/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2844
         Available space (kbytes) :     259276
         ID                       :  879001605
         Device/File Name         :  +SYSTEMDG
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

    因为votedisk file的位置具体指向一个ASM DISK，所以后续我们会需要crsctl replace votedisk， 我们现在重启LINUX OS:    

[root@maclean1 rules.d]# init 6

rebooting ............

[root@maclean1 dev]# ls -l *asm*
brw-rw---- 1 grid asmadmin 8,  16 Jul 15 04:15 rasm-diskb
brw-rw---- 1 grid asmadmin 8,  32 Jul 15 04:15 rasm-diskc
brw-rw---- 1 grid asmadmin 8,  48 Jul 15 04:15 rasm-diskd
brw-rw---- 1 grid asmadmin 8,  64 Jul 15 04:15 rasm-diske
brw-rw---- 1 grid asmadmin 8,  80 Jul 15 04:15 rasm-diskf
brw-rw---- 1 grid asmadmin 8,  96 Jul 15 04:15 rasm-diskg
brw-rw---- 1 grid asmadmin 8, 112 Jul 15 04:15 rasm-diskh
brw-rw---- 1 grid asmadmin 8, 128 Jul 15 04:15 rasm-diski

    重启后自动获得了形如/dev/rasm-disk*的ASM DISK,查阅ASM日志可以发现css服务仍搜索/dev/asm*路径来获得ASM DISK，但是这将导致找不到任何有效的ASM DISK:  

more /g01/grid/app/11.2.0/grid/log/maclean1/cssd/ocssd.log

2012-07-15 04:17:45.208: [ SKGFD][1099548992]Discovery with str:/dev/asm*:

2012-07-15 04:17:45.208: [   SKGFD][1099548992]UFS discovery with :/dev/asm*:

2012-07-15 04:17:45.208: [   SKGFD][1099548992]OSS discovery with :/dev/asm*:

2012-07-15 04:17:45.208: [ CSSD][1099548992]clssnmvDiskVerify: Successful discovery of 0 disks
2012-07-15 04:17:45.208: [    CSSD][1099548992]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2012-07-15 04:17:45.208: [    CSSD][1099548992]clssnmvFindInitialConfigs: No voting files found
2012-07-15 04:17:45.208: [    CSSD][1099548992](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found.
Retrying discovery in 15 seconds
2012-07-15 04:17:45.251: [    CSSD][1096661312]clssgmExecuteClientRequest(): type(37) size(80) only connect and
exit messages are allowed before lease acquisition proc(0x26a8ba0) client((nil))
2012-07-15 04:17:45.251: [    CSSD][1096661312]clssgmDeadProc: proc 0x26a8ba0
2012-07-15 04:17:45.251: [    CSSD][1096661312]clssgmDestroyProc: cleaning up proc(0x26a8ba0) con(0xfe6) skgpid
ospid 3751 with 0 clients, refcount 0
2012-07-15 04:17:45.252: [    CSSD][1096661312]clssgmDiscEndpcl: gipcDestroy 0xfe6
2012-07-15 04:17:45.829: [    CSSD][1096661312]clssscSelect: cookie accept request 0x2318ea0
2012-07-15 04:17:45.829: [    CSSD][1096661312]clssgmAllocProc: (0x2659480) allocated
2012-07-15 04:17:45.830: [    CSSD][1096661312]clssgmClientConnectMsg: properties of cmProc 0x2659480 - 1,2,3,4,5
2012-07-15 04:17:45.830: [    CSSD][1096661312]clssgmClientConnectMsg: Connect from con(0x114e) proc(0x2659480) pid(3751)
version 11:2:1:4, properties: 1,2,3,4,5
2012-07-15 04:17:45.830: [    CSSD][1096661312]clssgmClientConnectMsg: msg flags 0x0000
2012-07-15 04:17:45.939: [    CSSD][1096661312]clssscSelect: cookie accept request 0x253ddd0
2012-07-15 04:17:45.939: [    CSSD][1096661312]clssscevtypSHRCON: getting client with cmproc 0x253ddd0
2012-07-15 04:17:45.939: [    CSSD][1096661312]clssgmRegisterClient: proc(3/0x253ddd0), client(61/0x26877b0)
2012-07-15 04:17:45.939: [    CSSD][1096661312]clssgmExecuteClientRequest(): type(6) size(684) only connect and exit messages are