Suse 11下多路径及udev配置

    最近给客户基于SuSe 11 SP3下多路径部署Oracle 10g RAC。SuSe 11下用10g，也算一朵奇葩，连篇文档都比较难找，谁叫Oracle太贵呢。下面主要是描述了在该环境下如何去配置多路径。由于10g下的ocr与votingdisk不能直接存放到asm磁盘，所以依旧要使用raw设备方式来保存。下文供大家参考。

 

一、查看当前scsi设备及获取设备wwid
#使用 cat /proc/partitions或者/sbin/sfdisk命令查看当前系统中的scsi设备
suse11a:~ # more /proc/partitions
major minor  #blocks  name

   8        0  877264896 sda
   8        1     200812 sda1
   8        2  104856255 sda2
   8        3  104856255 sda3
   8        4  667340100 sda4
   8       16 2097152000 sdb
   8       17 2097150976 sdb1
               .....
   8      240    1048576 sdp
  65        0    1048576 sdq
               .....
  65       48 2097152000 sdt
  65       49 2097150976 sdt1
 253        0 2097152000 dm-0
 253        1 2097152000 dm-1
 253        2 1751121920 dm-2
 253        3    1048576 dm-3
 253        4    1048576 dm-4
 253        5 1751120896 dm-5
 253        6 2097150976 dm-6
 253        7 2097150976 dm-7
 
#获取多路径设备的wwid，使用下面的命令来获取当前系统中多路径设备的wwid
for i in `cat /proc/partitions | awk {'print $4'} |grep sd`; do echo "### $i: `/lib/udev/scsi_id --whitelist  --replace-whitespace /dev/$i`"; done

### sda: 3600605b007030d301a192f2d13c017a0
### sda1: 3600605b007030d301a192f2d13c017a0
     .....................
### sdt: 360022a11000cb6b10a902db200000000
### sdt1: 360022a11000cb6b10a902db200000000
#获取后，过滤本地磁盘及重复的wwid

 

二、配置mulitpatch.conf文件
#配置mulitpatch.conf文件的主要目的是用于使用有意义，用户自定义的设备名称，持久化多路径设备
#这个步骤是与配置单路径设备最大的区别
#下面是已经配置过后的multipath.conf文件
suse11a:~ # cp /etc/multipath.conf /etc/multipath.conf.bak
suse11a:~ # vi /etc/multipath.conf
suse11a:~ # grep -v ^# /etc/multipath.conf
defaults {
        user_friendly_names yes
}
blacklist {
        devnode ^hd[a-z]
}
multipaths {
      multipath {
              wwid                    36200bc710099b9190ba6fc350000000a
              alias                   mpatha
       }

      multipath {
              wwid                    36200bc710099b9190ba7109b0000000b
              alias                   mpathb
       }
      multipath {
              wwid                    36200bc710099b9190ab1155300000009
              alias                   mpathc
       }
      multipath {
              wwid                    360022a11000cb6b10a902db200000000
              alias                   mpathd
       }
       multipath {
              wwid                    360022a11000cb6b10a9116d000000002
              alias                   mpathe
       }
}

 

三、校验多路径设备
#使用下面的命令用于多路径配置生效
suse11a:~ # service multipathd stop
suse11a:~ # service multipathd start

#下面校验多路径设备
suse11a:~ # multipath -ll | grep mpath
mpathe (360022a11000cb6b10a9116d000000002) dm-1 HUAWEI  ,S5500T         
mpathd (360022a11000cb6b10a902db200000000) dm-0 HUAWEI  ,S5500T         
mpathc (36200bc710099b9190ab1155300000009) dm-2 HUAWEI  ,S2600T         
mpathb (36200bc710099b9190ba7109b0000000b) dm-4 HUAWEI  ,S2600T         
mpatha (36200bc710099b9190ba6fc350000000a) dm-3 HUAWEI  ,S2600T

#下面是设备的详细信息
suse11a:~ # multipath -ll
mpathe (360022a11000cb6b10a9116d000000002) dm-1 HUAWEI  ,S5500T         
size=2.0T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 5:0:0:1 sdc 8:32  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 5:0:3:1 sdk 8:160 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 6:0:2:1 sds 65:32 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 6:0:3:1 sdu 65:64 active ready running
mpathd (360022a11000cb6b10a902db200000000) dm-0 HUAWEI  ,S5500T         
size=2.0T features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 5:0:0:0 sdb 8:16  active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 5:0:3:0 sdj 8:144 active ready running
|-+- policy='service-time 0' prio=1 status=enabled
| `- 6:0:2:0 sdr 65:16 active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 6:0:3:0 sdt 65:48 active ready running
            .........

suse11a:~ # dmsetup ls |sort
mpatha  (253:3)
mpathb  (253:4)
mpathc  (253:2)
mpathc_part1    (253:5)
mpathd  (253:0)
mpathd_part1    (253:6)
mpathe  (253:1)
mpathe_part1    (253:7)
   
#以下命令均可以用于校验多路径设备
# ll /dev/disk/by-id/
# ls -l /dev/dm-*
# ll /dev/mapper/

#/dev/dm-*下面的为动态名，即系统重启后会发生变化，此部分用于os 内部使用。
#/dev/mapper/下面查看到的内容为持久化名称，也即是我们在mulitpatch.conf文件中定义的。

   
四、配置RAW
#在Oracle 10g R2 rac安装的过程中，OUI不能够校验共享的快涉笔，因此需要使用raw。Oracle 11g R1后可以直接使用多路径设备。
#使用udev来管理raw，编辑raw文件，如下
suse11a:~ # cat /etc/raw | grep -v \#
raw1:mapper/mpatha
raw2:mapper/mpathb
raw3:mapper/mpathc
raw4:mapper/mpathd
raw5:mapper/mpathe

#重启raw以及配置开机启动raw
suse11a:~ # rcraw start
suse11a:~ # chkconfig raw on
suse11a:~ # raw -qa
/dev/raw/raw1:  bound to major 253, minor 3
/dev/raw/raw2:  bound to major 253, minor 4
/dev/raw/raw3:  bound to major 253, minor 2
/dev/raw/raw4:  bound to major 253, minor 0
/dev/raw/raw5:  bound to major 253, minor 1

#测试raw的可访问性
# dd if=/dev/zero of=/dev/raw/raw1 bs=1024 count=100
100+0 records in
100+0 records out
102400 bytes (102 kB) copied, 0.762352 seconds, 134 kB/s

#配置权限，该权限用于oracle用户访问裸设备
# Author : Leshami
# Blog   : http://blog.csdn.net/leshami
suse11a:~ # more /etc/udev/rules.d/99-raw.rules
KERNEL=="raw[1-2]*", GROUP="oinstall", MODE="664"
KERNEL=="raw[3-5]*", OWNER="oracle", GROUP="oinstall", MODE="664"

 

五、安装clusterware
#安装clusterware时，关于ocr和votingdisk，选择/dev/raw/raw1，/dev/raw/raw2，如果有配置冗余，此处应该为多个裸设备。
#校验安转好clusterware后裸设备的使用情况
suse11a:oracle:ora10g1 > ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          2
         Total space (kbytes)     :    1048296
         Used space (kbytes)      :       4592
         Available space (kbytes) :    1043704
         ID                       :  892902908
         Device/File Name         : /dev/raw/raw1
                                    Device/File integrity check succeeded

                                    Device/File not configured

         Cluster registry integrity check succeeded

suse11a:oracle:ora10g1 > crsctl query  css votedisk
 0.     0    /dev/raw/raw2

located 1 votedisk(s). 
#对于ASM磁盘可以通过dbca来创建，直接使用/dev/raw/raw*以及/dev/mapper/*。当然suse 11支持asmlib，也可以通过asmlib方式来访问asm磁盘。