【ASM】udev简介及配置、多路径（multipath）等

【ASM】udev简介及配置、多路径等

---

---

【ASM】Oracle ASM + 11gR2 + RHEL6.5 安装 请参考： http://blog.itpub.net/26736162/viewspace-1205206/

使用OpenFiler来模拟存储配置RAC中ASM共享盘及多路径（multipath）的测试 请参考 ： http://blog.itpub.net/26736162/viewspace-2132858/

（一）udev是什么？如何配置udev？

udev是Linux 2.6内核里的一个功能，它替代了原来的devfs，成为当前Linux默认的设备管理工具。udev以守护进程的形式运行，通过侦听内核发出来的uevent来管理/dev目录下的设备文件。

在Linux中，所有的设备都是以设备文件的形式存在的。在早期的Linux版本中，“/dev”目录包含了所有可能出现的设备的设备文件，所以，Linux用户很难在这些大量的设备文件中找到匹配条件的设备文件。现在，udev只为那些连接到Linux操作系统的设备产生设备文件，并且udev能通过定义一个udev规则（rule）来产生匹配设备属性的设备文件，这些设备属性可以是内核设备名称、总线路径、厂商名称、型号、序列号或者磁盘大小等等。当设备添加/删除时，udev的守护进程侦听来自内核的uevent，以此添加或者删除/dev下的设备文件，所以udev只为已经连接的设备产生设备文件，而不会在/dev下产生大量虚无的设备文件。

配置udev有如下几个步骤：

1.确认在所有RAC节点上已经安装了必要的udev包

[root@rh2 ~]# rpm -qa|grep udev

udev-095-14.21.el5

2.通过scsi_id获取设备的块设备的唯一标识名，假设系统上已有LUN sdc-sde

for i in c d e ;

do

echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id --whitelisted  --device=/dev/\$name\",RESULT==\"`scsi_id --whitelisted  --device=/dev/sd$i`\",NAME=\"asm-disk$i\",OWNER=\"grid\",GROUP=\"asmadmin\",MODE=\"0660\""

done

3.创建必要的udev配置文件

首先切换到配置文件目录：

[root@rh2 ~]# cd /etc/udev/rules.d

定义必要的规则配置文件：

[root@rh2 rules.d]# cat 99-oracle-asmdevices.rules

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted  --device=/dev/$name",RESULT=="14f504e46494c455232326c6c76442d4361634f2d4d4f4d41",NAME="asm-diskc",OWNER="grid",GROUP="asmadmin",MODE="0660"

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted  --device=/dev/$name",RESULT=="14f504e46494c455232326c6c76442d4361634f2d4d4f4d41",NAME="asm-diskd",OWNER="grid",GROUP="asmadmin",MODE="0660"

KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted  --device=/dev/$name",RESULT=="14f504e46494c455242674c7079392d753750482d63734443",NAME="asm-diske",OWNER="grid",GROUP="asmadmin",MODE="0660"

需要注意的是，一个KERNEL就是一行，不能换行。

4.将该规则文件拷贝到其它节点上

[root@rh2 rules.d]# scp 99-oracle-asmdevices.rules Other_node:/etc/udev/rules.d

5.在所有节点上启动udev服务，或者重启服务器即可

[root@rh2 rules.d]# /sbin/udevcontrol reload_rules

[root@rh2 rules.d]# /sbin/start_udev

Starting udev:                                            [  OK  ]

6.检查设备是否到位

[root@rh2 rules.d]# cd /dev

[root@rh2 dev]# ls -l asm-disk*

brw-rw---- 1 grid asmadmin 8,  64 Jul 10 17:31 asm-diskc

brw-rw---- 1 grid asmadmin 8, 208 Jul 10 17:31 asm-diskd

brw-rw---- 1 grid asmadmin 8, 224 Jul 10 17:31 asm-diske

配置完成后也可以使用udevadm进行测试，这里不再详述。

---

---

UDEV简介及配置过程  

原文地址：UDEV简介及配置过程 作者：kuqlan

在Linux环境下安装Oracle11g RAC时，OS层面配置好多路径软件后（multipath），下一步就需要配置udev或asmlib来处理共享分区（Lun），以便Orace ASM能够看到这些分区（Lun）。Linux6后，UDEV配置比asmlib配置起来相对简单。关于Multipath的配置说明，在上一文章(Linux MultiPath多路径软件实施说明 )已经进行了说明。接下来将要说明UDEV的配置操作。

udev 简介

     简单的讲，通过UDEV配置，能够让Oracle对磁盘名进行持久化并改变磁盘访问权限为grid:asmadmin。这样在ASM的配置过程中能够看到磁盘。今后增加磁盘也不会改变原有的磁盘名称。

           在存储行业中，用户总是有这样的要求：linux系统中原来有一块SCSI硬盘，系统分配的设备文件是/dev/sda。现在新增加了一个外置的磁盘阵列，通过SCSI卡连接。在接上这个磁盘阵列后，/dev/sda变成了磁盘阵列的硬盘了，原来内置的SCSI硬盘变成了 /dev/sdb，如何将设备文件名固定下来呢？

         在linux kernel 2.6内核之前，我们只能和用户说，这个比较麻烦，因为/dev/sda等文件都是linux内核自动分配的。很难固定下来，除非你更改加载SCSI卡驱动程序的顺序，而且对于其他的即插即用设备，如USB设备等都不适用。

升级到linux kernel 2.6内核后，这个问题已经可以通过linux kernel 2.6内核新的sysfs文件系统和udev程序来解决了。udev是一种工具，它能够根据系统中的硬件设备的状态动态更新设备文件，包括完成对设备文件的创建、删除和刷新。

udev原理和结构

1. 当udev从kernel收到设备add/remove 事件后, udev 将解析:

§        /etc/udev/rules.d目录中的用户自定义规则文件

§        使用自定义规则输出（可选的）

§        从/sys中查询相应信息

2. 根据解析的信息，udev汇集了处理设备命名：

§        确定将创建什么设备文件(device file)或符号链接（symlink)

§        确定如何设置设备文件属性

§        确定后续操作

配置udev

          所有的udev配置文件被放置在/etc/udev目录下，所有的配置文件均采用文本格式方便用户配置修改，在udev读取过程中将忽略配置文件中以#号开头的行。

udev的全局控制配置文件是/etc/udev/udev.conf，它定义了udev的全局配置参数。/etc/udev/udev.conf文件配置项目前只有3个：

• udev_root ── 定义在那个目录中创建所需的设备文件，默认位置为/dev目录。
• udev_rules ── 定义从那个配置文件或配置目录中读取udev规则，默认位置为 /etc/udev/rules.d目录
• udev_log ── 定义在传递log到syslog系统日志时的优先级，在调试udev的时候也可以run-time的使用udevcontrol命令修改这个优先级。 默认优先级为err，其它可选优先级为info和debug。

udev规则

1. 文件名位置及命名方法

• 默认放置在/etc/udev/rules.d目录下
• 命名为.rules,例如

1. 40multipath.rules
2. 60net.rules
3. bluetooth.rules

2. 规则结构

• value[,....]value[,....]

例如:

ACTION=="add",SUBSYSTEM=="mmc", UN+="modprobe mmc_block"

ACTION=="add", KERNEL=="sda", RUN+="/bin/raw  /dev/raw/raw1 %N"

3. 规则文件将在第一次预读后缓冲

• 当规则文件被修改后时间戳的更新会强制udev从读规则文件，如果想强制重读可以使用touch命令更新时间戳。

udev通过读取规则文件来决定如何命名或重命名设备和执行附加操作。一般情况下，规则文件放置在/etc/udev/rules.d目录下以 .rules 结尾的方式命名。如果配置文件不止一个，udev将按其文件名的词汇顺序读取每个规则配置文件。通常，我们以两位的数字开头定义udev规则文件，比如:

40multipath.rules和60net.rules。40multipath.rules 规则文件会在60net.rules规则文件之前被读取，因为按照词汇顺序4在6之前。缺省的udev规则存放在/etc/udev/rules.d/50udev.rules文件中。它包含一些例子，和一些默认的规则以提供一种devfs风格的/dev层。通常，我们希望自定义的规则先于默认的规则被运行，所以我建议你新增。

关于/etc/udev/rules.d/下新增配置文件的具体内容，与多路径配置时形成磁盘分区名称、类型、uuid等有关联，在此引用Deploying Oracle RAC 11g R2 Database on Red Hat EnterpriseLinux 6 的相关介绍，最后以生产系统为例进行说明：

99-oracle-asmdevices.rules文件的配置过程

1. On the first node of the Oracle RAC cluster, as the root user, identify the Device

Mapper Universally Unique IDentifier (DM_UUID) for each device mapper volume. The

example below shows the DM_UID for the partitions of the volumes labeled

ocrvote1,ocrvote2,ocrvote3,db1,db2,fra, and redo.

# for i in ocrvote1p1 ocrvote2p1 ocrvote3p1 db1p1 db2p1 frap1 redop1; do

printf "%s %s\n" "$i" "$(udevadm info --query=all --name=/dev/mapper/$i |grep -i dm_uuid)"; done

其中ocrvote1p1 ocrvote2p1 ocrvote3p1 db1p1 db2p1 frap1 redop1为multipath形成的裸设备分区名。在OS下执行以上文件后，会生成如下内容（根据实际环境有所不同）：

ocrvote1p1 E: DM_UUID=part1-mpath-3600c0ff000dabfe596a0f65101000000

ocrvote2p1 E: DM_UUID=part1-mpath-3600c0ff000dabfe5a2a0f65101000000

ocrvote3p1 E: DM_UUID=part1-mpath-3600c0ff000dabfe5b4a0f65101000000

db1p1 E: DM_UUID=part1-mpath-3600c0ff000d7e7a854a0f65101000000

db2p1 E: DM_UUID=part1-mpath-3600c0ff000dabfe562a0f65101000000

frap1 E: DM_UUID=part1-mpath-3600c0ff000d7e7a874a0f65101000000

redop1 E: DM_UUID=part1-mpath-3600c0ff000dabfe585a0f65101000000

2. Create a file labeled 99-oracle-asmdevices.rules within /etc/udev/rules.d/

3. Within 99-oracle-asmdevices.rules file, create rules for each device similar to the

example below:

/etc/udev/rules.d/99-oracle-asmdevices.rules

KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-

3600c0ff000dabfe5f4d8515101000000",OWNER="grid",GROUP="asmadmin",MODE="06

60"

To understand the rule above, it can be read as follows:

If any dm- device matches the DM_UUID of part1-mpath-

3600c0ff000dabfe5f4d8515101000000, assign to that dm- device to be owned by the

grid user and part of the asmadmin group with the permission mode set to 0660.

The 0660 value provides read and write permissions to the user grid and owner

asmadmin.

4. Save the file labeled 99-oracle-asmdevices.rules

5. Copy the 99-oracle-asmdevices.rules file to each node within the Oracle RAC

Database 11g Release 2 cluster using the scp command and enter the appropriate

password credentials for the other nodes. The example below shows how to copy the

file to node two of the Oracle RAC Database 11g Release 2 cluster.

# scp /etc/udev/rules.d/99-oracle-asmdevices.rules db-oraclenode2:/

etc/udev/rules.d/

Warning: Permanently added 'db-oracle-node2,10.16.142.52' (RSA) to the

list of known hosts.

root@db-oracle-node2's password:

99-oracleasmdevices.rules 100% 833 0.8KB/s 00:00

6. On each node within the Oracle RAC Database cluster, locate the dm- device for each

Oracle related partition. An example of how to find the dm- device for each partition is

to run the following command:

# for i in db1p1 db2p1 frap1 redop1 ocrvote1p1 ocrvote2p1 ocrvote3p1; do

printf "%s %s\n" "$i" "$(ls -ll /dev/mapper/$i)"; done

db1p1 lrwxrwxrwx. 1 root root 8 Aug 1 15:21 /dev/mapper/db1p1 -> ../dm-14

db2p1 lrwxrwxrwx. 1 root root 8 Aug 1 15:22 /dev/mapper/db2p1 -> ../dm-15

frap1 lrwxrwxrwx. 1 root root 8 Aug 1 15:22 /dev/mapper/frap1 -> ../dm-17

redop1 lrwxrwxrwx. 1 root root 8 Aug 1 15:22 /dev/mapper/redop1 -> ../dm-16

ocrvote1p1 lrwxrwxrwx. 1 root root 8 Aug 1 15:22 /dev/mapper/ocrvote1p1-> ../dm-18

ocrvote2p1 lrwxrwxrwx. 1 root root 8 Aug 1 15:22 /dev/mapper/ocrvote2p1-> ../dm-19

ocrvote3p1 lrwxrwxrwx. 1 root root 8 Aug 1 15:22 /dev/mapper/ocrvote3p1-> ../dm-20

7. On each node within the Oracle RAC Database cluster, apply and test the rules for

each dm- device created within the 99-oracle-asmdevices.rules by running a udevadm

test on each device. The example below demonstrates a udevadm test on dm-11.

# udevadm test /sys/block/dm-11

[ ... Output Abbreviated ... ]

udevadm_test: DM_NAME=db1p1

udevadm_test: DM_UUID=part1-mpath-3600c0ff000d7e7a86485ac5101000000

udevadm_test: DM_SUSPENDED=0

udevadm_test: DEVLINKS=/dev/mapper/db1p1 /dev/disk/by-id/dm-name-db1p1

/dev/disk/by-id/dm-uuid-part1-mpath-3600c0ff000d7e7a86485ac5101000000

/dev/block/253:11

udevadm_test: ID_FS_TYPE=oracleasm

8. Confirm each device has the desired permissions on each node within the cluster.

# ls -lh /dev/dm-*

brw-rw----. 1 grid asmadmin 253, 14 Aug 1 16:02 /dev/dm-14

brw-rw----. 1 grid asmadmin 253, 15 Aug 1 16:02 /dev/dm-15

brw-rw----. 1 grid asmadmin 253, 16 Aug 1 16:02 /dev/dm-16

brw-rw----. 1 grid asmadmin 253, 17 Aug 1 16:02 /dev/dm-17

brw-rw----. 1 grid asmadmin 253, 18 Aug 1 16:02 /dev/dm-18

brw-rw----. 1 grid asmadmin 253, 19 Aug 1 16:03 /dev/dm-19

brw-rw----. 1 grid asmadmin 253, 20 Aug 1 16:02 /dev/dm-20

NOTE: If the desired permissions are not visibile, please reboot the particular node

from the Oracle RAC Database cluster.

理解以上内容的基础上，将99-oracle-asmdevices.rules文件内容的配置可以简化为如下：

在第一节点通过fdisk格式化磁盘，做成分区形成：

fdisk /dev/mapper/mpathb

fdisk /dev/mapper/mpathc

fdisk /dev/mapper/mpathd

fdisk /dev/mapper/mpathe

.....

格式化fdisk /dev/mapper/mpathb，fdisk /dev/mapper/mpathc，fdisk /dev/mapper/mpathd，fdisk /dev/mapper/mpathe

等设备，形成/dev/mapper/mpathcp1，fdisk /dev/mapper/mpathdp1，fdisk /dev/mapper/mpathep1   ...

作为专业一点儿的DBA，我们也可以在multipath配置中设置alias，如下：

multipath {

wwid 3600c0ff000d7e7a854a0f65101000000

alias data01p1

}

二节点扫描设备：

# partprobe

编写UDEV文件（双节点都要配置）

编写UDEV权限文件：vi /etc/udev/rules.d/99-oracle-asmdevices.rules

PROGRAM="/bin/chown grid:oinstall /dev/mapper/orcvotep1"

PROGRAM="/bin/chown grid:oinstall /dev/mapper/data01p1"

PROGRAM="/bin/chown grid:oinstall /dev/mapper/data02p1"

PROGRAM="/bin/chown grid:oinstall /dev/mapper/data03p1"

PROGRAM="/bin/chown grid:oinstall /dev/mapper/data04p1"

PROGRAM="/bin/chown grid:oinstall /dev/mapper/data05p1"

PROGRAM="/bin/chown grid:oinstall /dev/mapper/fra01"

PROGRAM="/bin/chown grid:oinstall /dev/mapper/fra02"

PROGRAM="/bin/chown grid:oinstall /dev/mapper/fra03"

PROGRAM="/bin/chown grid:oinstall /dev/mapper/fra04"

PROGRAM="/bin/chown grid:oinstall /dev/mapper/fra05"

PROGRAM="/bin/chmod 0660 /dev/mapper/orcvotep1"

PROGRAM="/bin/chmod 0660 /dev/mapper/data01p1"

PROGRAM="/bin/chmod 0660 /dev/mapper/data02p1"

PROGRAM="/bin/chmod 0660 /dev/mapper/data03p1"

PROGRAM="/bin/chmod 0660 /dev/mapper/data04p1"

PROGRAM="/bin/chmod 0660 /dev/mapper/data05p1"

PROGRAM="/bin/chmod 0660 /dev/mapper/fra01"

PROGRAM="/bin/chmod 0660 /dev/mapper/fra02"

PROGRAM="/bin/chmod 0660 /dev/mapper/fra03"

PROGRAM="/bin/chmod 0660 /dev/mapper/fra04"

PROGRAM="/bin/chmod 0660 /dev/mapper/fra05"

双节点重新启动UDEV：

#start_udev

---

---

Oracle ASM on RHEL 6 with udev and multipath

As the root user, install the device-mapper-multipath package using the yum package manager

[root@vzwc1 ~]# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 6.4 (Santiago)
[root@vzwc1 ~]# uname -p
x86_64

[root@vzwc1 ~]# yum install device-mapper-multipath

 

Copy the multipath.conf file found within /usr/share/doc/device-mapper-multipath-0.4.9/ to /etc/
[root@vzwc1 ~]# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/multipath.conf

 

Capture the scsi id of the local disks on the system.

[root@vzwc2 ~]# for i in b c d e f g h i j
> do
> scsi_id –whitelisted –replace-whitespace –device=/dev/sd$i
> done
1ATA_VBOX_HARDDISK_VBda7290c1-228b4bfb
1ATA_VBOX_HARDDISK_VBcb76adf7-43d27c31
1ATA_VBOX_HARDDISK_VB2077c81c-7cfe5922
1ATA_VBOX_HARDDISK_VB7282bfc8-69601726
1ATA_VBOX_HARDDISK_VBa0072176-9e9ab0a0
1ATA_VBOX_HARDDISK_VB2a7b18e2-04053806
1ATA_VBOX_HARDDISK_VB40939671-89967d5e
1ATA_VBOX_HARDDISK_VB664cb850-2065525c
1ATA_VBOX_HARDDISK_VBa498ea69-53cdb110
[root@vzwc2 ~]# 

 

Enter the above values into /etc/multipath and give friendly aliases to uuid

[root@vzwc1 ~]# cat /etc/multipath.conf
blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z]"
        wwid 1ATA_VBOX_HARDDISK_VB82d3887f-a892e124         #exclude sda wwid
}
defaults {
        user_friendly_names no
        getuid_callout "/sbin/scsi_id –whitelisted –replace-whitespace –device=/dev/%n"
}
multipaths {
        multipath {
                wwid                    1ATA_VBOX_HARDDISK_VBda7290c1-228b4bfb
                alias                   ocrvotedisk1
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                failback                immediate
                rr_weight               priorities
                no_path_retry           5
        }
        multipath {
                wwid                    1ATA_VBOX_HARDDISK_VBcb76adf7-43d27c31
                alias                   ocrvotedisk2
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                failback                immediate
                rr_weight               priorities
                no_path_retry           5
        }
        multipath {
                wwid                    1ATA_VBOX_HARDDISK_VB2077c81c-7cfe5922
                alias                   ocrvotedisk3
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                failback                immediate
                rr_weight               priorities
                no_path_retry           5
        }
        multipath {
                wwid                    1ATA_VBOX_HARDDISK_VB7282bfc8-69601726
                alias                   ocrvotedisk4
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                failback                immediate
                rr_weight               priorities
                no_path_retry           5
        }
        multipath {
                wwid                    1ATA_VBOX_HARDDISK_VBa0072176-9e9ab0a0
                alias                   ocrvotedisk5
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                failback                immediate
                rr_weight               priorities
                no_path_retry           5
        }
        multipath {
                wwid                    1ATA_VBOX_HARDDISK_VB2a7b18e2-04053806
                alias                   data1
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                failback                immediate
                rr_weight               priorities
                no_path_retry           5
        }
        multipath {
                wwid                    1ATA_VBOX_HARDDISK_VB40939671-89967d5e
                alias                   data2
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                failback                immediate
                rr_weight               priorities
                no_path_retry           5
        }
        multipath {
                wwid                    1ATA_VBOX_HARDDISK_VB664cb850-2065525c
                alias                   fra1
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                failback                immediate
                rr_weight               priorities
                no_path_retry           5
        }
        multipath {
                wwid                    1ATA_VBOX_HARDDISK_VBa498ea69-53cdb110
                alias                   fra2
                path_grouping_policy    multibus
                path_selector           "round-robin 0"
                failback                immediate
                rr_weight               priorities
                no_path_retry           5
        }
}

 

Restart multipathd service

[root@vzwc1 ~]# service multipathd restart
ok
Stopping multipathd daemon: [  OK  ]
Starting multipathd daemon: [  OK  ]

[root@vzwc1 ~]# chkconfig multipathd on

 

Check if multipath is setup properly

[root@vzwc2 ~]# multipath -ll
fra2 (1ATA_VBOX_HARDDISK_VBa498ea69-53cdb110) dm-8 ATA,VBOX HARDDISK
size=12G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 11:0:0:0 sdj 8:144 active ready running
ocrvotedisk5 (1ATA_VBOX_HARDDISK_VBa0072176-9e9ab0a0) dm-4 ATA,VBOX HARDDISK
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 7:0:0:0  sdf 8:80  active ready running
fra1 (1ATA_VBOX_HARDDISK_VB664cb850-2065525c) dm-7 ATA,VBOX HARDDISK
size=12G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 10:0:0:0 sdi 8:128 active ready running
ocrvotedisk4 (1ATA_VBOX_HARDDISK_VB7282bfc8-69601726) dm-3 ATA,VBOX HARDDISK
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 6:0:0:0  sde 8:64  active ready running
ocrvotedisk3 (1ATA_VBOX_HARDDISK_VB2077c81c-7cfe5922) dm-2 ATA,VBOX HARDDISK
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 5:0:0:0  sdd 8:48  active ready running
ocrvotedisk2 (1ATA_VBOX_HARDDISK_VBcb76adf7-43d27c31) dm-1 ATA,VBOX HARDDISK
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 4:0:0:0  sdc 8:32  active ready running
ocrvotedisk1 (1ATA_VBOX_HARDDISK_VBda7290c1-228b4bfb) dm-0 ATA,VBOX HARDDISK
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 3:0:0:0  sdb 8:16  active ready running
data2 (1ATA_VBOX_HARDDISK_VB40939671-89967d5e) dm-6 ATA,VBOX HARDDISK
size=20G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 9:0:0:0  sdh 8:112 active ready running
data1 (1ATA_VBOX_HARDDISK_VB2a7b18e2-04053806) dm-5 ATA,VBOX HARDDISK
size=20G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 8:0:0:0  sdg 8:96  active ready running

 

[root@vzwc1 ~]# ls -l /dev/mapper/
total 0
crw-rw—- 1 root root 10, 236 Sep 24 13:56 control
lrwxrwxrwx 1 root root       7 Sep 24 13:57 data1 -> ../dm-5
lrwxrwxrwx 1 root root       7 Sep 24 13:57 data2 -> ../dm-6
lrwxrwxrwx 1 root root       7 Sep 24 13:56 fra1 -> ../dm-7
lrwxrwxrwx 1 root root       7 Sep 24 13:56 fra2 -> ../dm-8
lrwxrwxrwx 1 root root       7 Sep 24 13:57 ocrvotedisk1 -> ../dm-0
lrwxrwxrwx 1 root root       7 Sep 24 13:56 ocrvotedisk2 -> ../dm-1
lrwxrwxrwx 1 root root       7 Sep 24 13:56 ocrvotedisk3 -> ../dm-2
lrwxrwxrwx 1 root root       7 Sep 24 13:56 ocrvotedisk4 -> ../dm-3
lrwxrwxrwx 1 root root       7 Sep 24 13:56 ocrvotedisk5 -> ../dm-4

 

[root@vzwc1 ~]# for i in ocrvotedisk1 ocrvotedisk2 ocrvotedisk3 ocrvotedisk4 ocrvotedisk5 data1 data2 fra1 fra2
> do
> printf "%s %s\n" "$i" "$(ls -ll /dev/mapper/$i)"
> done
ocrvotedisk1 lrwxrwxrwx 1 root root 7 Sep 24 13:57 /dev/mapper/ocrvotedisk1 -> ../dm-0
ocrvotedisk2 lrwxrwxrwx 1 root root 7 Sep 24 13:56 /dev/mapper/ocrvotedisk2 -> ../dm-1
ocrvotedisk3 lrwxrwxrwx 1 root root 7 Sep 24 13:56 /dev/mapper/ocrvotedisk3 -> ../dm-2
ocrvotedisk4 lrwxrwxrwx 1 root root 7 Sep 24 13:56 /dev/mapper/ocrvotedisk4 -> ../dm-3
ocrvotedisk5 lrwxrwxrwx 1 root root 7 Sep 24 13:56 /dev/mapper/ocrvotedisk5 -> ../dm-4
data1 lrwxrwxrwx 1 root root 7 Sep 24 13:57 /dev/mapper/data1 -> ../dm-5
data2 lrwxrwxrwx 1 root root 7 Sep 24 13:57 /dev/mapper/data2 -> ../dm-6
fra1 lrwxrwxrwx 1 root root 7 Sep 24 13:56 /dev/mapper/fra1 -> ../dm-7
fra2 lrwxrwxrwx 1 root root 7 Sep 24 13:56 /dev/mapper/fra2 -> ../dm-8

Determine partitioned alias for target device

[root@vzwc1 ~]# dmsetup ls|sort
data1   (252:5)
data2   (252:6)
fra1    (252:7)
fra2    (252:8)
ocrvotedisk1    (252:0)
ocrvotedisk2    (252:1)
ocrvotedisk3    (252:2)
ocrvotedisk4    (252:3)
ocrvotedisk5    (252:4)

 

Edit /etc/udev/rules.d/12-dm-permissions.rules file to set the above device ownership as grid:asmadmin

[root@vzwc1 ~]# cat /etc/udev/rules.d/12-dm-permissions.rules
ENV{DM_NAME}=="ocrvotedisk1", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="ocrvotedisk2", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="ocrvotedisk3", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="ocrvotedisk4", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="ocrvotedisk5", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="data1", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="data2", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="fra1", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="fra2", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"

 

Restart multipathd service

[root@vzwc1 ~]# /etc/init.d/multipathd restart
ok
Stopping multipathd daemon: [  OK  ]
Starting multipathd daemon: [  OK  ]

 

Check if permissions are correct

[root@vzwc1 ~]# ls -l /dev/dm-*
brw-rw—- 1 grid asmadmin 252, 0 Sep 24 16:35 /dev/dm-0
brw-rw—- 1 grid asmadmin 252, 1 Sep 24 16:35 /dev/dm-1
brw-rw—- 1 grid asmadmin 252, 2 Sep 24 16:35 /dev/dm-2
brw-rw—- 1 grid asmadmin 252, 3 Sep 24 16:35 /dev/dm-3
brw-rw—- 1 grid asmadmin 252, 4 Sep 24 16:35 /dev/dm-4
brw-rw—- 1 grid asmadmin 252, 5 Sep 24 16:35 /dev/dm-5
brw-rw—- 1 grid asmadmin 252, 6 Sep 24 16:35 /dev/dm-6
brw-rw—- 1 grid asmadmin 252, 7 Sep 24 16:35 /dev/dm-7
brw-rw—- 1 grid asmadmin 252, 8 Sep 24 16:35 /dev/dm-8

 

Check if oracle ASM symbolic links exists for multipath devices

[root@vzwc2 ~]# ls -l /dev/iscsi/*
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-data1 -> ../dm-5
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-data2 -> ../dm-6
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-fra1 -> ../dm-7
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-fra2 -> ../dm-8
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-ocrvotedisk1 -> ../dm-0
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-ocrvotedisk2 -> ../dm-1
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-ocrvotedisk3 -> ../dm-2
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-ocrvotedisk4 -> ../dm-3
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-ocrvotedisk5 -> ../dm-4

 

Configure ASM disk change discovery path

 

[grid@vzwc2 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
–  —–    —————–                ——— ———
 1. ONLINE   1374f266ca874f15bfde7ca23d073e5f (/dev/mapper/ocrvotedisk1) [CRSDG]
 2. ONLINE   4145a926ff0d4f6dbf63e2c3695c580e (/dev/mapper/ocrvotedisk2) [CRSDG]
 3. ONLINE   6f3ab4bbe0c14f97bfb20c91820d898b (/dev/mapper/ocrvotedisk3) [CRSDG]
 4. ONLINE   8c973dd3f1da4f0dbf2080e04fcf9bd7 (/dev/mapper/ocrvotedisk4) [CRSDG]
 5. ONLINE   1bed4a3c6f624fabbfbe2564e81df250 (/dev/mapper/ocrvotedisk5) [CRSDG]
Located 5 voting disk(s).
[grid@vzwc2 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Wed Sep 24 17:17:22 2014

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> select * from v$active_instances;

INST_NUMBER INST_NAME
———– ——————–
          1 vzwc1:+ASM1
          2 vzwc2:+ASM2

SQL> select path from v$asm_disk;

PATH
——————————————————————————–
/dev/mapper/data2
/dev/mapper/fra2
/dev/mapper/fra1
/dev/mapper/data1
/dev/mapper/ocrvotedisk2
/dev/mapper/ocrvotedisk3
/dev/mapper/ocrvotedisk1
/dev/mapper/ocrvotedisk4
/dev/mapper/ocrvotedisk5

9 rows selected.

---

---

Oracle ASM on RHEL 5 with multipath

As the root user, install the device-mapper-multipath package using the yum package manager

[root@db11g ~]# cat /etc/redhat-release   
Red Hat Enterprise Linux Server release 5.9 (Tikanga)

[root@db11g ~]# yum install device-mapper*  
Loaded plugins: rhnplugin, security  
This system is not registered with ULN.  
You can use up2date –register to register.  
ULN support will be disabled.  
Setting up Install Process  
Package device-mapper-multipath-0.4.9-56.0.3.el5.x86_64 already installed and latest version  
Package device-mapper-multipath-libs-0.4.9-56.0.3.el5.x86_64 already installed and latest version  
Package device-mapper-event-1.02.67-2.el5.x86_64 already installed and latest version  
Package device-mapper-1.02.67-2.el5.x86_64 already installed and latest version  
Package device-mapper-1.02.67-2.el5.i386 already installed and latest version  
Nothing to do  
[root@db11g ~]#   
[root@db11g ~]#   
[root@db11g ~]#   
[root@db11g ~]#   
[root@db11g ~]# rpm -qa|grep device-mapper  
device-mapper-multipath-libs-0.4.9-56.0.3.el5  
device-mapper-event-1.02.67-2.el5  
device-mapper-1.02.67-2.el5  
device-mapper-1.02.67-2.el5  
device-mapper-multipath-0.4.9-56.0.3.el5  

 

Capture the scsi id of the local disks on the system

[root@db11g ~]# /sbin/scsi_id -g -u -s /block/sdb  
SATA_VBOX_HARDDISK_VB3c0bb909-10aab3a0_  
[root@db11g ~]# /sbin/scsi_id -g -u -s /block/sdc  
SATA_VBOX_HARDDISK_VB3edd7dc1-fd831bf4_  

 

Get to the grid user's uid and gid, here the gid is asmadmin group

[root@db11g ~]# cat /etc/passwd|grep grid  
grid:x:1100:1000:Grid Infrastructure Owner:/home/grid:/bin/bash  
[root@db11g ~]# cat /etc/group|grep asmadmin  
asmadmin:x:1200:grid,oracle  
[root@db11g ~]# id -a grid  
uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1200(asmadmin),1201(asmdba),1202(asmoper),1300(dba)  

 

Enter the above values into /etc/multipath.conf

[root@db11g ~]# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/  
[root@db11g ~]# cat /etc/multipath.conf  
defaults {  
user_friendly_names no  
queue_without_daemon no  
flush_on_last_del yes  
max_fds max  
}   
blacklist {  
devnode "^hd[a-z]"  
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"  
devnode "^cciss.*"  
}     
devices {  
        device {  
                vendor                  "OPNFILER "  
                product                 "LUN"  
                path_grouping_policy    group_by_prio  
                features                "3 queue_if_no_path pg_init_retries 50"  
                getuid_callout          "/sbin/scsi_id -g -u -s /block/%n"  
                path_checker            tur  
                path_selector           "round-robin 0"  
                hardware_handler        "1 alua"  
                failback                immediate  
                rr_weight               uniform  
                rr_min_io               128  
        }  
}    
multipaths {  
        multipath {  
                wwid                    SATA_VBOX_HARDDISK_VB3c0bb909-10aab3a0_   
                alias                   datadisk  
                uid                     1100                                                 
                gid                     1200                                                
        }  
        multipath {  
                wwid                    SATA_VBOX_HARDDISK_VB3edd7dc1-fd831bf4_     
                alias                   fradisk                                            
                uid                     1100                                                 
                gid                     1200                                                 
         }  
}  

 

Start multipathd service

[root@db11g ~]# modprobe dm-multipath  
[root@db11g ~]# modprobe dm-round-robin  
[root@db11g ~]# chkconfig multipathd start  
[root@db11g ~]# chkconfig multipathd on  
[root@db11g ~]# /etc/init.d/multipathd start  
Starting multipathd daemon: [  OK  ]  
[root@db11g ~]# ps -ef|grep multipathd  
root      3084     1  0 13:16 ?        00:00:00 /sbin/multipathd  
root      3186  2877  0 13:16 pts/0    00:00:00 grep multipathd  
[root@db11g ~]# multipath -F  
[root@db11g ~]# multipath -v2  
Jan 15 13:17:40 | SATA_VBOX_HARDDISK_VBa64c2685-cc2f3dd0_: ignoring map  
create: datadisk (SATA_VBOX_HARDDISK_VB3c0bb909-10aab3a0_) undef ATA,VBOX HARDDISK  
size=16G features='0' hwhandler='0' wp=undef  
`-+- policy='round-robin 0' prio=1 status=undef  
  `- 1:0:0:0 sdb 8:16 undef ready running  
create: fradisk (SATA_VBOX_HARDDISK_VB3edd7dc1-fd831bf4_) undef ATA,VBOX HARDDISK  
size=10G features='0' hwhandler='0' wp=undef  
`-+- policy='round-robin 0' prio=1 status=undef  
  `- 2:0:0:0 sdc 8:32 undef ready running  
[root@db11g ~]#   
[root@db11g ~]# multipath -ll  
fradisk (SATA_VBOX_HARDDISK_VB3edd7dc1-fd831bf4_) dm-3 ATA,VBOX HARDDISK  
size=10G features='0' hwhandler='0' wp=rw  
`-+- policy='round-robin 0' prio=1 status=active  
  `- 2:0:0:0 sdc 8:32 active ready running  
datadisk (SATA_VBOX_HARDDISK_VB3c0bb909-10aab3a0_) dm-2 ATA,VBOX HARDDISK  
size=16G features='0' hwhandler='0' wp=rw  
`-+- policy='round-robin 0' prio=1 status=active  
  `- 1:0:0:0 sdb 8:16 active ready running  
  
[root@db11g ~]# ll /dev/mapper/*disk  
brw-rw—- 1 grid asmadmin 252, 2 Jan 15 13:17 /dev/mapper/datadisk  
brw-rw—- 1 grid asmadmin 252, 3 Jan 15 13:17 /dev/mapper/fradisk 

 

---

---

About Me

............................................................................................................................... ● 本文整理自网络 ● 本文在itpub（http://blog.itpub.net/26736162）、博客园（http://www.cnblogs.com/lhrbest）和个人微信公众号（xiaomaimiaolhr）上有同步更新 ● 本文itpub地址：http://blog.itpub.net/26736162/abstract/1/ ● 本文博客园地址：http://www.cnblogs.com/lhrbest ● 本文pdf版及小麦苗云盘地址：http://blog.itpub.net/26736162/viewspace-1624453/ ● 数据库笔试面试题库及解答：http://blog.itpub.net/26736162/viewspace-2134706/ ● QQ群：230161599     微信群：私聊 ● 联系我请加QQ好友(646634621)，注明添加缘由 ● 于 2017-07-01 09:00 ~ 2017-07-31 22:00 在魔都完成 ● 文章内容来源于小麦苗的学习笔记，部分整理自网络，若有侵权或不当之处还请谅解 ● 版权所有，欢迎分享本文，转载请保留出处 ............................................................................................................................... 拿起手机使用微信客户端扫描下边的左边图片来关注小麦苗的微信公众号：xiaomaimiaolhr，扫描右边的二维码加入小麦苗的QQ群，学习最实用的数据库技术。

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/26736162/viewspace-2135271/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/26736162/viewspace-2135271/