RH436 UNIT 4 MULTIPATHING

Introduction

Unit Goal

        Describe multipathing

        Configure dm-multipath

Unit Sections

        What is Multipathing?

        Configuring Multipathing

        Testing Multipathing

Hands-On Activities

        Configure Multipathing

        Testing Multipathing

Unit Test

        Configuring Multipathing

What is Multipathing?

        所谓多路径是实现在服务器与磁盘阵列之间配置多条物理连接。多路径的优点有二,其一是在一条路径瘫痪时不会影响连通性,另外可以改善对存储的读写性能。

        如下图,服务器有两个HBA,都是独立连接到光纤交换机,每一条连接都连接着磁盘阵列上独立的控制器上。

image

        Red Hat Enterprise Linux 6 支持使用dm-multipath,这种方式是使用kernel磁盘映射生成一个虚拟的磁盘设备。通过multipathd的进程和multipath命令进行管理。

        必要的binaries,daemons,and kernel modules可以再device-mapper-multipath 包中找到。

        安装device-mapper-multipath软件包,配置并启动后,会创建/dev/mapper,多路径的SYMLINK会被创建在其中,名称可以为mpatchN[pM],或者使用WWID,管理员也可以自定义多路径设备的名称。

Important

        多路径只是提供接入的冗余,防护连接出现断开。如果存储本身出现问题,即使配置多路径还是无法访问存储。

        通过配置文件创建不同的路径连接存储,这些Path会被整合到不同的组中,当一个组为fails状态的时候会跳到另外一个组保证可以正常访问存储。

What is multipathing?

1. Multipathing can provide..    A,B

a. Redundancy in storage access

b. Increased performance

c. Storage redundancy

d. RAID

2. The package that provides multipathing capabilities is called..   C

a. dm-multipath

b. dm-multipathd

c. divice-mapper-multipath

d. multipathd

3. Devices nodes for multipathed storage are normally created under..   C

a. /dev/

b. /dev/multipath/

c. /dev/mapper/

d. /dev/dm-multipath

Configuring Multipathing

        配置多路径之前先确保device-mapper-multipath package已安装。

        配置两个网卡作为多路径的访问路径。

eth1      Link encap:Ethernet  HWaddr 00:0C:29:B3:DE:A4 
          inet addr:172.16.1.4  Bcast:172.16.1.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:feb3:dea4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:28 errors:0 dropped:0 overruns:0 frame:0
          TX packets:16 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1756 (1.7 KiB)  TX bytes:1000 (1000.0 b)

eth2      Link encap:Ethernet  HWaddr 00:0C:29:B3:DE:AE 
          inet addr:172.16.2.4  Bcast:172.16.2.255  Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:feb3:deae/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:13 errors:0 dropped:0 overruns:0 frame:0
          TX packets:24 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:780 (780.0 b)  TX bytes:1440 (1.4 KiB)

        安装完device-mapper-multipath 软件包之后开始配置multipath的配置文件,最简单的方式是使用mpathconf工具进行配置。

        如果已经/etc/multipath.conf已经存在,那么可以直接使用mpathconf进行配置,如果不存在,可以再/usr/share/doc/device-mapper-multipath-*下面找到multipath.conf的模板。也可以通过mpathconf命令创建默认模板。

        创建默认配置,启动和激活multipathd进程,可以使用以下命令。

mpathconf --enable --with_multipathd y --with_chkconfig y

--enable
       Removes any line that blacklists all device nodes from the /etc/multipath.conf blacklist section.

--with_multipathd { y | n }
       If set to y, this runs service multipathd start to start the multipathd daemon on --enable, service mul-
       tipathd stop to start the multipathd daemon on --disable, and service multipathd reload  to  reconfigure
       multipathd on --user_frindly_names and --find_multipaths.  This option is set to n by default.

        添加此参数会使用service multipathd start启动multipathd服务,前提是参数为y ,如果为n则为停止服务,默认为n。

--with_chkconfig { y | n }
       If  set to y, this runs chkconfig multipathd on to set multipathd to start automatically on --enable and
       chkconfig multipathd off to stop multipathd for starting automatically on --disable.  This option is set
       to y by default.

        如果为y,则执行chkconfig multipathd on设置开机启动,如果为n则为off关闭开机启动,默认为y。

Note

        关于―user_friendly_names 参数,如果设置为y,则使用友好名称作为多路径磁盘的名称,如果设置为n,则为使用WWID作为SYMLINK的名称。使用mpathconf �Cuser_friendly_names y|n 进行设置,通过此命令会直接修改配置文件。

[root@node1 nodes]# mpathconf --user_friendly_names n

## Use user friendly names, instead of using WWIDs as names.
defaults {
        user_friendly_names no
}
##

Disk /dev/mapper/1IET\x20\x20\x20\x20\x2000010001: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00001618

                                        Device Boot      Start         End      Blocks   Id  System

如果设置为y ,重启后结果为

Disk /dev/mapper/mpatha: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00001618

             Device Boot      Start         End      Blocks   Id  System

The multipath.conf Configuration File

        在multipath.conf配置文件中包含以下5部分内容

blacklist{}

        此列表中的devices将被排除在多路径的拓扑之外。for example,在配置文件中添加以下行

        blacklist {
                device{
                wwid    1IET_00010001
                }
        }

        重启多路径进程,重新登录target,发现物mpatha设备已消失。

blacklist_exceptions{}

        此列表为允许的device加入多路径组,及时它同时存在于blacklist之中。

defaults{}

        此列表内存储针对所有多路径的默认设置,除非他有设置相同属性在devices{},或者multipaths{}之中。

devices{}

        此列表内存储关于vendor,product的基本信息,此列表会覆盖defaults{}列表里面的内容,但是不会覆盖multipaths{}里面的内容。(可通过正则表达式匹配sysfs中的内容)

multipaths{}

        此列表存储多路径的基本设置,它的内容会覆盖defaults{}和devices{}中的内容。多路径会确认WWIDs上的信息。(获取可使用getuid_callout函数)

以下会对各列表做解释和举例

Blacklisting

        将设备列入多路径的黑名单可以使用multipath.conf配置文件的blacklist{}部分。你可以再blacklist中使用通配符匹配多数设备,然后使用blacklist_exceptions{}排除要加入多路径的设备。在Blacklist中可以使用device node或者WWID进行匹配。例如:

blacklist {
        devnode "sdb"
}

        设置之后,重启多路径服务,刚才的mapper就不存在了。

[root@node1 ~]# fdisk -l

Disk /dev/sda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a996c

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          26      204800   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              26        1332    10485760   83  Linux
/dev/sda3            1332        1462     1048576   82  Linux swap / Solaris

Disk /dev/sdb: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00001618

   Device Boot      Start         End      Blocks   Id  System
[root@node1 ~]#

Defaults

        在multipath.conf配置文件中的defaults{}进行设置属性被视为多路径的默认属性。如果想查询defaults可以配置哪些属性可以通过/usr/share/doc/device-mapper-multipath-*/multipath.conf.defaults找到。

常用的设置如下:

path_selector(路径选择器):此算法决定在group中哪一个path作为下一个I/O。默认情况下使用“round-robin 0” (循环模式),将I/O分布在group中的每个paths之中。rr_min_io_rq设置为一定量的I/O,默认为1000,在I/O达到1000之后会切换到下一个path。queue-length 0为判断当前请求数最少的path作为下一个请求的目标。service-time 0为通过算法计算服务响应请求时间最短的路径并发送。

path_grouping_policy(路径组策略): 指定默认分组策略适用于没有指定的多路径,这些值可能包括:

        failover(故障转移):每个路径在独立的组中。

        multibus(多总线):所有路径在一个组中,这种方式可以支持active-active的连接模式。

path_checker(路径检查):用来指定判断路径是否存活的方式。一般设置为directio或者readsector模式,如果有特殊设备需要使用指定的checker,则在devices{}中添加覆盖defaults{}中的设置。

        directio(直连): 通过路径对目标设备的第一块扇区进行I/O测试

        readsector(读取扇区):通过路径读取目标的第一块扇区

user_friendly_names(用户友好名称):这项设置决定多路径节点是否使用user_friendly_names,如果为yes,则就会设置别名为mpathN,否则会使用设备的WWID作为SYMLINKS。

Important:mpathconf 将改变multipath.conf中的defaults设置,但是不会对multipathd进程做出改变。

The devices{} Section

        在devices{}列表中可以对一些特殊设备做设置从而覆盖掉defaults{}中相同的属性。在devices{}列表中可以使用devices{}作为子列表对特殊设备进行定义。常见的设备的一些属性在多路径进程中已经内置存在,如果没有也可以添加一个device{}进行配置。范例:

devices {
        device {
                vendor "MegaHyperSuperStorage"
                product "BAS"
                revision "513/B"
                features "1 queue_if_no_path"
                path_grouping_policy multibus
                path_checker tur
        }
}

        其中features行表示如果没有path可用,仍然对I/O请求进行接收和加入队列。

The multipaths{} section

        在multipaths{}可以定义特殊的多路径设备。可以使用这个对特殊的多路径定制path_grouping策略。通常使用Multipaths定义多路径的alias,为了区分不同的多路径设备。下列范例匹配WWI为1234567890abcdef的设备alias为clusterstorage,path_selector为queue-length 0 (路径选择器,发送下一个I/O请求到队列最少的路径中)。

multipaths {
        multipath {
                wwid "1234567890abcdef"
                alias "clusterstorage"
                path_selector "queue-length 0"
        }
}

Configure Multipathing

Lab Overview: 此实验你将配置多路径,在node1上配置两个路径对target

Success Criteria:完成目标,成功在node1上激活iSCSI目标,目标有两条iscsi路径

Before you begin…

Make sure that the iSCSI target define on node4 is still running

1. 重新安装node1虚拟机,使用lab-build-cluster -1命令(课本实验环境使用KVM实现,按照要求部署实验环境科免去此步骤)

2. 在node4上添加acl,使得node1和node3的网卡都可以成功登陆,设置之后需要重启tgtd服务,如果还有initiator登陆,则使用force-restart参数强制重启。

ACL information:
    ALL

        这里设置ACL为ALL,所有用户都可登陆。

3. 设置node1的iSCSI initiator name为 iqn.2012-04.com.example:node1

vim /etc/iscsi/initiatorname.iscsi   添加 InitiatorName=iqn.2012-04.com.example:node1

重启iscsi服务后在node4上使用tgt-admin -s查看  (可能有缓存造成无法更新状态,重启node1之后生效)

System information:
    Driver: iscsi
    State: ready
I_T nexus information:
    I_T nexus: 7
        Initiator: iqn.2012-04.com.example:node1
        Connection: 0
            IP Address: 172.16.1.1
    I_T nexus: 8
        Initiator: iqn.2012-04.com.example:node1
        Connection: 0
            IP Address: 172.16.2.1

4.  在node1上设置default iscsi tmeouts为2秒。

[root@node1 ~]# vim /etc/iscsi/iscsid.conf

node.session.timeo.replacement_timeout = 2

# To specify the time to wait for login to complete, edit the line.
# The value is in seconds and the default is 15 seconds.
node.conn[0].timeo.login_timeout = 2

# To specify the time to wait for logout to complete, edit the line.
# The value is in seconds and the default is 15 seconds.
node.conn[0].timeo.logout_timeout = 2

# Time interval to wait for on connection before sending a ping.
node.conn[0].timeo.noop_out_interval = 2

# To specify the time to wait for a Nop-out response before failing
# the connection, edit this line. Failing the connection will
# cause IO to be failed back to the SCSI layer. If using dm-multipath
# this will cause the IO to be failed to the multipath layer.
node.conn[0].timeo.noop_out_timeout = 2

# To specify the time to wait for abort response before
# failing the operation and trying a logical unit reset edit the line.
# The value is in seconds and the default is 15 seconds.
node.session.err_timeo.abort_timeout = 2

# To specify the time to wait for a logical unit response
# before failing the operation and trying session re-establishment
# edit the line.
# The value is in seconds and the default is 30 seconds.
node.session.err_timeo.lu_reset_timeout = 2

# To specify the time to wait for a target response
# before failing the operation and trying session re-establishment
# edit the line.
# The value is in seconds and the default is 30 seconds.
node.session.err_timeo.tgt_reset_timeout = 2

如果已经Login,则需要在节点信息中修改

cd /var/lib/iscsi/nodes/iqn.2013-11.com.example.node4\:example/

5. 在node1上通过两块网卡discovery并log in 到node4的target中。

Disk /dev/sdd: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sde: 1073 MB, 1073741824 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

6. 确保node1上已经安装device-mapper-multipath 软件包。

[root@node1 172.16.1.4,3260,1]# rpm -qa|grep device-mapper-multipath
device-mapper-multipath-libs-0.4.9-56.el6.x86_64
device-mapper-multipath-0.4.9-56.el6.x86_64
[root@node1 172.16.1.4,3260,1]#

7. 创建Multipathd默认配置文档,不使用user friendly names 并不启动进程。

mpathconf --enable --with_multipathd n --user_friendly_names n

[root@node1 172.16.1.4,3260,1]# service multipathd status
multipathd is stopped

## Use user friendly names, instead of using WWIDs as names.
defaults {
        user_friendly_names no
}
##

8. 添加以下行到defaults{}之中,确保多路径使用WWID,不会造成混乱。

## Use user friendly names, instead of using WWIDs as names.
defaults {
        user_friendly_names no
        getuid_callout "lib/udev/scsi_id --replace-whitespace --whitelisted --device=/dev/%n"
}
##

9. 启动multipathd进程

[root@node1 ~]# service multipathd start
Starting multipathd daemon: [  OK  ]

发现多了一块设备

Disk /dev/mapper/1IET_00010001: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Testing Multipathing

Lab Overview:本次实验需要创建一个分区在多路径设备上,创建一个文件系统并测试多路径设置。

Success Criteria:成功指标为成功挂载到文件系统上,并保证每一条path可以访问。

Lab Outline : In this exercise you will create a partition and a file system on multipathed storage . mount it , and test multipathing.

Before you begin…

        确保node4上面的target访问正常,并且在之前的实验中配置好了多路径在node1上。

1. 创建128Mib 的分区在多路径设备上

[root@node1 ~]# fdisk -cu /dev/mapper/1IET_00010001
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xa4a7cfad.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First sector (2048-2097151, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-2097151, default 2097151): +128M

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

[root@node1 ~]# fdisk -l /dev/mapper/1IET_00010001

Disk /dev/mapper/1IET_00010001: 1073 MB, 1073741824 bytes
114 heads, 33 sectors/track, 557 cylinders
Units = cylinders of 3762 * 512 = 1926144 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xa4a7cfad

                     Device Boot      Start         End      Blocks   Id  System
/dev/mapper/1IET_00010001p1               1          71      131072   83  Linux

2. 确保系统已经识别到新的分区

[root@node1 ~]# partprobe; multipath �Cr

[root@node1 ~]# ls -rlt /dev/mapper/1IET_00010001p1
lrwxrwxrwx. 1 root root 7 Nov  6 23:05 /dev/mapper/1IET_00010001p1 -> ../dm-0

3. 创建ext4文件系统并将分区挂载到/mnt下。

[root@node1 ~]# mkfs.ext4 /dev/mapper/1IET_00010001p1

[root@node1 ~]# mount /dev/mapper/1IET_00010001p1 /mnt

4. 通过multipath �Cll 命令查看多路径状态

[root@node1 ~]# multipath -ll
1IET_00010001 dm-1 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 8:0:0:1 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 7:0:0:1 sde 8:64 active ready running

5. 关闭正在active的网卡,再查看状态。

[root@node1 mnt]# multipath -ll
1IET_00010001 dm-1 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 8:0:0:1 sdd 8:48 active ready  running
`-+- policy='round-robin 0' prio=0 status=enabled
  `- 7:0:0:1 sde 8:64 failed faulty running  

[root@node1 mnt]# touch test
[root@node1 mnt]# ls -rlt
total 13
drwx------. 2 root root 12288 Nov  6 23:07 lost+found
-rw-r--r--. 1 root root     0 Nov  6 23:19 test
[root@node1 mnt]#

6. 重新启动网卡,看是否恢复正常。

[root@node1 mnt]# multipath -ll
1IET_00010001 dm-1 IET,VIRTUAL-DISK
size=1.0G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 8:0:0:1 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
  `- 7:0:0:1 sde 8:64 active ready running

Configuring Multipathing

Lab Overview: 配置多路径设备,自定义设备别名。

Success Criteria:成功条件为为多路径设备创建自定义名称。

添加配置到多路径配置文件

multipaths {
        multipath {
        wwid "1IET     00010001"
        alias "clusterstorage"
        }
}

重启服务后可以看到

Disk /dev/mapper/clusterstorage: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xa4a7cfad

                     Device Boot      Start         End      Blocks   Id  System

你可能感兴趣的:(unit,4,MULTIPATHING,RH436)