一台机器装openfiler充当存储,另外一台当作客户端,主机名为server.xzxj.edu.cn.
openfile的安装这里不做具体介绍,完成多路径在openfiler端必须存在两块网卡,这里如下:
eth0:192.168.1.5
eth1:192.168.1.6
openfiler中的iscsi target配置如图所示:
server.xzxj.edu.cn端的配置如下:
启动iscsi进程:service iscsi start;chkconfig iscsi on
首先发现openfiler端的iscsi目标,其次是登录到iscsi目标上,如图所示:
因为要测试linux自带的多路径软件,所以这里要登录两次。两次的IP是不同的,但iqn名称是一致的。这里是我写的一个脚本,所以显示是这样子的。
现在已经登录上去了,可以使用fdisk -l命令查看是否已经识别到磁盘了:
[root@server ~]# fdisk -l
Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 2349 18763920 8e Linux LVM
/dev/sda3 2350 2610 2096482+ 82 Linux swap / Solaris
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 2349 18763920 8e Linux LVM
/dev/sda3 2350 2610 2096482+ 82 Linux swap / Solaris
Disk
/dev/sdj: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk /dev/sdj doesn't contain a valid partition table
Disk
/dev/sdk: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk /dev/sdk doesn't contain a valid partition table
Disk
/dev/sdl: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk /dev/sdl doesn't contain a valid partition table
Disk
/dev/sdm: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk /dev/sdm doesn't contain a valid partition table
Disk
/dev/sdn: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk /dev/sdn doesn't contain a valid partition table
Disk
/dev/sdo: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk /dev/sdo doesn't contain a valid partition table
Disk
/dev/sdp: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk /dev/sdp doesn't contain a valid partition table
Disk
/dev/sdq: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk /dev/sdq doesn't contain a valid partition table
看到了吧,这里多出了8块盘。在openfiler中只map了四次,为什么这里是8块而不是4块呢?因为在上面提过,使用两个IP登录两次iscsi target,所以这里有两块是重复的。
下面是多路径软件device-mapper-multipath的配置文件具体配置,如下所示:
完了后,启动multipathd进程:
service multipathd start
chkconfig multipathd on
启动之后,使用multipath -ll可以查看是否成功:
看到了吧,重复的磁盘现在绑定到一起了。
给这几块盘分区,创建文件系统,使用fdisk /dev/dm-1或者fdisk /dev/mapper/openfiler_vol01即可。如下所示:
[root@server45 ~]# fdisk /dev/dm-3
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
The number of cylinders for this disk is set to 1305.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): p
Disk /dev/dm-3: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1305, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1305, default 1305): +5120M
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1305, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1305, default 1305): +5120M
Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)
Selected partition 1
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (624-1305, default 624):
Using default value 624
Last cylinder or +size or +sizeM or +sizeK (624-1305, default 1305):
Using default value 1305
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (624-1305, default 624):
Using default value 624
Last cylinder or +size or +sizeM or +sizeK (624-1305, default 1305):
Using default value 1305
Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): fd
Changed system type of partition 2 to fd (Linux raid autodetect)
Partition number (1-4): 2
Hex code (type L to list codes): fd
Changed system type of partition 2 to fd (Linux raid autodetect)
Command (m for help): w
The partition table has been altered!
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
分区完结果如下:
[root@server45 ~]# fdisk -l
Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 2349 18763920 8e Linux LVM
/dev/sda3 2350 2610 2096482+ 82 Linux swap / Solaris
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 2349 18763920 8e Linux LVM
/dev/sda3 2350 2610 2096482+ 82 Linux swap / Solaris
Disk /dev/sdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 623 5004216 fd Linux raid autodetect
/dev/sdb2 624 1305 5478165 fd Linux raid autodetect
/dev/sdb1 1 623 5004216 fd Linux raid autodetect
/dev/sdb2 624 1305 5478165 fd Linux raid autodetect
Disk /dev/sdc: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 1 623 5004216 fd Linux raid autodetect
/dev/sdc2 624 1305 5478165 fd Linux raid autodetect
/dev/sdc1 1 623 5004216 fd Linux raid autodetect
/dev/sdc2 624 1305 5478165 fd Linux raid autodetect
Disk /dev/sdd: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdd1 1 623 5004216 fd Linux raid autodetect
/dev/sdd2 624 1305 5478165 fd Linux raid autodetect
/dev/sdd1 1 623 5004216 fd Linux raid autodetect
/dev/sdd2 624 1305 5478165 fd Linux raid autodetect
Disk /dev/sde: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sde1 1 623 5004216 fd Linux raid autodetect
/dev/sde2 624 1305 5478165 fd Linux raid autodetect
/dev/sde1 1 623 5004216 fd Linux raid autodetect
/dev/sde2 624 1305 5478165 fd Linux raid autodetect
Disk /dev/sdf: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdf1 1 623 5004216 fd Linux raid autodetect
/dev/sdf2 624 1305 5478165 fd Linux raid autodetect
/dev/sdf1 1 623 5004216 fd Linux raid autodetect
/dev/sdf2 624 1305 5478165 fd Linux raid autodetect
Disk /dev/sdg: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdg1 1 623 5004216 fd Linux raid autodetect
/dev/sdg2 624 1305 5478165 fd Linux raid autodetect
/dev/sdg1 1 623 5004216 fd Linux raid autodetect
/dev/sdg2 624 1305 5478165 fd Linux raid autodetect
Disk /dev/sdh: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdh1 1 623 5004216 fd Linux raid autodetect
/dev/sdh2 624 1305 5478165 fd Linux raid autodetect
/dev/sdh1 1 623 5004216 fd Linux raid autodetect
/dev/sdh2 624 1305 5478165 fd Linux raid autodetect
Disk /dev/sdi: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdi1 1 623 5004216 fd Linux raid autodetect
/dev/sdi2 624 1305 5478165 fd Linux raid autodetect
/dev/sdi1 1 623 5004216 fd Linux raid autodetect
/dev/sdi2 624 1305 5478165 fd Linux raid autodetect
Disk /dev/dm-1: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/dm-1p1 1 623 5004216 fd Linux raid autodetect
/dev/dm-1p2 624 1305 5478165 fd Linux raid autodetect
/dev/dm-1p1 1 623 5004216 fd Linux raid autodetect
/dev/dm-1p2 624 1305 5478165 fd Linux raid autodetect
Disk /dev/dm-2: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/dm-2p1 1 623 5004216 fd Linux raid autodetect
/dev/dm-2p2 624 1305 5478165 fd Linux raid autodetect
/dev/dm-2p1 1 623 5004216 fd Linux raid autodetect
/dev/dm-2p2 624 1305 5478165 fd Linux raid autodetect
Disk /dev/dm-3: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/dm-3p1 1 623 5004216 fd Linux raid autodetect
/dev/dm-3p2 624 1305 5478165 fd Linux raid autodetect
/dev/dm-3p1 1 623 5004216 fd Linux raid autodetect
/dev/dm-3p2 624 1305 5478165 fd Linux raid autodetect
Disk /dev/dm-4: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/dm-4p1 1 623 5004216 fd Linux raid autodetect
/dev/dm-4p2 624 1305 5478165 fd Linux raid autodetect
/dev/dm-4p1 1 623 5004216 fd Linux raid autodetect
/dev/dm-4p2 624 1305 5478165 fd Linux raid autodetect
这里的/dev/dm-[1,2,3,4]p[1,2]就是分区1和2。
查看下/dev/mapper目录下有何不同:
[root@server45 ~]# ll /dev/mapper/
total 0
crw------- 1 root root 10, 63 Apr 17 2010 control
brw-rw---- 1 root disk 253, 1 Apr 17 10:05 openfiler_vol01
brw-rw---- 1 root disk 253, 2 Apr 17 09:58 openfiler_vol02
brw-rw---- 1 root disk 253, 3 Apr 17 09:58 openfiler_vol03
brw-rw---- 1 root disk 253, 4 Apr 17 09:58 openfiler_vol04
brw-rw---- 1 root disk 253, 0 Apr 17 09:43 vg-lv01
[root@server45 ~]# fdisk -l /dev/mapper/openfiler_vol01
Disk /dev/mapper/openfiler_vol01: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/mapper/openfiler_vol01p1 1 623 5004216 fd Linux raid autodetect
/dev/mapper/openfiler_vol01p2 624 1305 5478165 fd Linux raid autodetect
/dev/mapper/openfiler_vol01p1 1 623 5004216 fd Linux raid autodetect
/dev/mapper/openfiler_vol01p2 624 1305 5478165 fd Linux raid autodetect
[root@server45 ~]# ll /dev/dm-*
brw-rw---- 1 root root 253, 1 Apr 17 09:58 /dev/dm-1
brw-rw---- 1 root root 253, 2 Apr 17 10:06 /dev/dm-2
brw-rw---- 1 root root 253, 3 Apr 17 10:07 /dev/dm-3
brw-rw---- 1 root root 253, 4 Apr 17 10:07 /dev/dm-4
brw-rw---- 1 root root 253, 1 Apr 17 09:58 /dev/dm-1
brw-rw---- 1 root root 253, 2 Apr 17 10:06 /dev/dm-2
brw-rw---- 1 root root 253, 3 Apr 17 10:07 /dev/dm-3
brw-rw---- 1 root root 253, 4 Apr 17 10:07 /dev/dm-4
如果要了解iscsi多路径的访问机制,可以参考我的另外一篇文章: 深入了解iSCSI的2种多路径访问机制