[ Centos 7 iscsi搭建 及 1台客户端同时挂载多台iscsi服务端问题 ]

一、什么是iscsi

    iSCSI技术是一种由IBM公司研究开发的，是一个供硬件设备使用的可以在IP协议的上层运行的SCSI指令集，这种指令集合可以实现在IP网络上运行SCSI协议，使其能够在诸如高速千兆以太网上进行路由选择。iSCSI技术是一种新储存技术，该技术是将现有SCSI接口与以太网络(Ethernet)技术结合，使服务器可与使用IP网络的储存装置互相交换资料。

 

二、需求分析

    最近在做虚拟化，需要通过一个主机挂载来自两个iscsi服务器的targets，如图：

 

三、实现思路
    
    问题：刚开始觉得这个应该问题不大，挂载第一台iscsi服务器的时候没有碰到问题，但是在挂载第二台iscsi时，发现怎样都无法挂载，挂载报错。
    最终找到了正确的解决方法，在centos7/RHEL7上挂载时，两台iscsi服务器端的target命名需要一致，下面进行搭建演示：

   

    系统：centos 7.2
        iscsi客户端：192.168.1.156
        iscsi服务端：192.168.1.157
        iscsi服务端：192.168.1.158

    首先要确保3台服务器的selinux和firewalld关闭

[root@iscsi-client ~]# setenforce 0 ; systemctl stop firewalld ; systemctl disable firewalld

    iscsi服务端配置如下：

[root@iscsi-server1 ~]# lsblk 
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0               2:0    1    4K  0 disk 
sda               8:0    0   20G  0 disk 
├─sda1            8:1    0  500M  0 part /boot
└─sda2            8:2    0 19.5G  0 part 
  ├─centos-root 253:0    0 17.5G  0 lvm  /
  └─centos-swap 253:1    0    2G  0 lvm  [SWAP]
sdb               8:16   0   10G  0 disk 
sr0              11:0    1    4G  0 rom

使用sdb作为target。为sdb进行分区并使用lvm进行发布出去，使用lvm的好处是后期便于扩展，在rhel7和centos7中，因为采用的是xfs文件系统，
lvm只能扩展，不能缩小，这点要牢记。

[root@iscsi-server1 ~]# fdisk /dev/sdb 
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x966a38a7.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): 
Using default response p
Partition number (1-4, default 1): 
First sector (2048-20971519, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519): 
Using default value 20971519
Partition 1 of type Linux and of size 10 GiB is set

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@iscsi-server1 ~]# lsblk 
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0               2:0    1    4K  0 disk 
sda               8:0    0   20G  0 disk 
├─sda1            8:1    0  500M  0 part /boot
└─sda2            8:2    0 19.5G  0 part 
  ├─centos-root 253:0    0 17.5G  0 lvm  /
  └─centos-swap 253:1    0    2G  0 lvm  [SWAP]
sdb               8:16   0   10G  0 disk 
└─sdb1            8:17   0   10G  0 part 
sr0              11:0    1    4G  0 rom

以上就为sdb做了分区，接下来将sdb1创建为独立的lvm

# 创建卷组
[root@iscsi-server1 ~]# vgcreate vg_iscsi_1_156 /dev/sdb1 
  Physical volume "/dev/sdb1" successfully created
  Volume group "vg_iscsi_1_156" successfully created

# 创建逻辑分区

[root@iscsi-server1 ~]# lvcreate -l +2559 -n lv_iscsi_1_156 vg_iscsi_1_156
  Logical volume "lv_iscsi_1_156" created.
[root@iscsi-server1 ~]# lvs
  LV             VG             Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root           centos         -wi-ao---- 17.47g                                                    
  swap           centos         -wi-ao----  2.00g                                                    
  lv_iscsi_1_156 vg_iscsi_1_156 -wi-a----- 10.00g 

lvm已经创建完毕，接下来创建target

# 安装target工具和服务
[root@iscsi-server1 ~]# yum install scsi-target-utils -y


# 开始创建target
[root@iscsi-server1 ~]# targetcli 
targetcli shell version 2.1.fb41
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 0]
  o- loopback ......................................................................................................... [Targets: 0]

创建块设备，这里要注意，因为我们使用的lvm，因此不能将sdb添加到块设备，而是将逻辑卷添加进去

/> backstores/block create node1.disk /dev/sdb1 
Cannot configure StorageObject because device /dev/sdb1 is already in use
/> backstores/block create node1.disk /dev/vg_iscsi_1_156/lv_iscsi_1_156 
Created block storage object node1.disk using /dev/vg_iscsi_1_156/lv_iscsi_1_156.

在创建wwn的时候，又踩了一个坑。
    wwn命名必须 iqn.2017-01.com.xxx:server   注意这个月份 2017-01 这里必须要01 如果写成1就会报错。

# 错误写法
/> iscsi/ create iqn.2017-1.com.node1:server
WWN not valid as: iqn, naa, eui

# 正确写法
/> iscsi/ create iqn.2017-01.com.node1:server
Created target iqn.2017-01.com.node1:server.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.

# 创建acl，在创建acl这里一定要注意，如果准备在同一客户端挂载两个iscsi这里的acl命名一定要一致
/> iscsi/iqn.2017-01.com.node1:server/tpg1/acls create iqn.2017-01.com.node1:client
Created Node ACL for iqn.2017-01.com.node1:client/> iscsi/iqn.2017-01.com.node1:server/tpg1/luns create /backstores/block/node1.disk 
Created LUN 0.
Created LUN 0->0 mapping in node ACL iqn.2017-01.com.node1:client

# 删除自动生成的监听
/> iscsi/iqn.2017-01.com.node1:server/tpg1/portals/ delete 0.0.0.0 3260
Deleted network portal 0.0.0.0:3260

# 创建绑定固定ip地址的监听
/> iscsi/iqn.2017-01.com.node1:server/tpg1/portals/ create 192.168.1.157 3260
Using default IP port 3260
Created network portal 192.168.1.157:3260.
/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 1]
  | | o- node1.disk ............................................ [/dev/vg_iscsi_1_156/lv_iscsi_1_156 (10.0GiB) write-thru activated]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2017-01.com.node1:server ...................................................................................... [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 1]
  |     | o- iqn.2017-01.com.node1:client ......................................................................... [Mapped LUNs: 1]
  |     |   o- mapped_lun0 ............................................................................ [lun0 block/node1.disk (rw)]
  |     o- luns .......................................................................................................... [LUNs: 1]
  |     | o- lun0 .......................................................... [block/node1.disk (/dev/vg_iscsi_1_156/lv_iscsi_1_156)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 192.168.1.157:3260 ............................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]
/> saveconfig 
Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json
/> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json

最后一定要执行 saveconfig 再退出

 

另一台iscsi服务端配置一样，targetcli如下：

/> ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 1]
  | | o- node1:disk ............................................ [/dev/vg_iscsi_1_156/lv_iscsi_1_156 (10.0GiB) write-thru activated]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2017-01.com.node1:server ...................................................................................... [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 1]
  |     | o- iqn.2017-01.com.node1:client ......................................................................... [Mapped LUNs: 1]
  |     |   o- mapped_lun0 ............................................................................ [lun0 block/node1:disk (rw)]
  |     o- luns .......................................................................................................... [LUNs: 1]
  |     | o- lun0 .......................................................... [block/node1:disk (/dev/vg_iscsi_1_156/lv_iscsi_1_156)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 192.168.1.158:3260 ............................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]

其中acl命名一定要和server1保持一致


这里要注意，target服务建议要配置开机启动。在搭建的时候，有出现过没有开机启动配置丢失的情况。

[root@iscsi-server2 ~]# systemctl start target ; systemctl enable target
Created symlink from /etc/systemd/system/multi-user.target.wants/target.service to /usr/lib/systemd/system/target.service.

 

iscsi客户端配置：

[root@iscsi-client yum.repos.d]# yum install iscsi* -y

1. 修改/etc/iscsi/initiatorname.iscsi文件

[root@iscsi-client ~]# vim /etc/iscsi/initiatorname.iscsi

# 这里InitiatorName 后面就是iscsi服务端通过targetcli创建的iscsi acl命名，因为要挂载两个iscsi所以名称必须一致
InitiatorName=iqn.2017-01.com.node1:client

2. 启动服务

# 这里客户端依然建议开机启动
[root@iscsi-client ~]# systemctl start iscsid ; systemctl enable iscsid

3. 挂载iscsi

# 挂载192.168.1.157，挂载之前必须要发现该iscsi
[root@iscsi-client ~]# iscsiadm -m discovery -t st -p 192.168.1.157
192.168.1.157:3260,1 iqn.2017-01.com.node1:server
[root@iscsi-client ~]# iscsiadm -m node -T iqn.2017-01.com.node1:server -p 192.168.1.157 -l
Logging in to [iface: default, target: iqn.2017-01.com.node1:server, portal: 192.168.1.157,3260] (multiple)
Login to [iface: default, target: iqn.2017-01.com.node1:server, portal: 192.168.1.157,3260] successful.


# 挂载192.168.1.158，挂载之前必须要发现该iscsi
[root@iscsi-client ~]# iscsiadm -m discovery -t st -p 192.168.1.158
192.168.1.158:3260,1 iqn.2017-01.com.node1:server
[root@iscsi-client ~]# iscsiadm -m node -T iqn.2017-01.com.node1:server -p 192.168.1.158 -l
Logging in to [iface: default, target: iqn.2017-01.com.node1:server, portal: 192.168.1.158,3260] (multiple)
Login to [iface: default, target: iqn.2017-01.com.node1:server, portal: 192.168.1.158,3260] successful.

查看是否生效：

[root@iscsi-client ~]# lsblk 
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0               2:0    1    4K  0 disk 
sda               8:0    0   20G  0 disk 
├─sda1            8:1    0  500M  0 part /boot
└─sda2            8:2    0 19.5G  0 part 
  ├─centos-root 253:0    0 17.5G  0 lvm  /
  └─centos-swap 253:1    0    2G  0 lvm  [SWAP]
sdb               8:16   0   10G  0 disk 
sdc               8:32   0   10G  0 disk 
sr0              11:0    1    4G  0 rom  /mnt/iso

sdb 和 sdc都已经生成，说明挂载成功

在配置完成iscsi客户端的时候，建议进行重启服务器进行验证

# 重启成功，查看没问题。
[root@iscsi-client ~]# lsblk 
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0               2:0    1    4K  0 disk 
sda               8:0    0   20G  0 disk 
├─sda1            8:1    0  500M  0 part /boot
└─sda2            8:2    0 19.5G  0 part 
  ├─centos-root 253:0    0 17.5G  0 lvm  /
  └─centos-swap 253:1    0    2G  0 lvm  [SWAP]
sdb               8:16   0   10G  0 disk 
sdc               8:32   0   10G  0 disk 
sr0              11:0    1    4G  0 rom 

 

 

四、对iscsi进行扩容的操作

对192.168.1.157这台主机添加一块硬盘，并扩容客户端iscsi的容量

192.168.1.157 iscsi服务端操作如下：

# 已经为157添加了一块新的硬盘 sdc，首先将sdc加入到lvm中
[root@iscsi-server1 ~]# lsblk 
NAME                              MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0                                 2:0    1    4K  0 disk 
sda                                 8:0    0   20G  0 disk 
├─sda1                              8:1    0  500M  0 part /boot
└─sda2                              8:2    0 19.5G  0 part 
  ├─centos-root                   253:0    0 17.5G  0 lvm  /
  └─centos-swap                   253:1    0    2G  0 lvm  [SWAP]
sdb                                 8:16   0   10G  0 disk 
└─sdb1                              8:17   0   10G  0 part 
  └─vg_iscsi_1_156-lv_iscsi_1_156 253:2    0   10G  0 lvm  
sdc                                 8:32   0   10G  0 disk 
sr0                                11:0    1    4G  0 rom 


# 对sdc进行分区
[root@iscsi-server1 ~]# fdisk /dev/sdc 
Welcome to fdisk (util-linux 2.23.2).

Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x68a23069.

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): 
Using default response p
Partition number (1-4, default 1): 
First sector (2048-20971519, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519): 
Using default value 20971519
Partition 1 of type Linux and of size 10 GiB is set

Command (m for help): 
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@iscsi-server1 ~]# 
[root@iscsi-server1 ~]# lsblk 
NAME                              MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0                                 2:0    1    4K  0 disk 
sda                                 8:0    0   20G  0 disk 
├─sda1                              8:1    0  500M  0 part /boot
└─sda2                              8:2    0 19.5G  0 part 
  ├─centos-root                   253:0    0 17.5G  0 lvm  /
  └─centos-swap                   253:1    0    2G  0 lvm  [SWAP]
sdb                                 8:16   0   10G  0 disk 
└─sdb1                              8:17   0   10G  0 part 
  └─vg_iscsi_1_156-lv_iscsi_1_156 253:2    0   10G  0 lvm  
sdc                                 8:32   0   10G  0 disk 
└─sdc1                              8:33   0   10G  0 part 
sr0                                11:0    1    4G  0 rom

# 将sdc1添加到之前lvm卷组vg_iscsi_1_156里
[root@iscsi-server1 ~]# vgextend vg_iscsi_1_156 /dev/sdc1 
  Physical volume "/dev/sdc1" successfully created
  Volume group "vg_iscsi_1_156" successfully extended

# 扩展逻辑卷
[root@iscsi-server1 ~]# lvresize -l +2559 -n /dev/vg_iscsi_1_156/lv_iscsi_1_156 
  Size of logical volume vg_iscsi_1_156/lv_iscsi_1_156 changed from 10.00 GiB (2559 extents) to 19.99 GiB (5118 extents).
  Logical volume lv_iscsi_1_156 successfully resized.

# 重读下逻辑卷大小
[root@iscsi-server1 ~]# resize2fs /dev/vg_iscsi_1_156/lv_iscsi_1_156 
resize2fs 1.42.9 (28-Dec-2013)
resize2fs: Device or resource busy while trying to open /dev/vg_iscsi_1_156/lv_iscsi_1_156
Couldn't find valid filesystem superblock.

到此，iscsi服务端扩容操作完毕。

192.168.1.156 iscsi客户端操作如下：

重新加载sdb和sdc容量，可以看到，sdc已经变成了20G

[root@iscsi-client ~]# lsblk 
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0               2:0    1    4K  0 disk 
sda               8:0    0   20G  0 disk 
├─sda1            8:1    0  500M  0 part /boot
└─sda2            8:2    0 19.5G  0 part 
  ├─centos-root 253:0    0 17.5G  0 lvm  /
  └─centos-swap 253:1    0    2G  0 lvm  [SWAP]
sdb               8:16   0   10G  0 disk 
sdc               8:32   0   20G  0 disk 
sr0              11:0    1    4G  0 rom

这里验证一个问题，因为挂载了两台iscsi，尝试多次重启，看看sdb和sdc会不会出现互换设备名的情况

经过3次重启的验证，并没有出现过互换设备名的情况，但是在挂载的时候，这里还是建议使用UUID
我在虚拟机上分区后无法查看到UUID，物理设备做iscsi服务端是可以查看的。
[root@db1 ~]# blkid /dev/vda1
/dev/vda1: UUID="e1b26164-3ec5-435a-bf0c-b9f4c0989941" TYPE="xfs"

至此，已经完成了一台服务器上挂载两个iscsi服务端的问题。

 

五、故障问题

在 192.168.1.156上查看到 sdb 所对应的是server 192.168.1.158

测试1：当sdb分区并挂载至某个目录

当158 down掉，客户端会有怎样的提示，如下验证：

dmesg 报错信息如下：

ls 挂载目录错误如下：

很明显的I/O错误。

解决方案：再次启动server 192.168.1.158, target开机自动启动，客户端需要重新挂载到目录才会恢复正常。

 

测试2：当sdb分区并挂载到某个目录做为web程序存放点，故障时，看web会返回什么状态码

通过apache测试，当iscsi服务器down掉，web服务器并不会马上故障。大概5分钟后报错

查看错误日志：

 

所以，当iscsi客户端出现IO错误时，先查看磁盘空间是否不足，再次检查下iscsi服务端网络通信是否正常。

转载于:https://www.cnblogs.com/hukey/p/6245614.html