[rhel6.5]ISCSI配置过程详解

• 关于数据存储的几种方式
  
  • 1.DAS:表示直接访问存储,例如本机上面的磁盘
  • 2.SAN:存储区域网,来自局域网里面其他存储设备提供的磁盘;
  • 3.NAS:Network Attached Storge:表示来自NAS提供的文件系统,只能够独立使用,不可以进行格式化;
• 关于NAS以及SAN
  
  • NAS:Network Attached Storage 表示网络附加存储,提供的是文件系统上面的服务,可以提供锁控制等机制,NFS ,SAMBA, FTP
  • SAN:Storage Area Networks,表示的含义是存储区域网,通过提供块设备,来提供服务,通常包括SCSI等;
  • ISCSI:表示的含义是Internet SCSI 主要通过TCP/IP技术将存储设备通过iSCSI target,做成提供磁盘设备的服务端,在通过iscsi Initiator做成iscsi target` 客户端;
• ISCSI主要包含两个部分:
  
  • iSCSI target:存储设备的服务器端,存放磁盘或者RAID设备,也可以用于将Linux主机仿真成为Linux设备,用于提供其他主机使用的磁盘;
  • iSCSI initiator:能够使用target设备的Client 通常是服务器,Client必须使用这个功能才能够连接到iSCSI提供的磁盘;
• iSCSI Target的设置:
  
  • 所需要的软件:
    
    • scsi-target-utils:用来将Linux系统仿真成为iSCSI target的功能;
    • iscsi-target-utils:挂在来自target的磁盘到Linux本机上面;
  • scsi-target-utils提供的配置文件:
    
    • /etc/sysconfig/tgtd: 主配置文件用于设置共享的磁盘文件以及磁盘文件的格式;
    • /usr/sbin/tgt-admin: 在线查询,删除target等功能的设置工具;
    • /usr/sbin/tgt-setup-lun:建立target以及设置共享磁盘以及可以使用的客户端等工具软件;
    • /usr/sbin/tgtadm: 手动直接管理的管理员工具;
    • /usr/sbin/tgtd: 提供scsi-target服务的主程序;
    • /usr/sbin/tgtimg: 搭建预计共享的影响文件设备的工具;
• scsi-target支持的共享设备类型:
  
  • 1.使用dd命令建立的大型文件,不需要事先进行格式化
  • 2.使用单一的分区;
  • 3.使用但以完整的磁盘,无需事先分区;
  • 4.使用磁盘阵列进行共享;
  • 5.使用软件磁盘阵列(software RAID)共享成单一磁盘;
  • 6.使用LVM的LV设备;

• 接下来进行规划

  1.server23.com作为scsi-target,
  2.server22.com作为iscsi-initiator,用来事先SCSI设备的访问;

  1. 这里的共享计划使用dd命令生成的大型文件,普通的分区[/dev/vdb1]以及使用LVM[//dev/vdb2]卷组的方式;

• 首先来准备共享的设备文件

• 1.首先建立500MB的一个大型文件

[root@server23 ~]# mkdir /srv/iscsi
[root@server23 ~]# dd if=/dev/zero of=/srv/iscsi/disk1.img bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 8.46649 s, 61.9 MB/s
[root@server23 ~]# chcon -Rv -t tgtd_var_lib_t /srv/iscsi/  //用于修改文件的安全上下文,比关闭selinux更加安全,selinux以后会补充说明;
changing security context of `/srv/iscsi/disk1.img'
changing security context of `/srv/iscsi/'
[root@server23 ~]# ls -lh /srv/iscsi/disk1.img 
-rw-r--r--. 1 root root 500M 3月  11 10:47 /srv/iscsi/disk1.img

• 建立分区,[关于kvm添加磁盘的使用方法参照以前的文章]

[root@server23 ~]# fdisk  /dev/vdb 
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xff1fad05.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): p

Disk /dev/vdb: 3221 MB, 3221225472 bytes
16 heads, 63 sectors/track, 6241 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xff1fad05

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-6241, default 1): 
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-6241, default 6241): +1G

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): 83
Changed system type of partition 1 to 8e (Linux)

Command (m for help): p

Disk /dev/vdb: 3221 MB, 3221225472 bytes
16 heads, 63 sectors/track, 6241 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xff1fad05

   Device Boot      Start         End      Blocks   Id  System
/dev/vdb1               1        2082     1049296+  83  Linux 

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (2083-6241, default 2083): 
Using default value 2083
Last cylinder, +cylinders or +size{K,M,G} (2083-6241, default 6241): +1G

Command (m for help): t
Partition number (1-4): 2
Hex code (type L to list codes): 8e
Changed system type of partition 2 to 8e (Linux LVM)
Command (m for help): p

Disk /dev/vdb: 3221 MB, 3221225472 bytes
16 heads, 63 sectors/track, 6241 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xff1fad05

   Device Boot      Start         End      Blocks   Id  System
/dev/vdb1               1        2082     1049296+  83  Linux
/dev/vdb2            2083        4164     1049328   8e  Linux LVM

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@server23 ~]# partx /dev/vdb
# 1:        63-  2098655 (  2098593 sectors,   1074 MB)
# 2:   2098656-  4197311 (  2098656 sectors,   1074 MB)
# 3:         0-       -1 (        0 sectors,      0 MB)
# 4:         0-       -1 (        0 sectors,      0 MB)

• 建立LV设备

[root@server23 ~]# pvcreate /dev/vdb2 
  Physical volume "/dev/vdb2" successfully created

[root@server23 ~]# vgcreate server /dev/vdb2 
  Volume group "server" successfully created
[root@server23 ~]# vgscan 
  Reading all physical volumes.  This may take a while...
  Found volume group "server" using metadata type lvm2
  Found volume group "VolGroup" using metadata type lvm2
[root@server23 ~]# lvcreate -l 255 -n iscsi server
  Logical volume "iscsi" created
[root@server23 ~]# lvscan 
  ACTIVE            '/dev/server/iscsi' [1020.00 MiB] inherit
  ACTIVE            '/dev/VolGroup/lv_root' [7.61 GiB] inherit
  ACTIVE            '/dev/VolGroup/lv_swap' [920.00 MiB] inherit

• 关于共享磁盘的文件写法

iqn.yyyy-mm.:identifier
iqn.年-月.域名的反转写法:共享的target的名称
例如:
iqn.2018-03.com.linux:testtarget

• 对于每一个target来说可以包含多个磁盘设备,通常称为逻辑单位编号 LUN [Logical Unit Number]
• 来修改主配置文件/etc/tgt/targets.conf

<target iqn.2018-03.com.server23:testscsi>
        backing-store /srv/iscsi/disk1.img
        backing-store /dev/vdb1
        backing-store /dev/server/iscsi

    initiator-address 172.25.23.0/24
    incominguser westos westos
    write-cache off
</target>

• 上面这个文件的配置语法

<target iqn.相关设备target的名称>
 backing-store /虚拟设备的完整路径名称  <=====自动进行LUN编号 LUN1
 backing-store /虚拟设备的完整路径名称  <=====自动进行LUN编号 LUN1
 backing-store /虚拟设备的完整路径名称  <=====自动进行LUN编号 LUN1
target>

• 关于这个文件里面一些参数的说明
  
  • 1.backing-store和direct-store的区别:在设置设备时,如果需要使用整块设备来作为iscci就可以使用direct-store,否则不建议使用;
  • 2.initiator-address:如果需要限制使用target的客户端来源,如果不填写,就表示任何用户都可以访问,也可以使用iptables来进行限制;
  • 3.incominguser:关于用户帐号密码的设置,可以通过帐号和密码来进行访问,后面依次是帐号和密码;
  • 4.write-cache:tgtd是否使用缓存来加快速度,但是存在数据丢失的风险;
• 接下来启动target并且查看端口以及磁盘信息

[root@server23 ~]# vim /etc/tgt/targets.conf 
[root@server23 ~]# /etc/init.d/tgtd start
Starting SCSI target daemon:                               [  OK  ]
[root@server23 ~]# chkconfig tgtd  on         
如果需要开机自动启动 
[root@server23 ~]# chkconfig --list tgtd
tgtd            0:off   1:off   2:on    3:on    4:on    5:on    6:off

• 同时防火墙存在需要添加防火墙例外,防火墙以后会特别说明
• 查看端口是否正启动

[root@server23 ~]# netstat -tlunp | grep tgt
tcp        0      0 0.0.0.0:3260                0.0.0.0:*                   LISTEN      1617/tgtd           
tcp        0      0 :::3260                     :::*                        LISTEN      1617/tgtd 

• 使用使用命令来查看target信息

[root@server23 ~]# tgt-admin --show
Target 1: iqn.2018-03.com.server23:testscsi
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
        LUN: 1
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: beaf11
            Size: 1070 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: rdwr
            Backing store path: /dev/server/iscsi
            Backing store flags: 
        LUN: 2
            Type: disk
            SCSI ID: IET     00010002
            SCSI SN: beaf12
            Size: 1074 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: rdwr
            Backing store path: /dev/vdb1
            Backing store flags: 
        LUN: 3
            Type: disk
            SCSI ID: IET     00010003
            SCSI SN: beaf13
            Size: 524 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: rdwr
            Backing store path: /srv/iscsi/disk1.img
            Backing store flags: 
    Account information:
        westos
    ACL information:
        172.25.23.0/24

• * 接下来配置iscsi-initiator*

• 关于initiator的软件架构
  

  • 需要安装的软件iscsi-initiator-utils.x86_64
    

  /etc/iscsi/iscsid.conf： 主配置文件，用于配置连接iSCSI target的配置;
  /sbin/iscsid: 启动iSCSI initiator的主要配置文件程序;
  /sbin/iscsiadm: 用来管理iSCSI initiator的主要设置程序;
  /sbin/iscsid: 让本机成为iSCSI initiator的主要服务;

首先修改配置文件,并且需要添加密码信息

[root@server22 ~]# vim /etc/iscsi/iscsid.conf
更改或者添加
node.session.auth.username = westos
node.session.auth.password = westos
discovery.sendtargets.auth.username = westos
discovery.sendtargets.auth.password = westos

* 启动服务并且进行连接尝试

[root@server22 ~]# /etc/init.d/iscsi
iscsi   iscsid  
[root@server22 ~]# /etc/init.d/iscsi start
[root@server22 ~]# iscsiadm -m discovery -t sendtargets -p 172.25.23.23
Starting iscsid:                                           [  OK  ]
iscsiadm: cannot make connection to 172.25.23.23: No route to host
iscsiadm: cannot make connection to 172.25.23.23: No route to host
iscsiadm: cannot make connection to 172.25.23.23: No route to host
iscsiadm: cannot make connection to 172.25.23.23: No route to host

• 上面的问题多半是因为没有添加防火墙例外,或者网络不同导致的;
• 查看启动过程中是否存在错误信息

(Jun 06, 2013)
Mar 11 15:13:54 server22 kernel: iscsi: registered transport (bnx2i)
Mar 11 15:13:54 server22 kernel: iscsi: registered transport (be2iscsi)
Mar 11 15:13:54 server22 kernel: In beiscsi_module_init, tt=ffffffffa0461e80
Mar 11 15:13:54 server22 iscsid: iSCSI logger with pid=1488 started!
Mar 11 15:13:54 server22 iscsid: iSCSI daemon with pid=1489 started!

• 使用命令探测target连接

[root@server22 ~]# iscsiadm -m discovery -t sendtargets -p 172.25.23.23
172.25.23.23:3260,1 iqn.2018-03.com.server23:testscsi

• 重新启动服务,然后执行登陆操作

[root@server22 ~]# /etc/init.d/iscsi restart
Stopping iscsi:                                            [  OK  ]
Starting iscsi:                                            [  OK  ]
[root@server22 ~]# iscsiadm -m node -T iqn.2018-03.com.server23:testscsi --login

• 使用命令查看新增的设备

[root@server22 ~]# fdisk -l

Disk /dev/vda: 9663 MB, 9663676416 bytes
16 heads, 63 sectors/track, 18724 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c4b5f

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *           3        1018      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/vda2            1018       18725     8924160   8e  Linux LVM
Partition 2 does not end on cylinder boundary.

Disk /dev/mapper/VolGroup-lv_root: 8170 MB, 8170504192 bytes
255 heads, 63 sectors/track, 993 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/VolGroup-lv_swap: 964 MB, 964689920 bytes
255 heads, 63 sectors/track, 117 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sda: 1069 MB, 1069547520 bytes
33 heads, 62 sectors/track, 1020 cylinders
Units = cylinders of 2046 * 512 = 1047552 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sdb: 1074 MB, 1074479616 bytes
34 heads, 61 sectors/track, 1011 cylinders
Units = cylinders of 2074 * 512 = 1061888 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/sdc: 524 MB, 524288000 bytes
17 heads, 59 sectors/track, 1020 cylinders
Units = cylinders of 1003 * 512 = 513536 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

• 通过iscsi共享的设备被识别为/dev/sda等设备;
  接下来如果需要进行使用,就可以通过格式化等方式来使用iscsi设备

• 1.需要注意的是提供服务为的主机target主机必须开机比iscsi initiator开机要早;

• 更新添加删除target的方法

• 可能需要使用以下几个命令

[root@server22 ~]# iscsiadm -m node -T iqn.2018-03.com.server23:testscsi --logout
Logging out of session [sid: 1, target: iqn.2018-03.com.server23:testscsi, portal: 172.25.23.23,3260]
Logout of [sid: 1, target: iqn.2018-03.com.server23:testscsi, portal: 172.25.23.23,3260] successful.
表示退出登陆,但是不会删除/var/lib/iscsi/nodes/iqn.2018-03.com.server23
[root@server22 ~]# ll /var/lib/iscsi/nodes/iqn.2018-03.com.server23\:testscsi/
total 4
drw-------. 2 root root 4096 3月  11 15:15 172.25.23.23,3260,1
删除|添加|更新target的方法
[root@server22 ~]# iscsiadm -m node -o [delete|new|update] -T targetname

• 如果执行了删除操作,上面的文件就不存在了

[root@server22 ~]# iscsiadm -m node -o delete -T iqn.2018-03.com.server23:testscsi
[root@server22 ~]# iscsiadm -m node
iscsiadm: No records found

• 暂时没有遇到比较大的错误,就不进行错误的总结了;