iSCSI 远程块存储 （共享设备）
一，概述
SCSI(Internet SCSI) 支持从客户端 ( 发起端 ) 通过 IP 向远程服务器上的 iSCSI 存储设备 ( 目标 ) 发送 SCSI 命令。
iSCSI限定名称用于确定发起端和目标 , 并采用 iqn.yyyy-mm.{reverse domain}:label 的格式
默认情况下 , 网络通信是至 iSCSI 目标上的端口 3260/tcp 的明文端口ISCSI 名称设定
iSCSI 发起端 : 需要访问原始 SAN 存储的客户端
iSCSI 目标 : 从 iSCSI 服务器提供的远程硬盘磁盘 , 或“目标门户”
iSCSI 目标门户 : 通过网络向发起端提供目标的服务器。
SCSI(Internet SCSI) 支持从客户端 ( 发起端 ) 通过 IP 向远程服务器上的 iSCSI 存储设备 ( 目标 ) 发送 SCSI 命令。
iSCSI限定名称用于确定发起端和目标 , 并采用 iqn.yyyy-mm.{reverse domain}:label 的格式
默认情况下 , 网络通信是至 iSCSI 目标上的端口 3260/tcp 的明文端口ISCSI 名称设定
iSCSI 发起端 : 需要访问原始 SAN 存储的客户端
iSCSI 目标 : 从 iSCSI 服务器提供的远程硬盘磁盘 , 或“目标门户”
iSCSI 目标门户 : 通过网络向发起端提供目标的服务器。
IQN:“iSCSI 限定名称”。每个发起端和目标需要唯一名称进行标识最好的做法是使用一个在 Internet 上可能独一无二的名称
二，安装配置iscsi
服务端
[root@server ~]# fdisk -l >>>>>查看磁盘
Disk /dev/vda: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00013f3e
Device Boot Start End Blocks Id System
/dev/vda1 * 2048 20970332 10484142+ 83 Linux
Disk /dev/vdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@server ~]# fdisk /dev/vdb >>>>>>建立一个磁盘分区用来做共享分区
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0xce7e4760.
Command (m for help): p
Disk /dev/vdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xce7e4760
Device Boot Start End Blocks Id System
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (2048-20971519, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519): +2G
Partition 1 of type Linux and of size 2 GiB is set
Command (m for help): p >>>>>打印分区表
Disk /dev/vdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xce7e4760
Device Boot Start End Blocks Id System
/dev/vdb1 2048 4196351 2097152 83 Linux
Command (m for help): wq
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@server ~]# partprobe >>>>>>同步分区表
Warning: Unable to open /dev/sr0 read-write (Read-only file system). /dev/sr0 has been opened read-only.
Warning: Unable to open /dev/sr0 read-write (Read-only file system). /dev/sr0 has been opened read-only.
Warning: Unable to open /dev/sr0 read-write (Read-only file system). /dev/sr0 has been opened read-only.
[root@server ~]# yum install targetcli -y >>>>>>安装iscsi软件管理包
[root@server ~]# systemctl status target
target.service - Restore LIO kernel target configuration
Loaded: loaded (/usr/lib/systemd/system/target.service; disabled)
Active: inactive (dead)
[root@server ~]# systemctl start target >>>>>>启动
[root@server ~]# systemctl enable target
ln -s '/usr/lib/systemd/system/target.service' '/etc/systemd/system/multi-user.target.wants/target.service'
[root@server ~]# targetcli
Warning: Could not load preferences file /root/.targetcli/prefs.bin.
targetcli shell version 2.1.fb34
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/> ls >>>>>查看
o- / ......................................................................... [...]
o- backstores .............................................................. [...]
| o- block .................................................. [Storage Objects: 0]
| o- fileio ................................................. [Storage Objects: 0]
| o- pscsi .................................................. [Storage Objects: 0]
| o- ramdisk ................................................ [Storage Objects: 0]
o- iscsi ............................................................ [Targets: 0]
o- loopback ......................................................... [Targets: 0]
/> /backstores/block create westos:storage1 /dev/vdb1 >>>>>把/dev/vdb1作为磁盘分区，起名为westos:storage1
Created block storage object westos:storage1 using /dev/vdb1.
/> /iscsi create iqn.2018-06.com.example:storage1 >>>>>创建iscsi限定名称 格式为iqn.年-月.域名反写:label标示
Created target iqn.2018-06.com.example:storage1.
Created TPG 1.
/> ls >>>>>再次查看
o- / ......................................................................... [...]
o- backstores .............................................................. [...]
| o- block .................................................. [Storage Objects: 1]
| | o- westos:storage1 ............... [/dev/vdb1 (2.0GiB) write-thru deactivated]
| o- fileio ................................................. [Storage Objects: 0]
| o- pscsi .................................................. [Storage Objects: 0]
| o- ramdisk ................................................ [Storage Objects: 0]
o- iscsi ............................................................ [Targets: 1]
| o- iqn.2018-06.com.example:storage1 .................................. [TPGs: 1]
| o- tpg1 ............................................... [no-gen-acls, no-auth]
| o- acls .......................................................... [ACLs: 0]
| o- luns .......................................................... [LUNs: 0]
| o- portals .................................................... [Portals: 0]
o- loopback ......................................................... [Targets: 0]
######/iscsi/iqn.2018-06.com.example:storage1/tpg1/下面的三个都要设置
/> /iscsi/iqn.2018-06.com.example:storage1/tpg1/acls create iqn.2018-06.com.example:westoskey >>>>>创建acl密钥
Created Node ACL for iqn.2018-06.com.example:westoskey
/> /iscsi/iqn.2018-06.com.example:storage1/tpg1/luns create /backstores/block/westos:storage1 >>>>>密钥和设备关联
Created LUN 0.
Created LUN 0->0 mapping in node ACL iqn.2018-06.com.example:westoskey
/> /iscsi/iqn.2018-06.com.example:storage1/tpg1/portals create 172.25.254.200 >>>>开放端口
Using default IP port 3260
Created network portal 172.25.254.200:3260.
/> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json >>>>>提示配置信息的保存位置
注意：这个里面的命令可以tab显示出来，help查看命令格式
客户端
[root@client ~]# fdisk -l >>>>>>查看是否有共享磁盘
Disk /dev/vda: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00013f3e
Device Boot Start End Blocks Id System
/dev/vda1 * 2048 20970332 10484142+ 83 Linux
Disk /dev/vdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@client ~]# yum search iscsi
Loaded plugins: langpacks
================================= N/S matched: iscsi ==================================
iscsi-initiator-utils.i686 : iSCSI daemon and utility programs
iscsi-initiator-utils.x86_64 : iSCSI daemon and utility programs
iscsi-initiator-utils-iscsiuio.x86_64 : Userspace configuration daemon required for
: some iSCSI hardware
libiscsi.i686 : iSCSI client library
libiscsi.x86_64 : iSCSI client library
Name and summary matches only, use "search all" for everything.
[root@client ~]# yum install iscsi-initiator-utils.x86_64 -y >>>>>>安装客户端软件
[root@server ~]# systemctl status firewalld >>>>>>>服务端关闭防火墙
firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled)
Active: inactive (dead)
[root@client ~]# iscsiadm -m discovery -t st -p 172.25.254.200 >>>>>>>客户端发现设备
####-t 类型 -p IP端口
[root@client ~]# iscsiadm -m node -T iqn.2018-06.com.example:storage1 -p 172.25.254.200 -l >>>>>>>客户端登陆
####-T iqn共享设备名称 -p IP端口 -l 登陆
Logging in to [iface: default, target: iqn.2018-06.com.example:storage1, portal: 172.25.254.200,3260] (multiple)
iscsiadm: Could not login to [iface: default, target: iqn.2018-06.com.example:storage1, portal: 172.25.254.200,3260].
iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure)
iscsiadm: Could not log into all portals >>>>>>>登陆失败，要先修改配置文件，添加密钥
[root@client ~]# vim /etc/iscsi/initiatorname.iscsi
[root@client ~]# cat /etc/iscsi/initiatorname.iscsi >>>>>>>直接修改
InitiatorName=iqn.2018-06.com.example:westoskey
[root@client ~]# systemctl restart iscsid.service >>>>>>>重启
[root@client ~]# iscsiadm -m discovery -t st -p 172.25.254.200 >>>>>>>>发现
[root@client ~]# iscsiadm -m node -T iqn.2018-06.com.example:storage1 -p 172.25.254.200 -l >>>>>>>>登陆成功
Logging in to [iface: default, target: iqn.2018-06.com.example:storage1, portal: 172.25.254.200,3260] (multiple)
Login to [iface: default, target: iqn.2018-06.com.example:storage1, portal: 172.25.254.200,3260] successful.
[root@client ~]# fdisk -l >>>>>>>查看设备
Disk /dev/vda: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00013f3e
Device Boot Start End Blocks Id System
/dev/vda1 * 2048 20970332 10484142+ 83 Linux
Disk /dev/vdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sda: 2147 MB, 2147483648 bytes, 4194304 sectors >>>>>>出现共享设备
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 4194304 bytes
三，客户端使用共享磁盘
[root@client ~]# fdisk /dev/sda >>>>>>对共享磁盘进行分区
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Device does not contain a recognized partition table
Building a new DOS disklabel with disk identifier 0x759cd7b5.
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p):
Using default response p
Partition number (1-4, default 1):
First sector (8192-4194303, default 8192):
Using default value 8192
Last sector, +sectors or +size{K,M,G} (8192-4194303, default 4194303): +500M
Partition 1 of type Linux and of size 500 MiB is set
Command (m for help): p
Disk /dev/sda: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 4194304 bytes
Disk label type: dos
Disk identifier: 0x759cd7b5
Device Boot Start End Blocks Id System
/dev/sda1 8192 1032191 512000 83 Linux
Command (m for help): wq
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@client ~]# partprobe >>>>>>同步数据表
Warning: Unable to open /dev/sr0 read-write (Read-only file system). /dev/sr0 has been opened read-only.
Warning: Unable to open /dev/sr0 read-write (Read-only file system). /dev/sr0 has been opened read-only.
Warning: Unable to open /dev/sr0 read-write (Read-only file system). /dev/sr0 has been opened read-only.
[root@client ~]# mkfs.xfs /dev/sda1 >>>>>>>格式化磁盘分区
meta-data=/dev/sda1 isize=256 agcount=8, agsize=16000 blks
= sectsz=512 attr=2, projid32bit=1
= crc=0
data = bsize=4096 blocks=128000, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=0
log =internal log bsize=4096 blocks=853, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
[root@client ~]# mount /dev/sda1 /mnt/ >>>>>>>手动挂载
[root@client ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda1 10473900 3159092 7314808 31% /
devtmpfs 927072 0 927072 0% /dev
tmpfs 942660 80 942580 1% /dev/shm
tmpfs 942660 17028 925632 2% /run
tmpfs 942660 0 942660 0% /sys/fs/cgroup
/dev/sda1 508588 25888 482700 6% /mnt四，客户端自动挂载
[root@client ~]# vim /etc/fstab
/dev/sda1 /mnt xfs defaults,_netdev 0 0 >>>>>>>一定要写上_netdev
系统启动时默认先加载/etc/fstab后启动网络服务，但是如果不先启动网络，那么客户端和服务端就无法通信，这个文件的内容也就无法完成，系统就会一直卡住。
[root@client ~]# reboot
[root@156 ~]# rht-vmctl poweroff desktop >>>>>>>第一次需要手动关机，再开机，否则会卡住关不了
Powering off desktop..
[root@156 ~]# rht-vmctl start desktop #####如果没有写_netdev
#####进入单用户界面，改变系统根
####进入/etc/fstab然后添加，exit两次退出
五，客户端卸载
[root@client ~]# yum install tree
[root@client ~]# tree /var/lib/iscsi/ >>>>>>查看树形结构
/var/lib/iscsi/
|-- ifaces
|-- isns
|-- nodes
| `-- iqn.2018-06.com.example:storage1
| `-- 172.25.254.200,3260,1
| `-- default
|-- send_targets
| `-- 172.25.254.200,3260
| |-- iqn.2018-06.com.example:storage1,172.25.254.200,3260,1,default -> /var/lib/iscsi/nodes/iqn.2018-06.com.example:storage1/172.25.254.200,3260,1
| `-- st_config
|-- slp
`-- static
[root@client ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/vda1 10473900 3159388 7314512 31% /
devtmpfs 927072 0 927072 0% /dev
tmpfs 942660 80 942580 1% /dev/shm
tmpfs 942660 17032 925628 2% /run
tmpfs 942660 0 942660 0% /sys/fs/cgroup
/dev/sda1 508588 25888 482700 6% /mnt
[root@client ~]# umount /mnt/ >>>>>>取消挂载
[root@client ~]# vim /etc/fstab >>>>>>删除开机自动挂载的命令
[root@client ~]# fdisk -l >>>>>>查看共享磁盘
Disk /dev/vda: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00013f3e
Device Boot Start End Blocks Id System
/dev/vda1 * 2048 20970332 10484142+ 83 Linux
Disk /dev/vdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sda: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 4194304 bytes
Disk label type: dos
Disk identifier: 0x759cd7b5
Device Boot Start End Blocks Id System
/dev/sda1 8192 1032191 512000 83 Linux >>>>>>设备仍然存在
[root@client ~]# iscsiadm -m node -T iqn.2018-06.com.example:storage1 -p 172.25.254.200 -u >>>>>> -u 表示unlogin
Logging out of session [sid: 1, target: iqn.2018-06.com.example:storage1, portal: 172.25.254.200,3260]
Logout of [sid: 1, target: iqn.2018-06.com.example:storage1, portal: 172.25.254.200,3260] successful.
[root@client ~]# fdisk -l
Disk /dev/vda: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00013f3e
Device Boot Start End Blocks Id System
/dev/vda1 * 2048 20970332 10484142+ 83 Linux
Disk /dev/vdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes >>>>>共享设备消失，只是暂时的
[root@client ~]# systemctl restart iscsi >>>>>>重启服务之后又会出现
[root@client ~]# fdisk -l
Disk /dev/vda: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00013f3e
Device Boot Start End Blocks Id System
/dev/vda1 * 2048 20970332 10484142+ 83 Linux
Disk /dev/vdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/sda: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 4194304 bytes
Disk label type: dos
Disk identifier: 0x759cd7b5
Device Boot Start End Blocks Id System
/dev/sda1 8192 1032191 512000 83 Linux >>>>>>共享设备仍然出现
[root@client ~]# iscsiadm -m node -T iqn.2018-06.com.example:storage1 -p 172.25.254.200 -u >>>>>>再次登出
Logging out of session [sid: 2, target: iqn.2018-06.com.example:storage1, portal: 172.25.254.200,3260]
Logout of [sid: 2, target: iqn.2018-06.com.example:storage1, portal: 172.25.254.200,3260] successful.
[root@client ~]# iscsiadm -m node -T iqn.2018-06.com.example:storage1 -p 172.25.254.200 -o delete
>>>>> -o operation 删除操作
[root@client ~]# tree /var/lib/iscsi//var/lib/iscsi/ >>>>> 查看树形结构
|-- ifaces
|-- isns
|-- nodes
|-- send_targets
| `-- 172.25.254.200,3260
| `-- st_config
|-- slp
`-- static
7 directories, 1 file
[root@client ~]# systemctl restart iscsi >>>>>重启服务
[root@client ~]# fdisk -l >>>>>查看设备
Disk /dev/vda: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00013f3e
Device Boot Start End Blocks Id System
/dev/vda1 * 2048 20970332 10484142+ 83 Linux >>>>>>共享设备消失
Disk /dev/vdb: 10.7 GB, 10737418240 bytes, 20971520 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes