target:
storage1:172.25.15.10
initiator:
iscsi1:172.25.15.14
iscsi2:172.25.15.15
用iscis方式实现共享存储
使用 iscsi-target 程序导出 iSCSI 设备需要完成如下步骤:
1. 下载 iscsi-target 程序,因为 RedHat 系统并没有自带这个程序的 rpm 包
2. 安装编译 iscsi-target 程序所需要的 gcc 、make 、kernel-devel 等编译环境 rpm 包
3. 编译 iscsi-target 并将其安装在系统中
4. 确定 iSCSI 导出的本地分区或本地存储设备已经就位
5. 编辑/etc/ietd.conf 配置文件,指定导出 iSCSI iqn 名和 LUN ID ,确定验证方式和
验证用用户名密码
6. 在/etc/initiator.allow 和/etc/initiator.deny 里设置访问控制规则
7. 启动/etc/init.d/iscsi-target 服务,导出 iSCSI 设备。
实验过程：
initiator：1.发现磁盘 2.登陆并导入磁盘 3.分区格式化挂接测试
[root@storage1-f15 ~]# iptables -F
[root@storage1-f15 ~]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]
[root@storage1-f15 ~]# getenforce
Enforcing
[root@storage1-f15 ~]# vim /etc/selinux/config
[root@storage1-f15 ~]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
[root@storage1-f15 ~]# date
Sun Oct 9 14:17:33 CST 2016
实验开始前设置实验环境
--------------------------
[root@storage1-f15 ~]# yum install scsi* -y #target端安装的软件
[root@storage1-f15 ~]# fdisk /dev/vd
vda vda1 vda2 vdb
[root@storage1-f15 ~]# fdisk /dev/vdb #新建一个逻辑分区，只能共享主分区和逻辑分区，扩展分区不能共享
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xb1011fea.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p #建立一个主分区
Partition number (1-4): 1
First cylinder (1-20805, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-20805, default 20805): +2G
#指定大小为2G
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@storage1-f15 ~]# ls /dev/vdb1
/dev/vdb1
[root@storage1-f15 ~]# vim /etc/tgt/targets.conf #修改配置文件指定共享盘或分区
backing-store /dev/vdb1
[root@storage1-f15 ~]# service tgtd start #启动服务
Starting SCSI target daemon: [ OK ]
[root@storage1-f15 ~]# tgtadm --lld iscsi --mode target --op show #查看共享盘
Target 1: iqn.2016-10.com.example.storage1-f15:vdb1.2G
......
--------------------------------------------------------------------------
[root@iscsi1-f15 ~]# iptables -F
[root@iscsi1-f15 ~]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]
[root@iscsi1-f15 ~]# getenforce
Enforcing
[root@iscsi1-f15 ~]# vim /etc/selinux/config
SELINUX=disabled
[root@iscsi1-f15 ~]# date
Sun Oct 9 14:19:19 CST 2016
[root@iscsi1-f15 ~]# yum install iscsi* -y #安装initiator端软件
[root@iscsi1-f15 ~]# iscsiadm -m discovery -t st -p 172.25.15.10 #发现 iSCSI 设备
Starting iscsid: [ OK ]
172.25.15.10:3260,1 iqn.2016-10.com.example.storage1-f15:vdb1.2G
[root@iscsi1-f15 ~]# iscsiadm -m node -T iqn.2016-10.com.example.storage1-f15:vdb1.2G -p 172.25.15.10 -l #登录并导入 iSCSI 设备
Logging in to [iface: default, target: iqn.2016-10.com.example.storage1-f15:vdb1.2G, portal: 172.25.15.10,3260] (multiple)
Login to [iface: default, target: iqn.2016-10.com.example.storage1-f15:vdb1.2G, portal: 172.25.15.10,3260] successful.
[root@iscsi1-f15 ~]# ls /dev/sd*
/dev/sda
[root@iscsi1-f15 ~]# fdisk /dev/sda #共享的分区在客户端显示为一块磁盘，可以进行创建分区的操作
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1009, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1009, default 1009):
Using default value 1009
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@iscsi1-f15 ~]# ls /dev/sd*
/dev/sda /dev/sda1
[root@iscsi1-f15 ~]# mkfs.ext4 /dev/sda1 #创建文件系统
[root@iscsi1-f15 ~]# mount /dev/sda1 /mnt #挂载做测试
[root@iscsi1-f15 ~]# cd /mnt
[root@iscsi1-f15 mnt]# echo 111 >file1
[root@iscsi1-f15 mnt]# cat file1
111
[root@iscsi1-f15 mnt]# cat file1
111
-------------------------------
[root@iscsi2-f15 ~]# iptables -F #iscsi2做同样的操作
[root@iscsi2-f15 ~]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[ OK ]
[root@iscsi2-f15 ~]# getenforce
Enforcing
[root@iscsi2-f15 ~]# vim /etc/selinux/config
[root@iscsi2-f15 ~]# SELINUX=disabled^C
[root@iscsi2-f15 ~]# date
Sun Oct 9 14:19:14 CST 2016
[root@iscsi2-f15 ~]# yum install iscsi* -y
[root@iscsi2-f15 ~]# iscsiadm -m discovery -t st -p 172.25.15.10
Starting iscsid: [ OK ]
172.25.15.10:3260,1 iqn.2016-10.com.example.storage1-f15:vdb1.2G
[root@iscsi2-f15 ~]# iscsiadm -m node -T iqn.2016-10.com.example.storage1-f15:vdb1.2G -p 172.25.15.10 -l
Logging in to [iface: default, target: iqn.2016-10.com.example.storage1-f15:vdb1.2G, portal: 172.25.15.10,3260] (multiple)
Login to [iface: default, target: iqn.2016-10.com.example.storage1-f15:vdb1.2G, portal: 172.25.15.10,3260] successful.
[root@iscsi2-f15 ~]# ls /dev/sd*
/dev/sda /dev/sda1
[root@iscsi2-f15 ~]# mount /dev/sda1 /mnt
[root@iscsi2-f15 ~]# cat /mnt/file1
111
[root@iscsi2-f15 ~]# echo 222 > /mnt/file1
[root@iscsi2-f15 ~]# cat /mnt/file1
222
这种方式一个文件修改后，在另一台主机上看不到更改后的内容
因为所用的文件系统是ext4，这是一种本地的文件系统，所以更改后不能同时生效
真实的生产环境中是不能用这种方式实现磁盘共享的
**********************************************************************************
**********************************************************************************
解决方法：
使用网路文件系统gfs2
使用这种文件系统，两个initiator端要实现通信，实现通信的方法是将initiator主机加入到一个集群中
GFS2 作为 RedHat
集群文件系统采用分布式元数据和多日志方式(multiple journal),并且 RedHat 仅支持 GFS2
文件系统作为高可用性集群文件系统部署使用。因为性能问题,从 RedHat Enterprise Linux 6
开始,GFS2 不再被支持作为单节点文件系统使用。
GFS2 命令
mount 描述
umount 卸载文件系统命令
fsck
fsck.gfs2
检查并修复卸载的文件系统(在未卸载的文件系统上使用此命
令是毁灭性的)
gfs2_grow
挂接文件系统命令
增大挂接的文件系统
gfs2_jadd 在某个挂接的文件系统中添加日志(节点)
mkfs.gfs2
mkfs -t gfs2
在存储设备上创建文件系统
gfs2_quota 在挂接的文件系统中管理配额
gfs2_tool 配置、调试文件系统或收集有关文件系统的信息
gfs2_edit 显示、输出或编辑文件系统内部结构。
chattr +j 在文件或目录上启用日志功能
setfacl /getfacl 设置或查看文件附加 ACL 访问权限
setattr/getattr 设置或查看文件的扩展权限
[root@iscsi1-f15 mnt]# yum install ricci luci -y #安装客户端和服务端
[root@iscsi1-f15 mnt]# service luci start #启动服务端服务
Adding following auto-detected host IDs (IP addresses/domain names), corresponding to `iscsi1-f15.example.com' address, to the configuration of self-managed certificate `/var/lib/luci/etc/cacert.config' (you can change them by editing `/var/lib/luci/etc/cacert.config', removing the generated certificate `/var/lib/luci/certs/host.pem' and restarting luci):
(none suitable found, you can still do it manually as mentioned above)
Generating a 2048 bit RSA private key
writing new private key to '/var/lib/luci/certs/host.pem'
Starting saslauthd: [ OK ]
Start luci... [ OK ]
Point your web browser to https://iscsi1-f15.example.com:8084 (or equivalent) to access luci
[root@iscsi1-f15 mnt]# service ricci start #启动客户端服务
Starting oddjobd: [ OK ]
generating SSL certificates... done
Generating NSS database... done
Starting ricci: [ OK ]
[root@iscsi1-f15 mnt]# passwd ricci #设置密码
Changing password for user ricci.
New password:
BAD PASSWORD: it is based on a dictionary word
Retype new password:
passwd: all authentication tokens updated successfully.
[root@iscsi1-f15 mnt]# vim /etc/hosts #修改hosts文件
172.25.15.14 iscsi1-f15.example.com
172.25.15.15 iscsi2-f15.example.com
-------------------------------------------------------------------------------
[root@iscsi2-f15 ~]# yum install ricci -y #安装客户端软件
[root@iscsi2-f15 ~]# service ricci start #启动客户端服务
Starting oddjobd: [ OK ]
generating SSL certificates... done
Generating NSS database... done
Starting ricci: [ OK ]
[root@iscsi2-f15 ~]# passwd ricci #设置密码
Changing password for user ricci.
New password:
BAD PASSWORD: it is based on a dictionary word
Retype new password:
passwd: all authentication tokens updated successfully.
[root@iscsi2-f15 ~]# vim /etc/hosts #修改hosts
172.25.15.14 iscsi1-f15.example.com
172.25.15.15 iscsi2-f15.example.com
--------------------------------------------------
浏览器https://iscsi1-f15.example.com:8084
创建一个集群加入iscsi1和iscsi2两台主机：如图
勾选，Download Packages Enable Share Stroage Support
-----------------------------------------------------
测试集群是否创建成功
[root@iscsi2-f15 ~]# cman_tool status #查看集群状态
Version: 6.2.0
Config Version: 1
Cluster Name: storage-f15
Cluster Id: 5242
Cluster Member: Yes
Cluster Generation: 8
Membership state: Cluster-Member
Nodes: 2
Expected votes: 1
Total votes: 2
Node votes: 1
Quorum: 1
Active subsystems: 9
Flags: 2node
Ports Bound: 0 11 177
Node name: iscsi2-f15.example.com
Node ID: 2
Multicast addresses: 239.192.20.142
Node addresses: 172.25.15.15
[root@iscsi2-f15 ~]# clustat #查看集群中的主机
Cluster Status for storage-f15 @ Sun Oct 9 16:28:40 2016
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
iscsi1-f15.example.com 1 Online
iscsi2-f15.example.com 2 Online, Local
[root@iscsi1-f15 mnt]# clustat
Cluster Status for storage-f15 @ Sun Oct 9 16:29:07 2016
Member Status: Quorate
Member Name ID Status
------ ---- ---- ------
iscsi1-f15.example.com 1 Online, Local
iscsi2-f15.example.com 2 Online
[root@iscsi1-f15 ~]# umount /mnt
[root@iscsi2-f15 ~]# umount /mnt
格式化磁盘创建文件系统
[root@iscsi1-f15 mnt]# mkfs.gfs2 -p lock_dlm -t storage-f15:test -j 2 /dev/sda1
This will destroy any data on /dev/sda1.
It appears to contain: Linux rev 1.0 ext4 filesystem data (needs journal recovery) (extents) (large files) (huge files)
Are you sure you want to proceed? [y/n] y
Device: /dev/sda1
Blocksize: 4096
Device Size 2.00 GB (523915 blocks)
Filesystem Size: 2.00 GB (523913 blocks)
Journals: 2
Resource Groups: 8
Locking Protocol: "lock_dlm"
Lock Table: "storage-f15:test"
UUID: 9fd77817-772b-93db-7875-96477a7f7e5f
挂载磁盘：
[root@iscsi1-f15 ~]# mount /dev/sda1 /mnt
[root@iscsi2-f15 ~]# mount /dev/sda1 /mnt
测试可以看出文件更改后可以互相生效
[root@iscsi1-f15 ~]# echo 111 > /mnt/file1
[root@iscsi2-f15 ~]# cat /mnt/file1
111
[root@iscsi2-f15 ~]# echo 222 > /mnt/file1
[root@iscsi2-f15 ~]# cat /mnt/file1
222
[root@iscsi1-f15 ~]# cat /mnt/file1
222
---------------------------------------------------------------------------
共享第二块硬盘：
[root@storage1-f15 ~]# fdisk /dev/vdb
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (4163-20805, default 4163):
Using default value 4163
Last cylinder, +cylinders or +size{K,M,G} (4163-20805, default 20805): +2G
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@storage1-f15 ~]# partx -a /dev/vdb #新建分区ls /dev/sd*读取不到执行
BLKPG: Device or resource busy
error adding partition 1
[root@storage1-f15 ~]# ls -a /dev/vd*
/dev/vda /dev/vda1 /dev/vda2 /dev/vdb /dev/vdb1 /dev/vdb2
[root@storage1-f15 ~]# vim /etc/tgt/targets.conf
[root@storage1-f15 ~]# service tgtd reload #重新加载服务
Updating SCSI target daemon configuration: [ OK ]
[root@iscsi1-f15 ~]# umount /mnt
[root@iscsi1-f15 ~]# iscsiadm -m node -u
Logging out of session [sid: 1, target: iqn.2016-10.com.example.storage1-f15:vdb1.2G, portal: 172.25.15.10,3260]
Logout of [sid: 1, target: iqn.2016-10.com.example.storage1-f15:vdb1.2G, portal: 172.25.15.10,3260] successful.
[root@iscsi1-f15 ~]# iscsiadm -m discovery -t st -p 172.25.15.10
172.25.15.10:3260,1 iqn.2016-10.com.example.storage1-f15:vdb1.2G
172.25.15.10:3260,1 iqn.2016-10.com.example.storage1-f15:vdb2.2G
[root@iscsi1-f15 ~]# iscsiadm -m node -l
Logging in to [iface: default, target: iqn.2016-10.com.example.storage1-f15:vdb1.2G, portal: 172.25.15.10,3260] (multiple)
Logging in to [iface: default, target: iqn.2016-10.com.example.storage1-f15:vdb2.2G, portal: 172.25.15.10,3260] (multiple)
Login to [iface: default, target: iqn.2016-10.com.example.storage1-f15:vdb1.2G, portal: 172.25.15.10,3260] successful.
Login to [iface: default, target: iqn.2016-10.com.example.storage1-f15:vdb2.2G, portal: 172.25.15.10,3260] successful.
[root@iscsi1-f15 ~]# ls /dev/sd*
/dev/sda /dev/sdb /dev/sdb1
-------------------------------------------------------------------------
常见问题？
共享磁盘后不能进行service tgtd restart 的操作，会导致客户机死机，等不可恢复的损坏
修改配置文件后可以使用service tgtd reload/force-reload
如果不行则必须对客户机作
1，umount /mnt #取消挂载
2,iscsiadm -m node -u #登出并撤销导入 iSCSI 设备
执行这两部后从才能stop tgtd操作
每台客户机都要操作，缺少一步都不可以
否则必须进行重启。才能重新挂载