RHCS+ISCSI+CLVM+GFS2 实现共享存储

环境介绍:

node1 (10.11.8.187) : target节点 --> 安装: corosync scsi-target-utils

node2 (10.11.8.186), node3(10.11.8.200) : initiator节点 --> 安装: corosync, iscsi-initiator-utils, gfs2-utils, lvm2-cluster

安装过程不再详述, 参照前文

target 配置:

[root@node1 ~]# tgtadm --lld iscsi --mode target --op show
Target 1: iqn.2016.com.shiina:storage.disk1
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
        I_T nexus: 1
            Initiator: iqn.2016.com.shiina:node2
            Connection: 0
                IP Address: 10.11.8.186
        I_T nexus: 2
            Initiator: iqn.1994-05.com.redhat:e6175c7b6952
            Connection: 0
                IP Address: 10.11.8.200
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
        LUN: 1
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: beaf11
            Size: 8590 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: rdwr
            Backing store path: /dev/sdb
            Backing store flags:
    Account information:
    ACL information:
        10.11.0.0/16

node2, node3 上发现并连接:

[root@node2 ~]# iscsiadm -m discovery -t sendtargets -p 10.11.8.187
10.11.8.187:3260,1 iqn.2016.com.shiina:storage.disk1
[root@node2 ~]# iscsiadm -m node -T iqn.2016.com.shiina:storage.disk1 -p 10.11.8.187 -l
Logging in to [iface: default, target: iqn.2016.com.shiina:storage.disk1, portal: 10.11.8.187,3260] (multiple)
Login to [iface: default, target: iqn.2016.com.shiina:storage.disk1, portal: 10.11.8.187,3260] successful.
[root@node2 ~]# fdisk -l
...
...
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1044     8385898+  83  Linux

配置RHCS, 添加一个集群: 详细操作见前文

配置CLVM:

lvm2-cluster 使用的已然是 lvm 程序, 但要让 lvm 支持集群需要修改 lvm 的配置

[root@node2 ~]# lvmconf --enable-cluster

也可以直接修改/etc/lvm/lvm.conf locking_type = 3

1.启动 clvmd 服务:

[root@node2 ~]# service clvmd start
Starting clvmd:
Activating VG(s):   2 logical volume(s) in volume group "vol0" now active
  clvmd not running on node node3
                                                           [  OK  ]
[root@node3 ~]# service clvmd start
Starting clvmd:
Activating VG(s):   2 logical volume(s) in volume group "vol0" now active
                                                           [  OK  ]

2.创建逻辑卷:

[root@node2 ~]# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created
[root@node2 ~]# vgcreate cluster_vg /dev/sdb
  Clustered volume group "cluster_vg" successfully created
[root@node2 ~]# lvcreate -L 2G -n clusterlv cluster_vg
  Logical volume "clusterlv" created.
[root@node2 ~]# lvs
  LV        VG         Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  clusterlv cluster_vg -wi-a-----  2.00g                                                    
  root      vol0       -wi-ao----  4.88g                                                    
  usr       vol0       -wi-ao---- 13.92g
[root@node3 ~]# lvs
  LV        VG         Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  clusterlv cluster_vg -wi-a-----  2.00g                                                    
  root      vol0       -wi-ao----  4.88g                                                    
  usr       vol0       -wi-ao---- 13.92g
mkfs.gfs2
  -j #: 指定日志区域的个数，有几个就能够被几个节点所挂载；
  -J #: 指定日志区域的大小，默认为128MB;
  -p {lock_dlm|lock_nolock}: 指定使用的锁类型
  -t : 锁表的名称，格式为clustername:locktablename, clustername为当前节点所在的集群的名称，locktablename要在当前集群惟一；
      当集群内有多个共享存储时, 用以区别锁的位置
gfs2-tool #工具, 查看调整参数等
gfs2_jadd -j #增加的日志区域数量
gfs2_grow #扩展gfs文件系统
[root@node2 ~]# mkfs.gfs2 -j 2 -p lock_dlm -t cluster:lvmstor /dev/cluster_vg/clusterlv
This will destroy any data on /dev/cluster_vg/clusterlv.
It appears to contain: symbolic link to `../dm-2'
 
Are you sure you want to proceed? [y/n] y
 
Device:                    /dev/cluster_vg/clusterlv
Blocksize:                 4096
Device Size                2.00 GB (524288 blocks)
Filesystem Size:           2.00 GB (524288 blocks)
Journals:                  2
Resource Groups:           8
Locking Protocol:          "lock_dlm"
Lock Table:                "cluster:lvmstor"
UUID:                      a3462622-2403-f3e4-b2ae-136b9274fa1d

挂载并测试文件系统:

[root@node2 ~]# mount -t gfs2 /dev/cluster_vg/clusterlv /mnt
[root@node2 ~]# touch /mnt/1.txt
[root@node2 ~]# ls /mnt/
1.txt
[root@node3 ~]# mount -t gfs2 /dev/cluster_vg/clusterlv /mnt
[root@node3 ~]# ls /mnt/
1.txt
 
[root@node3 ~]# touch /mnt/2.txt #node2,node3同时挂载时创建文件
[root@node2 ~]# ls /mnt/
1.txt  2.txt #正常查看