一,GFS2简介

GFS2是一个基于GFS的先进的集群文件系统,能够同步每台主机的集群文件系统的metadata,能够进行文件锁的管理,并且必须要redhat cluster suite支持,GFS2可以grow,进行容量的调整,不过这是在disk动态容量调整的支持下,也就是本文所要实现的CLVM。

实验环境:

192.168.30.119  tgtd.luojianlong.com  OS:Centos 6.4 x86_64   管理服务器  iscsi-target-server

192.168.30.115  node1.luojianlong.com OS:Centos 6.4 x86_64    iscsi-initiator

192.168.30.116  node2.luojianlong.com OS:Centos 6.4 x86_64    iscsi-initiator

192.168.30.117  node3.luojianlong.com OS:Centos 6.4 x86_64    iscsi-initiator

原理:

node1,node2,node3分别通过ISCSI-initiator登录并挂载tgtd服务器的存储设备,利用RHCS搭建GFS2高可用集群文件系统,且保证3个节点对存储设备能够同时读写访问。

下面是拓扑图:

RHCS,GFS2,ISCSI,CLVM实现共享可扩展集群存储_第1张图片



二,准备工作

分别设置4台服务器的hosts文件,以便能够解析对应节点,设置管理节点到各集群节点的ssh密钥无密码登录,关闭NetworkManager,设置开机不自动启动。


[root@tgtd ~]# cat /etc/hosts
192.168.30.115 node1.luojianlong.com node1
192.168.30.116 node2.luojianlong.com node2
192.168.30.117 node3.luojianlong.com node3
[root@tgtd ~]# ssh-copy-id -i node1
The authenticity of host 'node1 (192.168.30.115)' can't be established.
RSA key fingerprint is 66:2e:28:75:ba:34:5e:b1:40:66:af:ba:37:80:20:3f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node1,192.168.30.115' (RSA) to the list of known hosts.
root@node1's password:
Now try logging into the machine, with "ssh 'node1'", and check in:
  .ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.
[root@tgtd ~]# ssh-copy-id -i node2
The authenticity of host 'node2 (192.168.30.116)' can't be established.
RSA key fingerprint is 66:2e:28:75:ba:34:5e:b1:40:66:af:ba:37:80:20:3f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node2,192.168.30.116' (RSA) to the list of known hosts.
root@node2's password:
Now try logging into the machine, with "ssh 'node2'", and check in:
  .ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.
[root@tgtd ~]# ssh-copy-id -i node3
The authenticity of host 'node3 (192.168.30.117)' can't be established.
RSA key fingerprint is 66:2e:28:75:ba:34:5e:b1:40:66:af:ba:37:80:20:3f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node3,192.168.30.117' (RSA) to the list of known hosts.
root@node3's password:
Now try logging into the machine, with "ssh 'node3'", and check in:
  .ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.
[root@tgtd ~]# for I in {1..3}; do scp /etc/hosts node$I:/etc/; done
hosts                                                                                                    100%  129     0.1KB/s   00:00
hosts                                                                                                    100%  129     0.1KB/s   00:00
hosts


[root@tgtd ~]# for I in {1..3}; do ssh node$I 'service NetworkManager stop'; done
Stopping NetworkManager daemon: [  OK  ]
Stopping NetworkManager daemon: [  OK  ]
Stopping NetworkManager daemon: [  OK  ]
[root@tgtd ~]# for I in {1..3}; do ssh node$I 'chkconfig NetworkManager off'; done


关闭各节点的iptables,selinux服务


[root@tgtd ~]# for I in {1..3}; do ssh node$I 'service iptables stop'; done
[root@tgtd ~]# for I in {1..3}; do ssh node$I 'setenforce 0'; done

集群安装

RHCS的核心组件为cman和rgmanager,其中cman为基于openais的“集群基础架构层”,rgmanager为资源管理器。RHCS的集群中资源的配置需要修改其主配置文件/etc/cluster/cluster.xml实现,其仅安装在集群中的某一节点上即可,而cman和rgmanager需要分别安装在集群中的每个节点上。这里选择将此三个rpm包分别安装在了集群中的每个节点上

[root@tgtd ~]# for I in {1..3}; do ssh node$I 'yum -y install cman rgmanager'; done


为集群创建配置文件

RHCS的配置文件/etc/cluster/cluster.conf,其在每个节点上都必须有一份,且内容均相同,其默认不存在,因此需要事先创建,ccs_tool命令可以完成此任务。另外,每个集群通过集群ID来标识自身,因此,在创建集群配置文件时需要为其选定一个集群名称,这里假设其为tcluster。此命令需要在集群中的某个节点上执行

[root@tgtd ~]# for I in {1..3}; do ssh node$I 'ccs_tool create tcluster'; done

查看生成的配置文件的内容

[root@node1 cluster]# cat cluster.conf


  
  
  
  
  
    
    
  

#ccs_tool命令用于在线更新CCS的配置文件


为集群添加节点

RHCS集群需要配置好各节点及相关的fence设备后才能启动,因此,这里需要事先将各节点添加进集群配置文件。每个节点在添加进集群时,需要至少为其配置node id(每个节点的id必须惟一),ccs_tool的addnode子命令可以完成节点添加。将前面规划的三个集群节点添加至集群中,可以使用如下命令实现。

[root@node1 ~]# ccs_tool addnode -n 1 node1.luojianlong.com
[root@node1 ~]# ccs_tool addnode -n 2 node2.luojianlong.com
[root@node1 ~]# ccs_tool addnode -n 3 node3.luojianlong.com


查看已经添加完成的节点及相关信息:

[root@node1 ~]# ccs_tool lsnode
Cluster name: tcluster, config_version: 4
Nodename                        Votes Nodeid Fencetype
node1.luojianlong.com              1    1
node2.luojianlong.com              1    2
node3.luojianlong.com              1    3


复制配置文件到其他2个节点

[root@node1 ~]# scp /etc/cluster/cluster.conf node2:/etc/cluster/
[root@node1 ~]# scp /etc/cluster/cluster.conf node3:/etc/cluster/

启动集群

RHCS集群会等待各节点都启动后方才进入正常工作状态,因此,需要把集群各节点上的cman服务同时启动起来。这分别需要在各节点上执行如下命令

[root@tgtd ~]# for I in {1..3}; do ssh node$I 'service cman start'; done
Starting cluster:
   Checking if cluster has been disabled at boot... [  OK  ]
   Checking Network Manager... [  OK  ]
   Global setup... [  OK  ]
   Loading kernel modules... [  OK  ]
   Mounting configfs... [  OK  ]
   Starting cman... [  OK  ]
   Waiting for quorum... [  OK  ]
   Starting fenced... [  OK  ]
   Starting dlm_controld... [  OK  ]
   Tuning DLM kernel config... [  OK  ]
   Starting gfs_controld... [  OK  ]
   Unfencing self... [  OK  ]
   Joining fence domain... [  OK  ]
Starting cluster:
   Checking if cluster has been disabled at boot... [  OK  ]
   Checking Network Manager... [  OK  ]
   Global setup... [  OK  ]
   Loading kernel modules... [  OK  ]
   Mounting configfs... [  OK  ]
   Starting cman... [  OK  ]
   Waiting for quorum... [  OK  ]
   Starting fenced... [  OK  ]
   Starting dlm_controld... [  OK  ]
   Tuning DLM kernel config... [  OK  ]
   Starting gfs_controld... [  OK  ]
   Unfencing self... [  OK  ]
   Joining fence domain... [  OK  ]
Starting cluster:
   Checking if cluster has been disabled at boot... [  OK  ]
   Checking Network Manager... [  OK  ]
   Global setup... [  OK  ]
   Loading kernel modules... [  OK  ]
   Mounting configfs... [  OK  ]
   Starting cman... [  OK  ]
   Waiting for quorum... [  OK  ]
   Starting fenced... [  OK  ]
   Starting dlm_controld... [  OK  ]
   Tuning DLM kernel config... [  OK  ]
   Starting gfs_controld... [  OK  ]
   Unfencing self... [  OK  ]
   Joining fence domain... [  OK  ]


[root@tgtd ~]# for I in {1..3}; do ssh node$I 'service rgmanager start'; done
Starting Cluster Service Manager: [  OK  ]
Starting Cluster Service Manager: [  OK  ]
Starting Cluster Service Manager: [  OK  ]
[root@tgtd ~]# for I in {1..3}; do ssh node$I 'chkconfig rgmanager on'; done
[root@tgtd ~]# for I in {1..3}; do ssh node$I 'chkconfig cman on'; done


查看集群状态信息

[root@node1 ~]# clustat
Cluster Status for tcluster @ Tue Apr  1 16:45:23 2014
Member Status: Quorate
 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 node1.luojianlong.com                                               1 Online, Local
 node2.luojianlong.com                                               2 Online
 node3.luojianlong.com                                               3 Online


cman_tool的status子命令则以当前节点为视角来显示集群的相关信息

[root@node1 ~]# cman_tool status
Version: 6.2.0
Config Version: 4
Cluster Name: tcluster
Cluster Id: 10646
Cluster Member: Yes
Cluster Generation: 28
Membership state: Cluster-Member
Nodes: 3
Expected votes: 3
Total votes: 3
Node votes: 1
Quorum: 2
Active subsystems: 8
Flags:
Ports Bound: 0 177
Node name: node1.luojianlong.com
Node ID: 1
Multicast addresses: 239.192.41.191
Node addresses: 192.168.30.115

cman_tool的nodes子命令则可以列出集群中每个节点的相关信息

[root@node1 ~]# cman_tool nodes
Node  Sts   Inc   Joined               Name
   1   M     24   2014-04-01 16:37:57  node1.luojianlong.com
   2   M     24   2014-04-01 16:37:57  node2.luojianlong.com
   3   M     28   2014-04-01 16:38:11  node3.luojianlong.com


cman_tool的services子命令则可以列出集群中每个服务的相关信息

[root@node1 ~]# cman_tool services
fence domain
member count  3
victim count  0
victim now    0
master nodeid 1
wait state    none
members       1 2 3
dlm lockspaces
name          rgmanager
id            0x5231f3eb
flags         0x00000000
change        member 3 joined 1 remove 0 failed 0 seq 3,3
members       1 2 3

在tgtd server上安装scsi-target-utils

[root@tgtd ~]# yum -y install scsi-target-utils
[root@tgtd ~]# cp /etc/tgt/targets.conf /etc/tgt/targets.conf.bak

编辑target配置文件,定义target

[root@tgtd ~]# vi /etc/tgt/targets.conf
# 添加如下内容

        backing-store /dev/sdb
        initiator-address 192.168.30.0/24

[root@tgtd ~]# service  tgtd restart

backing-store:指定后端要共享的磁盘编号

initiator-address:授权客户端访问的网络地址

incominguser:设置登录用户的账号密码

启动target并查看

[root@tgtd ~]# tgtadm -L iscsi -m target -o show
Target 1: iqn.2014-04.com.luojianlong:target1
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: null
            Backing store path: None
            Backing store flags:
        LUN: 1
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: beaf11
            Size: 10737 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            Backing store type: rdwr
            Backing store path: /dev/sdb
            Backing store flags:
    Account information:
    ACL information:
        192.168.30.0/24


配置3个节点,使用iscsi-initiator登录tgtd服务的存储设备

[root@tgtd ~]# for I in {1..3}; do ssh node$I 'iscsiadm -m discovery -t st -p 192.168.30.119'; done
192.168.30.119:3260,1 iqn.2014-04.com.luojianlong:target1
192.168.30.119:3260,1 iqn.2014-04.com.luojianlong:target1
192.168.30.119:3260,1 iqn.2014-04.com.luojianlong:target1
[root@tgtd ~]# for I in {1..3}; do ssh node$I 'iscsiadm -m node -T iqn.2014-04.com.luojianlong:target1 -p 192.168.30.119:3260 -l'; done
Logging in to [iface: default, target: iqn.2014-04.com.luojianlong:target1, portal: 192.168.30.119,3260] (multiple)
Login to [iface: default, target: iqn.2014-04.com.luojianlong:target1, portal: 192.168.30.119,3260] successful.
Logging in to [iface: default, target: iqn.2014-04.com.luojianlong:target1, portal: 192.168.30.119,3260] (multiple)
Login to [iface: default, target: iqn.2014-04.com.luojianlong:target1, portal: 192.168.30.119,3260] successful.
Logging in to [iface: default, target: iqn.2014-04.com.luojianlong:target1, portal: 192.168.30.119,3260] (multiple)
Login to [iface: default, target: iqn.2014-04.com.luojianlong:target1, portal: 192.168.30.119,3260] successful.
[root@tgtd ~]# for I in {1..3}; do ssh node$I 'fdisk -l /dev/sdb'; done
Disk /dev/sdb: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdb: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdb: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


在其中一个节点上格式化一个分区

[root@node1 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x7ac42a91.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-10240, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-10240, default 10240): +5G
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@node1 ~]# fdisk /dev/sdb
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (5122-10240, default 5122):
Using default value 5122
Last cylinder, +cylinders or +size{K,M,G} (5122-10240, default 10240): +5G
Value out of range.
Last cylinder, +cylinders or +size{K,M,G} (5122-10240, default 10240): +4G
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@node1 ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 10.7 GB, 10737418240 bytes
64 heads, 32 sectors/track, 10240 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x7ac42a91
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        5121     5243888   83  Linux
/dev/sdb2            5122        9218     4195328   83  Linux


配置使用gfs2文件系统

[root@tgtd ~]# for I in {1..3}; do ssh node$I 'yum -y install gfs2-utils'; done

使用gfs2命令工具在之前创建好的/dev/sdb1上创建集群文件系统gfs2,可以使用如下命令

[root@node1 ~]# mkfs.gfs2 -j 3 -p lock_dlm -t tcluster:sdb1 /dev/sdb1
This will destroy any data on /dev/sdb1.
It appears to contain: Linux GFS2 Filesystem (blocksize 4096, lockproto lock_dlm)
Are you sure you want to proceed? [y/n] y
Device:                    /dev/sdb1
Blocksize:                 4096
Device Size                5.00 GB (1310972 blocks)
Filesystem Size:           5.00 GB (1310970 blocks)
Journals:                  3
Resource Groups:           21
Locking Protocol:          "lock_dlm"
Lock Table:                "tcluster:sdb1"
UUID:                      478dac97-c25f-5bc8-a719-0d385fea23e3

mkfs.gfs2为gfs2文件系统创建工具,其一般常用的选项有:


-b BlockSize:指定文件系统块大小,最小为512,默认为4096;

-J MegaBytes:指定gfs2日志区域大小,默认为128MB,最小值为8MB;

-j Number:指定创建gfs2文件系统时所创建的日志区域个数,一般需要为每个挂载的客户端指定一个日志区域;

-p LockProtoName:所使用的锁协议名称,通常为lock_dlm或lock_nolock之一;

-t LockTableName:锁表名称,一般来说一个集群文件系统需一个锁表名以便让集群节点在施加文件锁时得悉其所关联到的集群文件系统,锁表名称为clustername:fsname,其中的clustername必须跟集群配置文件中的集群名称保持一致,因此,也仅有此集群内的节点可访问此集群文件系统;此外,同一个集群内,每个文件系统的名称必须惟一。


格式化完成后,重启node1,node2,node3,不然无法挂载刚才创建的GFS2分区

[root@node1 ~]# mount /dev/sdb1 /mnt/
[root@node1 ~]# cp /etc/fstab /mnt/
# 在node2,node3上面也同时挂载/dev/sdb1
[root@node2 ~]# mount /dev/sdb1 /mnt/
[root@node3 ~]# mount /dev/sdb1 /mnt/
# 在node1上挂载目录中写入数据,检测node2,node3的挂载目录数据情况
[root@node2 mnt]# tail -f fstab
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/VolGroup-lv_root /                       ext4    defaults        1 1
UUID=db4bad23-32a8-44a6-bdee-1585ce9e13ac /boot                   ext4    defaults        1 2
/dev/mapper/VolGroup-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
[root@node3 mnt]# tail -f fstab
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/VolGroup-lv_root /                       ext4    defaults        1 1
UUID=db4bad23-32a8-44a6-bdee-1585ce9e13ac /boot                   ext4    defaults        1 2
/dev/mapper/VolGroup-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
[root@node1 mnt]# echo "hello" >> fstab
[root@node2 mnt]# tail -f fstab
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/VolGroup-lv_root /                       ext4    defaults        1 1
UUID=db4bad23-32a8-44a6-bdee-1585ce9e13ac /boot                   ext4    defaults        1 2
/dev/mapper/VolGroup-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
hello
[root@node3 mnt]# tail -f fstab
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/VolGroup-lv_root /                       ext4    defaults        1 1
UUID=db4bad23-32a8-44a6-bdee-1585ce9e13ac /boot                   ext4    defaults        1 2
/dev/mapper/VolGroup-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
hello

以上信息发现,node2,node3已经发现数据发生变化。


三,配置使用CLVM(集群逻辑卷)

在RHCS集群节点上安装lvm2-cluster

[root@tgtd ~]# for I in {1..3}; do ssh node$I 'yum -y install lvm2-cluster'; done

在RHCS的各节点上,为lvm启用集群功能

[root@tgtd ~]# for I in {1..3}; do ssh node$I 'lvmconf --enable-cluster'; done

为RHCS各节点启动clvmd服务

[root@tgtd ~]# for I in {1..3}; do ssh node$I 'service clvmd start'; done
Starting clvmd:
Activating VG(s):   2 logical volume(s) in volume group "VolGroup" now active
  clvmd not running on node node3.luojianlong.com
  clvmd not running on node node2.luojianlong.com
[  OK  ]
Starting clvmd:
Activating VG(s):   2 logical volume(s) in volume group "VolGroup" now active
  clvmd not running on node node3.luojianlong.com
[  OK  ]
Starting clvmd:
Activating VG(s):   2 logical volume(s) in volume group "VolGroup" now active
[  OK  ]

创建物理卷、卷组和逻辑卷,使用管理单机逻辑卷的相关命令即可

[root@node1 ~]# pvcreate /dev/sdb2
  Physical volume "/dev/sdb2" successfully created
[root@node1 ~]# pvs
  PV         VG       Fmt  Attr PSize  PFree
  /dev/sda2  VolGroup lvm2 a--  29.51g    0
  /dev/sdb2           lvm2 a--   4.00g 4.00g
# 此时,在另外的其它节点上也能够看到刚刚创建的物理卷

创建卷组和逻辑卷

[root@node1 ~]# vgcreate clustervg /dev/sdb2
  Clustered volume group "clustervg" successfully created
[root@node1 ~]# lvcreate -L 2G -n clusterlv clustervg
  Logical volume "clusterlv" created
[root@node1 ~]# lvs
  LV        VG        Attr       LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  lv_root   VolGroup  -wi-ao---- 25.63g                       
  lv_swap   VolGroup  -wi-ao----  3.88g                       
  clusterlv clustervg -wi-a-----  2.00g

在其他节点也能看到对应的逻辑卷

[root@tgtd ~]# for I in {1..3}; do ssh node$I 'lvs'; done
  LV        VG        Attr       LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  lv_root   VolGroup  -wi-ao---- 25.63g                       
  lv_swap   VolGroup  -wi-ao----  3.88g                       
  clusterlv clustervg -wi-a-----  2.00g                       
  LV        VG        Attr       LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  lv_root   VolGroup  -wi-ao---- 25.63g                       
  lv_swap   VolGroup  -wi-ao----  3.88g                       
  clusterlv clustervg -wi-a-----  2.00g                       
  LV        VG        Attr       LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  lv_root   VolGroup  -wi-ao---- 25.63g                       
  lv_swap   VolGroup  -wi-ao----  3.88g                       
  clusterlv clustervg -wi-a-----  2.00g

格式化逻辑卷

[root@node1 ~]# lvcreate -L 2G -n clusterlv clustervg
  Logical volume "clusterlv" created
[root@node1 ~]# lvs
  LV        VG        Attr       LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  lv_root   VolGroup  -wi-ao---- 25.63g                       
  lv_swap   VolGroup  -wi-ao----  3.88g                       
  clusterlv clustervg -wi-a-----  2.00g                       
[root@node1 ~]# mkfs.gfs2 -p lock_dlm -j 2 -t tcluster:clusterlv /dev/clustervg/clusterlv
This will destroy any data on /dev/clustervg/clusterlv.
It appears to contain: symbolic link to `../dm-2'
Are you sure you want to proceed? [y/n] y
Device:                    /dev/clustervg/clusterlv
Blocksize:                 4096
Device Size                2.00 GB (524288 blocks)
Filesystem Size:           2.00 GB (524288 blocks)
Journals:                  2
Resource Groups:           8
Locking Protocol:          "lock_dlm"
Lock Table:                "tcluster:clusterlv"
UUID:                      c8fbef88-970d-92c4-7b66-72499406fa9c


挂载逻辑卷

[root@node1 ~]# mount /dev/clustervg/clusterlv /media/
[root@node2 ~]# mount /dev/clustervg/clusterlv /media/
[root@node3 ~]# mount /dev/clustervg/clusterlv /media/
Too many nodes mounting filesystem, no free journals
# 发现node3挂载不了,因为刚才创建了2个journal,需要再添加一个
[root@node1 ~]# gfs2_jadd -j 1 /dev/clustervg/clusterlv
Filesystem:            /media
Old Journals           2
New Journals           3
# 然后挂载node3
[root@node3 ~]# mount /dev/clustervg/clusterlv /media/
[root@node1 ~]# df -hT
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
              ext4     26G  6.1G   18G  26% /
tmpfs        tmpfs    1.9G   38M  1.9G   2% /dev/shm
/dev/sda1     ext4    485M   65M  395M  15% /boot
/dev/sdb1     gfs2    5.1G  388M  4.7G   8% /mnt
/dev/mapper/clustervg-clusterlv
              gfs2    2.0G  388M  1.7G  19% /media

扩展逻辑卷


[root@node1 ~]# lvextend -L +2G /dev/clustervg/clusterlv
  Extending logical volume clusterlv to 4.00 GiB
  Logical volume clusterlv successfully resized
[root@node1 ~]# gfs2_grow /dev/clustervg/clusterlv
FS: Mount Point: /media
FS: Device:      /dev/dm-2
FS: Size:        524288 (0x80000)
FS: RG size:     65533 (0xfffd)
DEV: Size:       1048576 (0x100000)
The file system grew by 2048MB.
gfs2_grow complete.
[root@node1 ~]# df -hT
Filesystem    Type    Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
              ext4     26G  6.1G   18G  26% /
tmpfs        tmpfs    1.9G   38M  1.9G   2% /dev/shm
/dev/sda1     ext4    485M   65M  395M  15% /boot
/dev/sdb1     gfs2    5.1G  388M  4.7G   8% /mnt
/dev/mapper/clustervg-clusterlv
              gfs2    4.0G  388M  3.7G  10% /media

发现逻辑卷已经被扩展

到此,RHCS,GFS2,ISCSI,CLVM实现共享存储配置完毕。