CentOS 7 搭建GlusterFS
实验需求：
4台机器安装GlusterFS组成一个集群
客户端把docker registry存储到文件系统里
4个节点的硬盘空间不整合成一个硬盘空间，要求每个节点都存储一份，保证数据安全
环境规划
-----------------------------------------------------
server
-----------------------------------------------------
node1:192.168.56.11 主机名:glusterfs1
-----------------------------------------------------
node2:192.168.56.12 主机名:glusterfs2
-----------------------------------------------------
node3:192.168.56.13 主机名:glusterfs3
-----------------------------------------------------
node4:192.168.56.14 主机名:glusterfs4
-----------------------------------------------------
client
-----------------------------------------------------
192.168.56.20 主机名:client
-----------------------------------------------------
OS: BCLinux release 7.2 (Mars)
实验前准备：
所有主机关闭防火墙，SElinux修改hosts文件，能够互相解析
192.168.56.11 glusterfs1
192.168.56.12 glusterfs2
192.168.56.13 glusterfs3
192.168.56.14 glusterfs4
192.168.56.20 client
。SSH互信（最好选择主机名）
-->ssh-keygen -t rsa
-->ssh-copy-id -i /root/.ssh/id_rsa.pub root@glusterfs1
-->ssh-copy-id -i /root/.ssh/id_rsa.pub root@glusterfs2
-->ssh-copy-id -i /root/.ssh/id_rsa.pub root@glusterfs3
-->ssh-copy-id -i /root/.ssh/id_rsa.pub root@glusterfs4
-->ssh-copy-id -i /root/.ssh/id_rsa.pub root@client
一、安装
服务端
1.在glusterfs {1-4}节点上安装GlusrerFS软件包
。Glusterfs下载地址：
http://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-api-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-cli-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-client-xlators-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-debuginfo-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-fuse-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-geo-replication-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-libs-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-rdma-3.7.17-1.el7.x86_64.rpm
wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.17/CentOS/epel-7.2/x86_64/glusterfs-server-3.7.17-1.el7.x86_64.rpm
安装：
-->yum install libibverbs librdmacm -y
-->wget http://dl.fedoraproject.org/pub/epel/6/x86_64//userspace-rcu-0.7.7-1.el6.x86_64.rpm
-->rpm -ivh userspace-rcu-0.7.7-1.el6.x86_64.rpm
-->yum localinstall *.rpm -y
----------------------------------------------------------------------------------------------
报错：
需要：liburcu-bp.so.1()(64bit)
错误：依赖检测失败：
libibverbs.so.1()(64bit) 被 glusterfs-rdma-3.7.17-1.el7.x86_64 需要
libibverbs.so.1(IBVERBS_1.0)(64bit) 被 glusterfs-rdma-3.7.17-1.el7.x86_64 需要
libibverbs.so.1(IBVERBS_1.1)(64bit) 被 glusterfs-rdma-3.7.17-1.el7.x86_64 需要
librdmacm.so.1()(64bit) 被 glusterfs-rdma-3.7.17-1.el7.x86_64 需要
librdmacm.so.1(RDMACM_1.0)(64bit) 被 glusterfs-rdma-3.7.17-1.el7.x86_64 需要
liburcu-bp.so.1()(64bit) 被 glusterfs-server-3.7.17-1.el7.x86_64 需要
liburcu-cds.so.1()(64bit) 被 glusterfs-server-3.7.17-1.el7.x86_64 需要
解决方法：
-->yum install libibverbs librdmacm -y
-->wget http://dl.fedoraproject.org/pub/epel/6/x86_64//userspace-rcu-0.7.7-1.el6.x86_64.rpm
-->rpm -ivh userspace-rcu-0.7.7-1.el6.x86_64.rpm
----------------------------------------------------------------------------------------------
3.userspace-rcu下载地址：
-->wget http://dl.fedoraproject.org/pub/epel/7/x86_64/u/userspace-rcu-0.7.16-1.el7.x86_64.rpm
-->rpm -ivh serspace-rcu-0.7.16-1.el7.x86_64.rpm
安装：
-->yum localinstall *.rpm
。查看安装的版本
-->glusterfs -V
glusterfs 3.7.17 built on Nov 3 2016 14:29:48
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2013 Red Hat, Inc.
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
。启动glusterfs服务
-->systemctl enable glusterd.service
Created symlink from /etc/systemd/system/multi-user.target.wants/glusterd.service to /usr/lib/systemd/system/glusterd.service.
-->systemctl start gluterd.service
-->systemctl status glusterd.service
● glusterd.service - GlusterFS, a clustered file-system server
Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2017-07-09 16:20:51 CST; 30s ago
Process: 4684 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 4685 (glusterd)
CGroup: /system.slice/glusterd.service
└─4685 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO
Jul 09 16:20:48 glusterfs1 systemd[1]: Starting GlusterFS, a clustered file-system server...
Jul 09 16:20:51 glusterfs1 systemd[1]: Started GlusterFS, a clustered file-system server.
2.在glusterfs1节点上配置整个GlusterFS集群，把各个节点加入到集群
-->gluster peer probe glusterfs1
peer probe: success. Probe on localhost not needed
-->gluster peer probe glusterfs2
peer probe: success.
-->gluster peer probe glusterfs3
peer probe: success.
-->gluster peer probe glusterfs4
peer probe: success.
3.查看节点状态
-->gluster peer status
Number of Peers: 3
Hostname: glusterfs2
Uuid: 9149c992-e86c-436d-8da9-b2a7245289cb
State: Peer in Cluster (Connected)
Hostname: glusterfs3
Uuid: dcbe562d-5bbd-4ee8-8855-c48428bba7ef
State: Peer in Cluster (Connected)
Hostname: glusterfs4
Uuid: 677514f3-7c3c-4b8e-b93d-db3941485166
State: Peer in Cluster (Connected)
4.在glusterfs{1-4}上创建数据存储目录
-->mkdir -p /usr/local/share/models
5.在glusterfs1上创建GlusterFS磁盘
注意：
加上replica 4就是4个节点中，每个节点都要把数据存储一次，就是一个数据存储4份，每个节点一份
如果不加replica 4，就是4个节点的磁盘空间整合成一个硬盘，
-->gluster volume create models replica 4 glusterfs1:/usr/local/share/models glusterfs2:/usr/local/share/models glusterfs3:/usr/local/share/models glusterfs4:/usr/local/share/models force
volume create: models: success: please start the volume to access data
6.启动
-->gluster volume start models
volume start: models: success
----------------------------------------------------------------------------
客户端
1.部署GlusterFS客户端并mount GlusterFS文件系统
-->mkdir /usr/local/src/rpm
方法同服务器端（下载rpm安装)
-->yum install libibverbs librdmacm -y
-->wget http://dl.fedoraproject.org/pub/epel/6/x86_64//userspace-rcu-0.7.7-1.el6.x86_64.rpm
-->rpm -ivh userspace-rcu-0.7.7-1.el6.x86_64.rpm
-->yum localinstall *.rpm -y
最好按下面方法安装客户端包
-->rpm -ivh glusterfs-client-xlators-3.7.17-1.el7.x86_64.rpm glusterfs-fuse-3.7.17-1.el7.x86_64.rpm
-->mkdir -p /mnt/models
-->mount -t glusterfs -o ro glusterfs1:models /mnt/models/
2.查看效果
-->df -h
df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda2 ext4 50G 1.5G 46G 4% /
devtmpfs devtmpfs 904M 0 904M 0% /dev
tmpfs tmpfs 913M 0 913M 0% /dev/shm
tmpfs tmpfs 913M 8.8M 904M 1% /run
tmpfs tmpfs 913M 0 913M 0% /sys/fs/cgroup
/dev/sda6 ext4 35G 4.7G 29G 15% /var
/dev/sda3 ext4 9.8G 37M 9.2G 1% /home
/dev/sda1 ext4 477M 94M 354M 21% /boot
tmpfs tmpfs 183M 0 183M 0% /run/user/0
glusterfs1:models fuse.glusterfs 50G 1.5G 46G 4% /mnt/models
其他操作命令
。删除GlusterFS磁盘
-->gluster volume stop models 先停止
-->gluster volume delete models 再删除
。卸载GlusterFS磁盘
-->gluster peer detach glusterfs4
。ACL访问控制
-->gluster volume set models auth.allow 192.168.56.*,10.0.1.*
volume set: success
。添加GlusterFS节点
-->gluster peer probe sc2-log5
-->gluster peer probe sc2-log6
-->gluster volume add-brick models sc2-log5:/data/gluster sc2-log6:/data/gluster
。迁移GlusterFS数据
-->gluster volume remove-brick models sc2-log1:/usr/local/share/models sc2-log5:/usr/local/share/models start
-->gluster volume remove-brick models sc2-log1:/usr/local/share/models sc2-log5:/usr/local/share/models status
-->gluster volume remove-brick models sc2-log1:/usr/local/share/models sc2-log5:/usr/local/share/models commit
。修复GlusterFS数据(在节点1宕机的情况下)
-->gluster volume replace-brick models sc2-log1:/usr/local/share/models sc2-log5:/usr/local/share/models commit -force
-->gluster volume heal models full