GlusterFS日常运维小记

目前测试环境的OpenStack存储后端部分采用的GlusterFS，记下日常使用的一些命令方便其他同事查看。

运维命令####

创建卷

$ gluster volume create cinder-volumes replica 2 transport tcp host1:/brick/volume1 host2:/brick/volume1 host3:/brick/volume1 host4:/brick/volume1

卷设置

$ gluster volume cinder-volumes set nfs.disable on
$ gluster volume cinder-volumes set performance.readdir-ahead on
$ gluster volume cinder-volumes set transport.address-family inet 
$ gluster volume cinder-volumes set performance.read-ahead-page-count 8
$ gluster volume cinder-volumes set performance.io-thread-count 16
$ gluster volume cinder-volumes set network.ping-timeout 10
$ gluster volume cinder-volumes set performance.cache-size 8GB
$ gluster volume cinder-volumes set cluster.self-heal-daemon on
$ gluster volume cinder-volumes set cluster.heal-timeout 300
$ gluster volume cinder-volumes set performance.write-behind-window-size 256MB
$ gluster volume cinder-volumes set performance.io-cache off
$ gluster volume cinder-volumes set server.allow-insecure on

扩容卷

$ gluster volume add-brick cinder-volumes replica 2 host1:/brick/volume1 host2:/brick/volume1

#重建数据
$ gluster volume rebalance cinder-volumes start [force]

$ gluster volume rebalance cinder-volumes status

缩容卷

若是副本卷，则移除的Bricks数是replica的整数倍；stripe具有同样的要求.

$ gluster volume remove-brick cinder-volumes host1:/brick/volume1 host2:/brick/volume1

#执行start则开始迁移数据，系统会有负载
$ gluster volume remove-brick cinder-volumes start

$ gluster volume remove-brick cinder-volumes status

#不迁移数据，直接删除
$ gluster volume remove-brick cinder-volumes commit

替换卷

用host2的卷替换host1

$ gluster volume replace-brick cinder-volumes host1:/brick/volume1 host2:/brick/volume1 [force]

故障处理####

1.物理机数据盘硬盘故障

在正常的服务器上gluster的贡献盘文件系统属性，然后设置到新硬盘挂载贡献路径上；

#获取数据
$ getfattr -d -m '.*' /brick/volume1

getfattr: Removing leading '/' from absolute path names
# file: brick/volume1
trusted.gfid=0sAAAAAAAAAAAAAAAAAAAAAQ==
trusted.glusterfs.dht=0sAAAAAQAAAAA/////f////Q==
trusted.glusterfs.dht.commithash="3307144809"
trusted.glusterfs.volume-id=0sqcz7hnqRQ9asrnzx2DtUxQ==

#设置属性
$ setfattr -n trusted.gfid -v 0sAAAAAAAAAAAAAAAAAAAAAQ== /brick/volume2
$ setfattr -n trusted.glusterfs.dht -v 0sAAAAAQAAAAA/////f////Q== /brick/volume2
$ setfattr -n trusted.glusterfs.dht.commithash -v "3168624641" /brick/volume2
$ setfattr -n trusted.glusterfs.volume-id -v 0sqcz7hnqRQ9asrnzx2DtUxQ== /brick/volume2

重启glusterd服务，然后查看健康状态；

$ gluster volume heal cinder-volumes info

Brick host1:/brick/volume1
Status: Connected
Number of entries: 0

Brick host2:/brick/volume1
/data 
Status: Connected
Number of entries: 1        # 显示一个条目在修复，自动修复完成后会为 0