独立部署 GlusterFS+Heketi 实现 Kubernetes / OpenShift 共享存储

1，准备工作

1.1 硬件信息

主机名	IP地址
gfs1	192.168.160.131
gfs2	192.168.160.132
gfs3/heketi	192.168.160.133

• 20G 的裸盘 /dev/sdb

Disk /dev/sdb: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

1.2 环境准备

• 默认情况下，SELinux不允许从Pod写入远程Gluster服务器，要通过在每个节点上执行SELinux来写入GlusterFS卷

sudo setsebool -P virt_sandbox_use_fusefs on 
sudo setsebool -P virt_use_fusefs on

1.3 载入指定的个别模块

modprobe dm_snapshot
modprobe dm_mirror
modprobe dm_thin_pool

2，安装GlusterFS了

yum -y install glusterfs glusterfs-server glusterfs-fuse

2.1 需要为GlusterFS peers打开几个基本TCP端口，以便与OpenShift进行通信并提供存储：

firewall-cmd --add-port=24007-24008/tcp --add-port=49152-49664/tcp --add-port=2222/tcp
firewall-cmd --runtime-to-permanent

2.2 启动GlusterFS的daemon进程了：

systemctl enable glusterd
systemctl start glusterd

3，在GlusterFS的一台虚拟机上安装heketi

yum -y install heketi heketi-client

3.1 启动文件语法：

• /usr/lib/systemd/system/heketi.service

[Unit]
Description=Heketi Server

[Service]
Type=simple
WorkingDirectory=/var/lib/heketi
EnvironmentFile=-/etc/heketi/heketi.json
User=heketi
ExecStart=/usr/bin/heketi --config=/etc/heketi/heketi.json
Restart=on-failure
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=multi-user.target

3.2 重启 heketi

systemctl daemon-reload
systemctl start heketi

3.3 创建密钥并分发

ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''
chown heketi:heketi /etc/heketi/heketi_key
for i in gfs1 gfs2 gfs3 ;do ssh-copy-id -i /etc/heketi/heketi_key.pub $i ;done

3.4 配置heketi来使用SSH。 编辑/etc/heketi/heketi.json文件

      "executor":"ssh",
      "_sshexec_comment":"SSH username and private key file information",
      "sshexec":{
         "keyfile":"/etc/heketi/heketi_key",
         "user":"root",
         "port":"22",
         "fstab":"/etc/fstab"
      },

3.5 heketi将监听8080端口，添加防火墙规则：

firewall-cmd --add-port=8080/tcp
firewall-cmd --runtime-to-permanent

3.6 重启heketi：

systemctl enable heketi
systemctl restart heketi

3.7 测试 heketi 运行状态：

• curl http://gfs1:8080/hello

Hello from Heketi

3.8 配置 GlusterFS 存储池

• vim /etc/heketi/topology.json

{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "gfs1"
              ],
              "storage": [
                "192.168.160.131"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sdb"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "gfs2"
              ],
              "storage": [
                "192.168.160.132"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sdb"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "gfs3"
              ],
              "storage": [
                "192.168.160.133"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sdb"
          ]
        }
      ]
    }
  ]
}

3.9 创建 GlusterFS 存储池

export HEKETI_CLI_SERVER=http://gfs3:8080
heketi-cli --server=http://gfs3:8080 topology load --json=/etc/heketi/topology.json

• 输出信息

Creating cluster ... ID: d3a3f31dce28e06dbd1099268c4ebe84
    Allowing file volumes on cluster.
    Allowing block volumes on cluster.
    Creating node infra.test.com ... ID: ebfc1e8e2e7668311dc4304bfc1377cb
        Adding device /dev/sdb ... OK
    Creating node node1.test.com ... ID: 0ce162c3b8a65342be1aac96010251ef
        Adding device /dev/sdb ... OK
    Creating node node2.test.com ... ID: 62952de313e71eb5a4bfe5b76224e575
        Adding device /dev/sdb ...  OK

3.10 当前位于 gfs3， 查看集群信息

• heketi-cli topology info

Cluster Id: d3a3f31dce28e06dbd1099268c4ebe84

    File:  true
    Block: true

    Volumes:

    Nodes:

    Node Id: 0ce162c3b8a65342be1aac96010251ef
    State: online
    Cluster Id: d3a3f31dce28e06dbd1099268c4ebe84
    Zone: 1
    Management Hostnames: node1.test.com
    Storage Hostnames: 192.168.160.132
    Devices:
        Id:d6a5f0aba39a35d3d92f678dc9654eaa   Name:/dev/sdb            State:online    Size (GiB):19      Used (GiB):0       Free (GiB):19      
            Bricks:

    Node Id: 62952de313e71eb5a4bfe5b76224e575
    State: online
    Cluster Id: d3a3f31dce28e06dbd1099268c4ebe84
    Zone: 1
    Management Hostnames: node2.test.com
    Storage Hostnames: 192.168.160.133
    Devices:
        Id:dfd697f2215d2a304a44c5af44d352da   Name:/dev/sdb            State:online    Size (GiB):19      Used (GiB):0       Free (GiB):19      
            Bricks:

    Node Id: ebfc1e8e2e7668311dc4304bfc1377cb
    State: online
    Cluster Id: d3a3f31dce28e06dbd1099268c4ebe84
    Zone: 1
    Management Hostnames: infra.test.com
    Storage Hostnames: 192.168.160.131
    Devices:
        Id:e06b794b0b9f20608158081fbb5b5102   Name:/dev/sdb            State:online    Size (GiB):19      Used (GiB):0       Free (GiB):19      
            Bricks:

• heketi-cli node list

Id:0ce162c3b8a65342be1aac96010251ef Cluster:d3a3f31dce28e06dbd1099268c4ebe84
Id:62952de313e71eb5a4bfe5b76224e575 Cluster:d3a3f31dce28e06dbd1099268c4ebe84
Id:ebfc1e8e2e7668311dc4304bfc1377cb Cluster:d3a3f31dce28e06dbd1099268c4ebe84

• gluster peer status

Number of Peers: 2

Hostname: gfs2
Uuid: ae6e998a-92c2-4c63-a7c6-c51a3b7e8fcb
State: Peer in Cluster (Connected)
Other names:
gfs2

Hostname: gfs1
Uuid: c8c46558-a8f2-46db-940d-4b19947cf075
State: Peer in Cluster (Connected)

4，测试

4.1 测试创建volume

• heketi-cli --json volume create --size 3 --replica 3

{"size":3,"name":"vol_93060cd7698e9e48bd035f26bbfe57af","durability":{"type":"replicate","replicate":{"replica":3},"disperse":{"data":4,"redundancy":2}},"glustervolumeoptions":["",""],"snapshot":{"enable":false,"factor":1},"id":"93060cd7698e9e48bd035f26bbfe57af","cluster":"d3a3f31dce28e06dbd1099268c4ebe84","mount":{"glusterfs":{"hosts":["192.168.160.132","192.168.160.133","192.168.160.131"],"device":"192.168.160.132:vol_93060cd7698e9e48bd035f26bbfe57af","options":{"backup-volfile-servers":"192.168.160.133,192.168.160.131"}}},"blockinfo":{},"bricks":[{"id":"16b8ddb1f2b2d3aa588d4d4a52bb7f6b","path":"/var/lib/heketi/mounts/vg_e06b794b0b9f20608158081fbb5b5102/brick_16b8ddb1f2b2d3aa588d4d4a52bb7f6b/brick","device":"e06b794b0b9f20608158081fbb5b5102","node":"ebfc1e8e2e7668311dc4304bfc1377cb","volume":"93060cd7698e9e48bd035f26bbfe57af","size":3145728},{"id":"9e60ac3b7259c4e8803d4e1f6a235021","path":"/var/lib/heketi/mounts/vg_d6a5f0aba39a35d3d92f678dc9654eaa/brick_9e60ac3b7259c4e8803d4e1f6a235021/brick","device":"d6a5f0aba39a35d3d92f678dc9654eaa","node":"0ce162c3b8a65342be1aac96010251ef","volume":"93060cd7698e9e48bd035f26bbfe57af","size":3145728},{"id":"e3f5ec732d5a8fe4b478af67c9caf85b","path":"/var/lib/heketi/mounts/vg_dfd697f2215d2a304a44c5af44d352da/brick_e3f5ec732d5a8fe4b478af67c9caf85b/brick","device":"dfd697f2215d2a304a44c5af44d352da","node":"62952de313e71eb5a4bfe5b76224e575","volume":"93060cd7698e9e48bd035f26bbfe57af","size":3145728}]}

• heketi-cli volume list

Id:93060cd7698e9e48bd035f26bbfe57af    Cluster:d3a3f31dce28e06dbd1099268c4ebe84    Name:vol_93060cd7698e9e48bd035f26bbfe57af

• heketi-cli volume info 93060cd7698e9e48bd035f26bbfe57af

Name: vol_93060cd7698e9e48bd035f26bbfe57af
Size: 3
Volume Id: 93060cd7698e9e48bd035f26bbfe57af
Cluster Id: d3a3f31dce28e06dbd1099268c4ebe84
Mount: 192.168.160.132:vol_93060cd7698e9e48bd035f26bbfe57af
Mount Options: backup-volfile-servers=192.168.160.133,192.168.160.131
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3

• gluster volume list

vol_93060cd7698e9e48bd035f26bbfe57af

• gluster volume status

Status of volume: vol_93060cd7698e9e48bd035f26bbfe57af
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 192.168.160.132:/var/lib/heketi/mount
s/vg_d6a5f0aba39a35d3d92f678dc9654eaa/brick
_9e60ac3b7259c4e8803d4e1f6a235021/brick     49153     0          Y       30660
Brick 192.168.160.131:/var/lib/heketi/mount
s/vg_e06b794b0b9f20608158081fbb5b5102/brick
_16b8ddb1f2b2d3aa588d4d4a52bb7f6b/brick     49153     0          Y       21979
Brick 192.168.160.133:/var/lib/heketi/mount
s/vg_dfd697f2215d2a304a44c5af44d352da/brick
_e3f5ec732d5a8fe4b478af67c9caf85b/brick     49152     0          Y       61274
Self-heal Daemon on localhost               N/A       N/A        Y       61295
Self-heal Daemon on apps.test.com           N/A       N/A        Y       22000
Self-heal Daemon on 192.168.160.132         N/A       N/A        Y       30681
 
Task Status of Volume vol_93060cd7698e9e48bd035f26bbfe57af
------------------------------------------------------------------------------
There are no active volume tasks

• gluster volume info vol_93060cd7698e9e48bd035f26bbfe57af

Volume Name: vol_93060cd7698e9e48bd035f26bbfe57af
Type: Replicate
Volume ID: ca4a9854-a33c-40ab-86c7-0d0d34004454
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 192.168.160.132:/var/lib/heketi/mounts/vg_d6a5f0aba39a35d3d92f678dc9654eaa/brick_9e60ac3b7259c4e8803d4e1f6a235021/brick
Brick2: 192.168.160.131:/var/lib/heketi/mounts/vg_e06b794b0b9f20608158081fbb5b5102/brick_16b8ddb1f2b2d3aa588d4d4a52bb7f6b/brick
Brick3: 192.168.160.133:/var/lib/heketi/mounts/vg_dfd697f2215d2a304a44c5af44d352da/brick_e3f5ec732d5a8fe4b478af67c9caf85b/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

5，在OpenShift中使用Gluster

5.1 OpenShift 创建 StorageClass YAML文件：

• 编辑storage-class.yaml，其中resturl为heketi的url，volumetype: replicate:3为副本卷brick数量，建议为3
• cat storage-class.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: null
  name: gluster-heketi
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://gfs3:8080"
  restauthenabled: "true"
  volumetype: replicate:3

• 创建 StorageClass

oc create -f storage-class.yaml 

• 查看 StorageClass
• oc get sc

NAME             PROVISIONER               AGE
gluster-heketi   kubernetes.io/glusterfs   55m

5.2 OpenShift 创建 PVC YAML文件：

• cat pvc.yml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: test-pvc
spec:
 accessModes:
  - ReadWriteMany
 resources:
   requests:
        storage: 1Gi
 storageClassName: gluster-heketi

• 创建 PVC

oc create -f storage-class.yaml 

• 查看 PV，PVC
• oc get pv,pvc

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM           STORAGECLASS     REASON    AGE
persistentvolume/pvc-57362c7f-e6c2-11e9-8634-000c299365cc   1Gi        RWX            Delete           Bound     default/test1   gluster-heketi             57m

NAME                          STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
persistentvolumeclaim/test-pvc   Bound     pvc-57362c7f-e6c2-11e9-8634-000c299365cc  1Gi        RWX            gluster-heketi   57m

• 挂载到虚机

mount -t glusterfs 192.168.160.132:vol_b96d0e18cef937dd56a161ae5fa5b9cb /mnt

• df -h | grep vol_b96d0e18cef937dd56a161ae5fa5b9cb

192.168.160.132:vol_b96d0e18cef937dd56a161ae5fa5b9cb                                   1014M   43M  972M   5% /mnt

6，常用命令

查看集群节点：gluster pool list
查看集群状态(默认不显示当前主机)： gluster peer status
查看集群volume ：gluster volume list
查看volume 信息：gluster volume info 
查看volume状态：gluster volume stats 
强制启动volume：gluster volume start  force
查看volume需要修复的文件：gluster volume heal  info
启动完全修复：gluster volume heal  full
查看修复成功的文件：gluster volume heal  info healed
查看修复失败的文件：gluster volume heal  info heal-failed
查看脑裂文件：gluster volume heal  info split-brain

6.1 其它heketi客户端常用命令

heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv cluster list
heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv cluster info 
heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv node info 
heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv volume list
heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv volume create --size=1 --replica=2
heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv volume info 
heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv volume expand --volume= --expand-size=1
heketi-cli --server=http://localhost:8080 --user=admin --secret=kLd834dadEsfwcv volume delete 

6.2 初始化裸盘

• 每个glusterfs machine上挂一快同样大小的未格式化过磁盘。 如果已经格式化过，使用下面命令重置磁盘(假设磁盘挂在/dev/sdb)：

pvcreate --metadatasize=128M --dataalignment=256K /dev/sdb

7， GlusterFS集群故障处理

7.1 volume bricks掉线

• 查看volume状态：

gluster volume status 

• 当Online列有为N的表示当前bricks不在线，登录brick所在主机查看brick挂载

df -h |grep 

• 若未查看到需要重新mount

cat /etc/fstab |grep  |xargs -i mount {}

• 重新启动掉线的bricks

gluster volume start  force

7.2 bricks文件不一致修复

• 查看bricks是否存在文件不一致

gluster volume heal  info

• 启动自动修复

gluster volume heal  full

7.3 bricks脑裂修复

gluster volume heal  info

• 若存在Is in split-brain内容则发生脑裂

1)  选择较大的文件作为源修复
gluster volume heal  split-brain bigger-file 
2)  选择以最新的mtime作为源的文件
gluster volume heal  split-brain latest-mtime 
3)  选择副本中的砖块之一作为特定文件的源
gluster volume heal  split-brain source-brick  
4)  选择副本的一个brick作为所有文件的源
gluster volume heal  split-brain source-brick 

7.4 更换brick

• 将源bricks数据同步到新的bricks路径

gluster volume replace-brick  Server1:/home/gfs/r2_0 Server1:/home/gfs/r2_5 start

• 在数据迁移的过程中，可以查看替换任务是否完成

gluster volume replace-brick  Server1:/home/gfs/r2_0 Server1:/home/gfs/r2_5 status

• 在数据迁移结束之后，执行commit命令结束任务，则进行Brick替换。使用volume info命令可以查看到Brick已经被替换

gluster volume replace-brick  Server1:/home/gfs/r2_0 Server1:/home/gfs/r2_5 commit

8， heketi服务故障处理

• 报错：

[heketi] ERROR 2018/07/02 09:08:19 /src/github.com/heketi/heketi/apps/glusterfs/app.go:172: Heketi was terminated while performing one or more operations. Server may refuse to start as long as pending operations are present in the db.
heketi服务无法启动

1) 导出heketi的heketi.db文件，文件的路径在heketi.json文件里面
heketi db export --dbfile=/var/lib/heketi/heketi.db --jsonfile=/tmp/heketidb1.json
2) 打开导出的db文件，比如上文的/tmp/heketidb1.json，查找```pendingoperations```选项，找到之后把与它相关的内容删除
3) 将修改后的文件保存，切记要保存为json后缀。然后将db文件再按照如下命令导入
heketi db import --jsonfile=/tmp/succ.json --dbfile=/var/lib/heketi/heketi.db
4) 重启heketi 服务
systemctl start heketi

8，参考文档

• https://docs.gluster.org/en/latest/
• https://www.ctolib.com/docs/sfile/kubernetes-handbook/practice/storage-for-containers-using-glusterfs-with-openshift.html