GlusterFS容量管理

说明

Heketi允许管理一个或多个GlusterFS集群来添加和删除存储容量。对于gluster的集群扩容以及device删除增加、volume新建增加都通过heketi来统一管理.

heketi创建cluster集群，统一管理gluster集群的node,device,volume。

• cluster: gluster node组成的集群
• node: glusterd实例
• device: 裸磁盘设备，供volume使用
• volume: 需要被使用的卷

容量扩容

使用Heketi可以通过多种方式添加额外的存储容量。可以添加新device，增加集群大小或添加新的集群。

Adding new devices

添加更多设备时，请记住将设备添加为一组。例如，如果创建的卷使用副本为2，则应将device添加到两个节点（每个节点一个device）。如果使用副本3，则将device添加到三个节点。

使用Heketi cli，可以将单个device添加到节点，命令如下：

$ heketi-cli device add \
      --name=/dev/vdc1
      --node=3e098cb4407d7109806bb196d9e8f095

当一次添加多个设备的一种更简单的方法是将新设备添加到用于设置群集的拓扑文件(topology.json)中的节点描述中。然后重新运行该命令以加载新拓扑。下面是我们向节点添加新的/dev/vdc2磁盘的示例

$ cat /data/heketi/conf/topology.json
{
    "clusters": [
        {
            "nodes": [
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "k8s1"
                            ],
                            "storage": [
                                "10.111.209.188"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/vdc1",
                        "/dev/vdc2"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "k8s2"
                            ],
                            "storage": [
                                "10.111.209.189"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/vdc1",
                        "/dev/vdc2"
                    ]
                },
                {
                    "node": {
                        "hostnames": {
                            "manage": [
                                "k8s3"
                            ],
                            "storage": [
                                "10.111.209.190"
                            ]
                        },
                        "zone": 1
                    },
                    "devices": [
                        "/dev/vdc1",
                        "/dev/vdc2"
                    ]
                }
            ]
        }
    ]
}

我们为每台机器分别添加了/dev/vdc2磁盘.

$ heketi-cli topology load --json=/data/heketi/conf/topology.json
        Found node k8s1 on cluster 6fd6bf78b84315e12abcf8b6db6b1a40
                Found device /dev/vdc1
                Adding device /dev/vdc2 ... OK
        Found node k8s2 on cluster 6fd6bf78b84315e12abcf8b6db6b1a40
                Found device /dev/vdc1
                Adding device /dev/vdc2 ... OK
        Found node k8s3 on cluster 6fd6bf78b84315e12abcf8b6db6b1a40
                Found device /dev/vdc1
                Adding device /dev/vdc2 ... OK

Increasing cluster size

除了向现有节点添加新设备之外，还可以将新节点添加到集群中。与device一样，可以通过使用API，使用cli或修改拓扑文件将新节点添加到现有集群。

命令如下：

$ heketi-cli node add \
      --zone=1 \
      --cluster=6fd6bf78b84315e12abcf8b6db6b1a40 \
      --management-host-name=k8s4 \
      --storage-host-name=10.111.209.191
 
Node information:
Id: e0017385b683c10e4166492e78832d09
State: online
Cluster Id: 6fd6bf78b84315e12abcf8b6db6b1a40
Zone: 1
Management Hostname k8s4
Storage Hostname 10.111.209.191
 
$ heketi-cli device add \
      --name=/dev/vdc1 \
      --node=e0017385b683c10e4166492e78832d09
Device added successfully
 
$ heketi-cli device add \
      --name=/dev/vdc2 \
      --node=e0017385b683c10e4166492e78832d09
Device added successfully

更简单的方法是为拓扑文件文件添加新节点。添加新节点时，必须在现有节点之后添加此节点信息，以便Heketi cli确定这个新节点应该属于哪个集群。

命令如下:

$ heketi-cli topology load --json=/data/heketi/conf/topology.json
        Found node k8s1 on cluster 6fd6bf78b84315e12abcf8b6db6b1a40
                Found device /dev/vdc1
                Found device /dev/vdc2
        Found node k8s2 on cluster 6fd6bf78b84315e12abcf8b6db6b1a40
                Found device /dev/vdc1
                Found device /dev/vdc2
        Found node k8s3 on cluster 6fd6bf78b84315e12abcf8b6db6b1a40
                Found device /dev/vdc1
                Found device /dev/vdc2
        Creating node 192.168.10.105 ... ID: be0e8f7fba6ec1e5aa0337141f356013
                Adding device /dev/vdc1 ... OK
                Adding device /dev/vdc2 ... OK

Adding a new cluster

通过添加新的GlusterFS集群也可以增加存储容量。与之前创建集群一样，可以直接使用API或者使用heketi-cli手动添加群集，节点和device，或创建另一个拓扑文件来定义将组成此集群的新节点和设备。

容量缩容

Heketi也支持降低存储容量。这可以通过删除device，节点和集群来实现。可以使用API或使用heketi-cli执行这些更改。以下是如何从Heketi删除没有device被使用的命令：

$ heketi-cli topology info
  
Cluster Id: 6fe4dcffb9e077007db17f737ed999fe
  
    Volumes:
  
    Nodes:
  
        Node Id: 61d019bb0f717e04ecddfefa5555bc41
        State: online
        Cluster Id: 6fe4dcffb9e077007db17f737ed999fe
        Zone: 1
        Management Hostname: gprfc053.o.internal
        Storage Hostname: 172.18.10.53
        Devices:
                Id:e4805400ffa45d6da503da19b26baad6   Name:/dev/sdc            State:online    Size (GiB):279     Used (GiB):0       Free (GiB):279
                        Bricks:
                Id:ecc3c65e4d22abf3980deba4ae90238c   Name:/dev/sdd            State:online    Size (GiB):279     Used (GiB):0       Free (GiB):279
                        Bricks:
  
        Node Id: e97d77d0191c26089376c78202ee2f20
        State: online
        Cluster Id: 6fe4dcffb9e077007db17f737ed999fe
        Zone: 2
        Management Hostname: gprfc054.o.internal
        Storage Hostname: 172.18.10.54
        Devices:
                Id:3dc3b3f0dfd749e8dc4ee98ed2cc4141   Name:/dev/sdd            State:online    Size (GiB):279     Used (GiB):0       Free (GiB):279
                        Bricks:
                Id:4122bdbbe28017944a44e42b06755b1c   Name:/dev/sdc            State:online    Size (GiB):279     Used (GiB):0       Free (GiB):279
                        Bricks:
                Id:b5333d93446565243f1a7413be45292a   Name:/dev/sdb            State:online    Size (GiB):279     Used (GiB):0       Free (GiB):279
                        Bricks:
$
$ d=`heketi-cli topology info | grep Size | awk '{print $1}' | cut -d: -f 2`
$ for i in $d ; do
> heketi-cli device delete $i
> done
Device e4805400ffa45d6da503da19b26baad6 deleted
Device ecc3c65e4d22abf3980deba4ae90238c deleted
Device 3dc3b3f0dfd749e8dc4ee98ed2cc4141 deleted
Device 4122bdbbe28017944a44e42b06755b1c deleted
Device b5333d93446565243f1a7413be45292a deleted
$ heketi-cli node delete $node1
Node 61d019bb0f717e04ecddfefa5555bc41 deleted
$ heketi-cli node delete $node2
Node e97d77d0191c26089376c78202ee2f20 deleted
$ heketi-cli cluster delete $cluster
Cluster 6fe4dcffb9e077007db17f737ed999fe deleted