CEPH块存储管理

一、检查CEPH和集群各参数

1、检查ceph安装状态：

命令：ceph –s 或者 ceph status(显示结果一样)

示例：

root@node1:/home/ceph/ceph-cluster# ceph -s

       cluster 2f54214e-b6aa-44a4-910c-52442795b037

       health HEALTH_OK

       monmap e1: 1 mons at {node1=192.168.2.13:6789/0}, election epoch 1,quorum 0 node1

       osdmap e56: 5 osds: 5 up, 5 in

       pgmap v289: 192 pgs, 3 pools, 70988 kB data, 26 objects

           376 MB used, 134 GB / 134 GB avail

                 192 active+clean

2、检查集群健康状态：

命令：ceph –w

示例：

root@node1:/home/ceph/ceph-cluster# ceph -w

       cluster 2f54214e-b6aa-44a4-910c-52442795b037

       health HEALTH_OK

       monmap e1: 1 mons at {node1=192.168.2.13:6789/0}, election epoch 1,quorum 0 node1

       osdmap e56: 5 osds: 5 up, 5 in

       pgmap v289: 192 pgs, 3 pools, 70988 kB data, 26 objects

           376 MB used, 134 GB / 134 GB avail

                 192 active+clean

2016-09-08 09:40:18.084097 mon.0 [INF] pgmap v323: 192 pgs: 192active+clean; 8 bytes data, 182 MB used, 134 GB / 134 GB avail

3、检查cephmonitor仲裁状态

命令：ceph quorum_status --formatjson-pretty

示例：

root@node1:/home/ceph/ceph-cluster# ceph quorum_status --formatjson-pretty

{ "election_epoch": 1,

       "quorum": [

         0],

       "quorum_names": [

       "node1"],

      "quorum_leader_name": "node1",

      "monmap": { "epoch": 1,

      "fsid": "2f54214e-b6aa-44a4-910c-52442795b037",

      "modified": "0.000000",

      "created": "0.000000",

      "mons": [

           { "rank": 0,

              "name":"node1",

              "addr":"192.168.2.13:6789\/0"}]}}

4、导出cephmonitor信息

命令：ceph mon dump

示例：

root@node1:/home/ceph/ceph-cluster# ceph mon dump

dumped monmap epoch 1

epoch 1

fsid 2f54214e-b6aa-44a4-910c-52442795b037

last_changed 0.000000

created 0.000000

0: 192.168.2.13:6789/0 mon.node1

5、检查集群使用状态

    命令：ceph df

    示例：

root@node1:/home/ceph/ceph-cluster# ceph df

GLOBAL:

       SIZE     AVAIL     RAW USED     %RAW USED

       134G      134G         376M          0.27

POOLS:

       NAME         ID     USED      %USED     MAX AVAIL     OBJECTS

       data         0           0         0        45882M           0

       metadata     1           0         0        45882M           0

rbd          2      70988k      0.05        45882M          26

6、检查ceph monitor、osd和pg(配置组)状态

    命令：ceph mon stat、ceph osd stat、ceph pg stat

示例：

root@node1:/home/ceph/ceph-cluster# ceph mon stat

e1: 1 mons at {node1=192.168.2.13:6789/0}, election epoch 1, quorum0 node1

root@node1:/home/ceph/ceph-cluster# ceph osd stat

       osdmap e56: 5 osds: 5 up, 5 in

root@node1:/home/ceph/ceph-cluster# ceph pg stat

v289: 192 pgs: 192 active+clean; 70988 kB data, 376 MB used, 134 GB/ 134 GB avail

7、列表PG

    命令：ceph pg dump

示例：

root@node1:/home/ceph/ceph-cluster# ceph pg dump

dumped all in format plain

version 289

stamp 2016-09-08 08:44:35.249418

last_osdmap_epoch 56

last_pg_scan 1

full_ratio 0.95

nearfull_ratio 0.85

                             ……………

8、列表ceph存储池

    命令：ceph osd lspools

示例：

root@node1:/home/ceph/ceph-cluster# ceph osd lspools

0 data,1 metadata,2 rbd,

9、检查OSD的CRUSH map

    命令：ceph osd tree

示例：

root@node1:/home/ceph/ceph-cluster# ceph osd tree

# id      weight   type name     up/down       reweight

-1  0.15       root default

-2  0.06              host node2

0   0.03                     osd.0      up   1    

3   0.03                     osd.3      up   1    

-3  0.06              host node3

1   0.03                     osd.1      up   1    

4   0.03                     osd.4      up   1    

-4  0.03              host node1

2   0.03                     osd.2      up   1    

10、列表群集的认证秘钥：

    命令：ceph auth list

示例：

root@node1:/home/ceph/ceph-cluster# ceph auth list

installed auth entries:

osd.0

              key:AQCM089X8OHnIhAAnOnRZMuyHVcXa6cnbU2kCw==

              caps: [mon] allow profile osd

              caps: [osd] allow *

osd.1

              key:AQCU089X0KSCIRAAZ3sAKh+Fb1EYV/ROkBd5mA==

              caps: [mon] allow profile osd

              caps: [osd] allow *

osd.2

              key:AQAb1c9XWIuxEBAA3PredgloaENDaCIppxYTbw==

              caps: [mon] allow profile osd

              caps: [osd] allow *

osd.3

              key:AQBF1c9XuBOpMBAAx8ELjaH0b1qwqKNwM17flA==

              caps: [mon] allow profile osd

              caps: [osd] allow *

osd.4

              key:AQBc1c9X4LXCEBAAcq7UVTayMo/e5LBykmZZKg==

              caps: [mon] allow profile osd

              caps: [osd] allow *

client.admin

              key:AQAd089XMI14FRAAdcm/woybc8fEA6dH38AS6g==

              caps: [mds] allow

              caps: [mon] allow *

              caps: [osd] allow *

client.bootstrap-mds

              key:AQAd089X+GahIhAAgC+1MH1v0enAGzKZKUfblg==

              caps: [mon] allow profile bootstrap-mds

client.bootstrap-osd

              key: AQAd089X8B5wHBAAnrM0MQK3to1iBitDzk+LYA==

              caps: [mon] allow profile bootstrap-osd

二、块存储高级管理

1、创建块设备

命令：rbd create {image-name} --size{megabytes} --pool {pool-name} --image-format 2

注意：--image-format 2 用于指定format类型为2，不加则默认为1类型，保护快照功能仅支持2类型。1类型为淘汰类型，一般用2类型，这里演示用

示例：

root@node1:/home/ceph/ceph-cluster# rbd create zhangbo --size 2048--pool rbd

2、列出块设备

命令：rbd ls {pool-name}

示例：

root@node1:/home/ceph/ceph-cluster# rbd ls rbd

zhangbo

3、检索块信息

命令：rbd –image {image-name } info

rbd info {image-name}

示例：

root@node1:/home/ceph/ceph-cluster# rbd --image zhangbo  info

rbd image 'zhangbo':

               size 2048 MB in 512 objects

               order 22 (4096 kB objects)

               block_name_prefix: rb.0.5e56.2ae8944a

               format: 1

root@node1:/home/ceph/ceph-cluster# rbd info zhangbo

rbd image 'zhangbo':

               size 2048 MB in 512 objects

               order 22 (4096 kB objects)

               block_name_prefix: rb.0.5e56.2ae8944a

               format: 1

4、更改块大小

命令：rbd resize –image {image-name}–size {megabytes}

示例：

root@node1:/home/ceph/ceph-cluster# rbd resize --image zhangbo--size 4096

Resizing image: 100% complete...done.

root@node1:/home/ceph/ceph-cluster# rbd info zhangbo

rbd image 'zhangbo':

               size 4096 MB in 1024 objects

               order 22 (4096 kB objects)

               block_name_prefix: rb.0.5e56.2ae8944a

               format: 1

5、删除块设备

命令：rbd rm {image-name}

示例：

root@node1:/home/ceph/ceph-cluster# rbd rm zhangbo

Removing image: 100% complete...done.

root@node1:/home/ceph/ceph-cluster# rbd ls

6、映射块设备：

命令：rbd map {image-name} –pool{pool-name} –id {user-name}

示例：

root@node1:/home/ceph/ceph-cluster# rbd map zhangbo --pool rbd --idadmin

7、查看已映射块设备

命令：rbd showmapped

示例：

root@node1:/home/ceph/ceph-cluster# rbd showmapped

id pool image   snapdevice   

0  rbd  zhangbo -   /dev/rbd0

8、取消映射：

命令：rbd unmap/dev/rbd/{pool-name}/{image-name}

示例：

root@node1:/home/ceph/ceph-cluster# rbd unmap /dev/rbd/rbd/zhangbo

root@node1:/home/ceph/ceph-cluster# rbd showmapped

9、格式化：

命令：mkfs.ext4 /dev/rbd0

示例：

root@node1:/home/ceph/ceph-cluster# mkfs.ext4 /dev/rbd0

mke2fs 1.42.9 (4-Feb-2014)

Discarding device blocks: 完成                           

文件系统标签=

OS type: Linux

块大小=4096 (log=2)

分块大小=4096 (log=2)

Stride=1024 blocks, Stripe width=1024 blocks

262144 inodes, 1048576 blocks

52428 blocks (5.00%) reserved for the super user

第一个数据块=0

Maximum filesystem blocks=1073741824

32 block groups

32768 blocks per group, 32768 fragments per group

8192 inodes per group

Superblock backups stored on blocks:

            32768, 98304, 163840, 229376, 294912,819200, 884736

Allocating group tables: 完成                           

正在写入inode表: 完成                           

Creating journal (32768 blocks): 完成

Writing superblocks and filesystem accounting information: 完成

10、挂载

命令：mount /dev/rbd0 /mnt/{文件夹名}

示例：

root@node1:/home/ceph/ceph-cluster# mount /dev/rbd0/mnt/ceph-zhangbo/

root@node1:/home/ceph/ceph-cluster# df -h

文件系统        容量  已用 可用已用% 挂载点

udev            989M  4.0K 989M    1% /dev

tmpfs           201M  1.1M 200M    1% /run

/dev/sda5        19G  4.0G  14G   23% /

none            4.0K     0 4.0K    0% /sys/fs/cgroup

none            5.0M     0 5.0M    0% /run/lock

none           1001M   76K 1001M   1% /run/shm

none            100M   32K 100M    1% /run/user

/dev/sda1       9.3G   60M 8.8G    1% /boot

/dev/sda6        19G  67M   18G    1% /home

/dev/sdc1        27G  169M  27G    1% /var/lib/ceph/osd/ceph-2

/dev/rbd0       3.9G  8.0M 3.6G    1% /mnt/ceph-zhangbo

11、设置开机自动挂载（开机CEPH自动map和mount rbd块设备）

vim /etc/ceph/rbdmap

{poolname}/{imagename} id=client,keyring=/etc/ceph/ceph.client.keyring

rbd/zhangbo id=admin,keyring=/etc/ceph/ceph.client.admin.keyring

vim /etc/fstab

/dev/rbd/rbd/zhangbo /mnt/ceph-zhangbo xfs defaults,noatime,_netdev

12、块扩容

命令：rbd resize rbd/zhangbo –size 4096

rbd resize –image zhangbo –size 4096

支持文件系统在线扩容：resize2fs /dev/rbd0

13、使用块设备完整操作流程：

1、rbd createzhangbo --size 2048 --pool rbd

2、rbd mapzhangbo --pool rbd --id admin

3、mkfs.ext4/dev/rbd0

4、mount/dev/rbd0 /mnt/ceph-zhangbo/

5、设置开机自动挂载

6、文件系统在线扩容

rbd resize rbd/zhangbo --size 2048

resize2fs /dev/rbd0

7、umount/mnt/ceph-zhangbo

8、rbd unmap/dev/rbd/rbd/zhangbo

9、删除开机自动挂载添加的内容

10、rbd rmzhangbo

三、快照和克隆

1、创建快照：

命令：rbd –pool {pool-name} snapcreate –snap {snap-name} {image-name}

rbd snap create{pool-name}/{image-name}@{snap-name}

        示例：

root@node1:~# rbdsnap create rbd/zhangbo@zhangbo_snap

root@node1:~# rbdsnap ls rbd/zhangbo

SNAPID NAME            SIZE

           2zhangbo_snap 1024 MB

2、快照回滚

命令：rbd –pool {pool-name}snap sellback –snap {snap-name} {iname-name}

rbd snap rollback{pool-name}/{image-name}@{snap-name}

         示例：

root@node1:~# rbdsnap rollback rbd/zhangbo@zhangbo_snap

Rolling back tosnapshot: 100% complete...done.

3、清除快照（删除该块设备下所有的快照）

命令：rbd –pool{pool-name} snap purge {image-name}

rbd snap purge{pool-name}/{image-name}

        示例：

root@node1:~# rbdsnap ls rbd/zhangbo

SNAPID NAME             SIZE

          2zhangbo_snap  1024 MB

          3zhangbo_snap1 1024 MB

          4zhangbo_snap2 1024 MB

          5zhangbo_snap3 1024 MB

root@node1:~# rbdsnap purge rbd/zhangbo

Removing allsnapshots: 100% complete...done.

root@node1:~# rbdsnap ls rbd/zhangbo

root@node1:~#

4、删除快照（删除指定快照）

命令：rbd snap rm{pool-name}/{image-name}@(snap-name)

示例：

root@node1:~# rbdsnap ls rbd/zhangbo

SNAPID NAME             SIZE

           10 zhangbo_snap1 1024 MB

           11 zhangbo_snap2 1024 MB

           12 zhangbo_snap3 1024 MB

root@node1:~# rbdsnap rm rbd/zhangbo@zhangbo_snap2

root@node1:~# rbdsnap ls rbd/zhangbo

SNAPID NAME             SIZE

           10 zhangbo_snap1 1024 MB

12 zhangbo_snap31024 MB

5、列出快照：

命令：rbd –pool{pool-name} snap ls {image-name}

rbd snap ls{pool-name}/{image-name}

        示例：

root@node1:~# rbdsnap ls rbd/zhangbo

SNAPID NAME             SIZE

        16 zhangbo_snap1 1024 MB

        17 zhangbo_snap2 1024 MB

18 zhangbo_snap31024 MB

6、保护快照：

命令：rbd –pool {pool-name}snap pretect –image {image-name} –snap {snapshot-name}

rbd snap pretect{pool-name}/{image-name}@{snapshot-name}

        示例：

root@node1:~# rbdsnap protect rbd/zhangbo@zhangbo_snap2

root@node1:~# rbdsnap rm rbd/zhangbo@zhangbo_snap2

rbd: snapshot'zhangbo_snap2' is protected from removal.

2016-09-0814:05:03.874498 7f35bddad7c0 -1 librbd: removing snapshot from header failed:(16) Device or resource busy

7、取消保护快照

命令：rbd –pool{pool-name} snap unprotect –image {image-name} –snap {snapshot-name}

rbd snapunprotect {pool-name}/{image-name}@{snapshot-name}

        示例：

root@node1:~# rbdsnap unprotect rbd/zhangbo@zhangbo_snap2

root@node1:~# rbdsnap rm rbd/zhangbo@zhangbo_snap2

root@node1:~# rbdsnap ls rbd/zhangbo

SNAPID NAME             SIZE

        22 zhangbo_snap1 1024 MB

        24 zhangbo_snap3 1024 MB

8、快照克隆（需快照保护才能快照克隆）

注意：快照是只读的，而克隆是基于快照可读写的

命令：rbd clone{pool-name}/{parent-image}@{snap-name} {pool-name}/{child-image-name}

示例：

root@node1:~# rbdclone rbd/zhangbo@zhangbo_snap2 rbd/zhangbo-snap-clone

root@node1:~# rbdls

zhangbo

zhangbo-snap-clone

9、创建分层快照和克隆

命令：rbd createzhangbo --size 1024 --image-format 2

rbd snap create{pool-name}/{image-name}@{snap-name}

rbd snap pretect{pool-name}/{image-name}@{snapshot-name}

rbd clone{pool-name}/{parent-image}@{snap-name} {pool-name}/{child-image-name}

10、查看快照的克隆：

命令：rbd --pool{pool-name} children --image {image-name} --snap {snap-name}

rbd children{pool-name}/{image-name}@{snapshot-name}

        示例：

root@node1:~# rbdchildren rbd/zhangbo@zhangbo_snap2

rbd/zhangbo-snap-clone