ceph使用bluestore部署集群
首先安装 ceph的版本
# ceph -v
ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
对磁盘进行分区操作
# parted /dev/vdb -s mklabel gpt
# parted /dev/vdb -s mkpart primary 1 10G
# parted /dev/vdb -s mkpart primary 10G 15G
# parted /dev/vdb -s mkpart primary 15G 20G
# parted /dev/vdc -s mklabel gpt
# # parted /dev/vdc -s mkpart primary 1 10G
# parted /dev/vdc -s mkpart primary 10G 15G
# parted /dev/vdc -s mkpart primary 15G 20G
创建ceph集群的步骤
| uuidgen mkdir -p /data/mon/mon.0 monmaptool --create --add 0 127.0.0.1 --fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 /tmp/monmap ceph-mon -i 0 -c /etc/ceph/ceph.conf --mkfs --monmap /tmp/monmap --mon-data /data/mon/mon.0 --debug_mon 10 --fsid=a7f64266-0894-4f1e-a635-d0aeaca0e993 ceph-mon -i 0 ceph osd create mkdir -p /data/osd/osd.0 ceph-osd -i 0 --mkfs --osd-data=/data/osd/osd.0 -c /etc/ceph/ceph.conf --debug_osd 20 --mkkey ceph-osd -i 0 ceph osd create mkdir -p /data/osd/osd.1 ceph-osd -i 1 --mkfs --osd-data=/data/osd/osd.1 -c /etc/ceph/ceph.conf --debug_osd 20 --mkkey ceph-osd -i 1 ceph osd crush add-bucket unkownrack rack ceph osd tree ceph osd crush add-bucket host0 host ceph osd crush add-bucket host1 host ceph osd crush move host0 rack=unkownrack ceph osd crush move host1 rack=unkownrack ceph osd crush move unkownrack root=default ceph osd crush create-or-move osd.0 1.0 host=host0 rack=unkownrack root=default ceph osd crush create-or-move osd.1 1.0 host=host1 rack=unkownrack root=default |
ceph.conf的配置文件
| [global] max open files = 131072 log file = /var/log/ceph/ceph-$name.log pid file = /var/run/ceph/$name.pid auth supported = none auth cluster required = none auth service required = none auth client required = none osd pool default size = 1 enable experimental unrecoverable data corrupting features = bluestore rocksdb [mon] debug mon = 10 debug paxos = 0 mon data = /data/mon/mon.$id [mon.0] host = 127.0.0.1 mon addr = 127.0.0.1:6789 [osd] debug osd = 10 debug bluestore = 30 debug bluefs = 30 debug rocksdb = 30 debug bdev = 30 bluestore = true osd objectstore = bluestore bluestore fsck on mount = true bluestore block create = true bluestore block db size = 67108864 bluestore block db create = true bluestore block wal size = 134217728 bluestore block wal create = true [osd.0] host = 127.0.0.1 osd data = /data/osd/osd.0 bluestore block path = /dev/vdb1 bluestore block db path = /dev/vdb2 bluestore block wal path = /dev/vdb3 [osd.1] host = 127.0.0.1 osd data = /data/osd/osd.1 bluestore block path = /dev/vdc1 bluestore block db path = /dev/vdc2 bluestore block wal path = /dev/vdc3 |
# ceph -s
2016-09-09 08:02:10.477039 7fb9f7ce0700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-09-09 08:02:10.511301 7fb9f7ce0700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
cluster a7f64266-0894-4f1e-a635-d0aeaca0e993
health HEALTH_OK
monmap e1: 1 mons at {0=127.0.0.1:6789/0}
election epoch 3, quorum 0 0
osdmap e15: 2 osds: 2 up, 2 in
flags sortbitwise
pgmap v23: 64 pgs, 1 pools, 0 bytes data, 0 objects
2053 MB used, 17018 MB / 19072 MB avail
64 active+clean
创建过程中日志记录如下
| # mkdir -p /data/mon/mon.0 # monmaptool --create --add 0 127.0.0.1 --fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 /tmp/monmap monmaptool: monmap file /tmp/monmap monmaptool: set fsid to a7f64266-0894-4f1e-a635-d0aeaca0e993 monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) # ceph-mon -i 0 -c /etc/ceph/ceph.conf --mkfs --monmap /tmp/monmap --mon-data /data/mon/mon.0 --debug_mon 10 --fsid=a7f64266-0894-4f1e-a635-d0aeaca0e993 2016-09-09 07:53:34.564661 7fcb43dc14c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:53:34.565049 7fcb43dc14c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb ceph-mon: set fsid to a7f64266-0894-4f1e-a635-d0aeaca0e993 ceph-mon: created monfs at /data/mon/mon.0 for mon.0 # ceph-mon -i 0 2016-09-09 07:53:37.271989 7fec0bbef4c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:53:37.272325 7fec0bbef4c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:53:37.306821 7fec0bbef4c0 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb # # # ps aux | grep ceph root 7774 1.0 0.0 323128 15860 pts/0 Sl 07:53 0:00 ceph-mon -i 0 root 7836 0.0 0.0 112648 972 pts/0 R+ 07:53 0:00 grep --color=auto ceph # ceph -s 2016-09-09 07:53:45.908409 7fcca0282700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:53:45.944873 7fcca0282700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb cluster a7f64266-0894-4f1e-a635-d0aeaca0e993 health HEALTH_ERR no osds monmap e1: 1 mons at {0=127.0.0.1:6789/0} election epoch 3, quorum 0 0 osdmap e1: 0 osds: 0 up, 0 in flags sortbitwise pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects 0 kB used, 0 kB / 0 kB avail 64 creating # ceph osd create 2016-09-09 07:54:24.998055 7f528fcf1700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:54:25.031318 7f528fcf1700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 0 # mkdir -p /data/osd/osd.0 # ceph-osd -i 0 --mkfs --osd-data=/data/osd/osd.0 -c /etc/ceph/ceph.conf --debug_osd 20 --mkkey 2016-09-09 07:54:25.454949 7f1018023800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:54:25.455309 7f1018023800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:54:25.455391 7f1018023800 -1 WARNING: experimental feature 'bluestore' is enabled Please be aware that this feature is experimental, untested, unsupported, and may result in data corruption, data loss, and/or irreparable damage to your cluster. Do not use feature with important data. 2016-09-09 07:54:25.489072 7f1018023800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:54:25.516087 7f1018023800 -1 bluestore(/data/osd/osd.0) _read_fsid unparsable uuid 2016-09-09 07:54:26.322530 7f1018023800 -1 WARNING: experimental feature 'rocksdb' is enabled Please be aware that this feature is experimental, untested, unsupported, and may result in data corruption, data loss, and/or irreparable damage to your cluster. Do not use feature with important data. 2016-09-09 07:54:27.783304 7f1018023800 -1 WARNING: experimental feature 'rocksdb' is enabled Please be aware that this feature is experimental, untested, unsupported, and may result in data corruption, data loss, and/or irreparable damage to your cluster. Do not use feature with important data. 2016-09-09 07:54:29.592010 7f1018023800 -1 WARNING: experimental feature 'rocksdb' is enabled Please be aware that this feature is experimental, untested, unsupported, and may result in data corruption, data loss, and/or irreparable damage to your cluster. Do not use feature with important data. 2016-09-09 07:54:30.982027 7f1018023800 -1 created object store /data/osd/osd.0 for osd.0 fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 2016-09-09 07:54:30.982160 7f1018023800 -1 auth: error reading file: /data/osd/osd.0/keyring: can't open /data/osd/osd.0/keyring: (2) No such file or directory 2016-09-09 07:54:30.982553 7f1018023800 -1 created new key in keyring /data/osd/osd.0/keyring # ceph-osd -i 0 starting osd.0 at :/0 osd_data /data/osd/osd.0 /var/lib/ceph/osd/ceph-0/journal 2016-09-09 07:54:31.009434 7f11aa3e3800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:54:31.009552 7f11aa3e3800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:54:31.009654 7f11aa3e3800 -1 WARNING: experimental feature 'bluestore' is enabled Please be aware that this feature is experimental, untested, unsupported, and may result in data corruption, data loss, and/or irreparable damage to your cluster. Do not use feature with important data. # # ceph osd create 2016-09-09 07:54:31.131614 7fe2ec2af700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:54:31.165197 7fe2ec2af700 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 1 # mkdir -p /data/osd/osd.1 # ceph-osd -i 1 --mkfs --osd-data=/data/osd/osd.1 -c /etc/ceph/ceph.conf --debug_osd 20 --mkkey 2016-09-09 07:54:31.517213 7fe5e4f58800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:54:31.517578 7fe5e4f58800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:54:31.517653 7fe5e4f58800 -1 WARNING: experimental feature 'bluestore' is enabled Please be aware that this feature is experimental, untested, unsupported, and may result in data corruption, data loss, and/or irreparable damage to your cluster. Do not use feature with important data. 2016-09-09 07:54:31.550260 7fe5e4f58800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:54:31.615795 7fe5e4f58800 -1 bluestore(/data/osd/osd.1) _read_fsid unparsable uuid 2016-09-09 07:54:32.520344 7fe5e4f58800 -1 WARNING: experimental feature 'rocksdb' is enabled Please be aware that this feature is experimental, untested, unsupported, and may result in data corruption, data loss, and/or irreparable damage to your cluster. Do not use feature with important data. 2016-09-09 07:54:34.426150 7fe5e4f58800 -1 WARNING: experimental feature 'rocksdb' is enabled Please be aware that this feature is experimental, untested, unsupported, and may result in data corruption, data loss, and/or irreparable damage to your cluster. Do not use feature with important data. 2016-09-09 07:54:35.645626 7fe5e4f58800 -1 WARNING: experimental feature 'rocksdb' is enabled Please be aware that this feature is experimental, untested, unsupported, and may result in data corruption, data loss, and/or irreparable damage to your cluster. Do not use feature with important data. 2016-09-09 07:54:36.547734 7fe5e4f58800 -1 created object store /data/osd/osd.1 for osd.1 fsid a7f64266-0894-4f1e-a635-d0aeaca0e993 2016-09-09 07:54:36.547911 7fe5e4f58800 -1 auth: error reading file: /data/osd/osd.1/keyring: can't open /data/osd/osd.1/keyring: (2) No such file or directory 2016-09-09 07:54:36.548344 7fe5e4f58800 -1 created new key in keyring /data/osd/osd.1/keyring # ceph-osd -i 1 2016-09-09 07:54:40.320869 7fd5bad20800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:54:40.321260 7fd5bad20800 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb 2016-09-09 07:54:40.321385 7fd5bad20800 -1 WARNING: experimental feature 'bluestore' is enabled Please be aware that this feature is experimental, untested, unsupported, and may result in data corruption, data loss, and/or irreparable damage to your cluster. Do not use feature with important data. starting osd.1 at :/0 osd_data /data/osd/osd.1 /var/lib/ceph/osd/ceph-1/journal |
验证环境的正确性
# hexdump -C /dev/vdb1 | more
00000000 62 6c 75 65 73 74 6f 72 65 20 62 6c 6f 63 6b 20 |bluestore block |
00000010 64 65 76 69 63 65 0a 65 33 63 38 36 37 38 32 2d |device.e3c86782-|
00000020 30 34 30 63 2d 34 61 36 62 2d 62 31 38 63 2d 33 |040c-4a6b-b18c-3|
00000030 61 31 36 63 38 31 35 32 38 39 31 0a 01 01 28 00 |a16c8152891...(.|
00000040 00 00 e3 c8 67 82 04 0c 4a 6b b1 8c 3a 16 c8 15 |....g...Jk..:...|
00000050 28 91 00 00 00 54 02 00 00 00 31 fa d1 57 02 db |(....T....1..W..|
00000060 dd 1e 04 00 00 00 6d 61 69 6e 21 e3 31 19 00 00 |......main!.1...|
00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
[root@localhost osd.0]# ll
total 40
lrwxrwxrwx 1 root root 9 Sep 9 07:54 block -> /dev/vdb1
lrwxrwxrwx 1 root root 9 Sep 9 07:54 block.db -> /dev/vdb2
lrwxrwxrwx 1 root root 9 Sep 9 07:54 block.wal -> /dev/vdb3
-rw-r--r-- 1 root root 2 Sep 9 07:54 bluefs
-rw-r--r-- 1 root root 37 Sep 9 07:54 ceph_fsid
-rw-r--r-- 1 root root 37 Sep 9 07:54 fsid
-rw------- 1 root root 56 Sep 9 07:54 keyring
-rw-r--r-- 1 root root 8 Sep 9 07:54 kv_backend
-rw-r--r-- 1 root root 21 Sep 9 07:54 magic
-rw-r--r-- 1 root root 4 Sep 9 07:54 mkfs_done
-rw-r--r-- 1 root root 6 Sep 9 07:54 ready
-rw-r--r-- 1 root root 10 Sep 9 07:54 type
-rw-r--r-- 1 root root 2 Sep 9 07:54 whoami