openstack 与 ceph (osd 部署)
当 monitor 运行后, 你需要添加 OSD, 你的集群只有在获得足够的 OSD 数量用于复制对象时候才可能获得 active + clean 状态
例如 osd pool size = 2, 那么至少需要 2 个 OSD,
在启动 MONITOR 后,你的集群具有默认的 CURSH MAP, 当前 CURSH MAP 并没有 CEPH OSD 进程映射到 CEPH 节点
CEPH 提供 CEPH-DISK 工具, 可以准备一次磁盘, 分区或者目录用于 CEPH, ceph-disk 工具通过增加 INDEX 创建 OSD ID, 另外 ceph-disk
将会增加新的 OSD 到 CURSH MAP
目标
为每个电脑中的每个独立的磁盘都创建一个对应的 OSD, 当前数据备份副本数量为 31. 系统初始化
安装 Centos 7.1 版本操作系统
在 /etc/hosts 定义集群中所有主机名及 ip 地址
保证时间同步
确保 iptables, selinux 都处于关闭状态2. 准备磁盘
把打算用于创建 ceph 存储的磁盘进行分区, 并执行格式化, 参考该脚本, 并在每个节点上执行
#!/bin/bash
LANG=en_US
disk=`fdisk -l | grep ^Disk | grep sectors | grep sd | grep -v sda | awk -F[:\ ] '{print $2}' | sort`
yum install -y hdparm
for partition in $disk
do
dd if=/dev/zero of=$partition bs=1M count=100
parted -s $partition mklabel gpt
parted $partition mkpart primary xfs 1 100%
hdparm -z "$partition"1
mkfs.xfs -f -i size=512 "$partition"1
done3. 创建osd
每个独立的物理硬盘, 都已经完成格式化,
我们修改 /etc/fstab 把磁盘挂载到对应的目录中, 成为 CEPH 集群存储中的一部分
我们都会为每个独立的 磁盘创建一个独立的 OSD 与其对应,
#!/bin/bash
LANG=en_US
num=0
for ip in 240.30.128.55 240.30.128.56 240.30.128.57 240.30.128.73 240.30.128.74 240.30.128.75 240.30.128.76
do
diskpart=`ssh $ip "fdisk -l | grep GPT | grep -v sda" | awk '{print $1}' | sort`
for partition in $diskpart
do
ssh $ip "ceph osd create"
ssh $ip "mkdir /var/lib/ceph/osd/ceph-$num"
ssh $ip "echo $partition /var/lib/ceph/osd/ceph-$num xfs defaults 0 0 >> /etc/fstab"
let num++
done
ssh $ip "mount -a"
done重启, 验证挂载是否正常
4. ceph 配置文件
[global]
fsid = dc4f91c1-8792-4948-b68f-2fcea75f53b9
mon initial members = hh-yun-ceph-cinder015-128055, hh-yun-ceph-cinder017-128057, hh-yun-ceph-cinder024-128074
mon host = 10.199.128.55, 10.199.128.57, 10.199.128.74
public network = 192.168.209.0/24
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd journal size = 1024
filestore xattr use omap = true
osd pool default size = 2
osd pool default min size = 1
osd pool default pg num = 333
osd pool default pgp num = 333
osd crush chooseleaf type = 15. 初始化 OSD 数据目录
#!/bin/bash LANG=en_US num=0 for ip in 240.30.128.55 240.30.128.56 240.30.128.57 240.30.128.73 240.30.128.74 240.30.128.75 240.30.128.76 do diskpart=`ssh $ip "fdisk -l | grep GPT | grep -v sda" | awk '{print $1}' | sort` for partition in $diskpart do ssh $ip "ceph-osd -i $num --mkfs --mkkey --osd-uuid dc4f91c1-8792-4948-b68f-2fcea75f53b9" let num++ done done检测结果
[root@hh-yun-ceph-cinder015-128055 tmp]# ls /var/lib/ceph/osd/ceph*
/var/lib/ceph/osd/ceph-0:
ceph_fsid current fsid journal keyring magic ready store_version superblock whoami
/var/lib/ceph/osd/ceph-1:
ceph_fsid current fsid journal keyring magic ready store_version superblock whoami
/var/lib/ceph/osd/ceph-2:
ceph_fsid current fsid journal keyring magic ready store_version superblock whoami
/var/lib/ceph/osd/ceph-3:
ceph_fsid current fsid journal keyring magic ready store_version superblock whoami
/var/lib/ceph/osd/ceph-4:
ceph_fsid current fsid journal keyring magic ready store_version superblock whoami
/var/lib/ceph/osd/ceph-5:
ceph_fsid current fsid journal keyring magic ready store_version superblock whoami
/var/lib/ceph/osd/ceph-6:
ceph_fsid current fsid journal keyring magic ready store_version superblock whoami
/var/lib/ceph/osd/ceph-7:
ceph_fsid current fsid journal keyring magic ready store_version superblock whoami
/var/lib/ceph/osd/ceph-8:
ceph_fsid current fsid journal keyring magic ready store_version superblock whoami
/var/lib/ceph/osd/ceph-9:
ceph_fsid current fsid journal keyring magic ready store_version superblock whoami6. 注册 OSD 认证密钥
#!/bin/bash
LANG=en_US
num=0
for ip in 240.30.128.55 240.30.128.56 240.30.128.57 240.30.128.73 240.30.128.74 240.30.128.75 240.30.128.76
do
diskpart=`ssh $ip "fdisk -l | grep GPT | grep -v sda" | awk '{print $1}' | sort`
for partition in $diskpart
do
ssh $ip "ceph auth add osd.$num osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-$num/keyring"
let num++
done
done参考执行结果
[root@hh-yun-ceph-cinder015-128055 tmp]# ./authosd.sh
added key for osd.0
added key for osd.1
added key for osd.2
added key for osd.3
added key for osd.4
added key for osd.5
added key for osd.6
added key for osd.7
added key for osd.8
added key for osd.9
added key for osd.10
added key for osd.11
added key for osd.12
added key for osd.13
added key for osd.14
added key for osd.15
added key for osd.16
added key for osd.17
added key for osd.18
added key for osd.19
added key for osd.20
added key for osd.21
added key for osd.22
added key for osd.23
added key for osd.24
added key for osd.25
added key for osd.26
added key for osd.27
added key for osd.28
added key for osd.29
added key for osd.30
added key for osd.31
added key for osd.32
added key for osd.33
added key for osd.34
added key for osd.35
added key for osd.36
added key for osd.37
added key for osd.38
added key for osd.39
added key for osd.40
added key for osd.41
added key for osd.42
added key for osd.43
added key for osd.44
added key for osd.45
added key for osd.46
added key for osd.47
added key for osd.48
added key for osd.49
added key for osd.50
added key for osd.51
added key for osd.52
added key for osd.53
added key for osd.54
added key for osd.55
added key for osd.56
added key for osd.57
added key for osd.58
added key for osd.59
added key for osd.60
added key for osd.61
added key for osd.62
added key for osd.63
added key for osd.64
added key for osd.65
added key for osd.66
added key for osd.67
added key for osd.68
added key for osd.697. 添加 ceph 节点到 CURSH MAP 中, 并把 ceph 节点放置 ROOT 节点下
#!/bin/bash
for host in hh-yun-ceph-cinder015-128055 hh-yun-ceph-cinder016-128056 hh-yun-ceph-cinder017-128057 hh-yun-ceph-cinder023-128073 hh-yun-ceph-cinder024-128074 hh-yun-ceph-cinder025-128075 hh-yun-ceph-cinder026-128076
do
ceph osd crush add-bucket $host host
ceph osd crush move $host root=default
done参考执行结果
[root@hh-yun-ceph-cinder015-128055 tmp]# ./hostmap.sh
added bucket hh-yun-ceph-cinder015-128055 type host to crush map
moved item id -2 name 'hh-yun-ceph-cinder015-128055' to location {root=default} in crush map
added bucket hh-yun-ceph-cinder016-128056 type host to crush map
moved item id -3 name 'hh-yun-ceph-cinder016-128056' to location {root=default} in crush map
added bucket hh-yun-ceph-cinder017-128057 type host to crush map
moved item id -4 name 'hh-yun-ceph-cinder017-128057' to location {root=default} in crush map
added bucket hh-yun-ceph-cinder023-128073 type host to crush map
moved item id -5 name 'hh-yun-ceph-cinder023-128073' to location {root=default} in crush map
added bucket hh-yun-ceph-cinder024-128074 type host to crush map
moved item id -6 name 'hh-yun-ceph-cinder024-128074' to location {root=default} in crush map
added bucket hh-yun-ceph-cinder025-128075 type host to crush map
moved item id -7 name 'hh-yun-ceph-cinder025-128075' to location {root=default} in crush map
added bucket hh-yun-ceph-cinder026-128076 type host to crush map
moved item id -8 name 'hh-yun-ceph-cinder026-128076' to location {root=default} in crush map8. 管理 crush map
增加 OSD 到 CURSH MAP, 然后你就可以接收数据, 你同样可以重新编译 CURSH MAP, 添加 OSD 到磁盘, 添加主机到 CURSH MAP, 为磁盘添加设备
分配权重, 重新进行编译设定即可
#!/bin/bash
LANG=en_US
num=0
for ip in 240.30.128.55 240.30.128.56 240.30.128.57 240.30.128.73 240.30.128.74 240.30.128.75 240.30.128.76
do
diskpart=`ssh $ip "fdisk -l | grep GPT | grep -v sda" | awk '{print $1}' | sort`
for partition in $diskpart
do
hostname=`ssh $ip hostname -s`
ceph osd crush add osd.$num 1.0 root=default host=$hostname
let num++
done
done9. 启动 osd
#!/bin/bash
LANG=en_US
num=0
for ip in 240.30.128.55 240.30.128.56 240.30.128.57 240.30.128.73 240.30.128.74 240.30.128.75 240.30.128.76
do
diskpart=`ssh $ip "fdisk -l | grep GPT | grep -v sda" | awk '{print $1}' | sort`
for partition in $diskpart
do
ssh $ip "touch /var/lib/ceph/osd/ceph-$num/sysvinit"
ssh $ip "/etc/init.d/ceph start osd.$num"
let num++
done
done10. 校验状态
[root@hh-yun-ceph-cinder017-128057 ~]# ceph osd tree
# id weight type name up/down reweight
-1 70 root default -2 10 host hh-yun-ceph-cinder015-128055 0 1 osd.0 up 1
1 1 osd.1 up 1
2 1 osd.2 up 1
3 1 osd.3 up 1
4 1 osd.4 up 1
5 1 osd.5 up 1
6 1 osd.6 up 1
7 1 osd.7 up 1
8 1 osd.8 up 1
9 1 osd.9 up 1
-3 10 host hh-yun-ceph-cinder016-128056 10 1 osd.10 up 1
11 1 osd.11 up 1
12 1 osd.12 up 1
13 1 osd.13 up 1
14 1 osd.14 up 1
15 1 osd.15 up 1
16 1 osd.16 up 1
17 1 osd.17 up 1
18 1 osd.18 up 1
19 1 osd.19 up 1
-4 10 host hh-yun-ceph-cinder017-128057 20 1 osd.20 up 1
21 1 osd.21 up 1
22 1 osd.22 up 1
23 1 osd.23 up 1
24 1 osd.24 up 1
25 1 osd.25 up 1
26 1 osd.26 up 1
27 1 osd.27 up 1
28 1 osd.28 up 1
29 1 osd.29 up 1
-5 10 host hh-yun-ceph-cinder023-128073 30 1 osd.30 up 1
31 1 osd.31 up 1
32 1 osd.32 up 1
33 1 osd.33 up 1
34 1 osd.34 up 1
35 1 osd.35 up 1
36 1 osd.36 up 1
37 1 osd.37 up 1
38 1 osd.38 up 1
39 1 osd.39 up 1
-6 10 host hh-yun-ceph-cinder024-128074 40 1 osd.40 up 1
41 1 osd.41 up 1
42 1 osd.42 up 1
43 1 osd.43 up 1
44 1 osd.44 up 1
45 1 osd.45 up 1
46 1 osd.46 up 1
47 1 osd.47 up 1
48 1 osd.48 up 1
49 1 osd.49 up 1
-7 10 host hh-yun-ceph-cinder025-128075 50 1 osd.50 up 1
51 1 osd.51 up 1
52 1 osd.52 up 1
53 1 osd.53 up 1
54 1 osd.54 up 1
55 1 osd.55 up 1
56 1 osd.56 up 1
57 1 osd.57 up 1
58 1 osd.58 up 1
59 1 osd.59 up 1
-8 10 host hh-yun-ceph-cinder026-128076 60 1 osd.60 up 1
61 1 osd.61 up 1
62 1 osd.62 up 1
63 1 osd.63 up 1
64 1 osd.64 up 1
65 1 osd.65 up 1
66 1 osd.66 up 1
67 1 osd.67 up 1
68 1 osd.68 up 1
69 1 osd.69 up 1[root@hh-yun-ceph-cinder015-128055 tmp]# ceph -s
cluster dc4f91c1-8792-4948-b68f-2fcea75f53b9
health HEALTH_WARN too few pgs per osd (2 < min 20)
monmap e1: 3 mons at {hh-yun-ceph-cinder015-128055=240.30.128.55:6789/0,hh-yun-ceph-cinder017-128057=240.30.128.57:6789/0,hh-yun-ceph-cinder024-128074=240.30.128.74:6789/0}, election epoch 8, quorum 0,1,2 hh-yun-ceph-cinder015-128055,hh-yun-ceph-cinder017-128057,hh-yun-ceph-cinder024-128074
osdmap e226: 70 osds: 70 up, 70 in
pgmap v265: 192 pgs, 3 pools, 0 bytes data, 0 objects
74632 MB used, 254 TB / 254 TB avail
192 active+clean