openstack 与 ceph (monitor初始化)

Monitor 节点初始化
执行下面步骤

1. 操作系统初始化
   当前环境中, ceph 存储集群统一部署 centos 7.1 操作系统, 并在 /etc/hosts 中登记集群中的服务器主机名, 并完成时间同步

主机名验证

[root@hh-yun-ceph-cinder015-128055 ~]# hostname hh-yun-ceph-cinder015-128055.vclound.com [root@hh-yun-ceph-cinder017-128057 ~]# hostname hh-yun-ceph-cinder017-128057.vclound.com [root@hh-yun-ceph-cinder024-128074 ~]# hostname hh-yun-ceph-cinder024-128074.vclound.com

cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
240.30.128.55 hh-yun-ceph-cinder015-128055.vclound.com hh-yun-ceph-cinder015-128055
240.30.128.56 hh-yun-ceph-cinder016-128056.vclound.com hh-yun-ceph-cinder016-128056
240.30.128.57 hh-yun-ceph-cinder017-128057.vclound.com hh-yun-ceph-cinder017-128057
240.30.128.73 hh-yun-ceph-cinder023-128073.vclound.com hh-yun-ceph-cinder023-128073
240.30.128.74 hh-yun-ceph-cinder024-128074.vclound.com hh-yun-ceph-cinder024-128074
240.30.128.75 hh-yun-ceph-cinder025-128075.vclound.com hh-yun-ceph-cinder025-128075
240.30.128.76 hh-yun-ceph-cinder026-128076.vclound.com hh-yun-ceph-cinder026-128076

时间同步验证

[root@hh-yun-ceph-cinder024-128074 ~]# chronyc tracking
Reference ID    : 10.199.129.21 (hh-yun-129021.vclound.com)
Stratum         : 12
Ref time (UTC)  : Tue Jul 21 07:45:32 2015
System time     : 0.000002526 seconds fast of NTP time
Last offset     : 0.000005405 seconds
RMS offset      : 0.007192081 seconds
Frequency       : 18.428 ppm slow
Residual freq   : 0.001 ppm
Skew            : 0.060 ppm
Root delay      : 0.000141 seconds
Root dispersion : 0.011696 seconds
Update interval : 386.4 seconds
Leap status     : Normal

1. ceph 软件安装
   注: 假如之前安装过 ceph, 并计划清除之前的集群配置, 那么执行下面命令可以把 ceph 配置进行安装

清除方法: ceph-deploy purgedata {node-name},     ceph-deploy purge {node-name}

yum 源

yum install -y epel-release.noarch
echo "sslverify=0" >> /etc/yum.conf

ceph 软件安装

yum install -y ceph* bash-completion

1. 创建配置文件, 默认是 /etc/ceph/ceph.conf
   vim /etc/ceph/ceph.conf

[global]
fsid = dc4f91c1-8792-4948-b68f-2fcea75f53b9
mon initial members = hh-yun-ceph-cinder015-128055, hh-yun-ceph-cinder017-128057, hh-yun-ceph-cinder024-128074
mon host = 240.30.128.55, 240.30.128.57, 240.30.128.74
public network = 240.30.128.0/21
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd journal size = 1024
filestore xattr use omap = true
osd pool default size = 2
osd pool default min size = 1
osd pool default pg num = 333
osd pool default pgp num = 333
osd crush chooseleaf type = 1

1. 创建集群密钥对创建监控密钥

[root@hh-yun-ceph-cinder015-128055 tmp]# ceph-authtool --create-keyring /etc/ceph/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *' creating /etc/ceph/ceph.mon.keyring [root@hh-yun-ceph-cinder015-128055 tmp]# cat /etc/ceph/ceph.mon.keyring [mon.] key = AQCco7BVODvUJhAAz2PN+OQnLbD7o+Szq7/X9w== caps mon = "allow *"

注意: ceph-0.87 版本需要把文件 /etc/ceph/ceph.mon.keyring 重命名为 /etc/ceph/ceph.keyring 或 /etc/ceph/keyring 或 /etc/ceph/ceph.mon.(hostname).keyring 否则无法执行注册

1. 生成管理员密钥, 生成 client.admin 用户并增加用户密钥对

[root@hh-yun-ceph-cinder015-128055 tmp]# ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'
creating /etc/ceph/ceph.client.admin.keyring

[root@hh-yun-ceph-cinder015-128055 tmp]# ls -lh /etc/ceph/ceph.client.admin.keyring
-rw------- 1 root root 137 7月  23 16:20 /etc/ceph/ceph.client.admin.keyring

[root@hh-yun-ceph-cinder015-128055 tmp]# cat /etc/ceph/ceph.client.admin.keyring
[client.admin]
        key = AQC6o7BVWAskMhAAZyi8TEryY49lYA+2tEnOXw==
        auid = 0
        caps mds = "allow"
        caps mon = "allow *"
        caps osd = "allow *"

1. 增加 client.admin 密钥对到 ceph.mon.keyring 中

[root@hh-yun-ceph-cinder015-128055 tmp]# ceph-authtool /etc/ceph/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
importing contents of /etc/ceph/ceph.client.admin.keyring into /etc/ceph/ceph.mon.keyring

[root@hh-yun-ceph-cinder015-128055 tmp]# ls -lh /etc/ceph/ceph.client.admin.keyring /etc/ceph/ceph.mon.keyring
-rw------- 1 root root 137 7月  23 16:20 /etc/ceph/ceph.client.admin.keyring
-rw------- 1 root root 214 7月  23 16:20 /etc/ceph/ceph.mon.keyring

注意, 这里 /etc/ceph/ceph.mon.keyring 从之前的 77 字节变成 214 字节, 并自动添加如下的内容

[root@hh-yun-ceph-cinder015-128055 tmp]# cat /etc/ceph/ceph.mon.keyring
[mon.]
        key = AQCco7BVODvUJhAAz2PN+OQnLbD7o+Szq7/X9w==
        caps mon = "allow *"
[client.admin]
        key = AQC6o7BVWAskMhAAZyi8TEryY49lYA+2tEnOXw==
        auid = 0
        caps mds = "allow"
        caps mon = "allow *"
        caps osd = "allow *"

1. 使用主机名, IP 地址及 FSID 生成 monitor map 保存到 /etc/ceph/monmap 注意, 命令创建时候需要一次过把所有的 mon 节点都直接进行定义

[root@hh-yun-ceph-cinder015-128055 tmp]#  monmaptool --create --add  hh-yun-ceph-cinder015-128055 240.30.128.55 --add hh-yun-ceph-cinder017-128057 240.30.128.57 --add hh-yun-ceph-cinder024-128074 240.30.128.74 --fsid dc4f91c1-8792-4948-b68f-2fcea75f53b9 /etc/ceph/monmap
monmaptool: monmap file /etc/ceph/monmap
monmaptool: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9
monmaptool: writing epoch 0 to /etc/ceph/monmap (3 monitors)

1. 在对应的监控主机上创建默认数据目录

[root@hh-yun-ceph-cinder015-128055 /]# mkdir -p /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055 [root@hh-yun-ceph-cinder017-128057 ~]# mkdir -p /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057 [root@hh-yun-ceph-cinder024-128074 ~]# mkdir -p /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074

1. 定义监控进程与监控 MAP 及密钥对

[root@hh-yun-ceph-cinder015-128055 ~]# ceph-mon --mkfs -i hh-yun-ceph-cinder015-128055 --monmap /etc/ceph/monmap --keyring /etc/ceph/ceph.mon.keyring
ceph-mon: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9 ceph-mon: created monfs at /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055 for mon.hh-yun-ceph-cinder015-128055

把 /etc/ceph 目录下的文件: ceph.conf ceph.client.admin.keyring ceph.mon.keyring monmap 同步到另外两台 mon 节点中

[root@hh-yun-ceph-cinder015-128055 ceph]# scp ceph.* monmap hh-yun-ceph-cinder017-128057:/etc/ceph/.
ceph.client.admin.keyring                                                                   100%  137     0.1KB/s   00:00
ceph.conf                                                                                   100%  544     0.5KB/s   00:00
ceph.mon.keyring                                                                            100%  214     0.2KB/s   00:00
monmap                                                                                      100%  550     0.5KB/s   00:00

[root@hh-yun-ceph-cinder015-128055 ceph]# scp ceph.* monmap hh-yun-ceph-cinder024-128074:/etc/ceph/.
ceph.client.admin.keyring                                                                   100%  137     0.1KB/s   00:00
ceph.conf                                                                                   100%  544     0.5KB/s   00:00
ceph.mon.keyring                                                                            100%  214     0.2KB/s   00:00
monmap                                                                                      100%  550     0.5KB/s   00:00

为另外两台 mon 节点进行监控数据初始化

[root@hh-yun-ceph-cinder017-128057 ~]# ceph-mon --mkfs -i hh-yun-ceph-cinder017-128057 --monmap /etc/ceph/monmap --keyring /etc/ceph/ceph.mon.keyring
ceph-mon: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057 for mon.hh-yun-ceph-cinder017-128057
[root@hh-yun-ceph-cinder024-128074 ~]# ceph-mon --mkfs -i hh-yun-ceph-cinder024-128074 --monmap /etc/ceph/monmap --keyring /etc/ceph/ceph.mon.keyring
ceph-mon: set fsid to dc4f91c1-8792-4948-b68f-2fcea75f53b9
ceph-mon: created monfs at /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074 for mon.hh-yun-ceph-cinder024-128074

1. 分别启动三台 mon 节点

[root@hh-yun-ceph-cinder015-128055 ~]# touch /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055/done /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder015-128055/sysvinit
[root@hh-yun-ceph-cinder017-128057 ~]# touch /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057/done /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder017-128057/sysvinit
[root@hh-yun-ceph-cinder024-128074 ~]# touch /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074/done /var/lib/ceph/mon/ceph-hh-yun-ceph-cinder024-128074/sysvinit

[root@hh-yun-ceph-cinder015-128055 ~]# /etc/init.d/ceph start mon.hh-yun-ceph-cinder015-128055
=== mon.hh-yun-ceph-cinder015-128055 ===
Starting Ceph mon.hh-yun-ceph-cinder015-128055 on hh-yun-ceph-cinder015-128055...
Running as unit run-53061.service.
Starting ceph-create-keys on hh-yun-ceph-cinder015-128055...

[root@hh-yun-ceph-cinder017-128057 ~]# /etc/init.d/ceph start mon.hh-yun-ceph-cinder017-128057
=== mon.hh-yun-ceph-cinder017-128057 ===
Starting Ceph mon.hh-yun-ceph-cinder017-128057 on hh-yun-ceph-cinder017-128057...
Running as unit run-52736.service.
Starting ceph-create-keys on hh-yun-ceph-cinder017-128057...

[root@hh-yun-ceph-cinder024-128074 ~]# /etc/init.d/ceph start mon.hh-yun-ceph-cinder024-128074
=== mon.hh-yun-ceph-cinder024-128074 ===
Starting Ceph mon.hh-yun-ceph-cinder024-128074 on hh-yun-ceph-cinder024-128074...
Running as unit run-51440.service.
Starting ceph-create-keys on hh-yun-ceph-cinder024-128074...