Ceph-Ansible部署Ceph-N（基于ubuntu 18.04 LTS）
常用自动化部署方案比较
| 名称 | 实现语言 | 发布时间 | 架构 | DSL （区域特定语言） |
|---|---|---|---|---|
| SaltStack | Python | 2011 | C/S | yaml |
| Ansible | Python | 2012 | Agentless | yaml |
| Puppet | Ruby | 2009 | C/S | Puppet Language |
| Chef | Ruby | 2005 | C/S | 接近Ruby语法的Dsl |
环境准备
在部署 Ceph 存储集群之前，需要对Ceph 客户端和Ceph 节点进行一些基本的配置。安装一个ceph-ansible 管理节点和若干节点的Ceph存储集群来研究 Ceph 的基本特性。此处准备四个Ceph节点（其中adm节点充当ceph-ansible管理节点），以此构成 Ceph存储集群。
| 主机名称 | IP地址 | 操作系统 | 备注 |
|---|---|---|---|
| adm | 100.10.10.163 | ubuntu-18.04.3-server-amd64.iso | ceph-ansible管理节点 |
| N1 | 100.10.10.160/100.10.11.160 | ubuntu-18.04.3-server-amd64.iso | 专用节点 |
| N2 | 100.10.10.161/100.10.11.161 | ubuntu-18.04.3-server-amd64.iso | 专用节点 |
| N3 | 100.10.10.162/100.10.11.162 | ubuntu-18.04.3-server-amd64.iso | 专用节点 |
未特别说明，则操作需在每个节点上执行
添加keys
将密钥添加到系统的受信任密钥列表中，以避免出现安全警告。 对于主要发行版（例如luminous, mimic, nautilus）和开发发行版（release-name-rc1, release-name-rc2），请使用release.asc密钥。
要安装release.asc密钥，请执行以下操作：
# wget -q -O- 'http://100.10.10.163/ceph/keys/release.asc' | sudo apt-key add -
# wget -q -O- 'http://100.10.10.163/os/18.04_LTS/debs.asc' | sudo apt-key add -
# apt-key list
/etc/apt/trusted.gpg
--------------------
pub rsa4096 2015-09-15 [SC]
08B7 3419 AC32 B4E9 66C1 A330 E84A C2C0 460F 3994
uid [ unknown] Ceph.com (release key) <[email protected]>
添加CEPH
将Ceph软件包库添加到系统的APT来源列表中。 对于较新版本的Debian/Ubuntu，请在命令行上调用lsb_release -sc以获取简短的代号。
# cp /etc/apt/sources.list /etc/apt/sources.list.org
# > /etc/apt/sources.list
# sudo apt-add-repository 'deb http://100.10.10.163/ceph/debian-nautilus/ bionic main'
# apt-get update -y
修改/etc/hosts文件
# vim /etc/hosts
127.0.0.1 localhost
100.10.10.160 N1
100.10.10.161 N2
100.10.10.162 N3
100.10.10.163 adm
配置ssh免密登录
在adm节点执行以下命令，
# ssh-keygen -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:w3exfRZVT5yUhWGDS0KlJFSXkGOBRB52DY2wXJZfNdI
The key's randomart image is:
+---[RSA 2048]----+
| +O=@@o+*B@|
| +.OBo*.+E=|
| +..=.o ..|
| . o+ .|
| S . o . o|
| o . o |
| |
| |
| |
+----[SHA256]-----+
# ssh-copy-id 100.10.10.160
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '100.10.10.160'"
and check to make sure that only the key(s) you wanted were added.
# ssh-copy-id 100.10.10.161
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '100.10.10.161'"
and check to make sure that only the key(s) you wanted were added.
# ssh-copy-id 100.10.10.162
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '100.10.10.162'"
and check to make sure that only the key(s) you wanted were added.
# ssh-copy-id 100.10.10.163
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '100.10.10.163 (100.10.10.163)' can't be established.
ECDSA key fingerprint is SHA256:krdx3rwkZITeNgQBIDo4eY70tOTrMFhJ4c24RSGPQYI.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
[email protected]'s password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '100.10.10.163'"
and check to make sure that only the key(s) you wanted were added.
# scp /root/.ssh/id_rsa* 100.10.10.160:/root/.ssh/
id_rsa 100% 1679 1.6MB/s 00:00
id_rsa.pub 100% 393 481.8KB/s 00:00
# scp /root/.ssh/id_rsa* 100.10.10.161:/root/.ssh/
id_rsa 100% 1679 1.6MB/s 00:00
id_rsa.pub 100% 393 527.1KB/s 00:00
# scp /root/.ssh/id_rsa* 100.10.10.162:/root/.ssh/
id_rsa 100% 1679 1.5MB/s 00:00
id_rsa.pub
在每个节点上执行
# ssh 100.10.10.160
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Thu Jan 9 12:35:33 CST 2020
System load: 0.04 Processes: 311
Usage of /: 3.2% of 136.36GB Users logged in: 2
Memory usage: 0% IP address for enp3s0f0: 100.10.10.160
Swap usage: 0% IP address for enp3s0f1: 100.10.11.160
* Canonical Livepatch is available for installation.
- Reduce system reboots and improve kernel security. Activate at:
https://ubuntu.com/livepatch
0 packages can be updated.
0 updates are security updates.
Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings
Last login: Thu Jan 9 12:34:37 2020 from 100.10.10.162
root@N1:~# exit
logout
Connection to 100.10.10.160 closed.
# ssh 100.10.10.161
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Thu Jan 9 12:35:38 CST 2020
System load: 0.08 Processes: 305
Usage of /: 3.2% of 136.36GB Users logged in: 2
Memory usage: 0% IP address for enp4s0f0: 100.10.10.161
Swap usage: 0% IP address for enp4s0f1: 100.10.11.161
* Canonical Livepatch is available for installation.
- Reduce system reboots and improve kernel security. Activate at:
https://ubuntu.com/livepatch
0 packages can be updated.
0 updates are security updates.
Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings
Last login: Thu Jan 9 12:35:04 2020 from 100.10.10.162
root@N2:~# exit
logout
Connection to 100.10.10.161 closed.
# ssh 100.10.10.162
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Thu Jan 9 12:41:26 CST 2020
System load: 0.05 Processes: 299
Usage of /: 3.2% of 136.36GB Users logged in: 1
Memory usage: 0% IP address for enp3s0f0: 100.10.10.162
Swap usage: 0% IP address for enp3s0f1: 100.10.11.162
* Canonical Livepatch is available for installation.
- Reduce system reboots and improve kernel security. Activate at:
https://ubuntu.com/livepatch
0 packages can be updated.
0 updates are security updates.
Last login: Thu Jan 9 12:40:55 2020 from 100.10.10.162
root@N3:~# exit
logout
Connection to 100.10.10.162 closed.
# ssh 100.10.10.163
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-55-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Thu Jan 9 12:41:32 CST 2020
System load: 0.0 Processes: 320
Usage of /: 6.1% of 136.36GB Users logged in: 2
Memory usage: 1% IP address for enp3s0f0: 100.10.10.163
Swap usage: 0% IP address for enp3s0f1: 172.10.27.163
* Overheard at KubeCon: "microk8s.status just blew my mind".
https://microk8s.io/docs/commands#microk8s.status
* Canonical Livepatch is available for installation.
- Reduce system reboots and improve kernel security. Activate at:
https://ubuntu.com/livepatch
141 packages can be updated.
89 updates are security updates.
*** System restart required ***
Last login: Thu Jan 9 12:41:07 2020 from 100.10.10.162
root@ceph-n:~# exit
logout
Connection to 100.10.10.163 closed.
安装 CHRONY
在Ceph节点上安装CHRONY以防止时钟漂移引起的问题。
# timedatectl set-ntp false
# apt-get install chrony -y
# chronyd --version
# cp /etc/chrony/chrony.conf /etc/chrony/chrony.conf.org
设置chrony服务器
此处将adm节点作为CHRONY服务器
# vim /etc/chrony/chrony.conf
.....
pool ntp.ubuntu.com iburst maxsources 4
pool 0.ubuntu.pool.ntp.org iburst maxsources 1
pool 1.ubuntu.pool.ntp.org iburst maxsources 1
pool 2.ubuntu.pool.ntp.org iburst maxsources 2
.......
allow 100.10 ##允许同步时间的客户端所在网段
local stratum 10 ##设置源时钟等级
将配置文件中上述4个pool注释掉，在其中增加允许同步时间的IP网段段以及设置源时钟等级。
# systemctl enable chrony.service
# systemctl restart chrony.service
# systemctl status chrony.service
● chrony.service - chrony, an NTP client/server
Loaded: loaded (/lib/systemd/system/chrony.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-01-09 15:51:54 CST; 8s ago
Docs: man:chronyd(8)
man:chronyc(1)
man:chrony.conf(5)
Process: 25427 ExecStartPost=/usr/lib/chrony/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
Process: 25398 ExecStart=/usr/lib/systemd/scripts/chronyd-starter.sh $DAEMON_OPTS (code=exited, status=0/SUCCESS)
Main PID: 25422 (chronyd)
Tasks: 1 (limit: 4915)
CGroup: /system.slice/chrony.service
└─25422 /usr/sbin/chronyd
设置chrony客户端
此处N1、N2和N3节点为CHRONY客户端
# vim /etc/chrony/chrony.conf
.....
# pool ntp.ubuntu.com iburst maxsources 4
# pool 0.ubuntu.pool.ntp.org iburst maxsources 1
# pool 1.ubuntu.pool.ntp.org iburst maxsources 1
# pool 2.ubuntu.pool.ntp.org iburst maxsources 2
server 100.10.10.163 iburst
.......
其中100.10.10.163为CHRONY服务器的IP地址。
启动并确认chronyd服务
# systemctl enable chrony.service
Synchronizing state of chrony.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable chrony
# systemctl restart chrony.service
# systemctl status chrony.service
● chrony.service - chrony, an NTP client/server
Loaded: loaded (/lib/systemd/system/chrony.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-01-09 15:51:54 CST; 8s ago
Docs: man:chronyd(8)
man:chronyc(1)
man:chrony.conf(5)
Process: 25427 ExecStartPost=/usr/lib/chrony/chrony-helper update-daemon (code=exited, status=0/SUCCESS)
Process: 25398 ExecStart=/usr/lib/systemd/scripts/chronyd-starter.sh $DAEMON_OPTS (code=exited, status=0/SUCCESS)
Main PID: 25422 (chronyd)
Tasks: 1 (limit: 4915)
CGroup: /system.slice/chrony.service
└─25422 /usr/sbin/chronyd
确认时间同步状态
## 查看 chrony_servers 状态
# chronyc sources -v
## 查看 chrony_sync 状态
# chronyc sourcestats -v
## 查看 chrony_servers 是否在线
# chronyc activity -v
## 查看 chrony 详细信息
# chronyc tracking -v
Ceph-Ansible环境配置
下载Ceph-ansible
# wget -q -O ceph-ansible-stable-4.0.zip https://codeload.github.com/ceph/ceph-ansible/zip/stable-4.0
# unzip ceph-ansible-stable-4.0.zip
下载ansible
本次使用的是ansible-2.8.7，安装ansible需要一些依赖包，依赖包及其版本如下：
asn1crypto==0.24.0
cffi==1.13.2
cryptography==2.8
dnspython==1.15.0
enum34==1.1.6
httplib2==0.11.3
idna==2.6
ipaddress==1.0.23
Jinja2==2.10.3
MarkupSafe==1.1.1
netaddr==0.7.19
ply==3.11
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.19
pycrypto==2.6.1
PyYAML==5.3
six==1.13.0
真实的软件包如下：
-rw-r--r-- 1 root root 125326 Jan 8 15:23 Jinja2-2.10.3-py2.py3-none-any.whl
-rw-r--r-- 1 root root 24348 Jan 8 15:24 MarkupSafe-1.1.1-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root 268214 Jan 8 15:23 PyYAML-5.3.tar.gz
-rw-r--r-- 1 root root 12697144 Jan 8 15:23 ansible-2.8.7.tar.gz
-rw-r--r-- 1 root root 384920 Jan 8 15:24 cffi-1.13.2-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root 2260524 Jan 8 15:23 cryptography-2.8-cp27-cp27mu-manylinux1_x86_64.whl
-rw-r--r-- 1 root root 12427 Jan 8 15:23 enum34-1.1.6-py2-none-any.whl
-rw-r--r-- 1 root root 18159 Jan 8 15:23 ipaddress-1.0.23-py2.py3-none-any.whl
-rw-r--r-- 1 root root 1628451 Jan 8 15:23 netaddr-0.7.19-py2.py3-none-any.whl
-rw-r--r-- 1 root root 158295 Jan 8 15:24 pycparser-2.19.tar.gz
-rw-r--r-- 1 root root 10747 Jan 8 15:24 six-1.13.0-py2.py3-none-any.whl
不再赘述如何安装软件包
安装Ceph-N
1.创建主机清单
# cd ceph-ansible-stable-4.0
# vim hosts
[mons]
N1 monitor_address=100.10.10.160
N2 monitor_address=100.10.10.161
N3 monitor_address=100.10.10.162
[osds]
N[1:3]
[rgws]
N1 radosgw_address=100.10.10.160
N2 radosgw_address=100.10.10.161
N3 radosgw_address=100.10.10.162
[grafana-server]
N[1:3]
[mdss]
N[1:3]
[clients]
N[1:3]
[mgrs]
N[1:3]
2.拷贝group_vars/all.yml.sample 到group_vars/all.yml，并修改all.yml 文件，添加如下参数：
ceph_origin: repository
ceph_repository: community
ceph_mirror: http://100.10.10.163/ceph/
ceph_stable_key: "{{ ceph_mirror }}/keys/release.asc"
ceph_stable_release: nautilus
ceph_stable_repo: "{{ ceph_mirror }}/debian-{{ ceph_stable_release }}"
ceph_stable_distro_source: bionic
cephx: "true"
public_network: 100.10.10.0/24
cluster_network: 100.10.11.0/24
mon_host: 100.10.10.160,100.10.10.161,100.10.10.162
mon_initial_members: N1,N2,N3
monitor_interface: enp3s0f0
rbd_cache: "true"
rbd_cache_writethrough_until_flush: "true"
rbd_concurrent_management_ops: 20
rbd_client_directories: true
osd_objectstore: bluestore
osd_auto_discovery: true
osd_auto_discovery_exclude: "dm-*|loop*|md*|rbd*"
mds_max_mds: 1
radosgw_frontend_type: beast
radosgw_thread_pool_size: 512
radosgw_interface: "{{ monitor_interface }}"
email_address: [email protected]
dashboard_enabled: True
dashboard_protocol: http
dashboard_port: 8443
dashboard_admin_user: admin
dashboard_admin_password: toyou@123
grafana_admin_user: admin
grafana_admin_password: admin
grafana_uid: 472
grafana_datasource: Dashboard
grafana_dashboard_version: nautilus
grafana_port: 3000
grafana_allow_embedding: True
3.拷贝site.yml.sample 到site.yml（注释掉一些hosts）
# Defines deployment design and assigns role to server groups
- hosts:
- mons
- osds
- mdss
- rgws
# - nfss
# - rbdmirrors
- clients
- mgrs
# - iscsigws
# - iscsi-gws # for backward compatibility only!
- grafana-server
# - rgwloadbalancers
4.执行部署操作
# ansible-playbook -i hosts site.yml
清空集群
# cp infrastructure-playbooks/purge-cluster.yml purge-cluster.yml # 必须copy到项目根目录下
# ansible-playbook -i hosts purge-cluster.yml