三、单机Ceph手动安装

准备工作

主机：192.168.2.90 主机名：ceph0

操作系统：centos7

ceph版本：14.2.22-0.el7.x86_64

硬盘：两块硬盘，一块用来挂载ceph

主机设置

hostnamectl set-hostname ceph0
cat >> /etc/hosts << EOF
192.168.2.90 ceph0
EOF

#关闭selinux
setenforce 0
sed -ri 's#(SELINUX=).*#1disabled#g' /etc/selinux/config

# 时间同步
yum install -y chrony
systemctl start  chronyd
systemctl enable chronyd
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

服务器需要设置防火墙，测试环境可以将防火墙关闭

sudo firewall-cmd --zone=public --add-service=ceph-mon

sudo firewall-cmd --zone=public --add-service=ceph-mon --permanent

配置ceph源

这里注意centos版本，centos7对应el7,8系列对应el8

[ceph]
name=ceph
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/
gpgcheck=0
priority =1

[ceph-noarch]
name=cephnoarch
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch/
gpgcheck=0
priority =1

[ceph-source]
name=cephsource
baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/SRPMS/
gpgcheck=0
priority =1

下载安装包

yum install -y ceph

MON

初始化

ceph安装包install之后会自动建立文件夹/etc/ceph，我们需要在该文件夹建立配置文件ceph.conf：

• 生成uniqueID

uuidgen
fsid = {UUID} # fsid=ec15eb3e-eb66-4431-acda-428e91658560

• 添加监视器

mon initial members = {hostname}[,{hostname}] #mon initial members = ceph0

• 添加监视器IP信息

mon host = {ip-address}[,{ip-address}] #mon host = 192.168.2.90

• 配置秘钥

# 生成秘钥
sudo ceph-authtool --create-keyring /tmp/ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'
# 创建管理员
sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.admin.keyring --gen-key -n client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *'
# osd用户
sudo ceph-authtool --create-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring --gen-key -n client.bootstrap-osd --cap mon 'profile bootstrap-osd' --cap mgr 'allow r'
# 添加秘钥到ceph.mon.keyring文件
sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /etc/ceph/ceph.client.admin.keyring
sudo ceph-authtool /tmp/ceph.mon.keyring --import-keyring /var/lib/ceph/bootstrap-osd/ceph.keyring
# 修改ceph.mon.keyring拥有者
sudo chown ceph:ceph /tmp/ceph.mon.keyring

• 映射监视器

monmaptool --create --add {hostname} {ip-address} --fsid {uuid} /tmp/monmap
# monmaptool --create --add ceph0 192.168.2.90 --fsid ec15eb3e-eb66-4431-acda-428e91658560 /tmp/monmap

• 创建mon文件夹

sudo mkdir /var/lib/ceph/mon/{cluster-name}-{hostname}
#sudo -u ceph mkdir /var/lib/ceph/mon/ceph-ceph0

• 配置monitor daemon

sudo -u ceph ceph-mon [--cluster {cluster-name}] --mkfs -i {hostname} --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
#sudo -u ceph ceph-mon --mkfs -i ceph0 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring

• 完善ceph.conf

[global]
fsid = {cluster-id}
mon initial members = {hostname}[, {hostname}]
mon host = {ip-address}[, {ip-address}]
public network = {network}[, {network}]
cluster network = {network}[, {network}]
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd journal size = {n}
osd pool default size = {n}  # Write an object n times.
osd pool default min size = {n} # Allow writing n copies in a degraded state.
osd pool default pg num = {n}
osd pool default pgp num = {n}
osd crush chooseleaf type = {n}

例：

[global]
fsid = ec15eb3e-eb66-4431-acda-428e91658560
mon initial members = ceph0
mon host = 192.168.2.90
public network = 192.168.2.0/24
cluster network = 192.168.2.0/24
auth cluster required = cephx
auth service required = cephx
auth client required = cephx
osd journal size = 1024
osd pool default size = 1
osd pool default min size = 1
osd crush chooseleaf type = 0 # 单节点需改为0

实际测试过程中发现文件最后没有空行会出错，不知道是不是跟版本有关，待验证

启动monitor

sudo systemctl start ceph-mon@ceph0

查看状态

sudo ceph -s
cluster:
  id:     ec15eb3e-eb66-4431-acda-428e91658560
  health: HEALTH_OK

services:
  mon: 1 daemons, quorum ceph0
  mgr: foo(active, since 0h)
  osd: 0 osds: 0 up, 0 in

data:
  pools:   0 pools, 0 pgs
  objects: 0 objects, 0 bytes
  usage:   0 kB used, 0 kB / 0 kB avail
  pgs:

MGR

# 配置用户
ceph auth get-or-create mgr.$name mon 'allow profile mgr' osd 'allow *' mds 'allow *'

# 启动服务
ceph-mgr -i $name

Dashboard

yum -y install ceph-mgr-dashboard

# 开启服务
ceph mgr module enable dashboard

#dashboard需要配置ssl，先生成ssl证书
openssl req -new -nodes -x509 \
  -subj "/O=IT/CN=ceph-mgr-dashboard" -days 3650 \
  -keyout dashboard.key -out dashboard.crt -extensions v3_ca
  
ceph dashboard set-ssl-certificate -i dashboard.crt
ceph dashboard set-ssl-certificate-key -i dashboard.key

ceph mgr module disable dashboard
ceph mgr module enable dashboard

#可以选择配置dashboard的IP和端口，也可以不配置
ceph config set mgr mgr/dashboard/server_addr $IP
ceph config set mgr mgr/dashboard/server_port $PORT
ceph config set mgr mgr/dashboard/ssl_server_port $PORT

#查看服务地址
ceph mgr services
{
    "dashboard": "https://ceph1:8443/"
}

创建用户

新创建的dashboard是没有用户的，通过ac-user-create命令来创建一个用户，该命令需要先创建一个密码文件，并将密码输入

ceph dashboard ac-user-create  -i  administrator
# ceph dashboard ac-user-create admin -i /root/passwordfile administrator

OSD

这里使用BLUESTORE快速创建的方式，如果是多台服务器需要将monitor中生成的key复制到OSD服务器。

sudo ceph-volume lvm create --data {data-path}
# sudo ceph-volume lvm create --data /dev/sdb

MDS

初始化

# 创建文件夹
mkdir -p /var/lib/ceph/mds/{cluster-name}-{id}
# mkdir -p /var/lib/ceph/mds/ceph-ceph0

ceph-authtool --create-keyring /var/lib/ceph/mds/ceph-ceph0/keyring --gen-key -n mds.ceph0

ceph auth add mds.{id} osd "allow rwx" mds "allow *" mon "allow profile mds" -i /var/lib/ceph/mds/{cluster}-{id}/keyring
#ceph auth add mds.ceph0 osd "allow rwx" mds "allow *" mon "allow profile mds" -i /var/lib/ceph/mds/ceph-ceph0/keyring

修改ceph.conf

[mds.{id}]
host = {id}
#[mds.ceph0]
ceph-mds --cluster {cluster-name} -i {id} -m {mon-hostname}:{mon-port} [-f]
#ceph-mds --cluster ceph -i ceph0 -m 192.168.2.90:6789

结束

通过ceph -s查看状态