k8s学习笔记——ceph集群安装
如果//ubuntu系统
添加 release key
wget -q -O- 'https://download.ceph.com/keys/release.asc' | sudo apt-key add -添加Ceph软件包源
echo deb http://download.ceph.com/debian-octopus/ $(lsb_release -sc) main | sudo tee /etc/apt/sources.list.d/ceph.list更新仓库，并安装 ceph-deploy
sudo apt-get update && sudo apt-get install ceph-deploy在安装过程中会报如下错误：
[ceph_deploy][ERROR ] RuntimeError: AttributeError: module 'platform' has no attribute 'linux_distribution'
这是由于python3.7后不再支持platform.linux_distribution
//修改方法
修改/usr/lib/python3/dist-packages/ceph_deploy/hosts/remotes.py文件为如下所示
def platform_information(_linux_distribution=None):
""" detect platform information from remote host """
"""
linux_distribution = _linux_distribution or platform.linux_distribution
distro, release, codename = linux_distribution()
"""
distro = release = codename = None
try:
linux_distribution = _linux_distribution or platform.linux_distribution
distro, release, codename = linux_distribution()
except AttributeError:
pass
下面使用ceph-deploy进行安装
如下操作必须在刚创建的ssh免密用户下进行
如果过程中出现问题可以使用如下命令清除安装数据重新进行。
ceph-deploy purgedata {ceph-node} [{ceph-node}]
ceph-deploy forgetkeys用下列命令可以连 Ceph 安装包一起清除：
ceph-deploy purge {ceph-node} [{ceph-node}]在每个节点上运行如下命令，自动修复存在异常的deb
sudo apt --fix-broken install创建集群monitor节点
ceph-deploy new --cluster_network 192.168.11.128/25 --public_network 10.12.70.128/25 k8sceph70201创建成功后在当前目录下应该有ceph.conf 、ceph.mon.keyring、ceph-deploy-ceph.log三个文件
安装ceph其他节点
//ceph-deploy install admin-node node1 node2 node3
ceph-deploy install k8sceph70201 k8sceph70201 k8sceph70202 k8sceph70203配置初始 monitor(s)、并收集所有密钥：
ceph-deploy mon create-initial收集完后，当前目录下会多出如下文件
ceph.bootstrap-mds.keyring、ceph.bootstrap-rgw.keyring、ceph.bootstrap-osd.keyring、ceph.client.admin.keyring
把配置文件和 admin 密钥拷贝到管理节点和 Ceph 节点
ceph-deploy admin k8sceph70201 k8sceph70202 k8sceph70203确保你对 ceph.client.admin.keyring 有正确的操作权限,，对每个节点执行
sudo chmod +r /etc/ceph/ceph.client.admin.keyring如果如下报错就为操作权限问题
auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
部署管理器守护程序
ceph-deploy mgr create k8sceph70201添加硬盘
执行下列命令列举一节点上的磁盘：
//ceph-deploy disk list {node-name [node-name]...}
ceph-deploy disk list k8sceph70201//报错
[k8sceph70202][INFO ] Running command: sudo fdisk -l
[ceph_deploy][ERROR ] Traceback (most recent call last):
[ceph_deploy][ERROR ] File "/usr/lib/python3/dist-packages/ceph_deploy/util/decorators.py", line 69, in newfunc
[ceph_deploy][ERROR ] return f(*a, **kw)
[ceph_deploy][ERROR ] File "/usr/lib/python3/dist-packages/ceph_deploy/cli.py", line 166, in _main
[ceph_deploy][ERROR ] return args.func(args)
[ceph_deploy][ERROR ] File "/usr/lib/python3/dist-packages/ceph_deploy/osd.py", line 434, in disk
[ceph_deploy][ERROR ] disk_list(args, cfg)
[ceph_deploy][ERROR ] File "/usr/lib/python3/dist-packages/ceph_deploy/osd.py", line 375, in disk_list
[ceph_deploy][ERROR ] if line.startswith('Disk /'):
[ceph_deploy][ERROR ] TypeError: startswith first arg must be bytes or a tuple of bytes, not str
[ceph_deploy][ERROR ] //解决方法
sudo vim /usr/lib/python3/dist-packages/ceph_deploy/osd.py中
if line.startswith('Disk /'):
替换为
if line.startswith(b'Disk /'):
擦净硬盘
ceph-deploy disk zap k8sceph70201 /dev/sdb
//会删除硬盘上所有分区和数据创建OSDceph-deploy --overwrite-conf osd create --data /dev/sdb k8sceph70201
使用ceph -s或者 ceph health查看集群状态，正常显示
cluster:
id: d10b3028-b78d-4b23-bacb-ca26c0a35c99
health: HEALTH_OK
services:
mon: 3 daemons, quorum k8sceph70201,k8sceph70202,k8sceph70203 (age 2d)
mgr: k8sceph70201(active, since 3w)
mds: 1 up:standby
osd: 9 osds: 9 up (since 3w), 9 in (since 3w)
rgw: 1 daemon active (k8sceph70201)
task status:
data:
pools: 7 pools, 169 pgs
objects: 306 objects, 182 MiB
usage: 10 GiB used, 25 TiB / 25 TiB avail
pgs: 169 active+clean
ceph health detail //详情查看
//报错
sudo ceph -s
cluster:
id: d10b3028-b78d-4b23-bacb-ca26c0a35c99
health: HEALTH_WARN
mons are allowing insecure global_id reclaim//解决方法
ceph config set mon auth_allow_insecure_global_id_reclaim false