k8s使用ceph存储
文章目录
-
-
- 初始化操作
- k8s使用ceph rbd
-
- volume
- PV
-
- 静态pv
- 动态pv
- k8s使用cephfs
-
- volume
- 静态pv
-
初始化操作
ceph创建rbd存储池
ceph osd pool create k8s-data 32 32 replicated
ceph osd pool application enable k8s-data rbd
rbd pool init -p k8s-data
ceph添加授权，需要创建两个用户，一个挂载rbd时使用，另一个挂载cephfs时使用
ceph auth get-or-create client.k8s-user mon 'allow r' mds 'allow' osd 'allow * pool=k8s-data' -o /etc/ceph/ceph.client.k8s-user.keyring
ceph auth get-or-create client.k8s-cephfs-user mon 'allow r' mds 'allow' osd 'allow * pool=cephfs_data' -o /etc/ceph/ceph.client.k8s-cephfs-user.keyring
k8s集群中所有节点安装ceph-common
apt -y install ceph-common
将ceph配置文件和keyring文件拷贝至所有k8s集群中节点
scp /etc/ceph/ceph.conf /etc/ceph/ceph.client.k8s-user.keyring /etc/ceph/ceph.client.k8s-cephfs-user.keyring root@xxx:/etc/ceph/
k8s使用ceph rbd
k8s集群中的pod使用rbd时可以直接通过pod的volume进行挂载，也可以以pv形式挂载使用
volume
提前在存储池中创建image
rbd create volume1 -s 2G -p k8s-data
配置pod通过volume挂载volume1
apiVersion: v1
kind: Pod
metadata:
name: pod-with-rbd-volume
spec:
containers:
- name: busybox
image: busybox
imagePullPolicy: IfNotPresent
command: ["/bin/sh", "-c", "sleep 3600"]
volumeMounts:
- name: data-volume
mountPath: /data/
volumes:
- name: data-volume
rbd:
monitors: ["192.168.211.23:6789", "192.168.211.24:6789", "192.168.211.25:6789"] #指定mon节点地址
pool: k8s-data #指定pool
image: volume1 #指定要挂载的rbd image
user: k8s-user #指定挂载rbd image时使用的用户，默认admin用户
keyring: /etc/ceph/ceph.client.k8s-user.keyring #用户的keyring文件路径
fsType: xfs #指定挂载的rbd image格式化的文件系统类型，默认ext4
创建pod到集群中，等待pod就绪后可以进入pod验证
由于pod使用的的是宿主机内核，所以rbd image实际是在宿主机挂载的
另外keyring文件也可以保存在secret中，在pod中通过secret来引用keyring文件。
例如将k8s-user的keyring保存到secret中（注意，需要对用户的key先进行base64编码）：
apiVersion: v1
kind: Secret
metadata:
name: k8s-user-keyring
type: "kubernetes.io/rbd" #类型必须是这个
data:
key: "QVFCSWlvVmszNVh5RXhBQWhkK1lwb3k3VHhvQkswQ2VkRE1zcWc9PQo="
将pod创建到集群中，等待pod就绪后进入pod验证
配置pod通过secret引用keyring文件
apiVersion: v1
kind: Pod
metadata:
name: pod-use-scret-keyring
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: web
mountPath: /usr/share/nginx/html/
volumes:
- name: web
rbd:
monitors: ["192.168.211.23:6789", "192.168.211.24:6789", "192.168.211.25:6789"]
pool: k8s-data
image: volume2
user: k8s-user
secretRef: #通过secret引用keyring
name: k8s-user-keyring
fsType: xfs
PV
静态pv
静态pv的方式也需要提前在存储池中创建好image
创建pv
apiVersion: v1
kind: PersistentVolume
metadata:
name: rbd-pv
spec:
accessModes: ["ReadWriteOnce"]
capacity:
storage: 2Gi
persistentVolumeReclaimPolicy: Retain
rbd:
monitors: ["192.168.211.23:6789", "192.168.211.24:6789", "192.168.211.25:6789"]
pool: k8s-data
image: volume3
user: k8s-user
secretRef:
name: k8s-user-keyring
fsType: xfs
创建pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pvc
namespace: default
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 2Gi
pod使用pvc
apiVersion: v1
kind: Pod
metadata:
name: pod-use-rbd-pvc
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: web
mountPath: /usr/share/nginx/html/
volumes:
- name: web
persistentVolumeClaim:
claimName: rbd-pvc
readOnly: false
将pod创建到集群中，等待pod就绪后进入pod验证
动态pv
创建存储类
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rbd-storageclass
provisioner: kubernetes.io/rbd
reclaimPolicy: Retain
parameters:
monitors: 192.168.211.23:6789,192.168.211.24:6789,192.168.211.25:6789 #mon节点地址
adminId: k8s-user #用户名称，这个用户是用于在pool中创建image时使用
adminSecretName: k8s-user-keyring #用户keyring对应的secret
adminSecretNamespace: default #用户keyring对应的secret所在的名称空间
pool: k8s-data #存储池
userId: k8s-user #这个用户是用于挂载rbd image时使用
userSecretName: k8s-user-keyring
userSecretNamespace: default
fsType: xfs
imageFormat: "2" #创建的rbd image的格式
imageFeatures: "layering" #创建的rbd image启用的特性
创建pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: rbd-pvc-with-sc
namespace: default
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 2Gi
storageClassName: rbd-storageclass
pod使用动态pvc
apiVersion: v1
kind: Pod
metadata:
name: pod-use-rdynamic-pvc
spec:
containers:
- name: redis
image: redis
imagePullPolicy: IfNotPresent
volumeMounts:
- name: data
mountPath: /data/redis/
volumes:
- name: data
persistentVolumeClaim:
claimName: rbd-pvc-with-sc
readOnly: false
将pod创建到集群中，等待pod就绪后进入pod验证
k8s使用cephfs
cephfs可以同时挂载给多个pod使用，实现数据共享
volume
先将前面创建的ceph用户client.k8s-cephfs-user的keyring保存到secret中
key=$(cat /etc/ceph/ceph.client.k8s-cephfs-user.keyring |grep key|awk '{print $3}')
kubectl create secret generic k8s-cephfs-user-keyring --type=kubernetes.io/rbd --from-literal=key=$key
配置pod挂载cephfs
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: web-data
mountPath: /usr/share/nginx/html
volumes:
- name: web-data
cephfs:
monitors:
- 192.168.211.23:6789
- 192.168.211.24:6789
- 192.168.211.25:6789
path: /
user: k8s-cephfs-user
secretRef:
name: k8s-cephfs-user-keyring
在pod中验证挂载
在其中一个pod写入测试页面，从其他的pod访问测试页面验证数据共享
静态pv
创建cephfs pv和pvc
apiVersion: v1
kind: PersistentVolume
metadata:
name: cephfs-pv
spec:
accessModes: ["ReadWriteOnce"]
capacity:
storage: 2Gi
persistentVolumeReclaimPolicy: Retain
cephfs:
monitors: ["192.168.211.23:6789", "192.168.211.24:6789", "192.168.211.25:6789"]
path: /
user: k8s-cephfs-user
secretRef:
name: k8s-cephfs-user
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cephfs-pvc
namespace: default
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 2Gi
配置pod使用cephfs-pv
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-with-pvc
spec:
replicas: 2
selector:
matchLabels:
app: nginx
strage: cephfs-pvc
template:
metadata:
labels:
app: nginx
strage: cephfs-pvc
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: web
mountPath: /usr/share/nginx/html/
volumes:
- name: web
persistentVolumeClaim:
claimName: cephfs-pvc
readOnly: false
进入pod验证
目前k8s内置的cephfs存储插件还不能实现动态pv，可以通过ceph官方的ceph-csi插件来实现cephfs的动态pv功能：https://github.com/ceph/ceph-csi