kubernetes -- helm charts 开发: 19、卸载pv重新挂载pv的解决方法
卸载mongodb老的pv,挂载新的pv操作方法如下:
1 先查看原来的pv
kubectl get pv -n openstack | grep mongo
样例输出信息如下:
pvc-e353b86f-c971-11e9-a602-fa163ea65c3c 500Gi RWO Delete Bound openstack/mongodb-data-mongodb-0 general 9m
注意:
pv是pvc-e353b86f-c971-11e9-a602-fa163ea65c3c
pvc的名称是 mongodb-data-mongodb-0
2 查看pv的具体信息
kubectl get pv -n openstack pvc-e353b86f-c971-11e9-a602-fa163ea65c3c -o yaml
注意:
请将 pvc-e353b86f-c971-11e9-a602-fa163ea65c3c 替换为真实的pv名称
样例输出信息如下:
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: ceph.com/rbd
rbdProvisionerIdentity: ceph-rbd-provisioner-74d7556897-75hzh
creationTimestamp: 2019-08-28T08:57:54Z
name: pvc-e353b86f-c971-11e9-a602-fa163ea65c3c
resourceVersion: "122533"
selfLink: /api/v1/persistentvolumes/pvc-e353b86f-c971-11e9-a602-fa163ea65c3c
uid: ec0f3314-c971-11e9-a602-fa163ea65c3c
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 500Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: mongodb-data-mongodb-0
namespace: openstack
resourceVersion: "122488"
uid: e353b86f-c971-11e9-a602-fa163ea65c3c
persistentVolumeReclaimPolicy: Delete
rbd:
image: kubernetes-dynamic-pvc-e7a0686d-c971-11e9-8d2e-0a58c210020f
keyring: /etc/ceph/keyring
monitors:
- ceph-mon.ceph.svc.cluster.local:6789
pool: rbd
secretRef:
name: pvc-ceph-client-key
user: admin
storageClassName: general
status:
phase: Bound
3 删除pv,pvc之前需要先将使用pv,pvc的statefulset副本数设置为0
防止pod还在尝试挂载pv
kubectl scale statefulset -n openstack mongodb --replicas=0
4 删除pv
kubectl delete pv -n openstack pvc-e353b86f-c971-11e9-a602-fa163ea65c3c
注意:
将 pvc-e353b86f-c971-11e9-a602-fa163ea65c3c 替换为真实的pv名称
样例输出信息如下:
persistentvolume "pvc-e353b86f-c971-11e9-a602-fa163ea65c3c" deleted
5 删除pvc
kubectl delete pvc -n openstack mongodb-data-mongodb-0
注意:
将 mongodb-data-mongodb-0 替换为真实的pvc名称
6 删除rbd对应信息
先进入busybox的pod
kubectl exec -n openstack busybox-openstack-7944b6668f-mpg84 -it bash
注意:
将 busybox-openstack-7944b6668f-mpg84 替换为真实的名称
然后查看待删除的rbd信息
rbd ls rbd | grep kubernetes-dynamic-pvc-e7a0686d-c971-11e9-8d2e-0a58c210020f
注意:
请将 kubernetes-dynamic-pvc-e7a0686d-c971-11e9-8d2e-0a58c210020f 替换为
步骤 2 查看pv的具体信息中的
rbd:
image: kubernetes-dynamic-pvc-e7a0686d-c971-11e9-8d2e-0a58c210020f
中image字段对应的真实的值
样例输出信息如下:
kubernetes-dynamic-pvc-e7a0686d-c971-11e9-8d2e-0a58c210020f
删除rbd相关信息:
rbd rm rbd/kubernetes-dynamic-pvc-e7a0686d-c971-11e9-8d2e-0a58c210020f
注意:
请将 kubernetes-dynamic-pvc-e7a0686d-c971-11e9-8d2e-0a58c210020f 替换为
步骤 2 查看pv的具体信息中的
rbd:
image: kubernetes-dynamic-pvc-e7a0686d-c971-11e9-8d2e-0a58c210020f
中image字段对应的真实的值
样例输出信息如下:
Removing image: 16% complete...
等待完全删除后
7 恢复应用statefulset的副本数
kubectl scale statefulset -n openstack mongodb --replicas=1
等待一定时间，可以看到mongodb状态变成running。
问题原因:
类似问题:
https://jira.mongodb.org/browse/SERVER-30077
该问题的解决方法
Thank you for the report. Unfortunately this error indicates that the disk corruption affected more than just the WiredTiger.wt file. In this situation, I would recommend checking the integrity of your disks and then restoring from a backup or resyncing from another node if possible.
另外一个类似问题:
https://jira.mongodb.org/browse/SERVER-25794
解决方法:
We do not support the steps you have referenced. If your database cannot be recovered using the built-in repair or the attempted manual repair effort, I unfortunately don't have any further suggestions to offer.
I would recommend extreme caution if you are attempting to recover data using the wt binary, as it is not aware of the data structures used by MongoDB and may be using a different version of the WiredTiger engine.
Finally, please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. As such, I'm unable to provide additional support for data recovery on this forum beyond the initial repair attempt which was not successful.
总结:
出现问题的原因是mongodb使用的磁盘发生损坏，导致mongodb无法启动。
解决方法:
目前的workroud是手动卸载老的pv,给mongodb挂上新的pv。