起因
开发反馈habor镜像库登陆不了，初步查看是证书过期了。
解决方案
之前Harbor-helm部署镜像库文档可以回顾链接Kubernetes1.13.1集群集成Harbor-helm
1.首先新建新证书的secret
[root@elasticsearch01 harbor-helm]# kubectl create secret tls ingress-secret2021 --key minminmsnauto.key --cert minminmsnauto.crt 2.然后修改harbor-helm的value.yaml，把secretName替换下
[root@elasticsearch01 harbor-helm]# head -n 20 values.yaml
expose:
# Set the way how to expose the service. Set the type as "ingress",
# "clusterIP" or "nodePort" and fill the information in the corresponding
# section
type: ingress
tls:
# Enable the tls or not. Note: if the type is "ingress" and the tls
# is disabled, the port must be included in the command when pull/push
# images. Refer to https://github.com/goharbor/harbor/issues/5291
# for the detail.
enabled: true
# Fill the name of secret if you want to use your own TLS certificate
# and private key. The secret must contain keys named tls.crt and
# tls.key that contain the certificate and private key to use for TLS
# The certificate and private key will be generated automatically if
# it is not set
secretName: "ingress-secret2021"
# By default, the Notary service will use the same cert and key as
# described above. Fill the name of secret if you want to use a
# separated one. Only needed when the type is "ingress".3.最后使用helm upgrade更新版本
[root@elasticsearch01 harbor-helm]# helm upgrade minminmsn . -f values.yaml到这个时候应该能解决需求，可是事与愿违，不知道哪儿除了问题，这时登陆Harbor证书问题是解决了，但是项目及库访问不了提示内部错误，看Pod的运行状态也都是Running。
最后打算使用helm先delete掉再install，但是这样创建的harbor看起来一切正常，实际上是个初始化环境，是自动生成的新PV并没有原来的数据。此时发现原来的PV还在，下面就开始找PV恢复的方案。
调整PV状态
1.查询此时PV与PVC状态
[root@elasticsearch01 harbor-helm]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE 9h
pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 50Gi RWO Retain Released default/minminmsn-harbor-chartmuseum rbd 417d
pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Released default/minminmsn-harbor-jobservice rbd 417d
pvc-e7985b55-7ded-11e9-a09d-52540089b2b6 2000Gi RWO Retain Released default/minminmsn-harbor-registry rbd 417d
pvc-e7d38097-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Released default/database-data-minminmsn-harbor-database-0 rbd 417d
pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Released default/data-minminmsn-harbor-redis-0 rbd 417d
[root@elasticsearch01 harbor-helm]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-minminmsn-harbor-redis-0 Bound pvc-6cd422e4-c5f0-11ea-9386-52540089b2b6 20Gi RWO rbd 9h
database-data-minminmsn-harbor-database-0 Bound pvc-6ccda00b-c5f0-11ea-9386-52540089b2b6 20Gi RWO rbd 9h
minminmsn-harbor-chartmuseum Bound pvc-6c903857-c5f0-11ea-9386-52540089b2b6 50Gi RWO rbd 9h
minminmsn-harbor-jobservice Bound pvc-6c91d1a4-c5f0-11ea-9386-52540089b2b6 20Gi RWO rbd 9h
minminmsn-harbor-registry Bound pvc-6c92bfc0-c5f0-11ea-9386-52540089b2b6 500Gi RWO rbd 9h2.修改PV状态
先把PV的状态由Released改变成
备注：默认创建的PV的回收策略是Delete就是用完就删除，之前特意把RECLAIM POLICY改为了Retain，在线修改PV回收策略可以参考文档在线修改PV的回收策略，否则这里Helm Delete后就会自动删除PV，就没有后来这篇PV数据恢复操作了。
在线编辑PV,需要把其中claimRef这段删除，这样状态就可以变成Available了。
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: minminmsn-harbor-chartmuseum
namespace: default
resourceVersion: "91736092"
uid: b31ec8ca-c649-11ea-9386-52540089b2b6
persistentVolumeReclaimPolicy: Retain具体如下修改
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/bound-by-controller: "yes"
pv.kubernetes.io/provisioned-by: ceph.com/rbd
rbdProvisionerIdentity: ceph.com/rbd
creationTimestamp: "2019-05-24T06:33:55Z"
finalizers:
- kubernetes.io/pv-protection
name: pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6
resourceVersion: "91736100"
selfLink: /api/v1/persistentvolumes/pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6
uid: e7ade7f7-7ded-11e9-a09d-52540089b2b6
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 50Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: minminmsn-harbor-chartmuseum
namespace: default
resourceVersion: "91736092"
uid: b31ec8ca-c649-11ea-9386-52540089b2b6
persistentVolumeReclaimPolicy: Retain
rbd:
image: kubernetes-dynamic-pvc-e79b34d3-7ded-11e9-ac1b-02420afe4905
keyring: /etc/ceph/keyring
monitors:
- 10.0.4.8:6789
pool: rbd-k8s
secretRef:
name: ceph-secret
namespace: default
user: admin
storageClassName: rbd
volumeMode: Filesystem
status:
phase: Released
3.其他四个PV同样操作
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7985b55-7ded-11e9-a09d-52540089b2b6
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7d38097-7ded-11e9-a09d-52540089b2b6
[root@elasticsearch01 harbor-helm]# kubectl edit pv pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b64.查看效果
现在看PV的STATUS已经变成了Available，然后CLAIM也变空了，这样就可以在后面绑定使用了
[root@elasticsearch01 harbor-helm]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 50Gi RWO Retain Available rbd 417d
pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Available rbd 417d
pvc-e7985b55-7ded-11e9-a09d-52540089b2b6 2000Gi RWO Retain Available rbd 417d
pvc-e7d38097-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Available rbd 417d
pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Available rbd 417d创建PVC
1.先设置好PVC及PV对应关系
[root@elasticsearch01 yaml]# cat minminmsn.pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: minminmsn-harbor-registry
spec:
accessModes:
- ReadWriteOnce
storageClassName: "rbd"
resources:
requests:
storage: 2000Gi
volumeName: "pvc-e7985b55-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: minminmsn-harbor-jobservice
spec:
accessModes:
- ReadWriteOnce
storageClassName: "rbd"
resources:
requests:
storage: 20Gi
volumeName: "pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: minminmsn-harbor-chartmuseum
spec:
accessModes:
- ReadWriteOnce
storageClassName: "rbd"
resources:
requests:
storage: 50Gi
volumeName: "pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: database-data-minminmsn-harbor-database-0
spec:
accessModes:
- ReadWriteOnce
storageClassName: "rbd"
resources:
requests:
storage: 20Gi
volumeName: "pvc-e7d38097-7ded-11e9-a09d-52540089b2b6"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-minminmsn-harbor-redis-0
spec:
accessModes:
- ReadWriteOnce
storageClassName: "rbd"
resources:
requests:
storage: 20Gi
volumeName: "pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6"2.创建PVC
[root@elasticsearch01 yaml]# kubectl apply -f minminmsn.pvc
persistentvolumeclaim/minminmsn-harbor-registry created
persistentvolumeclaim/minminmsn-harbor-jobservice created
persistentvolumeclaim/minminmsn-harbor-chartmuseum created
persistentvolumeclaim/database-data-minminmsn-harbor-database-0 created
persistentvolumeclaim/data-minminmsn-harbor-redis-0 created3.检查PV与PVC
[root@elasticsearch01 yaml]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 50Gi RWO Retain Bound default/minminmsn-harbor-chartmuseum rbd 417d
pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Bound default/minminmsn-harbor-jobservice rbd 417d
pvc-e7985b55-7ded-11e9-a09d-52540089b2b6 2000Gi RWO Retain Bound default/minminmsn-harbor-registry rbd 417d
pvc-e7d38097-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Bound default/database-data-minminmsn-harbor-database-0 rbd 417d
pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6 20Gi RWO Retain Bound default/data-minminmsn-harbor-redis-0 rbd 417d
[root@elasticsearch01 yaml]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ceph-rbd-pv-claim Bound ceph-rbd-pv 20Gi RWO 540d
data-minminmsn-harbor-redis-0 Pending pvc-e7da3f3c-7ded-11e9-a09d-52540089b2b6 0 rbd 12s
database-data-minminmsn-harbor-database-0 Pending pvc-e7d38097-7ded-11e9-a09d-52540089b2b6 0 rbd 12s
minminmsn-harbor-chartmuseum Pending pvc-e7967cfe-7ded-11e9-a09d-52540089b2b6 0 rbd 12s
minminmsn-harbor-jobservice Pending pvc-e7974d1c-7ded-11e9-a09d-52540089b2b6 0 rbd 12s
minminmsn-harbor-registry Bound pvc-e7985b55-7ded-11e9-a09d-52540089b2b6 2000Gi RWO rbd 12s
[root@elasticsearch01 yaml]# kubectl describe pvc minminmsn-harbor-registry
Name: minminmsn-harbor-registry
Namespace: default
StorageClass: rbd
Status: Bound
Volume: pvc-e7985b55-7ded-11e9-a09d-52540089b2b6
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"minminmsn-harbor-registry","namespace":"default"},"spe...
pv.kubernetes.io/bind-completed: yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 2000Gi
Access Modes: RWO
VolumeMode: Filesystem
Events:
Mounted By: 使用Hlem重新部署Harbor镜像库
1.部署前先删除版本
[root@elasticsearch01 harbor-helm]# helm delete --purge minminmsn
helm delete --purge minminmsn
release "minminmsn" deleted2.修改Harbor-helm的values.yaml中PVC相关值
注意existingClaim: ""由空值改成上面生成的PVC名字，注意对应关系，其他不变，具体变更如下
persistence:
enabled: true
# Setting it to "keep" to avoid removing PVCs during a helm delete
# operation. Leaving it empty will delete PVCs after the chart deleted
resourcePolicy: "keep"
persistentVolumeClaim:
registry:
# Use the existing PVC which must be created manually before bound
existingClaim: "minminmsn-harbor-registry"
# Specify the "storageClass" used to provision the volume. Or the default
# StorageClass will be used(the default).
# Set it to "-" to disable dynamic provisioning
storageClass: "rbd"
subPath: ""
accessMode: ReadWriteOnce
size: 2000Gi
chartmuseum:
existingClaim: "minminmsn-harbor-chartmuseum"
storageClass: "rbd"
subPath: ""
accessMode: ReadWriteOnce
size: 50Gi
jobservice:
existingClaim: "minminmsn-harbor-jobservice"
storageClass: "rbd"
subPath: ""
accessMode: ReadWriteOnce
size: 20Gi
# If external database is used, the following settings for database will
# be ignored
database:
existingClaim: "database-data-minminmsn-harbor-database-0"
storageClass: "rbd"
subPath: ""
accessMode: ReadWriteOnce
size: 20Gi
# If external Redis is used, the following settings for Redis will
# be ignored
redis:
existingClaim: "data-minminmsn-harbor-redis-0"
storageClass: "rbd"
subPath: ""
accessMode: ReadWriteOnce
size: 20Gi3.重新部署
[root@elasticsearch01 harbor-helm]# helm install . --name minminmsn
NAME: minminmsn
LAST DEPLOYED: Wed Jul 15 11:18:13 2020
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
minminmsn-harbor-adminserver ClusterIP 10.254.58.23 80/TCP 1s
minminmsn-harbor-chartmuseum ClusterIP 10.254.154.44 80/TCP 1s
minminmsn-harbor-clair ClusterIP 10.254.25.107 6060/TCP 1s
minminmsn-harbor-core ClusterIP 10.254.56.153 80/TCP 1s
minminmsn-harbor-database ClusterIP 10.254.65.18 5432/TCP 1s
minminmsn-harbor-jobservice ClusterIP 10.254.81.97 80/TCP 1s
minminmsn-harbor-notary-server ClusterIP 10.254.99.90 4443/TCP 1s
minminmsn-harbor-notary-signer ClusterIP 10.254.175.105 7899/TCP 1s
minminmsn-harbor-portal ClusterIP 10.254.242.113 80/TCP 1s
minminmsn-harbor-redis ClusterIP 10.254.127.40 6379/TCP 1s
minminmsn-harbor-registry ClusterIP 10.254.158.222 5000/TCP,8080/TCP 1s
==> v1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
minminmsn-harbor-adminserver 1 1 1 0 1s
minminmsn-harbor-chartmuseum 1 1 1 0 1s
minminmsn-harbor-clair 1 0 0 0 1s
minminmsn-harbor-core 1 0 0 0 1s
minminmsn-harbor-jobservice 1 0 0 0 1s
minminmsn-harbor-notary-server 1 0 0 0 1s
minminmsn-harbor-notary-signer 1 0 0 0 1s
minminmsn-harbor-portal 1 0 0 0 1s
minminmsn-harbor-registry 1 0 0 0 1s
==> v1/StatefulSet
NAME DESIRED CURRENT AGE
minminmsn-harbor-database 1 1 1s
minminmsn-harbor-redis 1 1 1s
==> v1beta1/Ingress
NAME HOSTS ADDRESS PORTS AGE
minminmsn-harbor-ingress core-harbor.minminmsn.com,notary-harbor.minminmsn.com 80, 443 1s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
minminmsn-harbor-adminserver-b5d58db8c-wmrbd 0/1 ContainerCreating 0 1s
minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb 0/1 Pending 0 1s
minminmsn-harbor-clair-54465ff7dd-d7bxx 0/1 Pending 0 1s
minminmsn-harbor-core-587cc5d9b5-2xxl9 0/1 Pending 0 1s
minminmsn-harbor-jobservice-764bb697d-wsxqx 0/1 Pending 0 1s
minminmsn-harbor-notary-server-77fbb84fcc-2bw7c 0/1 Pending 0 1s
minminmsn-harbor-notary-signer-8466d68f5b-klv76 0/1 Pending 0 1s
minminmsn-harbor-database-0 0/1 Pending 0 1s
minminmsn-harbor-redis-0 0/1 Pending 0 1s
==> v1/Secret
NAME TYPE DATA AGE
minminmsn-harbor-adminserver Opaque 4 1s
minminmsn-harbor-chartmuseum Opaque 1 1s
minminmsn-harbor-core Opaque 4 1s
minminmsn-harbor-database Opaque 1 1s
minminmsn-harbor-jobservice Opaque 1 1s
minminmsn-harbor-registry Opaque 1 1s
==> v1/ConfigMap
NAME DATA AGE
minminmsn-harbor-adminserver 39 1s
minminmsn-harbor-chartmuseum 24 1s
minminmsn-harbor-clair 1 1s
minminmsn-harbor-core 1 1s
minminmsn-harbor-jobservice 1 1s
minminmsn-harbor-notary-server 5 1s
minminmsn-harbor-registry 2 1s
NOTES:
Please wait for several minutes for Harbor deployment to complete.
Then you should be able to visit the Harbor portal at https://core-harbor.minminmsn.com.
For more details, please visit https://github.com/goharbor/harbor.
3.查看新生成Pods的信息
[root@elasticsearch01 harbor-helm]# kubectl get pods
NAME READY STATUS RESTARTS AGE
minminmsn-harbor-adminserver-b5d58db8c-wmrbd 0/1 ContainerCreating 0 9s
minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb 0/1 ContainerCreating 0 9s
minminmsn-harbor-clair-54465ff7dd-d7bxx 0/1 Running 0 9s
minminmsn-harbor-core-587cc5d9b5-2xxl9 0/1 Running 0 9s
minminmsn-harbor-database-0 0/1 Init:0/1 0 9s
minminmsn-harbor-jobservice-764bb697d-wsxqx 0/1 ContainerCreating 0 9s
minminmsn-harbor-notary-server-77fbb84fcc-2bw7c 0/1 ContainerCreating 0 9s
minminmsn-harbor-notary-signer-8466d68f5b-klv76 0/1 ContainerCreating 0 9s
minminmsn-harbor-portal-64cf8b9b69-xm8nl 0/1 ContainerCreating 0 8s
minminmsn-harbor-redis-0 0/1 ContainerCreating 0 9s
minminmsn-harbor-registry-755746c5bb-q8m55 0/2 ContainerCreating 0 8s 再等2分钟查看就上恢复了
[root@elasticsearch01 harbor-helm]# kubectl get pods
NAME READY STATUS RESTARTS AGE
jenkins-0 1/1 Running 0 62d
rbd-provisioner-67b4857bcd-rjwlg 1/1 Running 0 61d
minminmsn-harbor-adminserver-b5d58db8c-wmrbd 1/1 Running 1 2m33s
minminmsn-harbor-chartmuseum-7c6b9d4977-94rhb 1/1 Running 0 2m33s
minminmsn-harbor-clair-54465ff7dd-d7bxx 1/1 Running 1 2m33s
minminmsn-harbor-core-587cc5d9b5-2xxl9 1/1 Running 1 2m33s
minminmsn-harbor-database-0 1/1 Running 0 2m33s
minminmsn-harbor-jobservice-764bb697d-wsxqx 1/1 Running 0 2m33s
minminmsn-harbor-notary-server-77fbb84fcc-2bw7c 1/1 Running 0 2m33s
minminmsn-harbor-notary-signer-8466d68f5b-klv76 1/1 Running 0 2m33s
minminmsn-harbor-portal-64cf8b9b69-xm8nl 1/1 Running 0 2m32s
minminmsn-harbor-redis-0 1/1 Running 0 2m33s
minminmsn-harbor-registry-755746c5bb-q8m55 2/2 Running 0 2m32s4.Harbor控制验证
证书更新了项目也恢复了
https://core-harbor.minminmsn.com/harbor/projects
