k3s集群监控（Rancher）删除之空间(namespace)cattle-system一直为Terminating状态解决方案

故障现象

使用Rancher2.0添加了k3s集群进行Kubernetes的相关web管理。

因删除k3s集群监控服务：删除Rancher依赖的namespace（cattle-system），状态一直是Terminating。

root@k3s:~# kubectl get namespace
NAME               STATUS        AGE
kube-system        Active        76d
kube-public        Active        76d
default            Active        76d
kube-node-lease    Active        76d
hollysys-edge      Active        49d
hollysys-project   Active        71d
monitoring         Active        51d
public             Active        71d
emojivoto          Active        72d
cattle-system      Terminating   34d

导致新的监控服务无法添加（Error from server (Forbidden): error when creating “STDIN”:）

root@k3s:~# curl --insecure -sfL https://***.***.***.**:9443/v3/import/kqxdpmckkj5xsgg696hhpx5xhchzh8nnq74kvczbvhbp5b5b5cl2s5.yaml | kubectl apply -f -
clusterrole.rbac.authorization.k8s.io/proxy-clusterrole-kubeapiserver created
clusterrolebinding.rbac.authorization.k8s.io/proxy-role-binding-kubernetes-master created
namespace/cattle-system unchanged
clusterrolebinding.rbac.authorization.k8s.io/cattle-admin-binding created
clusterrole.rbac.authorization.k8s.io/cattle-admin created
Error from server (Forbidden): error when creating "STDIN": serviceaccounts "cattle" is forbidden: unable to create new content in namespace cattle-system because it is being terminated
Error from server (Forbidden): error when creating "STDIN": secrets "cattle-credentials-855d8e8" is forbidden: unable to create new content in namespace cattle-system because it is being terminated
Error from server (Forbidden): error when creating "STDIN": deployments.apps "cattle-cluster-agent" is forbidden: unable to create new content in namespace cattle-system because it is being terminated
Error from server (Forbidden): error when creating "STDIN": daemonsets.apps "cattle-node-agent" is forbidden: unable to create new content in namespace cattle-system because it is being terminated

解决办法

kubectl patch namespace cattle-system -p '{"metadata":{"finalizers":[]}}' --type='merge' -n cattle-system
kubectl delete namespace cattle-system --grace-period=0 --force

kubectl patch namespace cattle-global-data -p '{"metadata":{"finalizers":[]}}' --type='merge' -n cattle-system
kubectl delete namespace cattle-global-data --grace-period=0 --force

kubectl patch namespace local -p '{"metadata":{"finalizers":[]}}' --type='merge' -n cattle-system

for resource in `kubectl api-resources --verbs=list --namespaced -o name | xargs -n 1 kubectl get -o name -n local`; do kubectl patch $resource -p '{"metadata": {"finalizers": []}}' --type='merge' -n local; done

kubectl delete namespace local --grace-period=0 --force

执行完成后查看namespace

root@k3s:~# kubectl get namespaces
NAME               STATUS   AGE
kube-system        Active   76d
kube-public        Active   76d
default            Active   76d
kube-node-lease    Active   76d
hollysys-edge      Active   49d
hollysys-project   Active   71d
monitoring         Active   51d
public             Active   71d
emojivoto          Active   72d

之后再rancher界面添加新的集群监控即可重新部署该k3s集群到新的rancher中进行web管理