记录一次kubectl出错事件

一、 问题

上午还在正常使用的Kubernetes集群 下午使用kubectl命令一直提示The connection to the server 172.16.77.53:6443 was refused - did you specify the right host or port?

二、排查

[docker@k8s-77-53 ~]$ kubectl get pods
The connection to the server 172.16.77.53:6443 was refused - did you specify the right host or port?

使用docker查看k8s相关容器及apiserver是否正常

[docker@k8s-77-53 ~]$ docker ps
CONTAINER ID        IMAGE                                            COMMAND                   CREATED             STATUS              PORTS                         NAMES
f1d6f5acd4b2        144ecb80a73c                                     "/usr/sbin/init"          32 hours ago        Up 32 hours                                       k8s_test_test-0_default_4c6c78ec-64a2-11e9-b297-6cae8b77ae08_0
e2079cd28387        k8s.gcr.io/pause:3.1                             "/pause"                  32 hours ago        Up 32 hours
b5a315d1477b        40a63db91ef8                                     "kube-apiserver --au…"    3 months ago        Up 3 months                                       k8s_kube-apiserver_kube-apiserver-k8s-77-53_kube-system_467247f0aca129ce9b864d2562c1e575_6
....

发现容器一切正常 业务系统也在正常运行 就是kubectl没办法用

排查k8s请求

[docker@k8s-77-53 ~]$ kubectl -v=8  get nodes
I0423 16:09:42.199244   26311 loader.go:359] Config loaded from file /etc/kubernetes/admin.conf
I0423 16:09:42.201189   26311 loader.go:359] Config loaded from file /etc/kubernetes/admin.conf
I0423 16:09:42.203029   26311 round_trippers.go:416] GET https://172.16.77.53:6443/api?timeout=32s
I0423 16:09:42.203145   26311 round_trippers.go:423] Request Headers:
I0423 16:09:42.203179   26311 round_trippers.go:426]     Accept: application/json, */*
I0423 16:09:42.203212   26311 round_trippers.go:426]     User-Agent: kubectl/v1.13.1 (linux/amd64) kubernetes/eec55b9
I0423 16:09:42.204660   26311 round_trippers.go:441] Response Status:  in 1 milliseconds
I0423 16:09:42.204727   26311 round_trippers.go:444] Response Headers:
I0423 16:09:42.204860   26311 cached_discovery.go:113] skipped caching discovery info due to Get https://172.16.77.53:6443/api?timeout=32s: proxyconnect tcp: dial tcp 172.16.77.36:8118: connect: connection refused

这里能看的出应该是宿主机代理出现了问题

三、解决问题

为了能正常使用kubectl 先把代理注释掉

[docker@k8s-77-53 ~]$ vi /etc/profile
...
[docker@k8s-77-52 ~]$ source /etc/profile

再使用kubectl看看是否正常

[docker@k8s-77-53 ~]$ kubectl get pods
The connection to the server 172.16.77.53:6443 was refused - did you specify the right host or port?

发现还是原来的错误

经过查看后发现 代理还在

[docker@k8s-77-53 ~]$ env | grep -i  proxy
http_proxy=http://172.16.77.36:8118
https_proxy=http://172.16.77.36:8118

使用unset取消

[docker@k8s-77-53 ~]$ unset http_proxy
[docker@k8s-77-53 ~]$ unset https_proxy

然后查看代理是否还在

[docker@k8s-77-53 ~]$ env | grep -i  proxy
[docker@k8s-77-53 ~]$

发现这次取消成功了，然后再查看kubectl是否能正常使用

[docker@k8s-77-53 ~]$ kubectl get pods
NAME                           READY   STATUS    RESTARTS   AGE
cp-schema-registry-0           1/1     Running   0          6d6h
test-0                         1/1     Running   0          31h

现在已可以正常使用
注意：上面那个代理是由于已经写入了系统变量，所以在source之后还在应用。必须得unset或者再重开个ssh窗口才能取消