K8s之kubectl命令行工具常用命令（三）

Kubectl是管理k8s集群的命令行工具，通过生成的json格式传递给apiserver进行创建、查看、管理的操作

注意：此处需要用到我们之前部署的K8s多节点的部署环境，如果还未部署的可以参考我的上篇文章：https://blog.csdn.net/JarryZho/article/details/104212822

常用命令行：

`查看帮助命令`
[root@master1 ~]# kubectl --help
kubectl controls the Kubernetes cluster manager.
Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/

Basic Commands (Beginner):
  create         Create a resource from a file or from stdin.
  expose         使用 replication controller, service, deployment 或者 pod 并暴露它作为一个 新的
Kubernetes Service
  run            在集群中运行一个指定的镜像
  set            为 objects 设置一个指定的特征

Basic Commands (Intermediate):
  explain        查看资源的文档
  get            显示一个或更多 resources
  edit           在服务器上编辑一个资源
  delete         Delete resources by filenames, stdin, resources and names, or by resources and label selector

Deploy Commands:
  rollout        Manage the rollout of a resource
  scale          为 Deployment, ReplicaSet, Replication Controller 或者 Job 设置一个新的副本数量
  autoscale      自动调整一个 Deployment, ReplicaSet, 或者 ReplicationController 的副本数量

Cluster Management Commands:
  certificate    修改 certificate 资源.
  cluster-info   显示集群信息
  top            Display Resource (CPU/Memory/Storage) usage.
  cordon         标记 node 为 unschedulable
  uncordon       标记 node 为 schedulable
  drain          Drain node in preparation for maintenance
  taint          更新一个或者多个 node 上的 taints

Troubleshooting and Debugging Commands:
  describe       显示一个指定 resource 或者 group 的 resources 详情
  logs           输出容器在 pod 中的日志
  attach         Attach 到一个运行中的 container
  exec           在一个 container 中执行一个命令
  port-forward   Forward one or more local ports to a pod
  proxy          运行一个 proxy 到 Kubernetes API server
  cp             复制 files 和 directories 到 containers 和从容器中复制 files 和 directories.
  auth           Inspect authorization

Advanced Commands:
  apply          通过文件名或标准输入流(stdin)对资源进行配置
  patch          使用 strategic merge patch 更新一个资源的 field(s)
  replace        通过 filename 或者 stdin替换一个资源
  wait           Experimental: Wait for a specific condition on one or many resources.
  convert        在不同的 API versions 转换配置文件

Settings Commands:
  label          更新在这个资源上的 labels
  annotate       更新一个资源的注解
  completion     Output shell completion code for the specified shell (bash or zsh)

Other Commands:
  alpha          Commands for features in alpha
  api-resources  Print the supported API resources on the server
  api-versions   Print the supported API versions on the server, in the form of "group/version"
  config         修改 kubeconfig 文件
  plugin         Provides utilities for interacting with plugins.
  version        输出 client 和 server 的版本信息

接下来我们以项目的形式讲其中的命令

其步骤如下：创建一>发布一>更新一>回滚一>删除

1.创建nginx

#语法：kubectl run NAME --image=image [--env="key=value"参数] [--port=port端口] [--replicas=replicas副本集] [--dry-run=bool状态] [--overrides=inline-json] [--command命令] -- [COMMAND] [args...] [options]`

`示例：`
[root@master1 k8s]# kubectl run nginx-deployment --image=nginx --port=80 --replicas=3
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
deployment.apps/nginx-deployment created
[root@master1 k8s]# kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
nginx-dbddb74b8-7hdfj               1/1     Running   0          4d18h
`nginx-deployment-5477945587-2dljt   1/1     Running   0          68s`
`nginx-deployment-5477945587-tt8vx   1/1     Running   0          68s`
`nginx-deployment-5477945587-wsb69   1/1     Running   0          68s`
#最后三个就是我们新创建的副本集

---

2.发布nginx service提供负载均衡的功能

#语法：kubectl expose (-f FILENAME | TYPE NAME) [--port=port群集之间内部通信的端口] [--protocol=TCP|UDP|SCTP] [--target-port对外暴露的端口=number-or-name] [--name=name指定名称] [--external-ip=external-ip-of-service] [--type=type指定类型] [options]

`示例：`
[root@master1 k8s]# kubectl expose deployment nginx-deployment --port=80 --target-port=80 --name=nginx-deployment-service --type=NodePort
service/nginx-deployment-service exposed

`查看发布`
[root@master1 k8s]# kubectl get pods,svc		#此处svc位service服务组件的缩写
NAME                                    READY   STATUS    RESTARTS   AGE
pod/nginx-dbddb74b8-7hdfj               1/1     Running   0          4d19h
pod/nginx-deployment-5477945587-2dljt   1/1     Running   0          13m
pod/nginx-deployment-5477945587-tt8vx   1/1     Running   0          13m
pod/nginx-deployment-5477945587-wsb69   1/1     Running   0          13m

NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
service/kubernetes      ClusterIP   10.0.0.1     <none>        443/TCP        6d19h
service/nginx-service   NodePort    10.0.0.242   <none>        80:48330/TCP   111s
#此时对外暴露的端口为48330

`查看资源对象简写`
[root@master1 k8s]# kubectl api-resources
NAME                              SHORTNAMES   APIGROUP                       NAMESPACED 
componentstatuses                 cs                                          false       
configmaps                        cm                                          true       
endpoints                         ep                                          true       
events                            ev                                          true       
limitranges                       limits                                      true       
namespaces                        ns                                          false       
nodes                             no                                          false       
persistentvolumeclaims            pvc                                         true       
persistentvolumes                 pv                                          false       
pods                              po                                          true      
replicationcontrollers            rc                                          true       
resourcequotas                    quota                                       true       
serviceaccounts                   sa                                          true       
services                          svc                                         true       
customresourcedefinitions         crd,crds     apiextensions.k8s.io           false       
daemonsets                        ds           apps                           true       
deployments                       deploy       apps                           true       
replicasets                       rs           apps                           true       
statefulsets                      sts          apps                           true       
horizontalpodautoscalers          hpa          autoscaling                    true       
cronjobs                          cj           batch                          true       
certificatesigningrequests        csr          certificates.k8s.io            false       
events                            ev           events.k8s.io                  true       
daemonsets                        ds           extensions                     true       
deployments                       deploy       extensions                     true       
ingresses                         ing          extensions                     true       
networkpolicies                   netpol       extensions                     true       
podsecuritypolicies               psp          extensions                     false       
replicasets                       rs           extensions                     true       
networkpolicies                   netpol       networking.k8s.io              true       
poddisruptionbudgets              pdb          policy                         true       
podsecuritypolicies               psp          policy                         false       
priorityclasses                   pc           scheduling.k8s.io              false       
storageclasses                    sc           storage.k8s.io                 false       

`查看关联后端的节点`
[root@master1 k8s]# kubectl get endpoints
NAME                       ENDPOINTS                                      AGE
kubernetes                 192.168.18.128:6443,192.168.18.132:6443        7d4h
nginx-deployment-service   172.17.32.4:80,172.17.40.2:80,172.17.40.3:80   17s

`网络状态详细信息`
[root@master1 ~]# kubectl get pods -o wide
NAME                                READY   STATUS    RESTARTS   AGE    IP            NODE             NOMINATED NODE
nginx-dbddb74b8-7hdfj               1/1     Running   0          5d5h   172.17.32.2   192.168.18.148   <none>
nginx-deployment-5477945587-2dljt   1/1     Running   0          10h    172.17.40.3   192.168.18.145   <none>
nginx-deployment-5477945587-tt8vx   1/1     Running   0          10h    172.17.40.2   192.168.18.145   <none>
nginx-deployment-5477945587-wsb69   1/1     Running   0          10h    172.17.32.4   192.168.18.148   <none>

`服务暴露的端口`
[root@master1 ~]# kubectl get svc
NAME                       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
kubernetes                 ClusterIP   10.0.0.1     <none>        443/TCP        7d5h
nginx-deployment-service   NodePort    10.0.0.50    <none>        80:48330/TCP   70m

`在node1操作，查看负载均衡端口48330`
`k8s里kube-proxy支持三种模式，在v1.8之前我们使用的是iptables以及userspace两种模式，在k8s1.8之后引入了ipvs模式`
[root@node1 ~]# yum install ipvsadm -y
[root@node1 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.18.148:48330 rr
  -> 172.17.32.4:80               Masq    1      0          0
  -> 172.17.40.2:80               Masq    1      0          0
  -> 172.17.40.3:80               Masq    1      0          0
#对外提供端口48330，调度算法为rr轮询

`在node2操作 同样安装ipvsadmin工具查看`
[root@node2 ~]# yum install ipvsadm -y
[root@node2 ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.18.145:48330 rr
  -> 172.17.32.4:80               Masq    1      0          0
  -> 172.17.40.2:80               Masq    1      0          0
  -> 172.17.40.3:80               Masq    1      0          0

我们现在宿主机中使用浏览器访问192.168.18.148:48330和192.168.18.145:48330都可以访问到nginx的主页，然后再查看日志

`在master1操作查看访问日志（注意：如果访问其他node无法访问检查proxy组件）`
[root@master1 ~]# kubectl get pods		#此时会有三个副本集
NAME                                READY   STATUS    RESTARTS   AGE
nginx-dbddb74b8-7hdfj               1/1     Running   0          5d5h
nginx-deployment-5477945587-2dljt   1/1     Running   0          10h
nginx-deployment-5477945587-tt8vx   1/1     Running   0          10h
nginx-deployment-5477945587-wsb69   1/1     Running   0          10h

`此时回头查看访问日志：`
[root@master1 ~]# kubectl logs nginx-deployment-5477945587-2dljt
172.17.40.1 - - [12/Feb/2020:11:44:46 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.131 Safari/537.36" "-"
172.17.40.1 - - [12/Feb/2020:11:44:46 +0000] "GET /favicon.ico HTTP/1.1" 404 555 "http://192.168.18.145:48330/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.131 Safari/537.36" "-"

[root@master1 ~]# kubectl logs nginx-deployment-5477945587-tt8vx

[root@master1 ~]# kubectl logs nginx-deployment-5477945587-wsb69
172.17.32.1 - - [12/Feb/2020:11:47:07 +0000] "GET / HTTP/1.1" 200 612 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.131 Safari/537.36" "-"
172.17.32.1 - - [12/Feb/2020:11:47:07 +0000] "GET /favicon.ico HTTP/1.1" 404 555 "http://192.168.18.148:48330/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.131 Safari/537.36" "-"

#此时nginx-deployment-5477945587-wsb69和nginx-deployment-5477945587-2dljt分别被访问了nginx的主页，说明业务已经成功发布，而且对方用的是轮询的方式访问多个Pod资源

---

3.更新nginx 为1.14版本

查看nginx版本步骤：F12打开开发者选项--》F5刷新访问---》找到network---》点击name---》找到headers头部信息

[root@master1 ~]# kubectl set image deployment/nginx-deployment nginx-deployment=nginx:1.14
deployment.extensions/nginx-deployment image updated

`处于动态监听状态，此时可以查看监控状态`
[root@master1 ~]# kubectl get pods -w
NAME                                READY   STATUS              RESTARTS   AGE
nginx-6c94d899fd-8pf48              1/1     Running             0          3m54s
nginx-deployment-5477945587-2dljt   1/1     Running             0          10h
nginx-deployment-5477945587-tt8vx   1/1     Running             0          10h
nginx-deployment-5477945587-wsb69   1/1     Running             0          10h
nginx-deployment-8f66bcd89-jncdr    0/1     ContainerCreating   0          7s
#此时哦我们可以按Ctrl+c中断监听，这样更新速度快

验证：我们再次重载宿主机中的nginx主页，此时在开发者选项中看到的头部信息里nginx的版本更新为了1.14.2

---

4.回滚nginx

`查看历史版本`
[root@master1 ~]# kubectl rollout history deployment/nginx-deployment
deployment.extensions/nginx-deployment
REVISION  CHANGE-CAUSE
1         <none>	#1.17版本
2         <none>	#1.14版本

`执行回滚到上一次`
[root@master1 ~]#  kubectl rollout undo deployment/nginx-deployment
deployment.extensions/nginx-deployment

`检查回滚状态`
[root@master1 ~]# kubectl rollout status deployment/nginx-deployment
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
deployment "nginx-deployment" successfully rolled out
#最终显示成功回滚

验证：再回到宿主机的浏览器刷新nginx主页，此时在页面开发者选择的头部信息中看到的nginx版本就恢复到了之前的1.17.8

---

5.删除nginx

`查看deployment`
[root@master1 ~]# kubectl get deploy
NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3         3         3            3           11h
[root@master1 ~]# kubectl delete deployment/nginx-deployment
deployment.extensions "nginx-deployment" deleted

[root@master1 ~]# kubectl get deploy
No resources found.
[root@master1 ~]# kubectl get pods
No resources found.

`删除服务SVC`
[root@master1 ~]# kubectl get svc
NAME                       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
kubernetes                 ClusterIP   10.0.0.1     <none>        443/TCP        7d6h
nginx-deployment-service   NodePort    10.0.0.50    <none>        80:48330/TCP   142m
[root@master1 ~]# kubectl delete svc/nginx-deployment-service
service "nginx-deployment-service" deleted	#此步骤为删除步骤
[root@master1 ~]# kubectl get svc
NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
kubernetes      ClusterIP   10.0.0.1     <none>        443/TCP        7d7h

---

6.其它

`查看具体资源的详细信息`
[root@master1 ~]# kubectl run nginx-deployment --image=nginx --port=80 --replicas=3
kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead.
deployment.apps/nginx-deployment created

[root@master1 ~]# kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
nginx-6c94d899fd-8pf48              1/1     Running   0          54m
nginx-deployment-5477945587-f5dsm   1/1     Running   0          15s
nginx-deployment-5477945587-hmgd2   1/1     Running   0          15s
nginx-deployment-5477945587-pl2hn   1/1     Running   0          15s

[root@master1 ~]# kubectl describe pod nginx-deployment-5477945587-f5dsm
Name:               nginx-deployment-5477945587-f5dsm
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               192.168.18.145/192.168.18.145
Start Time:         Wed, 12 Feb 2020 21:04:40 +0800
Labels:             pod-template-hash=5477945587
                    run=nginx-deployment
Annotations:        <none>
Status:             Running
IP:                 172.17.40.2
Controlled By:      ReplicaSet/nginx-deployment-5477945587
Containers:
  nginx-deployment:
    Container ID:   docker://670cb7230f200279b2accb344766e621ab97d279c8585cc27bd4e519dac9e677
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:ad5552c786f128e389a0263104ae39f3d3c7895579d45ae716f528185b36bc6f
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Wed, 12 Feb 2020 21:04:44 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-pbr9p (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-pbr9p:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-pbr9p
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From                     Message
  ----    ------     ----  ----                     -------
  Normal  Scheduled  19m   default-scheduler        Successfully assigned default/nginx-deployment-5477945587-f5dsm to 192.168.18.145
  Normal  Pulling    19m   kubelet, 192.168.18.145  pulling image "nginx"
  Normal  Pulled     19m   kubelet, 192.168.18.145  Successfully pulled image "nginx"
  Normal  Created    19m   kubelet, 192.168.18.145  Created container
  Normal  Started    19m   kubelet, 192.168.18.145  Started container

`查看deployment资源`
[root@master1 ~]# kubectl describe deployment/nginx-deployment
Name:                   nginx-deployment
Namespace:              default
CreationTimestamp:      Wed, 12 Feb 2020 21:04:40 +0800
Labels:                 run=nginx-deployment
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               run=nginx-deployment
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  run=nginx-deployment
  Containers:
   nginx-deployment:
    Image:        nginx
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   nginx-deployment-5477945587 (3/3 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  21m   deployment-controller  Scaled up replica set nginx-deployment-5477945587 to 3
  
`进入pod`
[root@master1 ~]# kubectl exec -it nginx-deployment-5477945587-f5dsm bash
root@nginx-deployment-5477945587-f5dsm:/#

问题集锦及原因：

`1:状态为NotReady`
[root@localhost bin]# kubectl get node
NAME              STATUS     ROLES    AGE     VERSION
192.168.195.150   NotReady   <none>   6d19h   v1.12.3
192.168.195.151   NotReady   <none>   6d18h   v1.12.3
#原因: node 节点无法连接apiserver
1)单节点：master节点	apiserver服务是否正常运行
systemctl restart kube-apiserver	失败
cat /var/log/messages	日志（第一次部署：检查证书）
2）多节点：如果api-server运行正常
检查：负载均衡中的VIP，检查nginx调度 四层转发模块配置

`2：kubectl get pods 查询podes资源状态`
NAME                     READY   STATUS        RESTARTS   AGE
nginx-7697996758-jg47q   0/1     Pending       0          93s
nginx-7697996758-k967k   0/1     Pending       0          93s
nginx-7697996758-p5n8m   0/1     Pending       0          93s
无法配置资源到node节点中进行创建（处于pending状态）
检查node节点中kubelet服务组件，kubeconfig配置


`3：客户无法访问pods发布的服务`
ipvsadm -L -n	负载均衡	对外提供的端口44888
#正常情况每个node应该是以下状态：
TCP  192.168.195.151:44888 rr
  -> 172.17.68.2:80               Masq    1      0          0    
  -> 172.17.68.3:80               Masq    1      0          0     
  -> 172.17.86.2:80               Masq    1      0          0 
#原因：kube-proxy组件服务