在现网情况下,业务升级常常需要停机,下面我们使用Deployment来进行不停机升级和回滚.
1.先使用创建多个nginx的pod
[root@node-1 pod]# cat nginx_deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
[root@node-1 pod]#
[root@node-1 pod]# kubectl create -f nginx_deployment.yaml
[root@node-1 pod]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-deployment-54f57cf6bf-lh2rv 1/1 Running 0 74s
nginx-deployment-54f57cf6bf-psbqx 1/1 Running 0 74s
nginx-deployment-54f57cf6bf-tgtms 1/1 Running 0 74s
2.现在我们对镜像从nginx:1.7.9升级到nginx:1.9.1
方法1:直接使用命令行修改镜像:
[root@node-1 pod]# kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1
deployment.apps/nginx-deployment image updated
方法2:使用kubectl edit deployment命令修改deployment的配置信息
[root@node-1 pod]# kubectl edit deployments.apps nginx-deployment
使用kubectl rollout status查看进度,正在
[root@node-1 pod]# kubectl rollout status deployment nginx-deployment
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment "nginx-deployment" rollout to finish: 1 old replicas are pending termination...
deployment "nginx-deployment" successfully rolled out
实际上deployment操作是的rs,nginx-deployment-54f57cf6bf 是旧的rs,nginx-deployment-56f8998dbc是新的deployment
[root@node-1 pod]# kubectl get rs
NAME DESIRED CURRENT READY AGE
centos-56d49f9c64 0 0 0 7d4h
centos-6b68465d79 1 1 1 7d4h
nginx-deployment-54f57cf6bf 0 0 0 11m
nginx-deployment-56f8998dbc 3 3 3 6m58s
3.系统如何升级的说明
更新的过程中,系统会创建临时的deployment对象,先把旧的deployment的replicas减1,新的deployment加1,直到旧的deployment的副本为0,升级完成.
使用kubectl edit deployments命令,你可以看到spec下的strategy一些关于升级的默认策略.
spec:
strategy:
rollingUpdate:
maxSurge: 25% #新旧副本pod的总数不超过配置数的比率,如果100个pod,那么运行中的和升级中的pod总和不超过125.
maxUnavailable: 25% #升级过程中不可用状态pod的上限,意思就是升级过程中确保处于running状态的pod是75%以上,再继续更新.
type: RollingUpdate#以滚动更新的方式逐个更新pod,如果是Recreate,系统会杀死所有的pod,重建pod.
4.回滚
现网中,如果发现新版本存在问题,需要回退,需要一系列的操作还原,还需要进行详细的测试,防止回退时配置错误.
deployment记录了升级前的配置,通过deployment,我们更容易将版本进行回退.
首先查看更新记录
[root@node-1 pod]# kubectl rollout history deployment nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
1 <none>
2 <none>
我们发现没有记录更新信息,原因是因为我们在创建deployment的时候没有配置记录命令.创建的时候加上–record参数
[root@node-1 pod]# kubectl create -f nginx_deployment.yaml --record
deployment.apps/nginx-deployment created
[root@node-1 pod]#
#为了模拟版本的问题,我们升级的镜像选择一个不存在的版本
[root@node-1 pod]# kubectl set image deployment/nginx-deployment nginx=nginx:2020
#查看状态,发现由于找不到镜像,会一直卡在这里
[root@node-1 pod]# kubectl rollout status deployment nginx-deployment
Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
#查看RS状态,发现新的RS一直在创建第一个pod,
[root@node-1 pod]# kubectl get rs
NAME DESIRED CURRENT READY AGE
nginx-deployment-54f57cf6bf 3 3 3 2m47s
nginx-deployment-569cb44c46 1 1 0 80s
#查看pod,发现有一个pod拉取镜像失败.
[root@node-1 pod]# kubectl get pod
NAME READY STATUS RESTARTS AGE
centos-6b68465d79-5w9df 1/1 Running 0 7d5h
nginx-deployment-54f57cf6bf-2dfh8 1/1 Running 0 3m57s
nginx-deployment-54f57cf6bf-57plj 1/1 Running 0 3m57s
nginx-deployment-54f57cf6bf-vjpwq 1/1 Running 0 3m57s
nginx-deployment-569cb44c46-g2bq7 0/1 ImagePullBackOff 0 2m30s
#发现错误,我们执行回滚,先查看更新记录
[root@node-1 pod]# kubectl rollout history deployment nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
1 kubectl create --filename=nginx_deployment.yaml --record=true
2 kubectl create --filename=nginx_deployment.yaml --record=true
#添加--revision=n的方式选择查看具体更新版本的信息
[root@node-1 pod]# kubectl rollout history deployment nginx-deployment --revision=1
deployment.apps/nginx-deployment with revision #1
Pod Template:
Labels: app=nginx
pod-template-hash=54f57cf6bf
Annotations: kubernetes.io/change-cause: kubectl create --filename=nginx_deployment.yaml --record=true
Containers:
nginx:
Image: nginx:1.7.9
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
[root@node-1 pod]# kubectl rollout history deployment nginx-deployment --revision=2
deployment.apps/nginx-deployment with revision #2
Pod Template:
Labels: app=nginx
pod-template-hash=569cb44c46
Annotations: kubernetes.io/change-cause: kubectl create --filename=nginx_deployment.yaml --record=true
Containers:
nginx:
Image: nginx:2020
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
[root@node-1 pod]#
#我们发现版本1是我们需要还原的版本,执行回滚,--to-revision=1为指定回滚版本,默认是上一个版本.
[root@node-1 pod]# kubectl rollout undo deployment nginx-deployment --to-revision=1
deployment.apps/nginx-deployment rolled back
[root@node-1 pod]#
#查看pod已还原为原来的状态了
[root@node-1 pod]# kubectl get pod
NAME READY STATUS RESTARTS AGE
centos-6b68465d79-5w9df 1/1 Running 0 7d5h
nginx-deployment-54f57cf6bf-2dfh8 1/1 Running 0 8m44s
nginx-deployment-54f57cf6bf-57plj 1/1 Running 0 8m44s
nginx-deployment-54f57cf6bf-vjpwq 1/1 Running 0 8m44s
如果镜像拉取正常,已经完成更新了,pod也运行正常,只是业务异常,可以回退吗?
[root@node-1 pod]# kubectl set image deployment/nginx-deployment nginx=nginx:1.9.1
deployment.apps/nginx-deployment image updated
#已更新到1.9.1版本
[root@node-1 pod]# kubectl describe deployments.apps nginx-deployment
Name: nginx-deployment
Namespace: default
CreationTimestamp: Tue, 04 Feb 2020 19:27:09 +0800
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 4
kubernetes.io/change-cause: kubectl create --filename=nginx_deployment.yaml --record=true
Selector: app=nginx
Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=nginx
Containers:
nginx:
Image: nginx:1.9.1
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-deployment-56f8998dbc (3/3 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 34m deployment-controller Scaled up replica set nginx-deployment-54f57cf6bf to 3
Normal ScalingReplicaSet 33m deployment-controller Scaled up replica set nginx-deployment-569cb44c46 to 1
Normal ScalingReplicaSet 26m deployment-controller Scaled down replica set nginx-deployment-569cb44c46 to 0
Normal ScalingReplicaSet 24m deployment-controller Scaled up replica set nginx-deployment-56f8998dbc to 1
Normal ScalingReplicaSet 24m deployment-controller Scaled down replica set nginx-deployment-54f57cf6bf to 2
Normal ScalingReplicaSet 24m deployment-controller Scaled up replica set nginx-deployment-56f8998dbc to 2
Normal ScalingReplicaSet 24m deployment-controller Scaled down replica set nginx-deployment-54f57cf6bf to 1
Normal ScalingReplicaSet 24m deployment-controller Scaled up replica set nginx-deployment-56f8998dbc to 3
Normal ScalingReplicaSet 24m deployment-controller Scaled down replica set nginx-deployment-54f57cf6bf to 0
#查看更新版本
[root@node-1 pod]# kubectl rollout history deployment nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
2 kubectl create --filename=nginx_deployment.yaml --record=true
3 kubectl create --filename=nginx_deployment.yaml --record=true
4 kubectl create --filename=nginx_deployment.yaml --record=true
[root@node-1 pod]# kubectl rollout history deployment nginx-deployment --revision=3
deployment.apps/nginx-deployment with revision #3
Pod Template:
Labels: app=nginx
pod-template-hash=54f57cf6bf
Annotations: kubernetes.io/change-cause: kubectl create --filename=nginx_deployment.yaml --record=true
Containers:
nginx:
Image: nginx:1.7.9
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
[root@node-1 pod]# kubectl rollout history deployment nginx-deployment --revision=4
deployment.apps/nginx-deployment with revision #4
Pod Template:
Labels: app=nginx
pod-template-hash=56f8998dbc
Annotations: kubernetes.io/change-cause: kubectl create --filename=nginx_deployment.yaml --record=true
Containers:
nginx:
Image: nginx:1.9.1
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
[root@node-1 pod]#
#我们回退到版本3
[root@node-1 pod]# kubectl rollout undo deployment nginx-deployment --to-revision=3
deployment.apps/nginx-deployment rolled back
[root@node-1 pod]# kubectl get pod
NAME READY STATUS RESTARTS AGE
centos-6b68465d79-5w9df 1/1 Running 0 7d5h
nginx-deployment-54f57cf6bf-7972m 1/1 Running 0 3s
nginx-deployment-54f57cf6bf-pdvp7 1/1 Running 0 7s
nginx-deployment-54f57cf6bf-v2rxd 1/1 Running 0 5s
nginx-deployment-56f8998dbc-cdnll 0/1 Terminating 0 27m
nginx-deployment-56f8998dbc-ljct7 0/1 Terminating 0 27m
nginx-deployment-56f8998dbc-pdtbj 0/1 Terminating 0 27m
[root@node-1 pod]#
#已回退到版本1.7.9
[root@node-1 pod]# kubectl describe deployments.apps nginx-deployment
Name: nginx-deployment
Namespace: default
CreationTimestamp: Tue, 04 Feb 2020 19:27:09 +0800
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 5
kubernetes.io/change-cause: kubectl create --filename=nginx_deployment.yaml --record=true
Selector: app=nginx
Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=nginx
Containers:
nginx:
Image: nginx:1.7.9
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-deployment-54f57cf6bf (3/3 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 36m deployment-controller Scaled up replica set nginx-deployment-569cb44c46 to 1
Normal ScalingReplicaSet 29m deployment-controller Scaled down replica set nginx-deployment-569cb44c46 to 0
Normal ScalingReplicaSet 27m deployment-controller Scaled up replica set nginx-deployment-56f8998dbc to 1
Normal ScalingReplicaSet 27m deployment-controller Scaled down replica set nginx-deployment-54f57cf6bf to 2
Normal ScalingReplicaSet 27m deployment-controller Scaled up replica set nginx-deployment-56f8998dbc to 2
Normal ScalingReplicaSet 27m deployment-controller Scaled down replica set nginx-deployment-54f57cf6bf to 1
Normal ScalingReplicaSet 27m deployment-controller Scaled up replica set nginx-deployment-56f8998dbc to 3
Normal ScalingReplicaSet 27m deployment-controller Scaled down replica set nginx-deployment-54f57cf6bf to 0
Normal ScalingReplicaSet 35s deployment-controller Scaled up replica set nginx-deployment-54f57cf6bf to 1
Normal ScalingReplicaSet 33s deployment-controller Scaled up replica set nginx-deployment-54f57cf6bf to 2
Normal ScalingReplicaSet 33s deployment-controller Scaled down replica set nginx-deployment-56f8998dbc to 2
Normal ScalingReplicaSet 31s (x2 over 37m) deployment-controller Scaled up replica set nginx-deployment-54f57cf6bf to 3
Normal ScalingReplicaSet 31s deployment-controller Scaled down replica set nginx-deployment-56f8998dbc to 1
Normal ScalingReplicaSet 29s deployment-controller Scaled down replica set nginx-deployment-56f8998dbc to 0
5.更新多种配置
在使用命令更新deployment配置的时候,更新就马上执行了,在需要更新多种配置的时候,是不方便的.
我们可以通过暂停deployment更新,配置好所有的更新配置之后,再执行统一更新.
#暂停更新
[root@node-1 pod]# kubectl rollout pause deployment nginx-deployment
deployment.apps/nginx-deployment paused
[root@node-1 pod]#
[root@node-1 pod]# kubectl set image deployment/nginx-deployment nginx=nginx:1.9.2
deployment.apps/nginx-deployment image updated
#没有立即进行更新
[root@node-1 pod]# kubectl rollout history deployment nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
2 kubectl create --filename=nginx_deployment.yaml --record=true
4 kubectl create --filename=nginx_deployment.yaml --record=true
5 kubectl create --filename=nginx_deployment.yaml --record=true
[root@node-1 pod]# kubectl rollout history deployment nginx-deployment --revision=5
deployment.apps/nginx-deployment with revision #5
Pod Template:
Labels: app=nginx
pod-template-hash=54f57cf6bf
Annotations: kubernetes.io/change-cause: kubectl create --filename=nginx_deployment.yaml --record=true
Containers:
nginx:
Image: nginx:1.7.9
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
[root@node-1 pod]#
#我们可以添加其他更新配置
[root@node-1 pod]# kubectl set resources deployment nginx-deployment -c=nginx --limits=cpu=200m,memory=512Mi
deployment.apps/nginx-deployment resource requirements updated
#未更新
[root@node-1 pod]# kubectl rollout history deployment nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
2 kubectl create --filename=nginx_deployment.yaml --record=true
4 kubectl create --filename=nginx_deployment.yaml --record=true
5 kubectl create --filename=nginx_deployment.yaml --record=true
[root@node-1 pod]#
#恢复更新
[root@node-1 pod]# kubectl rollout resume deployment nginx-deployment
deployment.apps/nginx-deployment resumed
[root@node-1 pod]# kubectl rollout history deployment nginx-deployment
deployment.apps/nginx-deployment
REVISION CHANGE-CAUSE
2 kubectl create --filename=nginx_deployment.yaml --record=true
4 kubectl create --filename=nginx_deployment.yaml --record=true
5 kubectl create --filename=nginx_deployment.yaml --record=true
6 kubectl create --filename=nginx_deployment.yaml --record=true
[root@node-1 pod]# kubectl rollout history deployment nginx-deployment --revision=6
deployment.apps/nginx-deployment with revision #6
Pod Template:
Labels: app=nginx
pod-template-hash=9f5dd848b
Annotations: kubernetes.io/change-cause: kubectl create --filename=nginx_deployment.yaml --record=true
Containers:
nginx:
Image: nginx:1.9.2
Port: 80/TCP
Host Port: 0/TCP
Limits:
cpu: 200m
memory: 512Mi
Environment: <none>
Mounts: <none>
Volumes: <none>
[root@node-1 pod]#
#更新完成
[root@node-1 pod]# kubectl describe deployments.apps nginx-deployment
Name: nginx-deployment
Namespace: default
CreationTimestamp: Tue, 04 Feb 2020 19:27:09 +0800
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 6
kubernetes.io/change-cause: kubectl create --filename=nginx_deployment.yaml --record=true
Selector: app=nginx
Replicas: 3 desired | 1 updated | 4 total | 3 available | 1 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=nginx
Containers:
nginx:
Image: nginx:1.9.2
Port: 80/TCP
Host Port: 0/TCP
Limits:
cpu: 200m
memory: 512Mi
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True ReplicaSetUpdated
OldReplicaSets: nginx-deployment-54f57cf6bf (3/3 replicas created)
NewReplicaSet: nginx-deployment-9f5dd848b (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 55m deployment-controller Scaled down replica set nginx-deployment-569cb44c46 to 0
Normal ScalingReplicaSet 53m deployment-controller Scaled up replica set nginx-deployment-56f8998dbc to 1
Normal ScalingReplicaSet 53m deployment-controller Scaled down replica set nginx-deployment-54f57cf6bf to 2
Normal ScalingReplicaSet 53m deployment-controller Scaled up replica set nginx-deployment-56f8998dbc to 2
Normal ScalingReplicaSet 53m deployment-controller Scaled up replica set nginx-deployment-56f8998dbc to 3
Normal ScalingReplicaSet 53m deployment-controller Scaled down replica set nginx-deployment-54f57cf6bf to 1
Normal ScalingReplicaSet 53m deployment-controller Scaled down replica set nginx-deployment-54f57cf6bf to 0
Normal ScalingReplicaSet 26m deployment-controller Scaled up replica set nginx-deployment-54f57cf6bf to 1
Normal ScalingReplicaSet 26m deployment-controller Scaled up replica set nginx-deployment-54f57cf6bf to 2
Normal ScalingReplicaSet 26m deployment-controller Scaled down replica set nginx-deployment-56f8998dbc to 2
Normal ScalingReplicaSet 26m (x2 over 63m) deployment-controller Scaled up replica set nginx-deployment-54f57cf6bf to 3
Normal ScalingReplicaSet 26m deployment-controller Scaled down replica set nginx-deployment-56f8998dbc to 1
Normal ScalingReplicaSet 26m deployment-controller Scaled down replica set nginx-deployment-56f8998dbc to 0
Normal ScalingReplicaSet 40s deployment-controller Scaled up replica set nginx-deployment-9f5dd848b to 1
6.deployment扩缩容
如果当前pod不足于处理业务,需要扩容;或者业务量缩减,需要进行缩容.
1)手动扩缩容
#我们将pod的副本数从3扩容到5
[root@node-1 pod]#
[root@node-1 pod]# kubectl get pod
NAME READY STATUS RESTARTS AGE
centos-6b68465d79-5w9df 1/1 Running 0 7d6h
nginx-deployment-9f5dd848b-4bbgv 1/1 Running 0 101s
nginx-deployment-9f5dd848b-6qkvf 1/1 Running 0 3m21s
nginx-deployment-9f5dd848b-rnm7s 1/1 Running 0 4m16s
[root@node-1 pod]#
[root@node-1 pod]# kubectl scale deployment nginx-deployment --replicas=5
deployment.apps/nginx-deployment scaled
[root@node-1 pod]# kubectl get pod
NAME READY STATUS RESTARTS AGE
centos-6b68465d79-5w9df 1/1 Running 0 7d6h
nginx-deployment-9f5dd848b-4bbgv 1/1 Running 0 2m1s
nginx-deployment-9f5dd848b-6qkvf 1/1 Running 0 3m41s
nginx-deployment-9f5dd848b-7r46p 1/1 Running 0 3s
nginx-deployment-9f5dd848b-cffn5 1/1 Running 0 3s
nginx-deployment-9f5dd848b-rnm7s 1/1 Running 0 4m36s
#我们将pod的副本数从5缩容到2
[root@node-1 pod]# kubectl scale deployment nginx-deployment --replicas=2
deployment.apps/nginx-deployment scaled
[root@node-1 pod]# kubectl get pod
NAME READY STATUS RESTARTS AGE
centos-6b68465d79-5w9df 1/1 Running 0 7d6h
nginx-deployment-9f5dd848b-4bbgv 0/1 Terminating 0 3m28s
nginx-deployment-9f5dd848b-6qkvf 1/1 Running 0 5m8s
nginx-deployment-9f5dd848b-7r46p 0/1 Terminating 0 90s
nginx-deployment-9f5dd848b-cffn5 0/1 Terminating 0 90s
nginx-deployment-9f5dd848b-rnm7s 1/1 Running 0 6m3s
[root@node-1 pod]# kubectl get pod
NAME READY STATUS RESTARTS AGE
centos-6b68465d79-5w9df 1/1 Running 0 7d6h
nginx-deployment-9f5dd848b-6qkvf 1/1 Running 0 5m14s
nginx-deployment-9f5dd848b-rnm7s 1/1 Running 0 6m9s
自动扩缩容较复杂,放在下一个博客写.