11-k8s结合Metrics-Server实现自动伸缩
k8s结合Metrics-Server实现自动伸缩
资源限制的介绍
requests.cpu是命名空间中所有容器的最大组合CPU请求（以毫秒为单位）。 在上面的示例中，您可以拥有50个具有10m请求的容器，5个具有100m请求的容器，甚至一个具有500m请求的容器。 只要命名空间中请求的总CPU和小于500m！
requests.memory是命名空间中所有容器的最大组合内存请求。 在上面的示例中，您可以拥有50个具有2MiB请求的容器，5个具有20MiB请求的容器，甚至是具有100MiB请求的单个容器。 只要命名空间中请求的总内存小于100MiB！
limits.cpu是命名空间中所有容器的最大组合CPU限制。 它就像requests.cpu，但是这里指的是限制。
limits.memory是命名空间中所有容器的最大组合内存限制。 它就像requests.memory，但是同样地这里指的是限制。
如果您使用的是生产和开发命名空间（与每个团队或服务的命名空间不同），则常见的模式是在生产命名空间上没有配额，在开发命名空间上则是没有严格的配额。 这使得生产能够在流量激增的情况下获取所需的所有资源。
Metrics-Server概念介绍
Metrics-Server是集群核心监控数据的聚合器，在k8s早期版本中，对资源的监控使用的是heapster的资源监控工具。但是从 Kubernetes 1.8 开始，Kubernetes 通过 Metrics API 获取资源使用指标，例如容器 CPU 和内存使用情况。这些度量指标可以由用户直接访问，例如通过使用kubectl top 命令，或者使用集群中的控制器，,因为k8s的api-server将所有的数据持久化到了etcd中，显然k8s本身不能处理这种频率的采集，而且这种监控数据变化快且都是临时数据，因此需要有一个组件单独处理他们
环境
注意：此环境为kubeadm搭建
如果是minikube搭建：需要开启Metrics-Server即可
minikube addons list
minikube addons enable metrics-server
[root@k8s-master ~]# docker -v
Docker version 19.03.5, build 633a0ea
[root@k8s-master ~]# kubelet --version
Kubernetes v1.18.0
一、修改配置安装Metrics-Server
1.检查 API Server 是否开启了 Aggregator Routing：查看 API Server 是否具有 --enable-aggregator-routing=true 选项。
如果没有则需要添加
[root@k8s-master ~]# ps -ef | grep apiserver
root 16619 16581 3 18:51 ? 00:02:06 kube-apiserver --advertise-address=192.168.10.101 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root 36741 35704 0 19:54 pts/0 00:00:00 grep --color=auto apiserver
#此处没有需要添加
2.修改每个 API Server 的 kube-apiserver.yaml 配置开启 Aggregator Routing：修改 manifests 配置后 API Server 会自动重启生效。
[root@k8s-master ~]# vim /etc/kubernetes/manifests/kube-apiserver.yaml
21 - --enable-bootstrap-token-auth=true
22 - --enable-aggregator-routing=true #在21行下方添加此行内容
23 - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
：wq #保存退出
二、安装metrics-server(v0.3.6)
1、下载yaml文件
方法一：互联网下载：
[root@k8s-master ~]# wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.6/components.yaml
修改配置文件
[root@k8s-master ~]# vim components.yaml
85 - name: metrics-server
86 image: k8s.gcr.io/metrics-server-amd64:v0.3.6
#将86行换为修改为阿里云的镜像地址
86 image: registry.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6
#在90行下方添加
88 args:
89 - --cert-dir=/tmp
90 - --secure-port=4443
91 - /metrics-server #添加内容
92 - --kubelet-preferred-address-types=InternalIP #添加内容
93 - --kubelet-insecure-tls #添加内容
方法二：手动编写（不需要修改）ps:因为我已经修改好了
[root@k8s-master ~]# vim components.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:aggregated-metrics-reader
labels:
rbac.authorization.k8s.io/aggregate-to-view: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
name: v1beta1.metrics.k8s.io
spec:
service:
name: metrics-server
namespace: kube-system
group: metrics.k8s.io
version: v1beta1
insecureSkipTLSVerify: true
groupPriorityMinimum: 100
versionPriority: 100
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metrics-server
namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: metrics-server
namespace: kube-system
labels:
k8s-app: metrics-server
spec:
selector:
matchLabels:
k8s-app: metrics-server
template:
metadata:
name: metrics-server
labels:
k8s-app: metrics-server
spec:
serviceAccountName: metrics-server
volumes:
# mount in tmp so we can safely use from-scratch images and/or read-only containers
- name: tmp-dir
emptyDir: {}
containers:
- name: metrics-server
image: registry.aliyuncs.com/google_containers/metrics-server-amd64:v0.3.6
imagePullPolicy: IfNotPresent
args:
- --cert-dir=/tmp
- --secure-port=4443
- /metrics-server
- --kubelet-preferred-address-types=InternalIP
- --kubelet-insecure-tls
ports:
- name: main-port
containerPort: 4443
protocol: TCP
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- name: tmp-dir
mountPath: /tmp
nodeSelector:
kubernetes.io/os: linux
kubernetes.io/arch: "amd64"
---
apiVersion: v1
kind: Service
metadata:
name: metrics-server
namespace: kube-system
labels:
kubernetes.io/name: "Metrics-server"
kubernetes.io/cluster-service: "true"
spec:
selector:
k8s-app: metrics-server
ports:
- port: 443
protocol: TCP
targetPort: main-port
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: system:metrics-server
rules:
- apiGroups:
- ""
resources:
- pods
- nodes
- nodes/stats
- namespaces
- configmaps
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
2、安装
[root@k8s-master ~]# kubectl apply -f components.yaml
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
serviceaccount/metrics-server created
deployment.apps/metrics-server created
service/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
3、查看metrics-server服务状态
[root@k8s-master ~]# kubectl get pod -n kube-system | grep metrics-server
metrics-server-59dd47f7d9-dhsjw 1/1 Running 0 34s
4、检查接口是否有异常
[root@k8s-master ~]# kubectl describe apiservice v1beta1.metrics.k8s.io
Name: v1beta1.metrics.k8s.io
Namespace:
Labels: <none>
Annotations: API Version: apiregistration.k8s.io/v1
Kind: APIService
Metadata:
Creation Timestamp: 2021-03-01T12:18:14Z
Resource Version: 13106
Self Link: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
UID: 0f3e9d6f-8618-4914-a687-9d5853b9a940
Spec:
Group: metrics.k8s.io
Group Priority Minimum: 100
Insecure Skip TLS Verify: true
Service:
Name: metrics-server
Namespace: kube-system
Port: 443
Version: v1beta1
Version Priority: 100
Status:
Conditions:
Last Transition Time: 2021-03-01T12:18:18Z
Message: all checks passed
Reason: Passed
Status: True
Type: Available
Events: <none>
5、执行以下命令，检查节点占用性能情况。
[root@k8s-master ~]# kubectl top nodes #查看node节点负载情况
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master 93m 4% 1176Mi 68%
k8s-node01 30m 1% 848Mi 49%
k8s-node02 20m 1% 850Mi 49%
[root@k8s-master ~]# kubectl top pods #使用次命令需要多等待一会，默认采集信息为每分钟采集一次
error: metrics not available yet
[root@k8s-master ~]# sleep 180 && kubectl top pods #等待180秒进行查看，前提是需要有pod在运行
NAME CPU(cores) MEMORY(bytes)
rc-demo-chw5t 0m 4Mi
rc-demo-mmzvc 0m 3Mi
rc-demo-v289z 0m 3Mi
# MEMORY(bytes) 内存信息 CPU(cores) cpu信息
二、生成测试pod
这里同时创建Service，是为了后续的模拟压力测试。
[root@k8s-master ~]# vim Deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: php-apache
spec:
selector:
matchLabels:
run: php-apache
replicas: 1
template:
metadata:
labels:
run: php-apache
spec:
containers:
- name: php-apache
image: "registry.cn-shenzhen.aliyuncs.com/cookcodeblog/hpa-example:latest"
ports:
- containerPort: 80
resources: #限制指标
limits:
cpu: 500m
requests:
cpu: 200m
---
apiVersion: v1
kind: Service
metadata:
name: php-apache
labels:
run: php-apache
spec:
ports:
- port: 80
selector:
run: php-apache
值得注意的是：要监控的pod必须进行资源限制：限制指标
创建deployment和servcie：
[root@k8s-master ~]# kubectl create -f Deployment.yaml
deployment.apps/php-apache created
service/php-apache created
验证:
[root@k8s-master ~]# kubectl get deployment php-apache
NAME READY UP-TO-DATE AVAILABLE AGE
php-apache 1/1 1 1 30s
[root@k8s-master ~]# kubectl get svc php-apache
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
php-apache ClusterIP 10.102.178.170 <none> 80/TCP 36s
[root@k8s-master ~]# kubectl get pods -l run=php-apache -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
php-apache-6b79cc8c8f-rtrv8 1/1 Running 0 55s 10.244.2.4 k8s-node02 <none> <none>
三、创建HPA
[root@k8s-master ~]# kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10
horizontalpodautoscaler.autoscaling/php-apache autoscaled
#默认创建的HPA名称和需要自动伸缩的对象名一致
# 可以通过--name来指定HPA名称
# 删除 hpa
# kubectl delete horizontalpodautoscalers.autoscaling {Deploymeng名称}
为deployment php-apache 创建HPA，其中最小副本数为1，最大副本数为10，保持该deployment的所有Pod的平均CPU使用率不超过50%。
在本例中，deployment的pod的resources.request.cpu为200m （200 milli-cores vCPU)，所以HPA将保持所有Pod的平均CPU使用率不超过100m。
可以通过kubectl top pods查看pod的CPU使用情况。
查看HPA：
[root@k8s-master ~]# kubectl get hpa php-apache
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 0%/50% 1 10 1 82s
#此时已经可以监控php-apache的CPU信息
如果TARGETS列值格式为
#HPA默认每15秒从Metrics Server取一下指标来判断是否要自动伸缩。
The Horizontal Pod Autoscaler is implemented as a control loop, with a period controlled by the controller manager’s --horizontal-pod-autoscaler-sync-period flag (with a default value of 15 seconds).
#------------------------------------------------------------------------------------
#Metrics Server采集指标的默认间隔为60秒。
Default 60 seconds, can be changed using metrics-resolution flag. We are not recommending setting values below 15s, as this is the resolution of metrics calculated within Kubelet.
四、模拟增加负载
打开一个新的终端，创建一个临时的pod load-generator，并在该pod中向php-apache服务发起不间断循环请求，模拟增加php-apache的负载（CPU使用率）。
[root@k8s-master ~]# kubectl run -i --tty load-generator --rm --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"
If you don't see a command prompt, try pressing enter.
OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!
#使用busybox 死循环每0.01秒请求一次网站页面
模拟压力测试几分钟后，观察HPA：
[root@k8s-master ~]# while sleep 5; do kubectl get hpa php-apache; done
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 250%/50% 1 10 4 12m
#使用循环语句每5秒查看一次HPA的信息
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 57%/50% 1 10 5 14m
#经过一段时间可以看出
#------------------------------------参数解释-----------------------------------
#TARGETS:平均负载57%，设置的平均负载大于50%就会增加pod数量
#MINPODS ：最少pod的副本数量为1
#MAXPODS ：最多pod的副本数量为10
#REPLICAS：当前的pod副本数量为5
#AGE ：运行时间
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 49%/50% 1 10 7 19m
#又经过一段时间当前的pod副本数量为7台
#此时的TARGETS不大于50就不会在增加pod副本数量了
查看当前pod
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
load-generator 1/1 Running 0 9m43s
php-apache-6b79cc8c8f-6gcsf 1/1 Running 0 9m7s
php-apache-6b79cc8c8f-jkz7q 1/1 Running 0 9m7s
php-apache-6b79cc8c8f-mzkwc 1/1 Running 0 7m6s
php-apache-6b79cc8c8f-n2hkk 1/1 Running 0 7m6s
php-apache-6b79cc8c8f-q6qg5 1/1 Running 0 9m7s
php-apache-6b79cc8c8f-qjbtl 1/1 Running 0 8m52s
php-apache-6b79cc8c8f-rtrv8 1/1 Running 0 23m
也就是HPA通过自动扩容到7个副本，来分摊了负载，使得所有Pod的平均CPU使用率保持（近似）在目标值。
可以通过kubectl describe hpa php-apache查看HPA自动伸缩的事件。
[root@k8s-master ~]# kubectl describe hpa php-apache
Name: php-apache
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Mon, 01 Mar 2021 20:34:46 +0800
Reference: Deployment/php-apache
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 44% (88m) / 50%
Min replicas: 1
Max replicas: 10
Deployment pods: 7 current / 7 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 11m horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 11m horizontal-pod-autoscaler New size: 5; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 9m36s horizontal-pod-autoscaler New size: 7; reason: cpu resource utilization (percentage of request) above target
#信息为当前cpu使用百分比大于设置阈值，触发自动伸缩，伸缩为7台
五、模拟减少负载
在运行load-generator的Terminal，按下Ctrl + C来终止进程
OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!OK!^C
pod "load-generator" deleted
pod default/load-generator terminated (Error)
[root@k8s-master ~]#
继续观察HPA
php-apache Deployment/php-apache 0%/50% 1 10 7 27m
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 0%/50% 1 10 7 27m
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 0%/50% 1 10 7 28m
#此时cpu平均负载为0但是要过了伸缩冷却时间5分钟后才能进行弹性伸缩
#如果观察HPA没有scale down，需要再等待一段时间。Kuberntes为了保证缩容时业务不中断，和防止频繁伸缩导致系统抖动，scaledown一次后需要等待一段时间才能再次scaledown，也叫伸缩冷却(cooldown)。默认伸缩冷却时间为5分钟。
--horizontal-pod-autoscaler-downscale-stabilization: The value for this option is a duration that specifies how long the autoscaler has to wait before another downscale operation can be performed after the current one has completed. The default value is 5 minutes (5m0s).
等待伸缩冷却时间为5分钟过后进行查看
[root@k8s-master ~]# kubectl get hpa php-apache
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 0%/50% 1 10 1 34m
#因为平均负载小于阈值，此时已经由原来的7台缩至1台，
通过kubectl describe hpa php-apache查看HPA自动伸缩的事件，可以看到“horizontal-pod-autoscaler New size: 1; reason: All metrics below target”的事件。
[root@k8s-master ~]# kubectl describe hpa php-apache
Name: php-apache
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Mon, 01 Mar 2021 20:34:46 +0800
Reference: Deployment/php-apache
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 0% (1m) / 50%
Min replicas: 1
Max replicas: 10
Deployment pods: 1 current / 1 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count fromcpu resource utilization (percentage of request)
ScalingLimited True TooFewReplicas the desired replica count is less than the minimum replica count
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 23m horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 23m horizontal-pod-autoscaler New size: 5; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 21m horizontal-pod-autoscaler New size: 7; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 3m23s horizontal-pod-autoscaler New size: 1; reason: All metrics below targe
#日志中显示因为所有指标都低于阈值，触发伸缩，由7台伸缩至一台