一、概述
HPA全称Horizontal Pod Autoscaling,即pod的水平自动扩展。HPA的操作对象是RC、RS或Deployment对应的Pod,根据观察到的CPU实际使用量与用户的期望值进行比对,做出是否需要增减实例数量的决策。
它根据Pod当前系统的负载来自动水平扩容,如果系统负载超过预定值,就开始增加Pod的个数,如果低于某个值,就自动减少Pod的个数。目前K8S的HPA只能根据CPU和内存去度量系统的负载,而且目前还依赖heapster去收集CPU的使用情况。
HPA通过定期(定期轮询的时间通过–horizontal-pod-autoscaler-sync-period选项来设置,默认的时间为30秒)通过Status.PodSelector来查询pods的状态,获得pod的CPU使用率。然后,通过现有pods的CPU使用率的平均值(计算方式是最近的pod使用量除以设定的每个Pod的CPU使用率限额,例最近一分钟从heapster中获得的平均值除以Pod的CPU limits值)跟目标使用率进行比较,并且在扩容时,还要遵循预先设定的副本数限制:MinReplicas <= Replicas <= MaxReplicas。
计算扩容后Pod的个数:sum(最近一分钟内某个Pod的CPU使用率/量的平均值)/CPU使用上限的整数+1
考虑到自动扩展的决策可能需要一段时间才会生效,甚至在短时间内会引入一些噪声。例如当pod所需要的CPU负荷过大,从而运行一个新的pod进行分流,在创建过程中,系统的CPU使用量可能会有一个攀升的过程。所以,在每一次作出决策后的一段时间内,将不再进行扩展决策。对于ScaleUp而言,这个时间段为3分钟,Scaledown为5分钟。
HPA允许一定范围内的CPU使用量的不稳定,只有avg(CurrentPodsConsumption) / Target小于90%或者大于110%时才会触发扩容或缩容,避免频繁扩容、缩容造成颠簸。
为了简便,选用了相对比率(90%的CPU资源)而不是0.6个CPU core来描述扩容、缩容条件。如果选择使用绝对度量,用户需要保证目标(限额)要比请求使用的低,否则,过载的Pod未必能够消耗那么多,从而自动扩容永远不会被触发:假设设置CPU为1个核,那么这个pod只能使用1个核,可能Pod在过载的情况下也不能完全利用这个核,所以扩容不会发生。在修改申请资源时,还有同时调整扩容的条件,比如将1个core变为1.2core,那么扩容条件应该同步改为1.2core,真是太麻烦了,与自动扩容的目标相悖。
例1:
[root@docker79 ~]# kubectl get pods
No resources found.
[root@docker79 ~]# kubectl run myapp --image=ikubernetes/myapp:v1 --replicas=1 --requests='cpu=50m,memory=256Mi' --limits='cpu=50m,memory=256Mi' --labels='app=myapp' --expose --port=80
service/myapp created
deployment.apps/myapp created
[root@docker79 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-6985749785-l4c87 1/1 Running 0 15s
[root@docker79 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 443/TCP 20d
myapp ClusterIP 10.100.44.181 80/TCP 20s
[root@docker79 ~]# kubectl autoscale deployment myapp --min=1 --max=4 --cpu-percent=40
horizontalpodautoscaler.autoscaling/myapp autoscaled
[root@docker79 ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
myapp Deployment/myapp /40% 1 4 0 6s
[root@docker79 ~]#
[root@docker79 ~]# kubectl patch svc myapp -p '{"spec":{"type":"NodePort"}}'
service/myapp patched
[root@docker79 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 443/TCP 20d
myapp NodePort 10.100.44.181 80:30193/TCP 4m
[root@docker79 ~]#
[root@docker79 ~]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
myapp Deployment/myapp 0%/40% 1 4 1 6m
[root@docker79 ~]#
以上创建hap,并定义了CPU使用率达到40%时进行扩展,下面开启 ab压测,然后使其自动扩展
[root@docker79 ~]# ab -c 100 -n 50000 http://docker78:30193/index.html
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking docker78 (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
......
[root@docker79 ~]# kubectl describe hpa
Name: myapp
Namespace: default
Labels:
Annotations:
CreationTimestamp: Mon, 17 Sep 2018 16:40:26 +0800
Reference: Deployment/myapp
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 102% (51m) / 40%
Min replicas: 1
Max replicas: 4
Deployment pods: 1 current / 3 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 3
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 14s horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target
[root@docker79 ~]#
[root@docker79 ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
myapp-6985749785-fvxbz 1/1 Running 0 1m
myapp-6985749785-l4c87 1/1 Running 0 14m
myapp-6985749785-xdmnw 1/1 Running 0 1m
[root@docker79 ~]#
例2,接上以manifests yaml的格式定义HorizontalPodAutoscaler ,并同时定义CPU、Memory资源的阀值,以便pod扩展:
[root@docker79 ~]# cd manifests/
[root@docker79 manifests]# mkdir hpa
[root@docker79 manifests]# cd hpa
[root@docker79 hpa]# vim hpa-demo-v2.yaml
[root@docker79 hpa]# cat hpa-demo-v2.yaml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: myapp-hpa-v2
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: myapp
minReplicas: 1
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 55
- type: Resource
resource:
name: memory
targetAverageValue: 50Mi
[root@docker79 hpa]#
[root@docker79 hpa]# kubectl delete hpa myapp
horizontalpodautoscaler.autoscaling "myapp" deleted
[root@docker79 hpa]#
[root@docker79 hpa]# kubectl apply -f hpa-demo-v2.yaml
horizontalpodautoscaler.autoscaling/myapp-hpa-v2 created
[root@docker79 hpa]# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
myapp-hpa-v2 Deployment/myapp /50Mi, /55% 1 5 0 6s
[root@docker79 hpa]#
再次开启压测
[root@docker79 ~]# ab -c 100 -n 30000 http://docker77:30193/index.html
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking docker77 (be patient)
Completed 3000 requests
Completed 6000 requests
......
[root@docker79 hpa]# kubectl describe hpa
Name: myapp-hpa-v2
Namespace: default
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"autoscaling/v2beta1","kind":"HorizontalPodAutoscaler","metadata":{"annotations":{},"name":"myapp-hpa-v2","namespace":"default"},"spec":{...
CreationTimestamp: Mon, 17 Sep 2018 17:30:53 +0800
Reference: Deployment/myapp
Metrics: ( current / target )
resource memory on pods: 3442688 / 50Mi
resource cpu on pods (as a percentage of request): 68% (34m) / 55%
Min replicas: 1
Max replicas: 5
Deployment pods: 2 current / 3 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededRescale the HPA controller was able to update the target scale to 3
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 5m horizontal-pod-autoscaler New size: 2; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 9s horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target
[root@docker79 hpa]#
仔细观察hpa 的信息,发现已经扩展了两个new pods。
例3:
pods 可以自定义一些资源指标,并将其输出成Restful风格的API,并支持让prometheus 读取、认识这些自定义的指标值 ,然后hpa 就可以根据这些指标值 进行扩展或收缩。如下所示:
[root@docker79 hpa]# cat hpa-demo-custom.yaml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: myapp-hpa-v2
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: myapp
minReplicas: 1
maxReplicas: 5
metrics:
- type: Pods
pods:
metricName: http_requests
targetAverageValue: 800m
[root@docker79 hpa]#
由于hpa 我使用还不是特别熟练,所以简单总结这么多,后续慢慢补充。大家勿喷。