Pod自动扩容与缩容

Pod自动扩容/缩容

HPA介绍

001 Horizontal Pod Autoscaler（HPA，Pod水平自动伸缩）
002 根据资源利用率或者自定义指标自动调整Deployment的Pod副本数量，提供应用并发。
003 HPA不适于无法缩放的对象，例如DaemonSet。

1639534110940.png

HPA基本工作原理

001 Kubernetes 中的 Metrics Server 持续采集所有 Pod 副本的指标数据。
002 HPA 控制器通过 Metrics Server 的 API（聚合 API）获取这些数据，
003 基于用户定义的扩缩容规则进行计算，得到目标 Pod 副本数量。
004 当目标 Pod 副本数量与当前副本数量不同时，HPA 控制器就向 Pod 的Deployment控制器发起scale 操作，调整 Pod 的副本数量，完成扩缩容操作

1639534522974.png

使用HPA前提条件

001 启用Kubernetes API聚合层
002 相应的API已注册
•   对于资源指标（例如CPU、内存），将使用metrics.k8s.io API，一般由metrics-server提供
•   对于自定义指标（例如QPS），将使用custom.metrics.k8s.io API，由相关适配器（Adapter）服务提供

API聚合层

001 在 Kubernetes 1.7 版本引入了聚合层，允许第三方应用程序通过将自己注册到kube-apiserver上，仍然通过 API Server 的 HTTP URL 对新的 API 进行访问和操作。
002 为了实现这个机制，Kubernetes 在 kube-apiserver 服务中引入了一个API 聚合层（API Aggregation Layer），用于将扩展 API 的访问请求转发到用户服务的功能

1639535446518.png

启用聚合层

#用kubeadm 默认已开启
#二进制方式
[root@k8s-m1 ~]# vi /opt/kubernetes/cfg/kube-apiserver.conf

--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \
--proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \
--proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \
--requestheader-allowed-names=kubernetes \
--requestheader-extra-headers-prefix=X-Remote-Extra- \
--requestheader-group-headers=X-Remote-Group \
--requestheader-username-headers=X-Remote-User \
--enable-aggregator-routing=true \

基于资源指标

Metrics Server部署

[root@k8s-m1 hpa]# kubectl apply -f metrics-server.yaml

[root@k8s-m1 hpa]# kubectl get pod -n kube-system 
NAME                                      READY   STATUS    RESTARTS   AGE
......
metrics-server-b66888848-jqqww            1/1     Running   0          4m31s

查看指标并验证

[root@k8s-m1 hpa]# kubectl get apiservices
...
v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        5m7s

[root@k8s-m1 hpa]# kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes
{...}
[root@k8s-m1 hpa]# kubectl get --raw /apis/metrics.k8s.io/v1beta1/pods
{...}

#查看Node资源消耗
[root@k8s-m1 hpa]# kubectl top node
NAME        CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
k8s-m1      309m         7%     1044Mi          28%       
k8s-node1   142m         3%     377Mi           10%  

#查看Pod资源消耗
[root@k8s-m1 hpa]# kubectl top pod  -n kube-system
NAME                                      CPU(cores)   MEMORY(bytes)   
calico-kube-controllers-97769f7c7-wfvvd   2m           14Mi            
calico-node-76zz7                         69m          71Mi            
calico-node-lfjsk                         83m          74Mi            
coredns-6cc56c94bd-2zsgn                  5m           13Mi            
metrics-server-b66888848-jqqww            5m           21Mi 

#如果能正常显示资源消耗说明Metrics Server服务工作正常

应用部署

部署应用

[root@k8s-m1 hpa]# kubectl create deployment web --image=nginx --dry-run=client -o yaml > deployment.yaml
[root@k8s-m1 hpa]# vi deployment.yaml 
    spec:
      containers:
      - image: nginx
        name: nginx
        resources: 
          requests:
            cpu: 0.5

[root@k8s-m1 hpa]# kubectl apply -f deployment.yaml 
deployment.apps/web created


[root@k8s-m1 hpa]# kubectl expose deployment web --port=80 --target-port=80 
service/web exposed

[root@k8s-m1 hpa]# kubectl get pod,svc
NAME                       READY   STATUS    RESTARTS   AGE
pod/web-67bbbf48fc-cgflp   1/1     Running   0          5m42s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.0.0.1             443/TCP   2d21h
service/web          ClusterIP   10.0.0.129           80/TCP    5m8s

创建HPA

[root@k8s-m1 hpa]# kubectl autoscale deployment web --min=2 --max=10 --cpu-percent=80

[root@k8s-m1 hpa]# kubectl get hpa
NAME   REFERENCE        TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   /80%   2         10        1          17s

说明：为名为web的deployment创建一个HPA对象，目标CPU使用率为80%，副本数量配置为2到10之间

压测

yum install httpd-tools
# 总20w请求，并发1000
[root@k8s-m1 hpa]# ab -n 200000 -c 1000 http://10.0.0.129/index.html 

观察扩容状态

#3分钟后扩容
[root@k8s-m1 ~]# kubectl get hpa
NAME   REFERENCE        TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   0%/80%    2         10        4          11m
[root@k8s-m1 ~]# kubectl get pod
NAME                   READY   STATUS    RESTARTS   AGE
web-67bbbf48fc-5gkf8   1/1     Running   0          11m
web-67bbbf48fc-6xvdk   1/1     Running   0          4m14s
web-67bbbf48fc-cgflp   1/1     Running   0          18m
web-67bbbf48fc-ftvct   1/1     Running   0          4m14s

#5分钟后冷却
[root@k8s-m1 ~]# kubectl get hpa
NAME   REFERENCE        TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
web    Deployment/web   0%/80%    2         10        2          17m
[root@k8s-m1 ~]# kubectl get pod
NAME                   READY   STATUS    RESTARTS   AGE
web-67bbbf48fc-5gkf8   1/1     Running   0          18m
web-67bbbf48fc-cgflp   1/1     Running   0          25m

基于自定义指标

概述

001 为满足更多的需求，HPA也支持自定义指标，例如QPS、5xx错误状态码等
002 实现自定义指标由autoscaling/v2版本提供
003 对于自定义指标（例如QPS），将使用custom.metrics.k8s.io API，由相关适配器（Adapter）服务提供

1639547057253.png

基于qps实现pod动态扩容缩容案例

假设我们有一个网站，想基于每秒接收到的HTTP请求对其Pod进行
自动缩放，实现HPA大概步骤：
001 部署Prometheus
002 对应用暴露指标，部署应用，并让Prometheus采集暴露的指标
003 部署Prometheus Adapter
004 为指定HPA配置Prometheus Adapter
005 创建HPA
006 压测、验证

部署Prometheus

[root@k8s-m1 prometheus]# kubectl apply -f .

[root@k8s-m1 prometheus]# kubectl get pod -n kube-system
NAME                                      READY   STATUS    RESTARTS   AGE
...
prometheus-8474f8559d-pff4r               2/2     Running   0          4m43s

1639548227041.png

对应用暴露指标，部署应用，并让Prometheus采集暴露的指标

apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-flask-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: flask-app
  template:
    metadata:
      labels:
        app: flask-app
      # 声明Prometheus采集
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "80"
        prometheus.io/path: "/metrics"
    spec:
      containers:
      - image: lizhenliang/metrics-flask-app
        name: web
......
----------------------------------------------------------------------------------
[root@k8s-m1 metrics-app]# kubectl get pod
NAME                                 READY   STATUS    RESTARTS   AGE
metrics-flask-app-66c9995b58-b2pf5   1/1     Running   0          4m57s
metrics-flask-app-66c9995b58-mqthk   1/1     Running   0          4m57s
metrics-flask-app-66c9995b58-slgzm   1/1     Running   0          4m57s     

1639548836296.png

1639548983916.png

部署Prometheus Adapter

[root@k8s-m1 adapter]# ls
prometheus-adapter-configmap.yaml  prometheus-adapter.yaml
[root@k8s-m1 adapter]# kubectl apply -f .

[root@k8s-m1 adapter]# kubectl get pod -n kube-system
......
NAME                                      READY   STATUS    RESTARTS   AGE
prometheus-adapter-7f94cc997d-gdl9k       1/1     Running   0          6m34s

[root@k8s-m1 adapter]# kubectl get apiservices
v1beta1.custom.metrics.k8s.io    kube-system/prometheus-adapter   True        

[root@k8s-m1 adapter]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"
{...}

为指定HPA配置Prometheus Adapter

[root@k8s-m1 adapter]# vi prometheus-adapter.yaml 
#加入自定义指标

data:
  config.yaml: |
    rules:
    - seriesQuery: 'request_count_total{app="flask-app"}'
      resources:
        overrides:
          kubernetes_namespace: {resource: "namespace"}
          kubernetes_pod_name: {resource: "pod"}
      name:
        matches: "request_count_total"
        as: "qps"
      metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

----------------------------------------------------------------------------
部署
[root@k8s-m1 adapter]# kubectl apply -f prometheus-adapter.yaml 

[root@k8s-m1 adapter]# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/qps"
{"kind":"MetricValueList","apiVersion":"custom.metrics.k8s.io/v1beta1","metadata":{"selfLink":"/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/qps"},"items":[{"describedObject":{"kind":"Pod","namespace":"default","name":"metrics-flask-app-66c9995b58-b2pf5","apiVersion":"/v1"},"metricName":"qps","timestamp":"2021-12-15T07:48:11Z","value":"16m","selector":null},{"describedObject":{"kind":"Pod","namespace":"default","name":"metrics-flask-app-66c9995b58-mqthk","apiVersion":"/v1"},"metricName":"qps","timestamp":"2021-12-15T07:48:11Z","value":"16m","selector":null},{"describedObject":{"kind":"Pod","namespace":"default","name":"metrics-flask-app-66c9995b58-slgzm","apiVersion":"/v1"},"metricName":"qps","timestamp":"2021-12-15T07:48:11Z","value":"16m","selector":null}]}

----------------------------------------------------------------------------
配置描述：
• seriesQuery：Prometheus查询语句，查询应用系列指标。
• resources：Kubernetes资源标签映射到Prometheus标签。
• name：将Prometheus指标名称在自定义指标API中重命名，matches正则匹配，as指定新名称。
• metricsQuery：一个Go模板，对调用自定义指标API转换为Prometheus查询语句。


Adapter向Prometheus查询语句最终是：
sum(rate(request_count_total{app="flask-app", kubernetes_namespace="default"}[2m])) by (kubernetes_pod_name)

1639550395349.png

创建HPA

[root@k8s-m1 hpa]# vi hpa-v2-qps.yaml 
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: metrics-flask-app 
  namespace: default
spec:
  minReplicas: 1
  maxReplicas: 10
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: metrics-flask-app
  metrics:
  - type: Pods
    pods:
      metric:
        name: qps 
      target:
        type: AverageValue
        averageValue: 10000m

-------------------------------------------------------------------------
# 所有Pod平均值为10000m触发扩容，即每秒10个请求
[root@k8s-m1 hpa]# kubectl apply -f hpa-v2-qps.yaml

[root@k8s-m1 hpa]# kubectl get pod,svc
NAME                                     READY   STATUS    RESTARTS   AGE
pod/metrics-flask-app-66c9995b58-b2pf5   1/1     Running   2          109m
pod/metrics-flask-app-66c9995b58-mqthk   1/1     Running   2          109m
pod/metrics-flask-app-66c9995b58-slgzm   1/1     Running   2          109m

NAME                        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/metrics-flask-app   ClusterIP   10.0.0.119           80/TCP    109m

压力测试

[root@k8s-m1 hpa]# ab -n 300000 -c 1000 http://10.0.0.119/

[root@k8s-m1 hpa]# kubectl get pod        
NAME                                 READY   STATUS    RESTARTS   AGE
metrics-flask-app-66c9995b58-5gb77   1/1     Running   0          4m4s
metrics-flask-app-66c9995b58-b2pf5   1/1     Running   2          127m
metrics-flask-app-66c9995b58-jv2dr   1/1     Running   0          4m19s
metrics-flask-app-66c9995b58-lq8s4   1/1     Running   0          4m4s
metrics-flask-app-66c9995b58-mqthk   1/1     Running   2          127m
metrics-flask-app-66c9995b58-rq8zs   1/1     Running   0          4m19s
metrics-flask-app-66c9995b58-sgtkw   1/1     Running   0          4m4s
metrics-flask-app-66c9995b58-slgzm   1/1     Running   2          127m
metrics-flask-app-66c9995b58-vzzgt   1/1     Running   0          4m19s
metrics-flask-app-66c9995b58-xh9bb   1/1     Running   0          4m4s

[root@k8s-m1 hpa]# kubectl get hpa
NAME              REFERENCE                     TARGETS      MINPODS   MAXPODS REPLICAS 
metrics-flask-app Deployment/metrics-flask-app  258837m/10   1         10        10     



#5分钟后冷却
[root@k8s-m1 hpa]# kubectl get pod 
NAME                                 READY   STATUS    RESTARTS   AGE
metrics-flask-app-66c9995b58-b2pf5   1/1     Running   2          133m

[root@k8s-m1 hpa]# kubectl get hpa
NAME                REFERENCE                      TARGETS   MINPODS   MAXPODS REPLICAS
metrics-flask-app   Deployment/metrics-flask-app   16m/10    1         10      1