k8s对接云实现自动扩容pod

公众号关注 「SY技术小站」

设为「星标」，每天带你分享技术与生活！

介绍

在之前的文章我介绍了下 Custom Metric 怎么实现自动扩容的。k8s基于自定义指标实现自动扩容

实际上Kubernetes定义了三种不同的监控数据接口，分别是Resource Metric，Custom Metric以及External Metric。

一般来说Resource Metric是通过metrics-server采集；

Custom Metric是通过prometheus来实现自定义扩容。

External Metric就是针对云场景的了，比方说通过获取slb最大连接数来实现自动扩容。

下面我来说下具体怎么实现的。

部署

安装alibaba-cloud-metrics-adapter，以下是yaml文件

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: alibaba-cloud-metrics-adapter
  name: alibaba-cloud-metrics-adapter
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alibaba-cloud-metrics-adapter
  template:
    metadata:
      labels:
        app: alibaba-cloud-metrics-adapter
      name: alibaba-cloud-metrics-adapter
    spec:
      serviceAccountName: admin
      containers:
      - name: alibaba-cloud-metrics-adapter
        image: registry.cn-beijing.aliyuncs.com/acs/alibaba-cloud-metrics-adapter-amd64:v0.2.0-alpha-e8f8c17f
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 443
          name: https
        - containerPort: 8080
          name: http
        volumeMounts:
        - mountPath: /tmp
          name: temp-vol
        - name: tz-config
          mountPath: /etc/localtime
          readOnly: true
      volumes:
      - name: temp-vol
        emptyDir: {}
      - name: tz-config
        hostPath:
          path: /etc/localtime
---
apiVersion: v1
kind: Service
metadata:
  name: alibaba-cloud-metrics-adapter
  namespace: kube-system
spec:
  ports:
  - name: https
    port: 443
    targetPort: 443
  - name: http
    port: 80
    targetPort: 8080
  selector:
    app: alibaba-cloud-metrics-adapter
---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.external.metrics.k8s.io
spec:
  service:
    name: alibaba-cloud-metrics-adapter
    namespace: kube-system
  group: external.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-metrics-server-resources
rules:
- apiGroups:
  - external.metrics.k8s.io
  resources: ["*"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: external-metrics-resource-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-metrics-server-resources
subjects:
- kind: ServiceAccount
  name: horizontal-pod-autoscaler
  namespace: kube-system

可以使用下面的命令来检测是否生效了：

# kubectl get --raw="/apis/external.metrics.k8s.io/v1beta1" | jq
{
  "kind": "APIResourceList",
  "apiVersion": "v1",
  "groupVersion": "external.metrics.k8s.io/v1beta1",
  "resources": [
    {
      "name": "sls_ingress_qps",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "slb_l4_connection_utilization",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "slb_l7_qps",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "ahas_sentinel_total_qps",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "ahas_sentinel_avg_rt",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "k8s_workload_cpu_util",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "k8s_workload_memory_request",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    {
      "name": "k8s_workload_memory_cache",
      "singularName": "",
      "namespaced": true,
      "kind": "ExternalMetricValueList",
      "verbs": [
        "get"
      ]
    },
    。。。。。。。。。

简单说下各个指标的含义，方便之后去选择哪个指标去自动扩容。

❝

slb_l4_traffic_rx   每秒流入

slb_l4_packet_tx   每秒流入的数据包数

slb_l4_active_connection   活动连接

slb_l4_max_connection   最大连接数

slb_l7_qps   QPS

slb_l7_status_2xx   2xx个请求（每秒）

slb_l7_upstream_4xx   上游服务4xx请求（每秒）

sls_ingress_qps   特定入口路由的QPS

sls_ingress_inflow   入口流入带宽

k8s_workload_memory_usage  内存使用情况

k8s_workload_memory_rss  rss

❞

扩容例子

根据slb_l4_active_connection这个指标，实现自动扩容

apiVersion: apps/v1beta2 # for versions before 1.8.0 use apps/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment-basic
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9 # replace it with your exactly 
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
    - port: 80
      protocol: TCP
      targetPort: 80
  selector:
    app: nginx
  sessionAffinity: None
  type: LoadBalancer
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: slb-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta2
    kind: Deployment
    name: nginx-deployment-basic
  minReplicas: 5
  maxReplicas: 10
  metrics:
    - type: External
      external:
        metric:
          name: slb_l4_active_connection
          selector:
            matchLabels:
              # slb.instance.id: "lb-2ze2locy5fk8at1cfx47y"
              slb.instance.id: ""
              # slb.instance.port: "80"
              slb.instance.port: ""
        target:
          type: Value
          value: 100

这样就实现了通过External Metric自动扩容。

参考链接

https://github.com/AliyunContainerService/alibaba-cloud-metrics-adapter

往期精彩文章 

• ingress-nginx传输加密与认证
  

• ingress高可用

• k8s二进制升级从1.14.2至1.16.2

• k8s集群curl service的ip延迟响应高

• k8s集群数据备份与恢复

• ingress-nginx持久化日志

• ingress-nginx金丝雀发布

• k8s的pv与pvc
  

  微信群加入方式 

  欢迎关注作者公众号

点个在看少个 bug ????