使用k8s-prometheus-adapter实现HPA

环境：

kubernetes 1.11+/openshift3.11

---

自定义metric HPA原理：

首选需要注册一个apiservice(custom metrics API)。

当HPA请求metrics时，kube-aggregator(apiservice的controller)会将请求转发到adapter，adapter作为kubernentes集群的pod，实现了Kubernetes resource metrics API and custom metrics API，它会根据配置的rules从Prometheus抓取并处理metrics，在处理(如重命名metrics等)完后将metric通过custom metrics API返回给HPA。最后HPA通过获取的metrics的value对Deployment/ReplicaSet进行扩缩容。

adapter作为extension-apiserver，充当了代理kube-apiserver请求的功能。

如下是k8s-prometheus-adapter apiservice的定义，kube-aggregator通过下面的service将请求转发给adapter。v1beta1.custom.metrics.k8s.io是写在k8s-prometheus-adapter代码中的，因此不能任意改变。

apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.custom.metrics.k8s.io
spec:
  service:
    name: custom-metrics-apiserver
    namespace: custom-metrics
  group: custom.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100

---

部署：

• github下载k8s-prometheus-adapter

• 参照官方文档部署adapter：

  • pull镜像：directxman12/k8s-prometheus-adapter:latest，修改镜像tag并push到本地镜像仓库

  • 生成证书：运行如下shell脚本(来自官方)生成cm-adapter-serving-certs.yaml，并将其拷贝到manifests/目录下，该证书用于kube-aggregator与adapter通信时认证adapter。注意下面证书有效时间为5年(43800h)以及授权的域名。

    #!/usr/bin/env bash
    # exit immediately when a command fails
    set -e
    # only exit with zero if all commands of the pipeline exit successfully
    set -o pipefail
    # error on unset variables
    set -u
    
    # Detect if we are on mac or should use GNU base64 options
    case $(uname) in
            Darwin)
                b64_opts='-b=0'
                ;; 
            *)
                b64_opts='--wrap=0'
    esac
    
    go get -v -u github.com/cloudflare/cfssl/cmd/...
    
    export PURPOSE=metrics
    echo '{"signing":{"default":{"expiry":"43800h","usages":["signing","key encipherment","'${PURPOSE}'"]}}}' > "ca-config.json"
    
    export SERVICE_NAME=custom-metrics-apiserver
    export ALT_NAMES='"custom-metrics-apiserver.custom-metrics","custom-metrics-apiserver.custom-metrics.svc"'
    echo "{\"CN\":\"${SERVICE_NAME}\", \"hosts\": [${ALT_NAMES}], \"key\": {\"algo\": \"rsa\",\"size\": 2048}}" | \
            cfssl gencert -ca=ca.crt -ca-key=ca.key -config=ca-config.json - | cfssljson -bare apiserver
    
    cat <<-EOF > cm-adapter-serving-certs.yaml
    apiVersion: v1
    kind: Secret
    metadata:
      name: cm-adapter-serving-certs
    data:
      serving.crt: $(base64 ${b64_opts} < apiserver.pem)
      serving.key: $(base64 ${b64_opts} < apiserver-key.pem)
    EOF

    可以在custom-metrics-apiservice.yaml中设置insecureSkipTLSVerify: true时，kube-aggregator不会校验adapter的如上证书。如果需要启用校验，则需要在caBundle中添加openshift集群的ca证书(非openshift集群的自签证书会被认为是不可信任的证书)，将openshift集群master节点的/etc/origin/master/ca.crt进行base64转码黏贴到caBundle字段即可。

    base64 ca.crt

    也可以黏贴openshift集群master节点的/root/.kube/config文件中的clusters.cluster.certificate-authority-data字段

    • 创建命名空间：kubectl create namespace custom-metrics

  • openshift的kube-system下面可能没有role extension-apiserver-authentication-reader,如果不存在，则需要创建

    apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      annotations:
        rbac.authorization.kubernetes.io/autoupdate: "true"
      labels:
        kubernetes.io/bootstrapping: rbac-defaults
      name: extension-apiserver-authentication-reader
      namespace: kube-system
    rules:
    - apiGroups:
      - ""
      resourceNames:
      - extension-apiserver-authentication
      resources:
      - configmaps
      verbs:
      - get

  • 修改custom-metrics-apiserver-deployment.yaml的--prometheus-url字段，指向正确的prometheus

  • 创建其他组件：kubectl create -f manifests/

    在部署时会创建一个名为custom-metrics-resource-reader的clusterRole，用于授权adapter读取kubernetes cluster的资源，可以看到其允许读取的资源为namespaces/pods/services

    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: custom-metrics-resource-reader
    rules:
    - apiGroups:
      - ""
      resources:
      - namespaces
      - pods
      - services
      verbs:
      - get
      - list

• 部署demo：

  • 部署官方demo

    # cat sample-app.deploy.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: sample-app
      labels:
        app: sample-app
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: sample-app
      template:
        metadata:
          labels:
            app: sample-app
        spec:
          containers:
          - image: docker-local.art.aliocp.csvw.com/openshift3/autoscale-demo:v0.1.2
            name: metrics-provider
            ports:
            - name: http
              containerPort: 8080

  • 创建service

    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: sample-app
      name: sample-app
      namespace: custom-metrics
    spec:
      ports:
      - name: http
        port: 80
        protocol: TCP
        targetPort: 8080
      selector:
        app: sample-app
      type: ClusterIP

    在custom-metrics命名空间下验证可以获取到metrics

    curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP }')/metrics

• 部署serviceMonitor

  由于HPA需要用到namespace和pod等kubernetes的资源信息，因此需要使用servicemonitor注册方式来为metrics添加这些信息

  • openshift Prometheus operator对servicemonitor的限制如下

      serviceMonitorNamespaceSelector:
        matchExpressions:
        - key: openshift.io/cluster-monitoring
          operator: Exists
      serviceMonitorSelector:
        matchExpressions:
        - key: k8s-app
          operator: Exists

  • 因此需要给custom-metrics命名空间添加标签

    oc label namespace custom-metrics openshift.io/cluster-monitoring=true

  • 在openshift-monitoring命名空间中创建service-monitor

    # cat service-monitor.yaml
    kind: ServiceMonitor
    apiVersion: monitoring.coreos.com/v1
    metadata:
      name: sample-app
      labels:
        k8s-app: testsample
        app: sample-app
    spec:
      namespaceSelector:
        any: true
      selector:
        matchLabels:
          app: sample-app
      endpoints:
      - port: http

  • 添加权限

    oc adm policy add-cluster-role-to-user view system:serviceaccount:openshift-monitoring:prometheus-k8s
    
    oc adm policy add-role-to-user view system:serviceaccount:openshift-monitoring:prometheus-k8s -n custom-metrics

• 测试HPA

  • 创建HPA，表示1秒请求大于0.5个时开始扩容

    # cat sample-app-hpa.yaml
    kind: HorizontalPodAutoscaler
    apiVersion: autoscaling/v2beta1
    metadata:
      name: sample-app
    spec:
      scaleTargetRef:
        # point the HPA at the sample application
        # you created above
        apiVersion: apps/v1
        kind: Deployment
        name: sample-app
      # autoscale between 1 and 10 replicas
      minReplicas: 1
      maxReplicas: 10
      metrics:
      # use a "Pods" metric, which takes the average of the
      # given metric across all pods controlled by the autoscaling target
      - type: Pods
        pods:
          # use the metric that you used above: pods/http_requests
          metricName: http_requests_per_second
          # target 500 milli-requests per second,
          # which is 1 request every two seconds
          targetAverageValue: 500m

    通过oc describe hpa sample-app查看hpa是否运行正常

  • 持续执行命令curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP }')/metrics发出请求

  • 通过命令kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/custom-metrics/pods/*/http_requests_per_second"查看其对应的value值，当其值大于500m时开始扩容

    # oc get pod
    NAME                          READY     STATUS    RESTARTS   AGE
    sample-app-6d55487cdd-dc6qz   1/1       Running   0          18h
    sample-app-6d55487cdd-w6bbb   1/1       Running   0          5m
    sample-app-6d55487cdd-zbdbr   1/1       Running   0          5m

  • 过段时间，当kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/custom-metrics/pods/*/http_requests_per_second"的值持续低于500m时进行缩容，缩容时间由--horizontal-pod-autoscaler-downscale-stabilization指定，默认5分钟。

    提供oc get hpa的TARGETS字段可以查看扩缩容比例

    # oc get hpa
    NAME         REFERENCE               TARGETS    MINPODS   MAXPODS   REPLICAS   AGE
    sample-app   Deployment/sample-app   66m/500m   1         10        1          3h

---

Adapter config

部署adapter前需要配置adapter的rule，用于预处理metrics，默认配置为manifests/custom-metrics-config-map.yaml。adapter的配置主要分为4个:

• Discovery：指定需要处理的Prometheus的metrics。通过seriesQuery挑选需要处理的metrics集合，可以通过seriesFilters精确过滤metrics。

  seriesQuery可以根据标签进行查找(如下)，也可以直接指定metric name查找

  seriesQuery: '{__name__=~"^container_.*_total",container_name!="POD",namespace!="",pod_name!=""}'
  seriesFilters:
    - isNot: "^container_.*_seconds_total"

  seriesFilters：

  is: , 匹配包含该正则表达式的metrics.
  isNot: , 匹配不包含该正则表达式的metrics.

• Association：设置metric与kubernetes resources的映射关系，kubernetes resorces可以通过kubectl api-resources命令查看。overrides会将Prometheus metric label与一个kubernetes resource(下例为deployment)关联。需要注意的是该label必须是一个真实的kubernetes resource，如metric的pod_name可以映射为kubernetes的pod resource，但不能将container_image映射为kubernetes的pod resource，映射错误会导致无法通过custom metrics API获取正确的值。这也表示metric中必须存在一个真实的resource 名称，将其映射为kubernetes resource。

  resources:
    overrides:
      microservice: {group: "apps", resource: "deployment"}

• Naming：用于将prometheus metrics名称转化为custom metrics API所使用的metrics名称，但不会改变其本身的metric名称，即通过curl http://$(kubectl get service sample-app -o jsonpath='{ .spec.clusterIP }')/metrics获得的仍然是老的metric名称。如果不需要可以不执行这一步。

  # match turn any name _total to _per_second
  # e.g. http_requests_total becomes http_requests_per_second
  name:
    matches: "^(.*)_total$"
    as: "${1}_per_second"

  如本例中HPA后续可以通过/apis/{APIService-name}/v1beta1/namespaces/{namespaces-name}/pods/*/http_requests_per_second获取metrics

• Querying：处理调用custom metrics API获取到的metrics的value，该值最终提供给HPA进行扩缩容

  # convert cumulative cAdvisor metrics into rates calculated over 2 minutes
  metricsQuery: "sum(rate(<<.Series>>{<<.LabelMatchers>>,container_name!="POD"}[2m])) by (<<.GroupBy>>)"

  metricsQuery 字段使用Go template将URL请求转变为Prometheus的请求，它会提取custom metrics API请求中的字段，并将其划分为metric name,group-resource,以及group-resource中的一个或多个objects，对应如下字段：

  • Series: metric名称
  • LabelMatchers: 以逗号分割的objects，当前表示特定group-resource加上命名空间的label(如果该group-resource 是namespaced的)
  • GroupBy：以逗号分割的label的集合，当前表示LabelMatchers中的group-resource label

  假设metrics http_requests_per_second如下

  http_requests_per_second{pod="pod1",service="nginx1",namespace="somens"}
  http_requests_per_second{pod="pod2",service="nginx2",namespace="somens"}

  当调用kubectl get --raw "/apis/{APIService-name}/v1beta1/namespaces/somens/pods/*/http_request_per_second"时，metricsQuery字段的模板的实际内容如下：

  • Series: "http_requests_total"
  • LabelMatchers: "pod=~\"pod1|pod2",namespace="somens"
  • GroupBy:pod

  adapter使用字段rules和externalRules分别表示custom metrics和external metrics，如本例中

  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: adapter-config
    namespace: openshift-monitoring
  data:
    config.yaml: |
      externalRules:
      - seriesQuery: '{namespace!="",pod!=""}'
        seriesFilters: []
        resources:
          overrides:
            namespace:
              resource: namespace
            pod:
              resource: pod
        metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[22m])) by (<<.GroupBy>>)
      rules:
      - seriesQuery: '{namespace!="",pod!=""}'
        seriesFilters: []
        resources:
          overrides:
            namespace:
              resource: namespace
            pod:
              resource: pod
        name:
          matches: "^(.*)_total"
          as: "${1}_per_second"
        metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)

---

HPA的配置

HPA通常会根据type从aggregated APIs (metrics.k8s.io, custom.metrics.k8s.io, external.metrics.k8s.io)的资源路径上拉取metrics

HPA支持的metrics类型有4种(下述为v2beta2的格式)：

• resource：目前仅支持cpu和memory。target可以指定数值(targetAverageValue)和比例(targetAverageUtilization)进行扩缩容

  HPA从metrics.k8s.io获取resource metrics

• pods：custom metrics，这类metrics描述了pod类型，target仅支持按指定数值(targetAverageValue)进行扩缩容。targetAverageValue 用于计算所有相关pods上的metrics的平均值

  type: Pods
  pods:
    metric:
      name: packets-per-second
    target:
      type: AverageValue
      averageValue: 1k

  HPA从custom.metrics.k8s.io获取custom metrics

• object：custom metrics，这类metrics描述了相同命名空间下的(非pod)类型。target支持通过value和AverageValue进行扩缩容，前者直接将metric与target比较进行扩缩容，后者通过metric/相关的pod数目与target比较进行扩缩容

  type: Object
  object:
    metric:
      name: requests-per-second
    describedObject:
      apiVersion: extensions/v1beta1
      kind: Ingress
      name: main-route
    target:
      type: Value
      value: 2k

• external：kubernetes 1.10+。这类metrics与kubernetes集群无关(pods和object需要与kubernetes中的某一类型关联)。与object类似，target支持通过value和AverageValue进行扩缩容。由于external会尝试匹配所有kubernetes资源的metrics，因此实际中不建议使用该类型。

  HPA从external.metrics.k8s.io获取external metrics

  - type: External
    external:
      metric:
        name: queue_messages_ready
        selector: "queue=worker_tasks"
      target:
        type: AverageValue
        averageValue: 30

• 1.6版本支持多metrics的扩缩容，当其中一个metrics达到扩容标准时就会创建pod副本(当前副本

注：target的value的一个单位可以划分为1000份，每一份以m为单位，如500m表示1/2个单位。参见Quantity

kubernetes HPA的算法如下：

desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]

当使用targetAverageValue 或targetAverageUtilization时，currentMetricValue会取HPA指定的所有pods的metric的平均值

---

Kubernetes metrics的获取

假设注册的APIService为custom.metrics.k8s.io/v1beta1，在注册好APIService后HorizontalPodAutoscaler controller会从以/apis/custom.metrics.k8s.io/v1beta1为根API的路径上抓取metrics。metrics的API path可以分为namespaced和non-namespaced类型的。通过如下方式校验HPA是否可以获取到metrics：

namespaced

• 获取指定namespace下指定object类型和名称的metrics

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/{object-type}/{object-name}/{metric-name...}"

如获取monitor命名空间下名为grafana的pod的start_time_seconds metric

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitor/pods/grafana/start_time_seconds"

• 获取指定namespace下所有特定object类型的metrics

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/pods/*/{metric-name...}"

如获取monitor命名空间下名为所有pod的start_time_seconds metric

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitor/pods/*/start_time_seconds"

• 使用labelSelector可以选择带有特定label的object

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/{object-type}/{object-name}/{metric-name...}?labelSelector={label-name}"

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/pods/*/{metric-name...}?labelSelector={label-name}"

non-namespaced

non-namespaced和namespaced的类似，主要有node，namespace，PersistentVolume等。non-namespaced访问有些与custom metrics API描述不一致。

• 访问object为namespace的方式如下如下

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/metrics/{metric-name...}"

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/*/metrics/{metric-name...}"

• 访问node的方式如下

 kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/nodes/{node-name}/{metric-name...}"

---

DEBUG:

• 使用如下方式查看注册的APIService发现的所有rules

  kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1

  如果获取失败，可以看下使用oc get apiservice v1beta1.custom.metrics.k8s.io -oyaml查看status和message的相关信息

  如果获取到的resource为空，则需要校验deploy中的Prometheus url是否正确，是否有权限等

• 通过如下方式查看完整的请求过程(--v=8)

  kubectl get --raw “/apis/custom.metrics.k8s.io/v1beta1/namespaces/{namespace-name}/pods/*/{metric-name...}" --v=8 

• 如果上述过程正确，但获取到的items为空

  • 首先保证k8s-prometheus-adapter的参数--metrics-relist-interval设置值大于Prometheus的参数scrape_interval
  • 确保k8s-prometheus-adapter rules的seriesQuery规则可以抓取到Prometheus的数据
  • 确保k8s-prometheus-adapter rules的metricsQuery规则可以抓取到计算出数据，此处需要注意的是，如果使用到了计算某段时间的数据，如果时间设置过短，可能导致没有数据生成

---

TIPS:

• 官方提供了End-to-end walkthrough，但需要采集的metrics中包含pod和namespace label，否则在官方默认配置下无法采集到metrics。

• Configuration Walkthroughs一步步讲解了如何配置adapter config

• 在goland里面使用如下参数可以远程调试adapter：

  --secure-port=6443 --tls-cert-file=D:\adapter\serving.crt --tls-private-key-file=D:\adapter\serving.key --logtostderr=true --prometheus-url=${prometheus-url} --metrics-relist-interval=70s --v=10 --config=D:\adapter\config.yaml --lister-kubeconfig=D:\adapter\k8s-config.yaml --authorization-kubeconfig=D:\adapter\k8s-config.yaml --authentication-kubeconfig=D:\adapter\k8s-config.yaml

---

参考：

Kubernetes pod autoscaler using custom metrics

Kubernetes API Aggregation Setup — Nuts & Bolts

Configure the Aggregation Layer

Aggregation

Setup an Extension API Server

OpenShift下的JVM监控