k8s与HPA--通过 Prometheus adaptor 来自定义监控指标
自动扩展是一种根据资源使用情况自动扩展或缩小工作负载的方法。 Kubernetes中的自动缩放有两个维度：Cluster Autoscaler处理节点扩展操作，Horizontal Pod Autoscaler自动扩展部署或副本集中的pod数量。 Cluster Autoscaling与Horizontal Pod Autoscaler一起用于动态调整计算能力以及系统满足SLA所需的并行度。虽然Cluster Autoscaler高度依赖托管您的集群的云提供商的基础功能，但HPA可以独立于您的IaaS / PaaS提供商运营。
Horizontal Pod Autoscaler功能最初是在Kubernetes v1.1中引入的，并且从那时起已经发展了很多。 HPA缩放容器的版本1基于观察到的CPU利用率，后来基于内存使用情况。在Kubernetes 1.6中，引入了一个新的API Custom Metrics API，使HPA能够访问任意指标。 Kubernetes 1.7引入了聚合层，允许第三方应用程序通过将自己注册为API附加组件来扩展Kubernetes API。 Custom Metrics API和聚合层使Prometheus等监控系统可以向HPA控制器公开特定于应用程序的指标。
Horizontal Pod Autoscaler实现为一个控制循环，定期查询Resource Metrics API以获取CPU /内存等核心指标和针对特定应用程序指标的Custom Metrics API。
以下是为Kubernetes 1.9或更高版本配置HPA v2的分步指南。您将安装提供核心指标的Metrics Server附加组件，然后您将使用演示应用程序根据CPU和内存使用情况展示pod自动扩展。在本指南的第二部分中，您将部署Prometheus和自定义API服务器。您将使用聚合器层注册自定义API服务器，然后使用演示应用程序提供的自定义指标配置HPA。
在开始之前，您需要安装Go 1.8或更高版本并在GOPATH中克隆k8s-prom-hpa repo。
cd $GOPATH
git clone https://github.com/stefanprodan/k8s-prom-hpa部署 Metrics Server
kubernetes Metrics Server是资源使用数据的集群范围聚合器，是Heapster的后继者。度量服务器通过汇集来自kubernetes.summary_api的数据来收集节点和pod的CPU和内存使用情况。摘要API是一种内存高效的API，用于将数据从Kubelet / cAdvisor传递到度量服务器。
在HPA的第一个版本中，您需要Heapster来提供CPU和内存指标，在HPA v2和Kubernetes 1.8中，只有在启用horizontal-pod-autoscaler-use-rest-clients时才需要指标服务器。默认情况下，Kubernetes 1.9中启用了HPA rest客户端。 GKE 1.9附带预安装的Metrics Server。
在kube-system命名空间中部署Metrics Server：
kubectl create -f ./metrics-server一分钟后，度量服务器开始报告节点和pod的CPU和内存使用情况。
查看nodes metrics：
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" | jq .结果如下：
{
"kind": "NodeMetricsList",
"apiVersion": "metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/metrics.k8s.io/v1beta1/nodes"
},
"items": [
{
"metadata": {
"name": "ip-10-1-50-61.ec2.internal",
"selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-1-50-61.ec2.internal",
"creationTimestamp": "2019-02-13T08:34:05Z"
},
"timestamp": "2019-02-13T08:33:38Z",
"window": "30s",
"usage": {
"cpu": "78322168n",
"memory": "563180Ki"
}
},
{
"metadata": {
"name": "ip-10-1-57-40.ec2.internal",
"selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-1-57-40.ec2.internal",
"creationTimestamp": "2019-02-13T08:34:05Z"
},
"timestamp": "2019-02-13T08:33:42Z",
"window": "30s",
"usage": {
"cpu": "48926263n",
"memory": "554472Ki"
}
},
{
"metadata": {
"name": "ip-10-1-62-29.ec2.internal",
"selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/ip-10-1-62-29.ec2.internal",
"creationTimestamp": "2019-02-13T08:34:05Z"
},
"timestamp": "2019-02-13T08:33:36Z",
"window": "30s",
"usage": {
"cpu": "36700681n",
"memory": "326088Ki"
}
}
]
}查看pods metrics:
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" | jq .结果如下：
{
"kind": "PodMetricsList",
"apiVersion": "metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/metrics.k8s.io/v1beta1/pods"
},
"items": [
{
"metadata": {
"name": "kube-proxy-77nt2",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-77nt2",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:00Z",
"window": "30s",
"containers": [
{
"name": "kube-proxy",
"usage": {
"cpu": "2370555n",
"memory": "13184Ki"
}
}
]
},
{
"metadata": {
"name": "cluster-autoscaler-n2xsl",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/cluster-autoscaler-n2xsl",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:12Z",
"window": "30s",
"containers": [
{
"name": "cluster-autoscaler",
"usage": {
"cpu": "1477997n",
"memory": "54584Ki"
}
}
]
},
{
"metadata": {
"name": "core-dns-autoscaler-b4785d4d7-j64xd",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/core-dns-autoscaler-b4785d4d7-j64xd",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:08Z",
"window": "30s",
"containers": [
{
"name": "autoscaler",
"usage": {
"cpu": "191293n",
"memory": "7956Ki"
}
}
]
},
{
"metadata": {
"name": "spot-interrupt-handler-8t2xk",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/spot-interrupt-handler-8t2xk",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:04Z",
"window": "30s",
"containers": [
{
"name": "spot-interrupt-handler",
"usage": {
"cpu": "844907n",
"memory": "4608Ki"
}
}
]
},
{
"metadata": {
"name": "kube-proxy-t5kqm",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-t5kqm",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:08Z",
"window": "30s",
"containers": [
{
"name": "kube-proxy",
"usage": {
"cpu": "1194766n",
"memory": "12204Ki"
}
}
]
},
{
"metadata": {
"name": "kube-proxy-zxmqb",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-proxy-zxmqb",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:06Z",
"window": "30s",
"containers": [
{
"name": "kube-proxy",
"usage": {
"cpu": "3021117n",
"memory": "13628Ki"
}
}
]
},
{
"metadata": {
"name": "aws-node-rcz5c",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/aws-node-rcz5c",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:15Z",
"window": "30s",
"containers": [
{
"name": "aws-node",
"usage": {
"cpu": "1217989n",
"memory": "24976Ki"
}
}
]
},
{
"metadata": {
"name": "aws-node-z2qxs",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/aws-node-z2qxs",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:15Z",
"window": "30s",
"containers": [
{
"name": "aws-node",
"usage": {
"cpu": "1025780n",
"memory": "46424Ki"
}
}
]
},
{
"metadata": {
"name": "php-apache-899d75b96-8ppk4",
"namespace": "default",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/php-apache-899d75b96-8ppk4",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:08Z",
"window": "30s",
"containers": [
{
"name": "php-apache",
"usage": {
"cpu": "24612n",
"memory": "27556Ki"
}
}
]
},
{
"metadata": {
"name": "load-generator-779c5f458c-9sglg",
"namespace": "default",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/load-generator-779c5f458c-9sglg",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:34:56Z",
"window": "30s",
"containers": [
{
"name": "load-generator",
"usage": {
"cpu": "0",
"memory": "336Ki"
}
}
]
},
{
"metadata": {
"name": "aws-node-v9jxs",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/aws-node-v9jxs",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:00Z",
"window": "30s",
"containers": [
{
"name": "aws-node",
"usage": {
"cpu": "1303458n",
"memory": "28020Ki"
}
}
]
},
{
"metadata": {
"name": "kube2iam-m2ktt",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube2iam-m2ktt",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:11Z",
"window": "30s",
"containers": [
{
"name": "kube2iam",
"usage": {
"cpu": "1328864n",
"memory": "9724Ki"
}
}
]
},
{
"metadata": {
"name": "kube2iam-w9cqf",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube2iam-w9cqf",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:03Z",
"window": "30s",
"containers": [
{
"name": "kube2iam",
"usage": {
"cpu": "1294379n",
"memory": "8812Ki"
}
}
]
},
{
"metadata": {
"name": "custom-metrics-apiserver-657644489c-pk8rb",
"namespace": "monitoring",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/monitoring/pods/custom-metrics-apiserver-657644489c-pk8rb",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:04Z",
"window": "30s",
"containers": [
{
"name": "custom-metrics-apiserver",
"usage": {
"cpu": "22409370n",
"memory": "42468Ki"
}
}
]
},
{
"metadata": {
"name": "kube2iam-qghgt",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube2iam-qghgt",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:11Z",
"window": "30s",
"containers": [
{
"name": "kube2iam",
"usage": {
"cpu": "2078992n",
"memory": "16356Ki"
}
}
]
},
{
"metadata": {
"name": "spot-interrupt-handler-ps745",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/spot-interrupt-handler-ps745",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:10Z",
"window": "30s",
"containers": [
{
"name": "spot-interrupt-handler",
"usage": {
"cpu": "611566n",
"memory": "4336Ki"
}
}
]
},
{
"metadata": {
"name": "coredns-68fb7946fb-2xnpp",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-68fb7946fb-2xnpp",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:12Z",
"window": "30s",
"containers": [
{
"name": "coredns",
"usage": {
"cpu": "1610381n",
"memory": "10480Ki"
}
}
]
},
{
"metadata": {
"name": "coredns-68fb7946fb-9ctjf",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-68fb7946fb-9ctjf",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:13Z",
"window": "30s",
"containers": [
{
"name": "coredns",
"usage": {
"cpu": "1418850n",
"memory": "9852Ki"
}
}
]
},
{
"metadata": {
"name": "prometheus-7d4f6d4454-v4fnd",
"namespace": "monitoring",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/monitoring/pods/prometheus-7d4f6d4454-v4fnd",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:00Z",
"window": "30s",
"containers": [
{
"name": "prometheus",
"usage": {
"cpu": "17951807n",
"memory": "202316Ki"
}
}
]
},
{
"metadata": {
"name": "metrics-server-7cdd54ccb4-k2x7m",
"namespace": "kube-system",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/metrics-server-7cdd54ccb4-k2x7m",
"creationTimestamp": "2019-02-13T08:35:19Z"
},
"timestamp": "2019-02-13T08:35:04Z",
"window": "30s",
"containers": [
{
"name": "metrics-server-nanny",
"usage": {
"cpu": "144656n",
"memory": "5716Ki"
}
},
{
"name": "metrics-server",
"usage": {
"cpu": "568327n",
"memory": "16268Ki"
}
}
]
}
]
}基于CPU和内存使用情况的Auto Scaling
您将使用基于Golang的小型Web应用程序来测试Horizontal Pod Autoscaler（HPA）。
将podinfo部署到默认命名空间：
kubectl create -f ./podinfo/podinfo-svc.yaml,./podinfo/podinfo-dep.yaml使用NodePort服务访问podinfo，地址为http：//
接下来定义一个至少维护两个副本的HPA，如果CPU平均值超过80％或内存超过200Mi，则最多可扩展到10个：
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: podinfo
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: podinfo
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 80
- type: Resource
resource:
name: memory
targetAverageValue: 200Mi创建这个hpa:
kubectl create -f ./podinfo/podinfo-hpa.yaml几秒钟后，HPA控制器联系度量服务器，然后获取CPU和内存使用情况：
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
podinfo Deployment/podinfo 2826240 / 200Mi, 15% / 80% 2 10 2 5m为了增加CPU使用率，请使用rakyll / hey运行负载测试：
#install hey
go get -u github.com/rakyll/hey
#do 10K requests
hey -n 10000 -q 10 -c 5 http://:31198/ 您可以使用以下方式监控HPA事件：
$ kubectl describe hpa
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 7m horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 3m horizontal-pod-autoscaler New size: 8; reason: cpu resource utilization (percentage of request) above target 暂时删除podinfo。稍后将在本教程中再次部署它：
kubectl delete -f ./podinfo/podinfo-hpa.yaml,./podinfo/podinfo-dep.yaml,./podinfo/podinfo-svc.yaml部署 Custom Metrics Server
要根据自定义指标进行扩展，您需要拥有两个组件。一个组件，用于从应用程序收集指标并将其存储在Prometheus时间序列数据库中。第二个组件使用collect（k8s-prometheus-adapter）提供的指标扩展了Kubernetes自定义指标API。
您将在专用命名空间中部署Prometheus和适配器。
创建monitoring命名空间：
kubectl create -f ./namespaces.yaml在monitoring命名空间中部署Prometheus v2：
kubectl create -f ./prometheus生成Prometheus适配器所需的TLS证书：
make certs生成以下几个文件：
# ls output
apiserver.csr apiserver-key.pem apiserver.pem部署Prometheus自定义指标API适配器：
kubectl create -f ./custom-metrics-api
列出Prometheus提供的自定义指标：
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq .获取monitoring命名空间中所有pod的FS使用情况：
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/*/fs_usage_bytes" | jq .查询结果如下：
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/monitoring/pods/%2A/fs_usage_bytes"
},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "monitoring",
"name": "custom-metrics-apiserver-657644489c-pk8rb",
"apiVersion": "/v1"
},
"metricName": "fs_usage_bytes",
"timestamp": "2019-02-13T08:52:30Z",
"value": "94253056"
},
{
"describedObject": {
"kind": "Pod",
"namespace": "monitoring",
"name": "prometheus-7d4f6d4454-v4fnd",
"apiVersion": "/v1"
},
"metricName": "fs_usage_bytes",
"timestamp": "2019-02-13T08:52:30Z",
"value": "24576"
}
]
}基于custom metrics 自动伸缩
在默认命名空间中创建podinfo NodePort服务和部署：
kubectl create -f ./podinfo/podinfo-svc.yaml,./podinfo/podinfo-dep.yamlpodinfo应用程序公开名为http_requests_total的自定义指标。 Prometheus适配器删除_total后缀并将度量标记为计数器度量标准。
从自定义指标API获取每秒的总请求数：
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_requests" | jq .{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_requests"
},
"items": [
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "podinfo-6b86c8ccc9-kv5g9",
"apiVersion": "/__internal"
},
"metricName": "http_requests",
"timestamp": "2018-01-10T16:49:07Z",
"value": "901m"
},
{
"describedObject": {
"kind": "Pod",
"namespace": "default",
"name": "podinfo-6b86c8ccc9-nm7bl",
"apiVersion": "/__internal"
},
"metricName": "http_requests",
"timestamp": "2018-01-10T16:49:07Z",
"value": "898m"
}
]
}
建一个HPA，如果请求数超过每秒10个，将扩展podinfo部署：
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: podinfo
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: podinfo
minReplicas: 2
maxReplicas: 10
metrics:
- type: Pods
pods:
metricName: http_requests
targetAverageValue: 10
在默认命名空间中部署podinfo HPA：
kubectl create -f ./podinfo/podinfo-hpa-custom.yaml
几秒钟后，HPA从指标API获取http_requests值：
kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
podinfo Deployment/podinfo 899m / 10 2 10 2 1m
在podinfo服务上应用一些负载，每秒25个请求：
#install hey
go get -u github.com/rakyll/hey
#do 10K requests rate limited at 25 QPS
hey -n 10000 -q 5 -c 5 http://:31198/healthz
几分钟后，HPA开始扩展部署：
kubectl describe hpa
Name: podinfo
Namespace: default
Reference: Deployment/podinfo
Metrics: ( current / target )
"http_requests" on pods: 9059m / 10
Min replicas: 2
Max replicas: 10
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 2m horizontal-pod-autoscaler New size: 3; reason: pods metric http_requests above target
按照当前的每秒请求速率，部署永远不会达到10个pod的最大值。三个复制品足以使每个吊舱的RPS保持在10以下。
负载测试完成后，HPA会将部署缩到其初始副本：
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 5m horizontal-pod-autoscaler New size: 3; reason: pods metric http_requests above target
Normal SuccessfulRescale 21s horizontal-pod-autoscaler New size: 2; reason: All metrics below target
您可能已经注意到自动缩放器不会立即对使用峰值做出反应。默认情况下，度量标准同步每30秒发生一次，只有在最后3-5分钟内没有重新缩放时才能进行扩展/缩小。通过这种方式，HPA可以防止快速执行冲突的决策，并为Cluster Autoscaler提供时间。
结论
并非所有系统都可以通过单独依赖CPU /内存使用指标来满足其SLA，大多数Web和移动后端需要基于每秒请求进行自动扩展以处理任何流量突发。对于ETL应用程序，可以通过作业队列长度超过某个阈值等来触发自动缩放。通过使用Prometheus检测应用程序并公开正确的自动缩放指标，您可以对应用程序进行微调，以更好地处理突发并确保高可用性。