k8s Pod的自动水平伸缩（HPA）

HPA（Horizontal Pod Autoscaler ）

pod的自动水平伸缩

有了HPA，我们就不用为上面的问题而烦恼，HPA会帮我们自动完成pod的扩缩容。

当资源需求过高时，会自动创建出pod副本；当资源需求低时，会自动收缩pod副本数。

注意：首先必须确保集群中已经安装heapster的组件，否则无法获取集群内资源数据，无法进行以下操作。

原理：

通过集群内的资源监控系统（heapster），来获取集群中资源的使用状态。

根据CPU、内存、以及用户自定义的资源指标数据的使用量或连接数为参考依据，来制定一个临界点，一旦超出这个点，HPA就会自动创建出pod副本

版本说明：

通过kubectl api-versions可以看到，目前有3个版本：

autoscaling/v1            #只支持通过cpu为参考依据，来改变pod副本数
autoscaling/v2beta1       #支持通过cpu、内存、连接数以及用户自定义的资源指标数据为参考依据。
autoscaling/v2beta2       #同上，小的变动

查询版本：

kubectl explain hpa                                     ##默认查询到的是autoscaling/v1版本
kubectl explain hpa --api-version=autoscaling/v2beta1   ##如果使用其他版本，可以使用--api-version指明版本

一、部署监控组件
1、部署influxdb服务

[root@k8s-node1 heapster]# cat influxdb.yaml 
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: monitoring-influxdb
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: influxdb
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: influxdb
    spec:
      containers:
      - name: influxdb
        #image: gcr.io/google_containers/heapster-influxdb-amd64:v1.3.3
        image: mirrorgooglecontainers/heapster-influxdb-amd64:v1.3.3
        volumeMounts:
        - mountPath: /data
          name: influxdb-storage
      volumes:
      - name: influxdb-storage
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  labels:
    task: monitoring
    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
    # If you are NOT using this as an addon, you should comment out this line.
    # kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: monitoring-influxdb
  name: monitoring-influxdb
  namespace: kube-system
spec:
  ports:
  - port: 8086
    targetPort: 8086
    name: http
  selector:
    k8s-app: influxdb

2、部署heapster服务

[root@k8s-node1 heapster]# cat heapster.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: heapster
  namespace: kube-system
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: heapster
subjects:
  - kind: ServiceAccount
    name: heapster
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:heapster
  apiGroup: rbac.authorization.k8s.io
---

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: heapster
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        #image: gcr.io/google_containers/heapster-amd64:v1.5.1
        image: mirrorgooglecontainers/heapster-amd64:v1.5.1
        imagePullPolicy: IfNotPresent
        command:
        - /heapster
        - --source=kubernetes:https://kubernetes.default
        - --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
---
apiVersion: v1
kind: Service
metadata:
  labels:
    task: monitoring
    # For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)
    # If you are NOT using this as an addon, you should comment out this line.
    #kubernetes.io/cluster-service: 'true'
    kubernetes.io/name: Heapster
  name: heapster
  namespace: kube-system
spec:
  ports:
  - port: 80
    targetPort: 8082
  selector:
    k8s-app: heapster

3、部署grafana服务

[root@k8s-node1 heapster]# cat gra.yaml 
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: grafana-core
  namespace: kube-system
  labels:
    app: grafana
    component: core
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: grafana
        component: core
    spec:
      containers:
      - image: grafana/grafana:4.2.0
        name: grafana-core
        imagePullPolicy: IfNotPresent
        # env:
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 100Mi
        env:
          # The following env variables set up basic auth twith the default admin user and admin password.
          - name: GF_AUTH_BASIC_ENABLED
            value: "true"
          - name: GF_AUTH_ANONYMOUS_ENABLED
            value: "false"
          # - name: GF_AUTH_ANONYMOUS_ORG_ROLE
          #   value: Admin
          # does not really work, because of template variables in exported dashboards:
          # - name: GF_DASHBOARDS_JSON_ENABLED
          #   value: "true"
        readinessProbe:
          httpGet:
            path: /login
            port: 3000
          # initialDelaySeconds: 30
          # timeoutSeconds: 1
        volumeMounts:
        - name: grafana-persistent-storage
          mountPath: /var
      volumes:
      - name: grafana-persistent-storage
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: kube-system
  labels:
    app: grafana
    component: core
spec:
  type: NodePort
  ports:
    - port: 3000
      nodePort: 9004
  selector:
    app: grafana

4、部署测试服务nginx

[root@k8s-node1 hpa]# cat myapp.yaml 
apiVersion: v1
kind: Service
metadata:
  name: svc-hpa
  namespace: default
spec:
  selector:
    app: myapp
  type: NodePort  ##注意这里是NodePort，下面压力测试要用到。
  ports:
  - name: http
    port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      name: myapp-demo
      namespace: default
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: ikubernetes/myapp:v1
        imagePullPolicy: IfNotPresent
        ports:
        - name: http
          containerPort: 80
        resources:
          requests:
            cpu: 50m
            memory: 50Mi
          limits:
            cpu: 50m
            memory: 50Mi

5、部署hpa资源

[root@k8s-node1 hpa]# cat hpa.yaml 
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa-v2
  namespace: default
spec:
  minReplicas: 1         ##至少1个副本
  maxReplicas: 8         ##最多8个副本
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 50  ##注意此时是根据使用率，也可以根据使用量：targetAverageValue
  - type: Resource
    resource:
      name: memory
      targetAverageUtilization: 50  ##注意此时是根据使用率，也可以根据使用量：targetAverageValue

6、使用ab工具模拟压力测试：

ab -n 5000000 -c 100 http://192.168.29.176:36010/index.html

6.1、使用循环压力测试

kubectl run -i --tty load-generator --image=busybox /bin/sh
/ # while true; do wget -q -O- http://192.168.29.176:4000; done

6.1、等待一会后，查看hpa及pod数量情况

[root@k8s-node1 log]# kubectl get hpa
NAME           REFERENCE          TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
myapp-hpa-v2   Deployment/myapp   9%/50%, 78%/50%   1         8         2          15m

[root@k8s-node1 ~]# kubectl get pod
NAME                       READY     STATUS    RESTARTS   AGE
myapp-bd9cfd5fd-pj88q      1/1       Running   0          15m
myapp-bd9cfd5fd-zrqpt      1/1       Running   0          13s

7、排错
7.1、如果出现以下错误请按照此方法解决即可

[root@k8s-node1 log]# kubectl get hpa
NAME           REFERENCE          TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
myapp-hpa-v2   Deployment/myapp   /50%, /50%   1         8         2          15m

7.2、在kube-controller-manager启动文件中加入以下内容即可

--horizontal-pod-autoscaler-use-rest-clients=false

7.3、重启kube-controller-manager

systemctl daemon-reload
systemctl restart kube-controller-manager

7.4、再次查看hpa

[root@k8s-node1 log]# kubectl get hpa
NAME           REFERENCE          TARGETS          MINPODS   MAXPODS   REPLICAS   AGE
myapp-hpa-v2   Deployment/myapp   5%/50%, 0%/50%   1         8         1          10m