kubernetes 26、liveness和readiness的原理与使用

目标:
弄清楚readiness和liveness探针的原理

目录:
1 readiness和liveness探针
2 样例
3 总结

1 readiness和liveness探针
1.1 原理
liveness主要用来确定何时重启容器。liveness探测的结果会存储在livenessManager中。
kubelet在syncPod时，发现该容器的liveness探针检测失败时，会将其加入待启动的容器列表中，在之后的操作中会重新创建该容器。

readiness主要来确定容器是否已经就绪。只有当Pod中的容器都处于就绪状态，也就是pod的condition里的Ready为true时，kubelet才会认定该Pod处于就绪状态。而pod是否处于就绪状态的作用是控制哪些Pod应该作为service的后端。如果Pod处于非就绪状态，那么它们将会被从service的endpoint中移除。

1.2 用法
1)LivenessProbe: 判断容器是否健康，不健康，则删除容器
2)实现方式:
A)ExecAction: 在容器内部执行命令，退出码为0表示健康
B)TcpSocketAction:通过容器的IP地址和端口号执行TCP检查，若端口能被访问，则健康
C)HTTPGetAction:通过容器IP地址，端口号，路径调用HTTP Get方法，状态码>=200,<=400则健康
          readinessProbe:
            tcpSocket:
              port: {{ .Values.network.api.port }}
          livenessProbe:
            httpGet:
              port: {{ .Values.network.api.port }}

3)ReadinessProbe: 判断容器是否启动完成且准备接收请求

2 样例
2.1 deploy的yaml文件
{{- if .Values.manifests.deployment_api }}
{{- $envAll := . }}
{{- $dependencies := .Values.dependencies.api }}
{{- $mounts_ceilometer_api := .Values.pod.mounts.ceilometer_api.ceilometer_api }}
{{- $mounts_ceilometer_api_init := .Values.pod.mounts.ceilometer_api.init_container }}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ceilometer-api
spec:
  replicas: {{ .Values.pod.replicas.api }}
{{ tuple $envAll | include "helm-toolkit.snippets.kubernetes_upgrades_deployment" | indent 2 }}
  selector:
    matchLabels:
{{ tuple $envAll "ceilometer" "api" | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 6 }}
  template:
    metadata:
      labels:
{{ tuple $envAll "ceilometer" "api" | include "helm-toolkit.snippets.kubernetes_metadata_labels" | indent 8 }}
      annotations:
        configmap-bin-hash: {{ tuple "configmap-bin.yaml" . | include "helm-toolkit.utils.hash" }}
        configmap-etc-hash: {{ tuple "configmap-etc.yaml" . | include "helm-toolkit.utils.hash" }}
    spec:
      affinity:
{{ tuple $envAll "ceilometer" "api" | include "helm-toolkit.snippets.kubernetes_pod_anti_affinity" | indent 8 }}
      nodeSelector:
        {{ .Values.labels.api.node_selector_key }}: {{ .Values.labels.api.node_selector_value }}
      terminationGracePeriodSeconds: {{ .Values.pod.lifecycle.termination_grace_period.api.timeout | default "30" }}
      initContainers:
{{ tuple $envAll $dependencies $mounts_ceilometer_api_init | include "helm-toolkit.snippets.kubernetes_entrypoint_init_container" | indent 8 }}
      containers:
        - name: ceilometer-api
          image: {{ tuple .Values.images.tags "ceilometer_api" . | include "helm-toolkit.utils.update_image" }}
          imagePullPolicy: {{ .Values.images.pull_policy }}
{{ tuple $envAll $envAll.Values.pod.resources.api | include "helm-toolkit.snippets.kubernetes_resources" | indent 10 }}
          command:
            - /tmp/ceilometer-api.sh
            - start
          lifecycle:
            preStop:
              exec:
                command:
                  - /tmp/ceilometer-api.sh
                  - stop
          ports:
            - name: ce-api
              containerPort: {{ .Values.network.api.port }}
          readinessProbe:
            tcpSocket:
              port: {{ .Values.network.api.port }}
          livenessProbe:
            tcpSocket:
              port: {{ .Values.network.api.port }}
            initialDelaySeconds: 60
            periodSeconds: 60
          volumeMounts:
            - name: pod-etc-ceilometer
              mountPath: /etc/ceilometer
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/ceilometer.conf
              subPath: ceilometer.conf
              readOnly: true
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/api_paste.ini
              subPath: api_paste.ini
              readOnly: true
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/policy.json
              subPath: policy.json
              readOnly: true
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/event_definitions.yaml
              subPath: event_definitions.yaml
              readOnly: true
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/event_pipeline.yaml
              subPath: event_pipeline.yaml
              readOnly: true
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/pipeline.yaml
              subPath: pipeline.yaml
              readOnly: true
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/gnocchi_resources.yaml
              subPath: gnocchi_resources.yaml
              readOnly: true
            - name: ceilometer-etc
              mountPath: /etc/httpd/conf.d/wsgi-ceilometer.conf
              subPath: wsgi-ceilometer.conf
              readOnly: true
            - name: ceilometer-etc
              mountPath: /etc/ceilometer/mapping.json
              subPath: mapping.json
              readOnly: true
            - name: ceilometer-bin
              mountPath: /tmp/ceilometer-api.sh
              subPath: ceilometer-api.sh
              readOnly: true
{{ if $mounts_ceilometer_api.volumeMounts }}{{ toYaml $mounts_ceilometer_api.volumeMounts | indent 12 }}{{ end }}
      volumes:
        - name: pod-etc-ceilometer
          emptyDir: {}
        - name: ceilometer-etc
          configMap:
            name: ceilometer-etc
            defaultMode: 0444
        - name: ceilometer-bin
          configMap:
            name: ceilometer-bin
            defaultMode: 0555
{{ if $mounts_ceilometer_api.volumes }}{{ toYaml $mounts_ceilometer_api.volumes | indent 8 }}{{ end }}
{{- end }}

分析:
1)关于liveness和readiness内容如下:
          readinessProbe:
            tcpSocket:
              port: {{ .Values.network.api.port }}
          livenessProbe:
            tcpSocket:
              port: {{ .Values.network.api.port }}
            initialDelaySeconds: 60
            periodSeconds: 60

注意:
之所以liveness不用httpGet，类似如下内容:
            httpGet:
              port: {{ .Values.network.api.port }}
是因为ceilometer-api发送http get请求是需要携带keystone的token的，而这里ceilometer-api启动后，并不具备获取keystone token
的能力，因为如果使用httpGet,会导致这个请求失败，所以改用tcpSocket。
所以这里选择httpGet，还是tcpSocket需要根据实际项目情况来做对应处理。

2.2 deploy部署后的结果
kubectl get deploy -n openstack ceilometer-api -oyaml > ceilometer-api.yaml
cat ceilometer-api.yaml 
样例输出如下:
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
  creationTimestamp: "2020-06-05T07:42:41Z"
  generation: 2
  labels:
    application: ceilometer
    component: api
    release_group: ceilometer
  name: ceilometer-api
  namespace: openstack
  resourceVersion: "1024546"
  selfLink: /apis/apps/v1/namespaces/openstack/deployments/ceilometer-api
  uid: 22e39b29-a700-11ea-b7a9-b88303541ba2
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      application: ceilometer
      component: api
      release_group: ceilometer
  strategy:
    rollingUpdate:
      maxSurge: 3
      maxUnavailable: 70%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        configmap-bin-hash: 455ad0b60a283eec604bd8277d4ec1340f054960004ba61157b0c9274ae64ee3
        configmap-etc-hash: b3ee1495491b9d52a460742b47da39a13e8326cbf3f84f5939a93e2e95942f50
      creationTimestamp: null
      labels:
        application: ceilometer
        component: api
        release_group: ceilometer
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: release_group
                operator: In
                values:
                - ceilometer
              - key: application
                operator: In
                values:
                - ceilometer
              - key: component
                operator: In
                values:
                - api
            topologyKey: kubernetes.io/hostname
      containers:
      - command:
        - /tmp/ceilometer-api.sh
        - start
        image: hub.easystack.io/production/escloud-linux-source-ceilometer-api:5.1.0-alpha.190
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            exec:
              command:
              - /tmp/ceilometer-api.sh
              - stop
        livenessProbe:
          failureThreshold: 3
          initialDelaySeconds: 60
          periodSeconds: 60
          successThreshold: 1
          tcpSocket:
            port: 8777
          timeoutSeconds: 1
        name: ceilometer-api
        ports:
        - containerPort: 8777
          name: ce-api
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: 8777
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "2"
            memory: 2Gi
          requests:
            cpu: 100m
            memory: 124Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/ceilometer
          name: pod-etc-ceilometer
        - mountPath: /etc/ceilometer/ceilometer.conf
          name: ceilometer-etc
          readOnly: true
          subPath: ceilometer.conf
        - mountPath: /etc/ceilometer/api_paste.ini
          name: ceilometer-etc
          readOnly: true
          subPath: api_paste.ini
        - mountPath: /etc/ceilometer/policy.json
          name: ceilometer-etc
          readOnly: true
          subPath: policy.json
        - mountPath: /etc/ceilometer/event_definitions.yaml
          name: ceilometer-etc
          readOnly: true
          subPath: event_definitions.yaml
        - mountPath: /etc/ceilometer/event_pipeline.yaml
          name: ceilometer-etc
          readOnly: true
          subPath: event_pipeline.yaml
        - mountPath: /etc/ceilometer/pipeline.yaml
          name: ceilometer-etc
          readOnly: true
          subPath: pipeline.yaml
        - mountPath: /etc/ceilometer/gnocchi_resources.yaml
          name: ceilometer-etc
          readOnly: true
          subPath: gnocchi_resources.yaml
        - mountPath: /etc/httpd/conf.d/wsgi-ceilometer.conf
          name: ceilometer-etc
          readOnly: true
          subPath: wsgi-ceilometer.conf
        - mountPath: /etc/ceilometer/mapping.json
          name: ceilometer-etc
          readOnly: true
          subPath: mapping.json
        - mountPath: /tmp/ceilometer-api.sh
          name: ceilometer-bin
          readOnly: true
          subPath: ceilometer-api.sh
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - kubernetes-entrypoint
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: INTERFACE_NAME
          value: eth0
        - name: DEPENDENCY_SERVICE
          value: openstack:keystone-api,openstack:mariadb,openstack:mongodb,openstack:gnocchi-api
        - name: DEPENDENCY_JOBS
          value: ceilometer-db-init-mongodb,ceilometer-db-sync,ceilometer-ks-user,ceilometer-ks-endpoints
        - name: DEPENDENCY_DAEMONSET
        - name: DEPENDENCY_CONTAINER
        - name: COMMAND
          value: echo done
        image: hub.easystack.io/production/kubernetes-entrypoint:v0.2.1
        imagePullPolicy: IfNotPresent
        name: init
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      nodeSelector:
        openstack-control-plane: enabled
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 600
      volumes:
      - emptyDir: {}
        name: pod-etc-ceilometer
      - configMap:
          defaultMode: 292
          name: ceilometer-etc
        name: ceilometer-etc
      - configMap:
          defaultMode: 365
          name: ceilometer-bin
        name: ceilometer-bin
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2020-06-05T08:23:13Z"
    lastUpdateTime: "2020-06-10T08:06:21Z"
    message: ReplicaSet "ceilometer-api-696cb4c478" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2020-06-10T12:19:07Z"
    lastUpdateTime: "2020-06-10T12:19:07Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  observedGeneration: 2
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

 
3 总结
1)LivenessProbe: 判断容器是否健康，不健康，会将其加入待启动的容器列表中，在之后的操作中会重新创建该容器。，用于长时间运行的api服务挂了后的一种workround。
ReadinessProbe: 判断容器是否启动完成且准备接收请求，pod是否处于就绪状态的作用是控制哪些Pod应该作为service的后端。如果Pod处于非就绪状态，那么它们将会被从service的endpoint中移除。
实现方式:
A)ExecAction: 在容器内部执行命令，退出码为0表示健康
B)TcpSocketAction:通过容器的IP地址和端口号执行TCP检查，若端口能被访问，则健康
C)HTTPGetAction:通过容器IP地址，端口号，路径调用HTTP Get方法，状态码>=200,<=400则健康
          readinessProbe:
            tcpSocket:
              port: {{ .Values.network.api.port }}
          livenessProbe:
            httpGet:
              port: {{ .Values.network.api.port }}

2)建议给api的pod都加上liveness和readiness，作为long run环境下api服务异常的一种解决方法。

参考:
https://www.cnblogs.com/xuxinkun/p/11785521.html
kubernetes权威指南
https://blog.csdn.net/qingyuanluofeng/article/details/103077862