Kubernetes的有状态应用示例：使用StatefulSet部署Cassandra

环境

• RHEL 9.3
• Docker Community 24.0.7
• minikube v1.32.0

概述

本例展示了如何在Kubernetes上运行Apache Cassandra。Cassandra是一个数据库，需要永久性存储来提供数据持久性（应用状态）。在此示例中，自定义的Cassandra种子提供者使得数据库在新的Cassandra实例接入Cassandra集群时能够发现它们。

注意：Cassandra和Kubernetes都使用术语 node 来表示集群的成员。在本例中，StatefulSet的pod是Cassandra node，并且是Cassandra集群的成员（称为 ring ）。当这些pod在Kubernetes集群中运行时，Kubernetes control plane会把这些pod调度到Kubernetes的node上。

当 Cassandra node启动时，使用 seed 列表来引导发现ring中的其它node。本例部署了一个自定义的Cassandra种子提供者，使得数据库在新的Cassandra pod出现在Kubernetes集群中时，可以发现它们。

准备

Minikube默认需要2048MB内存和2个CPU。本例中使用默认资源配置运行Minikube会出现资源不足的错误。为避免这些错误，请使用以下设置启动Minikube：

$ minikube start --memory 5120 --cpus=4
  minikube v1.32.0 on Redhat 9.3
✨  Using the docker driver based on existing profile
❗  You cannot change the memory size for an existing minikube cluster. Please first delete the cluster.
❗  You cannot change the CPUs for an existing minikube cluster. Please first delete the cluster.
  Starting control plane node minikube in cluster minikube
  Pulling base image ...
  Restarting existing docker container for "minikube" ...
  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
  Configuring bridge CNI (Container Networking Interface) ...
  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
  Enabled addons: storage-provisioner, default-storageclass
  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

注意这两行：

❗  You cannot change the memory size for an existing minikube cluster. Please first delete the cluster.
❗  You cannot change the CPUs for an existing minikube cluster. Please first delete the cluster.

注：如果不管，按默认设置启动minikube，则在后面的步骤会报错，如：

$ kubectl get pods -l="app=cassandra"
Unable to connect to the server: http2: client connection lost

$ kubectl get nodes
Unable to connect to the server: net/http: TLS handshake timeout

要改变设置，貌似需要先删除cluster才行：

$ minikube delete
  Deleting "minikube" in docker ...
  Deleting container "minikube" ...
  Removing /home/ding/.minikube/machines/minikube ...
  Removed all traces of the "minikube" cluster.

然后再运行：

$ minikube start --memory 5120 --cpus=4
  minikube v1.32.0 on Redhat 9.3
✨  Automatically selected the docker driver
  Using Docker driver with root privileges
  Starting control plane node minikube in cluster minikube
  Pulling base image ...
❗  minikube was unable to download gcr.io/k8s-minikube/kicbase:v0.0.42, but successfully downloaded docker.io/kicbase/stable:v0.0.42 as a fallback image
  Creating docker container (CPUs=4, Memory=5120MB) ...
  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
  Verifying Kubernetes components...
  Enabled addons: storage-provisioner, default-storageclass
  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

为Cassandra创建headless service

以下 Service 用于在Cassandra pod和客户端之间进行DNS查找：

创建文件 cassandra-service.yaml 如下：

apiVersion: v1
kind: Service
metadata:
  labels:
    app: cassandra
  name: cassandra
spec:
  clusterIP: None
  ports:
  - port: 9042
  selector:
    app: cassandra

kubectl apply -f cassandra-service.yaml

$ kubectl get svc cassandra
NAME        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
cassandra   ClusterIP   None                 9042/TCP   24s

创建文件 cassandra-statefulset.yaml 如下：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra
  labels:
    app: cassandra
spec:
  serviceName: cassandra
  replicas: 3
  selector:
    matchLabels:
      app: cassandra
  template:
    metadata:
      labels:
        app: cassandra
    spec:
      terminationGracePeriodSeconds: 1800
      containers:
      - name: cassandra
        # image: gcr.io/google-samples/cassandra:v13
        image: docker.io/kaiding1/cassandra:v13
        imagePullPolicy: Always
        ports:
        - containerPort: 7000
          name: intra-node
        - containerPort: 7001
          name: tls-intra-node
        - containerPort: 7199
          name: jmx
        - containerPort: 9042
          name: cql
        resources:
          limits:
            cpu: "500m"
            memory: 1Gi
          requests:
            cpu: "500m"
            memory: 1Gi
        securityContext:
          capabilities:
            add:
              - IPC_LOCK
        lifecycle:
          preStop:
            exec:
              command: 
              - /bin/sh
              - -c
              - nodetool drain
        env:
          - name: MAX_HEAP_SIZE
            value: 512M
          - name: HEAP_NEWSIZE
            value: 100M
          - name: CASSANDRA_SEEDS
            value: "cassandra-0.cassandra.default.svc.cluster.local"
          - name: CASSANDRA_CLUSTER_NAME
            value: "K8Demo"
          - name: CASSANDRA_DC
            value: "DC1-K8Demo"
          - name: CASSANDRA_RACK
            value: "Rack1-K8Demo"
          - name: POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - /ready-probe.sh
          initialDelaySeconds: 15
          timeoutSeconds: 5
        # These volume mounts are persistent. They are like inline claims,
        # but not exactly because the names need to match exactly one of
        # the stateful pod volumes.
        volumeMounts:
        - name: cassandra-data
          mountPath: /cassandra_data
  # These are converted to volume claims by the controller
  # and mounted at the paths mentioned above.
  # do not use these in production until ssd GCEPersistentDisk or other ssd pd
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: fast
      resources:
        requests:
          storage: 1Gi
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: fast
provisioner: k8s.io/minikube-hostpath
parameters:
  type: pd-ssd

注：因为访问不了 gcr.io ，所以事先把image pull下来，并push到了可访问的位置。

$ kubectl apply -f cassandra-statefulset.yaml
statefulset.apps/cassandra created
storageclass.storage.k8s.io/fast created

验证Cassandra StatefulSet

注：启动pod特别慢，本例中花了半个多小时。

$ kubectl get statefulset cassandra
NAME        READY   AGE
cassandra   3/3     53m

$ kubectl get pods -l="app=cassandra"
NAME          READY   STATUS    RESTARTS   AGE
cassandra-0   1/1     Running   0          53m
cassandra-1   1/1     Running   0          47m
cassandra-2   1/1     Running   0          37m

$ kubectl exec -it cassandra-0 -- nodetool status
Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.244.0.4  142.7 KiB  32           72.8%             913518a2-c96e-4915-9b95-dc26c8c4e9a7  Rack1-K8Demo
UN  10.244.0.5  143.1 KiB  32           70.6%             e5689d71-a4b9-4a20-af0b-147ea2ec4c57  Rack1-K8Demo
UN  10.244.0.3  135.94 KiB  32           56.6%             af529f81-7da7-4129-b379-d41554390c6f  Rack1-K8Demo

修改Cassandra StatefulSet

kubectl edit statefulset cassandra

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: apps/v1
kind: StatefulSet
metadata:
  creationTimestamp: 2016-08-13T18:40:58Z
  generation: 1
  labels:
  app: cassandra
  name: cassandra
  namespace: default
  resourceVersion: "323"
  uid: 7a219483-6185-11e6-a910-42010a8a0fc0
spec:
  replicas: 3

把replica数量从3改为4，然后保存退出。

$ kubectl edit statefulset cassandra
statefulset.apps/cassandra edited

大约5分钟后，查看状态：

$ kubectl get pods -l="app=cassandra"
NAME          READY   STATUS    RESTARTS   AGE
cassandra-0   1/1     Running   0          101m
cassandra-1   1/1     Running   0          96m
cassandra-2   1/1     Running   0          85m
cassandra-3   1/1     Running   0          5m

$ kubectl get statefulset cassandra
NAME        READY   AGE
cassandra   4/4     101m

清理

删除或缩小StatefulSet不会删除与之关联的volume。这是出于安全考虑，因为与自动清除所有相关的StatefulSet资源相比，数据更有价值。

注意：根据Storage Class和回收策略，删除PVC可能导致相关联的volume也被删除。不要想当然认为PVC被删除后，还能访问数据。

$ grace=$(kubectl get pod cassandra-0 -o=jsonpath='{.spec.terminationGracePeriodSeconds}') \
  && kubectl delete statefulset -l app=cassandra \
  && echo "Sleeping ${grace} seconds" 1>&2 \
  && sleep $grace \
  && kubectl delete persistentvolumeclaim -l app=cassandra
statefulset.apps "cassandra" deleted
Sleeping 1800 seconds
persistentvolumeclaim "cassandra-data-cassandra-0" deleted
persistentvolumeclaim "cassandra-data-cassandra-1" deleted
persistentvolumeclaim "cassandra-data-cassandra-2" deleted
persistentvolumeclaim "cassandra-data-cassandra-3" deleted

$ kubectl delete service -l app=cassandra
service "cassandra" deleted

Cassandra容器环境变量

本例中的pod 使用来自Google容器registry的 gcr.io/google-samples/cassandra:v13 image。该Docker image基于debian-base，且包含 OpenJDK 8。

注：因为访问不了 gcr.io ，所以事先把image pull下来，并push到了可访问的位置。

该镜像包括Apache Debian repo的标准Cassandra安装。通过环境变量，你可以更改插入到 cassandra.yaml 中的值。

环境变量	缺省值
CASSANDRA_CLUSTER_NAME	'Test Cluster'
CASSANDRA_NUM_TOKENS	32
CASSANDRA_RPC_ADDRESS	0.0.0.0

参考

• https://kubernetes.io/docs/tutorials/stateful-application/cassandra