Kubernetes的有状态应用示例：使用StatefulSet部署Cassandra
文章目录
- 环境
- 概述
- 准备
- 为Cassandra创建headless service
- 验证Cassandra StatefulSet
- 修改Cassandra StatefulSet
- 清理
- Cassandra容器环境变量
- 参考
环境
- RHEL 9.3
- Docker Community 24.0.7
- minikube v1.32.0
概述
本例展示了如何在Kubernetes上运行Apache Cassandra。Cassandra是一个数据库，需要永久性存储来提供数据持久性（应用状态）。在此示例中，自定义的Cassandra种子提供者使得数据库在新的Cassandra实例接入Cassandra集群时能够发现它们。
注意：Cassandra和Kubernetes都使用术语 node 来表示集群的成员。在本例中，StatefulSet的pod是Cassandra node，并且是Cassandra集群的成员（称为 ring ）。当这些pod在Kubernetes集群中运行时，Kubernetes control plane会把这些pod调度到Kubernetes的node上。
当 Cassandra node启动时，使用 seed 列表来引导发现ring中的其它node。本例部署了一个自定义的Cassandra种子提供者，使得数据库在新的Cassandra pod出现在Kubernetes集群中时，可以发现它们。
准备
Minikube默认需要2048MB内存和2个CPU。本例中使用默认资源配置运行Minikube会出现资源不足的错误。为避免这些错误，请使用以下设置启动Minikube：
$ minikube start --memory 5120 --cpus=4
minikube v1.32.0 on Redhat 9.3
✨ Using the docker driver based on existing profile
❗ You cannot change the memory size for an existing minikube cluster. Please first delete the cluster.
❗ You cannot change the CPUs for an existing minikube cluster. Please first delete the cluster.
Starting control plane node minikube in cluster minikube
Pulling base image ...
Restarting existing docker container for "minikube" ...
Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
Configuring bridge CNI (Container Networking Interface) ...
Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
Enabled addons: storage-provisioner, default-storageclass
Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
注意这两行：
❗ You cannot change the memory size for an existing minikube cluster. Please first delete the cluster.
❗ You cannot change the CPUs for an existing minikube cluster. Please first delete the cluster.
注：如果不管，按默认设置启动minikube，则在后面的步骤会报错，如：
$ kubectl get pods -l="app=cassandra"
Unable to connect to the server: http2: client connection lost
$ kubectl get nodes
Unable to connect to the server: net/http: TLS handshake timeout
要改变设置，貌似需要先删除cluster才行：
$ minikube delete
Deleting "minikube" in docker ...
Deleting container "minikube" ...
Removing /home/ding/.minikube/machines/minikube ...
Removed all traces of the "minikube" cluster.
然后再运行：
$ minikube start --memory 5120 --cpus=4
minikube v1.32.0 on Redhat 9.3
✨ Automatically selected the docker driver
Using Docker driver with root privileges
Starting control plane node minikube in cluster minikube
Pulling base image ...
❗ minikube was unable to download gcr.io/k8s-minikube/kicbase:v0.0.42, but successfully downloaded docker.io/kicbase/stable:v0.0.42 as a fallback image
Creating docker container (CPUs=4, Memory=5120MB) ...
Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
Configuring bridge CNI (Container Networking Interface) ...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
Verifying Kubernetes components...
Enabled addons: storage-provisioner, default-storageclass
Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
为Cassandra创建headless service
以下 Service 用于在Cassandra pod和客户端之间进行DNS查找：
创建文件 cassandra-service.yaml 如下：
apiVersion: v1
kind: Service
metadata:
labels:
app: cassandra
name: cassandra
spec:
clusterIP: None
ports:
- port: 9042
selector:
app: cassandra
kubectl apply -f cassandra-service.yaml
$ kubectl get svc cassandra
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cassandra ClusterIP None 9042/TCP 24s
创建文件 cassandra-statefulset.yaml 如下：
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: cassandra
labels:
app: cassandra
spec:
serviceName: cassandra
replicas: 3
selector:
matchLabels:
app: cassandra
template:
metadata:
labels:
app: cassandra
spec:
terminationGracePeriodSeconds: 1800
containers:
- name: cassandra
# image: gcr.io/google-samples/cassandra:v13
image: docker.io/kaiding1/cassandra:v13
imagePullPolicy: Always
ports:
- containerPort: 7000
name: intra-node
- containerPort: 7001
name: tls-intra-node
- containerPort: 7199
name: jmx
- containerPort: 9042
name: cql
resources:
limits:
cpu: "500m"
memory: 1Gi
requests:
cpu: "500m"
memory: 1Gi
securityContext:
capabilities:
add:
- IPC_LOCK
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- nodetool drain
env:
- name: MAX_HEAP_SIZE
value: 512M
- name: HEAP_NEWSIZE
value: 100M
- name: CASSANDRA_SEEDS
value: "cassandra-0.cassandra.default.svc.cluster.local"
- name: CASSANDRA_CLUSTER_NAME
value: "K8Demo"
- name: CASSANDRA_DC
value: "DC1-K8Demo"
- name: CASSANDRA_RACK
value: "Rack1-K8Demo"
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
readinessProbe:
exec:
command:
- /bin/bash
- -c
- /ready-probe.sh
initialDelaySeconds: 15
timeoutSeconds: 5
# These volume mounts are persistent. They are like inline claims,
# but not exactly because the names need to match exactly one of
# the stateful pod volumes.
volumeMounts:
- name: cassandra-data
mountPath: /cassandra_data
# These are converted to volume claims by the controller
# and mounted at the paths mentioned above.
# do not use these in production until ssd GCEPersistentDisk or other ssd pd
volumeClaimTemplates:
- metadata:
name: cassandra-data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: fast
resources:
requests:
storage: 1Gi
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: fast
provisioner: k8s.io/minikube-hostpath
parameters:
type: pd-ssd
注：因为访问不了 gcr.io ，所以事先把image pull下来，并push到了可访问的位置。
$ kubectl apply -f cassandra-statefulset.yaml
statefulset.apps/cassandra created
storageclass.storage.k8s.io/fast created
验证Cassandra StatefulSet
注：启动pod特别慢，本例中花了半个多小时。
$ kubectl get statefulset cassandra
NAME READY AGE
cassandra 3/3 53m
$ kubectl get pods -l="app=cassandra"
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 53m
cassandra-1 1/1 Running 0 47m
cassandra-2 1/1 Running 0 37m
$ kubectl exec -it cassandra-0 -- nodetool status
Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 10.244.0.4 142.7 KiB 32 72.8% 913518a2-c96e-4915-9b95-dc26c8c4e9a7 Rack1-K8Demo
UN 10.244.0.5 143.1 KiB 32 70.6% e5689d71-a4b9-4a20-af0b-147ea2ec4c57 Rack1-K8Demo
UN 10.244.0.3 135.94 KiB 32 56.6% af529f81-7da7-4129-b379-d41554390c6f Rack1-K8Demo
修改Cassandra StatefulSet
kubectl edit statefulset cassandra
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: apps/v1
kind: StatefulSet
metadata:
creationTimestamp: 2016-08-13T18:40:58Z
generation: 1
labels:
app: cassandra
name: cassandra
namespace: default
resourceVersion: "323"
uid: 7a219483-6185-11e6-a910-42010a8a0fc0
spec:
replicas: 3
把replica数量从3改为4，然后保存退出。
$ kubectl edit statefulset cassandra
statefulset.apps/cassandra edited
大约5分钟后，查看状态：
$ kubectl get pods -l="app=cassandra"
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 101m
cassandra-1 1/1 Running 0 96m
cassandra-2 1/1 Running 0 85m
cassandra-3 1/1 Running 0 5m
$ kubectl get statefulset cassandra
NAME READY AGE
cassandra 4/4 101m
清理
删除或缩小StatefulSet不会删除与之关联的volume。这是出于安全考虑，因为与自动清除所有相关的StatefulSet资源相比，数据更有价值。
注意：根据Storage Class和回收策略，删除PVC可能导致相关联的volume也被删除。不要想当然认为PVC被删除后，还能访问数据。
$ grace=$(kubectl get pod cassandra-0 -o=jsonpath='{.spec.terminationGracePeriodSeconds}') \
&& kubectl delete statefulset -l app=cassandra \
&& echo "Sleeping ${grace} seconds" 1>&2 \
&& sleep $grace \
&& kubectl delete persistentvolumeclaim -l app=cassandra
statefulset.apps "cassandra" deleted
Sleeping 1800 seconds
persistentvolumeclaim "cassandra-data-cassandra-0" deleted
persistentvolumeclaim "cassandra-data-cassandra-1" deleted
persistentvolumeclaim "cassandra-data-cassandra-2" deleted
persistentvolumeclaim "cassandra-data-cassandra-3" deleted
$ kubectl delete service -l app=cassandra
service "cassandra" deleted
Cassandra容器环境变量
本例中的pod 使用来自Google容器registry的 gcr.io/google-samples/cassandra:v13 image。该Docker image基于debian-base，且包含 OpenJDK 8。
注：因为访问不了 gcr.io ，所以事先把image pull下来，并push到了可访问的位置。
该镜像包括Apache Debian repo的标准Cassandra安装。通过环境变量，你可以更改插入到 cassandra.yaml 中的值。
| 环境变量 | 缺省值 |
|---|---|
CASSANDRA_CLUSTER_NAME |
'Test Cluster' |
CASSANDRA_NUM_TOKENS |
32 |
CASSANDRA_RPC_ADDRESS |
0.0.0.0 |
参考
https://kubernetes.io/docs/tutorials/stateful-application/cassandra