StatefulSet 简单实践 Kubernetes
概述
在Kubernetes集群中部署MySQL和Mongodb的StatefulSet服务。
MySQL有官方文档的指引 + 其他网站博客的指引实现
Mongodb修改operator的Deployment进行简单的实现
MySQL-StatefulSet
参考官方文档：运行一个有状态的应用程序 | Kubernetes
深入剖析Kubernetes学习笔记：StatefulSet-MySQL集群（20） - 活的潇洒80 - 博客园 (cnblogs.com)
namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
labels:
kubernetes.io/metadata.name: mysql
name: mysql
spec:
finalizers:
- kubernetes
status:
phase: Active
mysql-class.yaml
创建动态存储卷：storeageclass。
存在部署依赖，需要先安装服务：nfs-subdir-external-provisioner
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-mysql
namespace: mysql
annotations:
storageclass.kubernetes.io/is-default-class: "false" ## 是否设置为默认的storageclass
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
#provisioner: nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
archiveOnDelete: "true"
pathPattern: "${.PVC.namespace}/${.PVC.name}/${.PVC.annotations.nfs.io/storage-path}"
# pathPattern: "${.PVC.namespace}/${.PVC.annotations.nfs.io/storage-path}"
# pathPattern: "${.PVC.namespace}-${.PVC.name}"
mysql-pvc.yaml
参考官方文档：存储类 | Kubernetes
存在部署依赖，需要先安装服务：nfs-subdir-external-provisioner
自动创建的，下面data-mysql-0为例子：
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-class: nfs-mysql
volume.beta.kubernetes.io/storage-provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
finalizers:
- kubernetes.io/pvc-protection
name: data-mysql-0
namespace: mysql
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Mi
storageClassName: nfs-mysql
volumeMode: Filesystem
configmap.yaml
参考文档：运行一个有状态的应用程序 | Kubernetes
参考文档：使用StatefulSet搭建MySQL8集群 - 简书 (jianshu.com)
目前使用的是5.7.43的mysql
apiVersion: v1
kind: ConfigMap
metadata:
name: mysql
namespace: mysql
labels:
app: mysql
data:
master.cnf: |
# Apply this config only on the master.
[client]
default-character-set=utf8mb4
[mysql]
default-character-set=utf8mb4
[mysqld]
# 打开binlog日志
log-bin
# binlog_expire_logs_seconds=2592000
max_connections=10000
# 在容器里面需要设置下时区
default-time-zone='+8:00'
character-set-client-handshake=FALSE
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
init_connect='SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci'
slave.cnf: |
# Apply this config only on slaves.
[client]
default-character-set=utf8mb4
[mysql]
default-character-set=utf8mb4
[mysqld]
# 机器设置只读
super-read-only
max_connections=10000
default-time-zone='+8:00'
character-set-client-handshake=FALSE
character-set-server=utf8mb4
collation-server=utf8mb4_unicode_ci
init_connect='SET NAMES utf8mb4 COLLATE utf8mb4_unicode_ci'
service.yaml
根据官方文档部署svc，无头服务 + ClusterIP
# 为 StatefulSet 成员提供稳定的 DNS 表项的无头服务（Headless Service）
apiVersion: v1
kind: Service
metadata:
name: mysql
namespace: mysql
labels:
app: mysql
app.kubernetes.io/name: mysql
spec:
ports:
- name: mysql
port: 3306
clusterIP: None
selector:
app: mysql
---
# 用于连接到任一 MySQL 实例执行读操作的客户端服务
# 对于写操作，你必须连接到主服务器：mysql-0.mysql
apiVersion: v1
kind: Service
metadata:
name: mysql-read
namespace: mysql
labels:
app: mysql
app.kubernetes.io/name: mysql
readonly: "true"
spec:
ports:
- name: mysql
port: 3306
selector:
app: mysql
secret.yaml
官方文档的 MYSQL_ALLOW_EMPTY_PASSWORD 是通过明文的方式传递，需要修复为secret的模式登录，参考文档：Secret | Kubernetes
创建命令：
kubectl create secret generic mysql-secret -n mysql --dry-run=client --from-literal=MYSQL_ALLOW_EMPTY_PASSWORD=1 -o yaml结果输出为：
apiVersion: v1
data:
MYSQL_ALLOW_EMPTY_PASSWORD: MQ==
kind: Secret
metadata:
creationTimestamp: null
name: mysql-secret
namespace: mysql
使用 Secret 安全地分发凭据 | Kubernetes 配置statefulset
...
env:
- name: MYSQL_ALLOW_EMPTY_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_ALLOW_EMPTY_PASSWORD
...详细参考下面的statefulset.yaml
StatefulSet.yaml
修改后的配置文件
与官方的文档主要差异如下：
- 配置了镜像拉取策略：imagePullPolicy: Always
- 密码文件使用secret的环境变量方式传递而非明文：secretKeyRef
- 使用的是image为MySQL:5.7.43
- 缩小了副本数：replicas: 2
- 添加了节点反亲和的策略：nodeAffinity + operator: NotIn
- 更新了镜像地址： image
- 使用了storeageclass的方式记录data：storageClassName + 更新了访问模式ReadWriteMany
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql
namespace: mysql
spec:
selector:
matchLabels:
app: mysql
app.kubernetes.io/name: mysql
serviceName: mysql
replicas: 2
template:
metadata:
labels:
app: mysql
app.kubernetes.io/name: mysql
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: NotIn
values:
- master01
initContainers:
- name: init-mysql
image: mysql:5.7.43
command:
- bash
- "-c"
- |
set -ex
# 基于 Pod 序号生成 MySQL 服务器的 ID。
[[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
echo [mysqld] > /mnt/conf.d/server-id.cnf
# 添加偏移量以避免使用 server-id=0 这一保留值。
echo server-id=$((100 + $ordinal)) >> /mnt/conf.d/server-id.cnf
# 将合适的 conf.d 文件从 config-map 复制到 emptyDir。
if [[ $ordinal -eq 0 ]]; then
cp /mnt/config-map/master.cnf /mnt/conf.d/
else
cp /mnt/config-map/slave.cnf /mnt/conf.d/
fi
volumeMounts:
- name: conf
mountPath: /mnt/conf.d
- name: config-map
mountPath: /mnt/config-map
- name: clone-mysql
image: reporsitory:5000/xtrabackup:1.0
imagePullPolicy: Always
command:
- bash
- "-c"
- |
set -ex
# 如果已有数据，则跳过克隆。
[[ -d /var/lib/mysql/mysql ]] && exit 0
# 跳过主实例（序号索引 0）的克隆。
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
[[ $ordinal -eq 0 ]] && exit 0
# 从原来的对等节点克隆数据。
ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
# 准备备份。
xtrabackup --prepare --target-dir=/var/lib/mysql
volumeMounts:
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
containers:
- name: mysql
image: mysql:5.7.43
env:
- name: MYSQL_ALLOW_EMPTY_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-secret
key: MYSQL_ALLOW_EMPTY_PASSWORD
ports:
- name: mysql
containerPort: 3306
volumeMounts:
- name: data
mountPath: /var/lib/mysql
subPath: mysql
- name: conf
mountPath: /etc/mysql/conf.d
resources:
requests:
cpu: 500m
memory: 1Gi
livenessProbe:
exec:
command: ["mysqladmin", "ping"]
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
exec:
# 检查我们是否可以通过 TCP 执行查询（skip-networking 是关闭的）。
command: ["mysql", "-h", "127.0.0.1", "-e", "SELECT 1"]
initialDelaySeconds: 5
periodSeconds: 2
timeoutSeconds: 1
- name: xtrabackup
image: reporsitory:5000/xtrabackup:1.0
imagePullPolicy: Always
ports:
- name: xtrabackup
containerPort: 3307
command:
- bash
- "-c"
- |
set -ex
cd /var/lib/mysql
# 确定克隆数据的 binlog 位置（如果有的话）。
if [[ -f xtrabackup_slave_info && "x$( change_master_to.sql.in
# 在这里要忽略 xtrabackup_binlog_info （它是没用的）。
rm -f xtrabackup_slave_info xtrabackup_binlog_info
elif [[ -f xtrabackup_binlog_info ]]; then
# 我们直接从主实例进行克隆。解析 binlog 位置。
[[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
rm -f xtrabackup_binlog_info xtrabackup_slave_info
echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
MASTER_LOG_POS=${BASH_REMATCH[2]}" > change_master_to.sql.in
fi
# 检查我们是否需要通过启动复制来完成克隆。
if [[ -f change_master_to.sql.in ]]; then
echo "Waiting for mysqld to be ready (accepting connections)"
until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done
echo "Initializing replication from clone position"
mysql -h 127.0.0.1 \
-e "$( Error: pvc的绑定错误
使用默认的官方的yaml文件会发生报错，需要配置PV-Claim:
一直报错：
修改配置PV-Claim。yaml文件：
这里应该不用修改PVC，因为pvc是自动创建，应该是修改statefulset中的storageClassName: nfs-mysql 实现修复。
Error: pause:3.6 镜像无法下载
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox i mage "registry.aliyuncs.com/google_containers/pause:3.6": failed to pull image "registry.aliyuncs.com/google_containers/pause:3.6": failed to pull and unpack imag e "registry.aliyuncs.com/google_containers/pause:3.6": failed to resolve reference "registry.aliyuncs.com/google_containers/pause:3.6": failed to do request: Head "https://registry.aliyuncs.com/v2/google_containers/pause/manifests/3.6": dial tcp: lookup registry.aliyuncs.com on [fe80::10%ens34]:53: read udp [fe80::d21d:3ae :ea99:27c6%ens34]:51322->[fe80::10%ens34]:53: i/o timeout
因为被调度到master01节点，该节点没有pause:3.6镜像，所以启动失败，添加反亲和策略：
将 Pod 指派给节点 | Kubernetes
解决办法如下：
Error: xtrabackup:1.0镜像下载失败
更换以下镜像源：
使用StatefulSet搭建MySQL8集群 - 简书 (jianshu.com)
mzmuer/xtrabackup:1.0又或者docker search:
但仍然是失败的，核心应该是修改： /etc/containerd/config.toml 。
因为目前集群使用的是containerd，镜像仓库是docker的registry，需要修改对应的配置：
- 在仓库主机中将镜像ist0ne/xtrabackup下载下来
- 使用docker tag 更新标签
- 从新推送到自己的镜像仓库中
Error: init-container 启动失败
官方文档使用的是：primary.cnf + replica.cnf
其他参考文档使用的是：master.cnf + slave.cnf
我参考的是国内其他参考文档的内容，所以要进行修改
Error:The node was low on resource: ephemeral-storage
临时存储空间不足，下载不了镜像
当时master02 --> master03 --> node04 --> node05 顺序集群自动逐步尝试部署。
Error: ErrImagePull
添加镜像拉取策略：imagePullPolicy: Always
修改containerd的配置文件，一般为：/etc/containerd/config.toml
随后重启服务： systemctl restart containerd
因为部署的是docker的镜像仓库：相关的daemon.json如图：
修复后：
Error: unknown variable 'binlog_expire_logs_seconds=2592000'
非官方的参考文档用的是mysql:8的镜像，其中配置文件写binlog_expire_logs_seconds=2592000
目前，我自己使用的镜像是MySQL:5.7.43的镜像，这里可能存在配置参数不支持的问题
修改配置文件：kubectl edit configmap mysql -n mysql
Error: pvc 一直处于pending状态
相关nfs服务已经处于资源吃紧的边缘：
缩减资源的使用，statefulset的副本需要缩小，由原来的3个缩小为2个。
成功结果：
验证：
参考文档
深入剖析Kubernetes学习笔记：StatefulSet-MySQL集群（20） - 活的潇洒80 - 博客园 (cnblogs.com)
kubernetes【statefulset搭建MySQL集群】 - 掘金 (juejin.cn)
使用pv + nfs服务器的案例：k8s-使用statefulset部署mysql主从集群5.7 - gg888666 - 博客园 (cnblogs.com)
使用storageclass + PV的案例：K8S 部署 Statefulset mysql-腾讯云开发者社区-腾讯云 (tencent.com)
有状态的节点控制器 -- StatefulSet 及其网络状态 (qq.com)
有状态的节点控制器 StatefulSet 的存储状态 (qq.com)
实战 Kubernetes StatefulSet -- MySQL 主从集群搭建-腾讯云开发者社区-腾讯云 (tencent.com)
Kubernetes：使用StatefulSet搭建MySQL集群（一主多从） - Cooper_73 - 博客园 (cnblogs.com)
Kubernetes：使用StatefulSet搭建MySQL集群（一主多从） - Cooper_73 - 博客园 (cnblogs.com)
Mongodb-StatefulSet
文件来源
Install and Configure the Kubernetes Operator — MongoDB Kubernetes Operator upcoming
Install the MongoDB Enterprise Kubernetes Operator — MongoDB Kubernetes Operator upcoming
mongodb-enterprise-kubernetes/mongodb-enterprise.yaml at master · mongodb/mongodb-enterprise-kubernetes (github.com)
crd.yaml
mongodb-enterprise-kubernetes/crds.yaml at master · mongodb/mongodb-enterprise-kubernetes (github.com)
来源于官网文件，使用kubectl apply -f 直接部署
mongodb-enterprise.yaml
参考的视频：statefulset 学亮-哔哩哔哩_Bilibili
将官方网站的deployment文件进行了修改，主要是：
- 增加了创建namespace的yaml部分
- 修改了Deployment为StatefulSet
- 增加了无头服务Service
---
apiVersion: v1
kind: Namespace
metadata:
labels:
kubernetes.io/metadata.name: mongodb
name: mongodb
spec:
finalizers:
- kubernetes
status:
phase: Active
---
# Source: enterprise-operator/templates/operator-roles.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: mongodb-enterprise-operator
namespace: mongodb
---
# Source: enterprise-operator/templates/operator-roles.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mongodb-enterprise-operator-mongodb-webhook
rules:
- apiGroups:
- "admissionregistration.k8s.io"
resources:
- validatingwebhookconfigurations
verbs:
- get
- create
- update
- delete
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- create
- update
- delete
---
# Source: enterprise-operator/templates/operator-roles.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mongodb-enterprise-operator-mongodb-webhook-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: mongodb-enterprise-operator-mongodb-webhook
subjects:
- kind: ServiceAccount
name: mongodb-enterprise-operator
namespace: mongodb
---
# Source: enterprise-operator/templates/operator-roles.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mongodb-enterprise-operator
namespace: mongodb
rules:
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- create
- update
- delete
- apiGroups:
- ""
resources:
- secrets
- configmaps
verbs:
- get
- list
- create
- update
- delete
- watch
- apiGroups:
- apps
resources:
- statefulsets
verbs:
- create
- get
- list
- watch
- delete
- update
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch
- delete
- deletecollection
- apiGroups:
- mongodb.com
verbs:
- "*"
resources:
- mongodb
- mongodb/finalizers
- mongodbusers
- opsmanagers
- opsmanagers/finalizers
- mongodbmulticluster
- mongodbmulticluster/finalizers
- mongodb/status
- mongodbusers/status
- opsmanagers/status
- mongodbmulticluster/status
---
# Source: enterprise-operator/templates/operator-roles.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mongodb-enterprise-operator
namespace: mongodb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: mongodb-enterprise-operator
subjects:
- kind: ServiceAccount
name: mongodb-enterprise-operator
namespace: mongodb
# This ClusterRoleBinding is necessary in order to use validating
# webhooks—these will prevent you from applying a variety of invalid resource
# definitions. The validating webhooks are optional so this can be removed if
# necessary.
---
# Source: enterprise-operator/templates/database-roles.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: mongodb-enterprise-appdb
namespace: mongodb
---
# Source: enterprise-operator/templates/database-roles.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: mongodb-enterprise-database-pods
namespace: mongodb
---
# Source: enterprise-operator/templates/database-roles.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: mongodb-enterprise-ops-manager
namespace: mongodb
---
# Source: enterprise-operator/templates/database-roles.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mongodb-enterprise-appdb
namespace: mongodb
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- apiGroups:
- ""
resources:
- pods
verbs:
- patch
- delete
- get
---
# Source: enterprise-operator/templates/database-roles.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mongodb-enterprise-appdb
namespace: mongodb
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: mongodb-enterprise-appdb
subjects:
- kind: ServiceAccount
name: mongodb-enterprise-appdb
namespace: mongodb
---
# Source: enterprise-operator/templates/operator.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mongodb-enterprise-operator
namespace: mongodb
spec:
serviceName: mongodb
replicas: 3
selector:
matchLabels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: mongodb-enterprise-operator
app.kubernetes.io/instance: mongodb-enterprise-operator
template:
metadata:
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/name: mongodb-enterprise-operator
app.kubernetes.io/instance: mongodb-enterprise-operator
spec:
serviceAccountName: mongodb-enterprise-operator
securityContext:
runAsNonRoot: true
runAsUser: 2000
containers:
- name: mongodb-enterprise-operator
image: "quay.io/mongodb/mongodb-enterprise-operator-ubi:1.22.0"
imagePullPolicy: Always
args:
- -watch-resource=mongodb
- -watch-resource=opsmanagers
- -watch-resource=mongodbusers
command:
- /usr/local/bin/mongodb-enterprise-operator
resources:
limits:
cpu: 1100m
memory: 1Gi
requests:
cpu: 500m
memory: 200Mi
env:
- name: OPERATOR_ENV
value: prod
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: CLUSTER_CLIENT_TIMEOUT
value: "10"
- name: IMAGE_PULL_POLICY
value: Always
# Database
- name: MONGODB_ENTERPRISE_DATABASE_IMAGE
value: quay.io/mongodb/mongodb-enterprise-database-ubi
- name: INIT_DATABASE_IMAGE_REPOSITORY
value: quay.io/mongodb/mongodb-enterprise-init-database-ubi
- name: INIT_DATABASE_VERSION
value: 1.0.19
- name: DATABASE_VERSION
value: 2.0.2
# Ops Manager
- name: OPS_MANAGER_IMAGE_REPOSITORY
value: quay.io/mongodb/mongodb-enterprise-ops-manager-ubi
- name: INIT_OPS_MANAGER_IMAGE_REPOSITORY
value: quay.io/mongodb/mongodb-enterprise-init-ops-manager-ubi
- name: INIT_OPS_MANAGER_VERSION
value: 1.0.12
# AppDB
- name: INIT_APPDB_IMAGE_REPOSITORY
value: quay.io/mongodb/mongodb-enterprise-init-appdb-ubi
- name: INIT_APPDB_VERSION
value: 1.0.18
- name: OPS_MANAGER_IMAGE_PULL_POLICY
value: Always
- name: AGENT_IMAGE
value: "quay.io/mongodb/mongodb-agent-ubi:12.0.25.7724-1"
- name: MONGODB_IMAGE
value: mongodb-enterprise-server
- name: MONGODB_REPO_URL
value: quay.io/mongodb
- name: MDB_IMAGE_TYPE
value: ubi8
- name: PERFORM_FAILOVER
value: "true"
---
apiVersion: v1
kind: Service
metadata:
name: mongodb
namespace: mongodb
spec:
selector:
app.kubernetes.io/name: mongodb-enterprise-operator
type: ClusterIP
clusterIP: None
ports:
- port: 27017
targetPort: 27017
成功部署的结果：
