7-2 k8s 示例-使用 StatefulSet 部署 MySQL 一主多从
更新时间：2023年4月
参考：运行一个有状态的应用程序 | Kubernetes
参考：MariaDB & K8s: How to replicate MariaDB in K8s - MariaDB.org
文章目录
-
- 部署 MySQL 主从（初版）
-
- Namespace
- StorageClass
- Service
- Secret
- ConfigMap
- StatefulSet
-
- 镜像选择
- 声明文件
- 检查
- 部署 MySQL 主从（改进）
-
- 镜像制作
- Namespace
- StorageClass
- Service
- Secret
- ConfigMap
- StatefulSet
- 检查
- 部署 ProxySQL（代理）
-
- PVC
- Deployment
- Service
- 配置 ProxySQL
-
- 配置 MySQL
- 配置 ProxySQL
- 检查
-
- 基础检查
- 验证读写分离
部署 MySQL 主从（初版）
Namespace
$ vim ./ns-mysql-demo.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: mysql-demo
应用声明
$ kubectl apply -f ns-mysql-demo.yaml
StorageClass
$ vim sc-nfs.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: sc-nfs
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
# 回收策略此处使用 删除，实验环境方便测试
reclaimPolicy: Delete
mountOptions:
- soft
- nfsvers=4.2
- noatime # 访问文件时不更新文件 inode 中的时间戳，高并发环境可提高性能
parameters:
# 根据 PVC 的namespace 和 PVC 名称来生成路径
pathPattern: "${.PVC.namespace}/${.PVC.name}"
archiveOnDelete: "true"
应用声明
$ kubectl apply -f sc-nfs.yaml
Service
$ vim hs-mysql-replica.yaml
---
apiVersion: v1
kind: Service
metadata:
name: hs-mysql-replica
namespace: mysql-demo
labels:
app: mysql
spec:
ports:
- name: server-port
port: 3306
clusterIP: None
selector:
app.kubernetes.io/name: mysql
应用声明
kubectl apply -f hs-mysql-replica.yaml
Secret
MySQL 的 root 用户密码
# 必须加 -n ，否则会把换行符也用 base64 编码
$ echo -n "qwert123.." | base64
cXdlcnQxMjMuLg==
$ vim secret-mysql-root-auth.yaml
---
apiVersion: v1
kind: Secret
metadata:
name: secret-mysql-root-auth
namespace: mysql-demo
type: Opaque
data:
root-password: cXdlcnQxMjMuLg==
应用声明
$ kubectl apply -f secret-mysql-root-auth.yaml
ConfigMap
$ vim configmap-mysql.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
name: configmap-mysql
namespace: mysql-demo
data:
primary.cnf: |
[mysqld]
gtid-mode=ON
enforce-gtid-consistency
replica.cnf: |
[mysqld]
gtid-mode=ON
enforce-gtid-consistency
read-only=ON
primary.sql: |
CREATE USER 'repluser'@'%' IDENTIFIED WITH mysql_native_password BY 'replsecret123..';
GRANT REPLICATION CLIENT,REPLICATION SLAVE ON *.* TO 'repluser'@'%';
CREATE DATABASE primary_db;
replica.sql: |
CHANGE MASTER TO
MASTER_HOST='mysql-replica-0.hs-mysql-replica',
MASTER_USER='repluser',
MASTER_PASSWORD='replsecret123..',
MASTER_AUTO_POSITION=1,
MASTER_CONNECT_RETRY=10;
应用声明
$ kubectl apply -f configmap-mysql.yaml
StatefulSet
镜像选择
使用官方镜像：mysql Tags | Docker Hub
镜像版本：mysql:8.0.32
默认配置文件位置：/etc/my.cnf、/etc/mysql/conf.d/
默认数据存储位置：/var/lib/mysql
其他重要目录：
/docker-entrypoint-initdb.d：放置在该路径的 SQL 语句，将作为初始化语句执行
声明文件
$ vim statefulset-mysql-replica.yaml
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql-replica
namespace: mysql-demo
labels:
app.kubernetes.io/name: mysql
spec:
serviceName: hs-mysql-replica
replicas: 3
selector:
matchLabels:
app.kubernetes.io/name: mysql
template:
metadata:
labels:
app.kubernetes.io/name: mysql
spec:
initContainers:
- name: init-mysql
image: mysql:8.0.32
imagePullPolicy: Always
command:
- bash
- "-c"
- |
set -ex
echo 'Starting init-mysql';
# Check config map to directory that already exists
# (but must be used as a volume for main container)
ls /mnt/config-map
# 获取主机名中的 id，以判断是 primary 还是 replica
[[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
# 复制对应的配置文件到相应目录
if [[ $ordinal -eq 0 ]]; then
# 复制配置文件到配置路径
cp /mnt/config-map/primary.cnf /etc/mysql/conf.d/server-id.cnf
# 复制初始化 SQL 语句到对应路径
cp /mnt/config-map/primary.sql /docker-entrypoint-initdb.d
else
cp /mnt/config-map/replica.cnf /etc/mysql/conf.d/server-id.cnf
cp /mnt/config-map/replica.sql /docker-entrypoint-initdb.d
fi
# 给 server-id 添加一个偏移量，防止 server-id=0
echo server-id=$((3000 + $ordinal)) >> /etc/mysql/conf.d/server-id.cnf
ls /etc/mysql/conf.d/
cat /etc/mysql/conf.d/server-id.cnf
volumeMounts:
- name: configmap-mysql
mountPath: /mnt/config-map
- name: initdb
mountPath: /docker-entrypoint-initdb.d
- name: mysql-config
mountPath: /etc/mysql/conf.d/
restartPolicy: Always
containers:
- name: mysql
image: mysql:8.0.32
env:
# 设置 root 用户密码
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: secret-mysql-root-auth
key: root-password
#
- name: MYSQL_INITDB_SKIP_TZINFO
value: "1"
# 设置时区
- name: TZ
value: "Asia/Shanghai"
ports:
- name: server-port
containerPort: 3306
# 设置运行 MySQL 的用户，默认 root
securityContext:
runAsUser: 65534
runAsGroup: 65534
# fsGroup: 3000
volumeMounts:
# 挂载配置（初始化容器筛选后的配置）
- name: mysql-config
mountPath: /etc/mysql/conf.d/
- name: initdb
mountPath: /docker-entrypoint-initdb.d
# 挂载数据卷
- name: mysql-data
mountPath: /var/lib/mysql
volumes:
- name: configmap-mysql
configMap:
name: configmap-mysql
- name: mysql-config
emptyDir: {}
- name: initdb
emptyDir: {}
volumeClaimTemplates:
- metadata:
name: mysql-data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: sc-nfs
resources:
requests:
storage: 200Mi
注意：
设置了运行容器的用户 ID 和组，默认为 root。若 NFS 存储侧开启了 root 权限压缩（root_squash）,则会出现 “chown: changing ownership of '/var/lib/mysql/': Operation not permitted” 问题，解决该问题方法有两个：
- NFS 存储不压缩 root 权限（
no_root_squash） - 配置容器的运行用户为非 root 用户（无需与 NFS 侧一致），配置后不可再更改运行用户
检查
检查 Pod 状态
$ kubectl get pods -l app.kubernetes.io/name=mysql -n mysql-demo
NAME READY STATUS RESTARTS AGE
mysql-replica-0 1/1 Running 0 2m44s
mysql-replica-1 1/1 Running 0 2m38s
检查主从状态
# 登录从节点
$ kubectl exec -it mysql-replica-1 -n mysql-demo bash
# 登录 mysql
bash-4.4$ mysql -u'root' -p'qwert123..'
# 检查从节点同步状态，IO 线程和 SQL 线程均正常运行
mysql> show slave status\G;
......
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
......
部署 MySQL 主从（改进）
上述版本有一个比较大的缺陷，当 MySQL 运行了一段时间后，binlog 进行了切换或删除，此时再扩充 MySQL 实例，会出现主从数据不一致的情况。为了解决这个问题，可以引入备份工具，从实例扩展时，先导入 主（primary) 实例实时的备份数据，再开启主从同步
镜像制作
创建 Dockerfile
$ vim Dockerfile
# 一阶段，下载解压 xtrabackup
FROM rockylinux:9.1 as build
MAINTAINER nemo "[email protected]"
ADD https://downloads.percona.com/downloads/Percona-XtraBackup-8.0/Percona-XtraBackup-8.0.32-26/binary/tarball/percona-xtrabackup-8.0.32-26-Linux-x86_64.glibc2.17-minimal.tar.gz /
RUN tar zxf percona-xtrabackup-8.0.32-26-Linux-x86_64.glibc2.17-minimal.tar.gz
# 二阶段，复制 xtrabackup 中需要的二进制文件和链接，安装 ncat
FROM rockylinux:9.1
MAINTAINER nemo "[email protected]"
ENV TZ=Asia/Shanghai
COPY --from=build /percona-xtrabackup-8.0.32-26-Linux-x86_64.glibc2.17-minimal/bin/* /usr/bin/
COPY --from=build /percona-xtrabackup-8.0.32-26-Linux-x86_64.glibc2.17-minimal/lib/* /usr/lib64/
RUN rpm -Uvh https://nmap.org/dist/ncat-7.93-1.x86_64.rpm
创建镜像
$ docker build registry.cn-hangzhou.aliyuncs.com/kmust/xtrabackup:8.0.32-26-1.generic .
上传到阿里云
$ docker push registry.cn-hangzhou.aliyuncs.com/kmust/xtrabackup:8.0.32-26-1.generic
Namespace
$ vim ./ns-mysql-xtra.yaml
---
apiVersion: v1
kind: Namespace
metadata:
name: mysql-xtra
应用声明
$ kubectl apply -f ns-mysql-xtra.yaml
StorageClass
$ vim sc-nfs.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: sc-nfs
provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
# 回收策略此处使用 删除，实验环境方便测试
reclaimPolicy: Delete
mountOptions:
- soft
- nfsvers=4.2
- noatime # 访问文件时不更新文件 inode 中的时间戳，高并发环境可提高性能
parameters:
# 根据 PVC 的namespace 和 PVC 名称来生成路径
pathPattern: "${.PVC.namespace}/${.PVC.name}"
archiveOnDelete: "true"
应用声明
$ kubectl apply -f sc-nfs.yaml
Service
Headless Service
$ vim hs-mysql.yaml
---
# 为 StatefulSet 成员提供稳定的 DNS 表项的无头服务（Headless Service）
apiVersion: v1
kind: Service
metadata:
name: hs-mysql
namespace: mysql-xtra
labels:
app: mysql
app.kubernetes.io/name: mysql
spec:
ports:
- name: mysql
port: 3306
clusterIP: None
selector:
app: mysql
---
应用声明
kubectl apply -f hs-mysql.yaml
普通 Service
$ vim svc-mysql-read.yaml
# 用于连接到任一 MySQL 实例执行读操作的客户端服务
# 对于写操作，则必须连接到主服务器：mysql-0.mysql
apiVersion: v1
kind: Service
metadata:
name: svc-mysql-read
namespace: mysql-xtra
labels:
app: mysql
app.kubernetes.io/name: mysql
readonly: "true"
spec:
ports:
- name: mysql
port: 3306
selector:
app: mysql
应用声明
$ kubectl apply -f svc-mysql-read.yaml
Secret
MySQL 的 root 用户密码
# 必须加 -n ，否则会把换行符也用 base64 编码
$ echo -n "qwert123.." | base64
cXdlcnQxMjMuLg==
$ vim secret-mysql-root-auth.yaml
---
apiVersion: v1
kind: Secret
metadata:
name: secret-mysql-root-auth
namespace: mysql-xtra
type: Opaque
data:
root-password: cXdlcnQxMjMuLg==
应用声明
$ kubectl apply -f secret-mysql-root-auth.yaml
ConfigMap
$ vim configmap-mysql.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
name: configmap-mysql
namespace: mysql-xtra
labels:
app: mysql
app.kubernetes.io/name: mysql
data:
primary.cnf: |
[mysqld]
gtid-mode=ON
enforce-gtid-consistency
replica.cnf: |
[mysqld]
gtid-mode=ON
enforce-gtid-consistency
relay-log=mysql-xtra-relay-bin
read-only=ON
primary.sql: |
# 创建复制账号
CREATE USER IF NOT EXISTS 'repluser'@'%' IDENTIFIED WITH mysql_native_password BY 'replsecret123..';
GRANT REPLICATION CLIENT,REPLICATION SLAVE ON *.* TO 'repluser'@'%';
CREATE DATABASE primary_db;
replica.sql: |
STOP SLAVE;
RESET SLAVE;
CHANGE MASTER TO
MASTER_HOST='mysql-xtra-0.hs-mysql',
MASTER_USER='repluser',
MASTER_PASSWORD='replsecret123..',
MASTER_AUTO_POSITION=1,
MASTER_CONNECT_RETRY=10;
START SLAVE;
general.sql: |
# 探针用户
CREATE USER IF NOT EXISTS 'probe-user'@'localhost' IDENTIFIED WITH mysql_native_password BY 'probe-pass';
GRANT EXECUTE ON *.* TO 'probe-user'@'localhost';
GRANT SELECT ON *.* TO 'probe-user'@'localhost';
# 备份用户
CREATE USER IF NOT EXISTS 'bkpuser'@'127.0.0.1' IDENTIFIED WITH mysql_native_password BY 'bkppass';
GRANT BACKUP_ADMIN, PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'bkpuser'@'127.0.0.1';
GRANT SELECT ON performance_schema.log_status TO 'bkpuser'@'127.0.0.1';
GRANT SELECT ON performance_schema.keyring_component_status TO bkpuser@'127.0.0.1';
GRANT SELECT ON performance_schema.replication_group_members TO bkpuser@'127.0.0.1';
# 刷新权限
FLUSH PRIVILEGES;
注：xtrabckup 备份用户需要的权限参考：Connection and privileges needed - Percona XtraBackup
应用声明
$ kubectl apply -f configmap-mysql.yaml
StatefulSet
总共运行 4 个 容器
-
两个初始化容器
init-conf和clone-datainit-conf用于筛选 MySQL 的配置（区分主从配置）clone-data使用ncat接收 MySQL 的xtrabackup备份数据
-
一个业务容器
mysql，运行 MySQL 实例 -
一个
sidecar容器，用于运行初始化 SQL ，并使用ncat为后续添加的 MySQL 从实例发送xtrabackup备份数据
$ vim statefulset-mysql-xtra.yaml
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mysql-xtra
namespace: mysql-xtra
labels:
app.kubernetes.io/name: mysql
spec:
selector:
matchLabels:
app: mysql
app.kubernetes.io/name: mysql
serviceName: hs-mysql
replicas: 3
template:
metadata:
labels:
app: mysql
app.kubernetes.io/name: mysql
spec:
restartPolicy: Always
# 初始化容器
initContainers:
##### 初始化容器 init-conf #####
- name: init-conf
image: mysql:8.0.32
imagePullPolicy: Always
command:
- bash
- "-c"
- |
set -ex
echo 'Starting init-mysql';
# Check config map to directory that already exists
# (but must be used as a volume for main container)
ls /mnt/config-map
# 获取主机名中的 id，以判断是 primary 还是 replica
[[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
# 复制对应的配置文件到相应目录
if [[ $ordinal -eq 0 ]]; then
# 复制配置文件到配置路径
cp /mnt/config-map/primary.cnf /etc/mysql/conf.d/server-id.cnf
# 复制初始化 SQL 语句到对应路径
cp /mnt/config-map/primary.sql /docker-entrypoint-initdb.d/
else
cp /mnt/config-map/replica.cnf /etc/mysql/conf.d/server-id.cnf
cp /mnt/config-map/replica.sql /docker-entrypoint-initdb.d/
fi
# 主从实例均要运行的语句
cp /mnt/config-map/general.sql /docker-entrypoint-initdb.d/
# 给 server-id 添加一个偏移量，防止 server-id=0
echo server-id=$((3000 + $ordinal)) >> /etc/mysql/conf.d/server-id.cnf
ls /etc/mysql/conf.d/
cat /etc/mysql/conf.d/server-id.cnf
volumeMounts:
- name: configmap-mysql
mountPath: /mnt/config-map
- name: init-sql
mountPath: /docker-entrypoint-initdb.d
- name: mysql-conf
mountPath: /etc/mysql/conf.d/
##### 初始化容器 clone-data #####
- name: clone-data
image: registry.cn-hangzhou.aliyuncs.com/kmust/xtrabackup:8.0.32-26-1.generic
imagePullPolicy: Always
command:
- bash
- "-c"
- |
set -ex
# 如果已有数据，则跳过克隆数据的步骤
[[ -d /var/lib/mysql/mysql ]] && exit 0
# 主实例（序号索引 0）跳过克隆数据调度步骤
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
[[ $ordinal -eq 0 ]] && exit 0
# 从比自身序号小一的实例上克隆数据
ncat --recv-only mysql-xtra-$(($ordinal-1)).hs-mysql 3307 | xbstream -x -C /var/lib/mysql-clone
# 还原预准备
xtrabackup --prepare --target-dir=/var/lib/mysql-clone
# 还原
xtrabackup --copy-back --target-dir=/var/lib/mysql-clone --datadir=/var/lib/mysql
volumeMounts:
- name: mysql-data-clone
mountPath: /var/lib/mysql-clone
- name: mysql-data
mountPath: /var/lib/mysql
#subPath: mysql
- name: mysql-conf
mountPath: /etc/mysql/conf.d
containers:
##### 业务容器 mysql #####
- name: mysql
image: mysql:8.0.32
imagePullPolicy: Always
env:
# 设置 root 用户密码
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: secret-mysql-root-auth
key: root-password
#
- name: MYSQL_INITDB_SKIP_TZINFO
value: "1"
# 设置时区
- name: TZ
value: "Asia/Shanghai"
ports:
- name: server-port
containerPort: 3306
# 设置运行 MySQL 的用户，默认 root
securityContext:
runAsUser: 65534
runAsGroup: 65534
# fsGroup: 3000
volumeMounts:
# 挂载配置（初始化容器筛选后的配置）
- name: mysql-conf
mountPath: /etc/mysql/conf.d/
- name: init-sql
mountPath: /docker-entrypoint-initdb.d
# 挂载数据卷
- name: mysql-data
mountPath: /var/lib/mysql
#subPath: mysql
# 探针
livenessProbe:
exec:
command:
- "/bin/bash"
- "-c"
- "mysqladmin -u'probe-user' -p'probe-pass' ping"
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 5
readinessProbe:
exec:
command:
- "bin/bash"
- "-c"
- "mysql -u'probe-user' -p'probe-pass' -e'SELECT 1 FROM dual' "
initialDelaySeconds: 30
periodSeconds: 10
timeoutSeconds: 2
##### sidecar 容器 xtrabackup #####
- name: xtrabackup
image: registry.cn-hangzhou.aliyuncs.com/kmust/xtrabackup:8.0.32-26-1.generic
imagePullPolicy: Always
ports:
- name: xtrabackup
containerPort: 3307
env:
# 设置 root 用户密码
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: secret-mysql-root-auth
key: root-password
command:
- bash
- "-c"
- |
set -ex
cd /var/lib/mysql
# 主实例（序号索引 0）跳过初始化 SQL文件的步骤，已经由容器初始化过，详见 MySQL 容器的 /entrypoint.sh
[[ `hostname` =~ -([0-9]+)$ ]]
ordinal=${BASH_REMATCH[1]}
if [ $ordinal -ne 0 ]; then
sleep 20s
#
echo "Waiting for mysqld to be ready (accepting connections)"
until mysql -uroot -p${MYSQL_ROOT_PASSWORD} -h127.0.0.1 -e "SELECT 1"; do sleep 1s; done
#
for sql_file in /docker-entrypoint-initdb.d/*.sql ;do
mysql -uroot -p${MYSQL_ROOT_PASSWORD} -h127.0.0.1 < ${sql_file}
done
fi
# 当对等点请求时，使用 xtrabackup 备份，并用 ncat 发送备份
exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
"xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=bkpuser --password=bkppass"
volumeMounts:
# 挂载配置（初始化容器筛选后的配置）
- name: mysql-conf
mountPath: /etc/mysql/conf.d/
- name: init-sql
mountPath: /docker-entrypoint-initdb.d
# 挂载数据卷
- name: mysql-data
mountPath: /var/lib/mysql
#subPath: mysql
volumes:
- name: configmap-mysql
configMap:
name: configmap-mysql
- name: mysql-conf
emptyDir: {}
- name: init-sql
emptyDir: {}
volumeClaimTemplates:
- metadata:
name: mysql-data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: sc-nfs
resources:
requests:
storage: 200Mi
- metadata:
name: mysql-data-clone
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: sc-nfs
resources:
requests:
storage: 200Mi
检查
检查 Pod 状态
$ kubectl get pods -n mysql-xtra
NAME READY STATUS RESTARTS AGE
mysql-xtra-0 2/2 Running 0 48m
mysql-xtra-1 2/2 Running 0 46m
mysql-xtra-2 2/2 Running 0 45m
检查主从状态
# 进入从节点
$ kubectl exec -it mysql-xtra-1 -n mysql-xtra bash
bash-4.4$ mysql -uroot -p'qwert123..'
mysql> SHOW SLAVE STATUS\G;
......
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
......
检查主写从读
运行一个 mysql 客户端
$ kubectl run -it --rm=true --image="registry.cn-hangzhou.aliyuncs.com/kmust/xtrabackup:8.0.32-26-1.generic" -n mysql-xtramysql-cli
连接主库写入数据
$ mysql -uroot -p'qwert123..' -h mysql-xtra-0.hs-mysql
# 测试创建数据库，创建用户
mysql> CREATE DATABASE testdb DEFAULT CHARSET utf8mb4;
mysql> CREATE USER 'testuser'@'%' IDENTIFIED BY 'qwert123';
mysql> GRANT ALL PRIVILEGES ON testdb.* TO 'testuser'@'%';
mysql> FLUSH PRIVILEGES;
# 测试创建表
mysql> USE testdb;
mysql> CREATE TABLE t_user01(
id int auto_increment primary key,
name varchar(40)
) ENGINE = InnoDB;
# 测试写入数据
BEGIN;
INSERT INTO t_user01 VALUES (1,'user01');
INSERT INTO t_user01 VALUES (2,'user02');
INSERT INTO t_user01 VALUES (3,'user03');
INSERT INTO t_user01 VALUES (4,'user04');
INSERT INTO t_user01 VALUES (5,'user05');
commit;
连接从库，测试读数据
$ mysql -u'testuser' -p'qwert123' -h svc-mysql-read
# 查看 hostname
mysql> show variables like '%hostname%';
+---------------+--------------+
| Variable_name | Value |
+---------------+--------------+select * from testdb.t_user01;
| hostname | mysql-xtra-2 |
+---------------+--------------+
1 row in set (0.01 sec)
# 测试读取数据
$ SELECT * FROM testdb.t_user01;
mysql> SELECT * FROM testdb.t_user01;
+----+--------+
| id | name |
+----+--------+
| 1 | user01 |
| 2 | user02 |
| 3 | user03 |
| 4 | user04 |
| 5 | user05 |
+----+--------+
5 rows in set (0.00 sec)
# 测试写入数据，返回错误提示：该库 read-only
mysql> INSERT INTO testdb.t_user01 VALUES (6,'user06');
ERROR 1290 (HY000): The MySQL server is running with the --read-only option so it cannot execute this statement
部署 ProxySQL（代理）
ProxySQL 用于代理 MySQL 主从实例，实现读写分离。此处仅部署一个 ProxySQL，如果需要部署 ProxySQL Cluster，可以参考：ProxySQL Cluster - ProxySQL
PVC
$ vim pvc-proxysql.yaml
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-proxysql
namespace: mysql-xtra
spec:
storageClassName: sc-nfs
accessModes:
- ReadWriteMany # 访问权限
resources:
requests:
storage: 100Mi # 空间大小
应用声明
$ kubectl apply -f pvc-proxysql.yaml
Deployment
$ vim deploy-proxysql.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: proxysql
namespace: mysql-xtra
labels:
app.kubernetes.io/name: proxysql
spec:
selector:
matchLabels:
app: proxysql
app.kubernetes.io/name: proxysql
replicas: 1
template:
metadata:
labels:
app: proxysql
app.kubernetes.io/name: proxysql
spec:
restartPolicy: Always
containers:
##### 业务容器 proxysql #####
- name: proxysql
image: proxysql/proxysql:2.5.1
imagePullPolicy: Always
env:
# 设置时区
- name: TZ
value: "Asia/Shanghai"
ports:
- name: admin
containerPort: 6032
- name: server
containerPort: 6033
- name: web
containerPort: 6080
volumeMounts:
# 挂载数据卷
- name: data
mountPath: /var/lib/proxysql
volumes:
- name: data
persistentVolumeClaim:
claimName: pvc-proxysql
应用声明
$ kubectl apply -f deploy-proxysql.yaml
Service
$ vim svc-proxysql.yaml
---
apiVersion: v1
kind: Service
metadata:
name: svc-proxysql
namespace: mysql-xtra
spec:
type: ClusterIP
ports:
- name: http
port: 6032
targetPort: 6032
protocol: TCP
- name: admin
port: 6033
targetPort: 6033
protocol: TCP
- name: web
port: 6080
targetPort: 6080
protocol: TCP
selector:
app.kubernetes.io/name: proxysql
应用声明
$ kubectl apply -f svc-proxysql.yaml
配置 ProxySQL
运行一个 MySQL Client
$ kubectl run -it --rm=true --image="registry.cn-hangzhou.aliyuncs.com/kmust/xtrabackup:8.0.32-26-1.generic" -n mysql-xtra mysql-cli
配置 MySQL
连接 MySQL 主实例
$ mysql -uroot -p'qwert123..' -h mysql-xtra-0.hs-mysql
创建用户并授权
ProxySQL 需要用户的权限，请参考：https://proxysql.com/documentation/backend-monitoring/
注：截止2023年4月（ ProxySQL 2.5 ）。ProxySQL 只支持
mysql_native_password。切记，在后端为 MySQL 8.0 的时候必须使用mysql_native_password作为用户密码插件，否则可能会出现密码验证不通过的情况
创建监控用户
# ProxySQL 需要两个用户，监控用户和连接用户
mysql> CREATE USER 'mysql-monitor'@'%' IDENTIFIED WITH mysql_native_password BY 'monitor-pass';
mysql> GRANT REPLICATION CLIENT ON *.* TO 'mysql-monitor'@'%';
mysql> FLUSH PRIVILEGES;
# 测试用户是否正常
$ mysql -u'mysql-monitor' -p'monitor-pass' -h mysql-xtra-0.hs-mysql -e 'select 1 from dual'
创建连接用户
# 连接用户为 proxysql，在 172.20.0.0/16（ Pod 的 IP 范围）内可以访问
mysql> CREATE USER 'proxysql'@'172.20.%.%' IDENTIFIED WITH mysql_native_password BY 'qwert123..';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'proxysql'@'172.20.%.%' WITH GRANT OPTION;
mysql> FLUSH PRIVILEGES;
# 测试用户是否正常
$ mysql -u'proxysql' -p'qwert123..' -h mysql-xtra-0.hs-mysql -e 'select 1 from dual'
配置 ProxySQL
此处使用正则匹配进行读写分离，实际生产中建议结合正则匹配和摘要一起使用，详情参考：How to set up ProxySQL Read/Write Split - ProxySQL
进入 ProxySQL 实例
$ kubectl exec -it proxysql-667c7d6c55-ctw7c -n mysql-xtra -- bash
连接管理界面
# 管理界面默认用户名密码为 admin/admin
$ mysql -u'admin' -p'admin' -h'127.0.0.1' -P6032 --prompt 'ProxySQL Admin> '
修改 MySQL 版本信息（对客户端连接的展示，可选）
ProxySQL Admin> UPDATE global_variables SET variable_value='8.0.32' WHERE variable_name='mysql-server_version';
配置主机组
注：ProxySQL 会检查 mysql 实例的
read_only值，然后动态调整该实例是属于读组还是写组，所以 mysql 实例的read_only必须配置正确
插入一组读写主机组，hostgroup_id = 10 表示写主机组，hostgroup_id = 20 表示读主机组
# 插入读写主机组
ProxySQL Admin> INSERT INTO mysql_replication_hostgroups (writer_hostgroup,reader_hostgroup,comment) VALUES (10,20,'mysql-xtra');
# 检查读写主机组
ProxySQL Admin> SELECT * FROM mysql_replication_hostgroups;
+------------------+------------------+------------+------------+
| writer_hostgroup | reader_hostgroup | check_type | comment |
+------------------+------------------+------------+------------+
| 10 | 20 | read_only | mysql-xtra |
+------------------+------------------+------------+------------+
添加后端服务器
将后端 MySQL 的节点信息添加到 ProxySQL 的 mysql_servers 表中，以管理 MySQL 后端服务器。并根据读写的分工设置 hostgroup_id
# 增加三个节点信息
ProxySQL Admin> INSERT INTO mysql_servers(hostgroup_id,hostname,port) VALUES (10,'mysql-xtra-0.hs-mysql',3306);
ProxySQL Admin> INSERT INTO mysql_servers(hostgroup_id,hostname,port) VALUES (20,'mysql-xtra-1.hs-mysql',3306);
ProxySQL Admin> INSERT INTO mysql_servers(hostgroup_id,hostname,port) VALUES (20,'mysql-xtra-2.hs-mysql',3306);
# 查询节点信息
ProxySQL Admin> SELECT * FROM mysql_servers;
+--------------+-----------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | gtid_port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-----------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10 | mysql-xtra-0.hs-mysql | 3306 | 0 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 20 | mysql-xtra-1.hs-mysql | 3306 | 0 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 20 | mysql-xtra-2.hs-mysql | 3306 | 0 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+-----------------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
加载、持久化 MySQL 服务器和 MySQL 复制组的配置
# 加载配置
ProxySQL Admin> LOAD MYSQL SERVERS TO RUNTIME;
# 持久化配置
ProxySQL Admin> SAVE MYSQL SERVERS TO DISK;
配置 MySQL 监控
ProxySQL 添加监控用户信息
注：监控用户只能有一个
# 配置监控用户（修改变量）
ProxySQL Admin> UPDATE global_variables SET variable_value='mysql-monitor' WHERE variable_name='mysql-monitor_username';
# 配置监控用户密码（修改变量）
ProxySQL Admin> UPDATE global_variables SET variable_value='monitor-pass' WHERE variable_name='mysql-monitor_password';
配置各种监控参数（根据需要进行配置，建议修改各种检查时间间隔）
# 配置监控参数（修改各种检查时间间隔）
ProxySQL Admin> UPDATE global_variables SET variable_value='5000' WHERE variable_name IN ('mysql-monitor_connect_interval','mysql-monitor_ping_interval','mysql-monitor_read_only_interval');
# 查看当前的监控参数
ProxySQL Admin> SELECT * FROM global_variables WHERE variable_name LIKE 'mysql-monitor_%';
+----------------------------------------------------------------------+----------------+
| variable_name | variable_value |
+----------------------------------------------------------------------+----------------+
| mysql-monitor_enabled | true |
| mysql-monitor_connect_timeout | 600 |
| mysql-monitor_ping_max_failures | 3 |
| mysql-monitor_ping_timeout | 1000 |
| mysql-monitor_read_only_max_timeout_count | 3 |
| mysql-monitor_replication_lag_group_by_host | false |
| mysql-monitor_replication_lag_interval | 10000 |
| mysql-monitor_replication_lag_timeout | 1000 |
| mysql-monitor_replication_lag_count | 1 |
| mysql-monitor_groupreplication_healthcheck_interval | 5000 |
| mysql-monitor_groupreplication_healthcheck_timeout | 800 |
| mysql-monitor_groupreplication_healthcheck_max_timeout_count | 3 |
| mysql-monitor_groupreplication_max_transactions_behind_count | 3 |
| mysql-monitor_groupreplication_max_transactions_behind_for_read_only | 1 |
| mysql-monitor_galera_healthcheck_interval | 5000 |
| mysql-monitor_galera_healthcheck_timeout | 800 |
| mysql-monitor_galera_healthcheck_max_timeout_count | 3 |
| mysql-monitor_replication_lag_use_percona_heartbeat | |
| mysql-monitor_query_interval | 60000 |
| mysql-monitor_query_timeout | 100 |
| mysql-monitor_slave_lag_when_null | 60 |
| mysql-monitor_threads_min | 8 |
| mysql-monitor_threads_max | 128 |
| mysql-monitor_threads_queue_maxsize | 128 |
| mysql-monitor_local_dns_cache_ttl | 300000 |
| mysql-monitor_local_dns_cache_refresh_interval | 60000 |
| mysql-monitor_local_dns_resolver_queue_maxsize | 128 |
| mysql-monitor_wait_timeout | true |
| mysql-monitor_writer_is_also_reader | true |
| mysql-monitor_username | mysql-monitor |
| mysql-monitor_password | monitor-pass |
| mysql-monitor_history | 600000 |
| mysql-monitor_connect_interval | 5000 |
| mysql-monitor_ping_interval | 5000 |
| mysql-monitor_read_only_interval | 5000 |
| mysql-monitor_read_only_timeout | 500 |
+----------------------------------------------------------------------+----------------+
加载、持久化 MySQL 变量配置
# 加载配置
ProxySQL Admin> LOAD MYSQL VARIABLES TO RUNTIME;
# 持久化配置
ProxySQL Admin> SAVE MYSQL VARIABLES TO DISK;
配置 MySQL 连接用户
配置 ProxySQL 连接到 MySQL 的 MySQL 用户，该用户同时也是客户端连接到 ProxySQL 的用户，配置默认为写组
# 配置 MySQL 用户
ProxySQL Admin> INSERT INTO mysql_users(username,password,default_hostgroup) VALUES ('proxysql','qwert123..',10);
# 查看用户信息
ProxySQL Admin> SELECT * FROM mysql_users;
加载、持久化 MySQL 用户配置
# 加载配置
ProxySQL Admin> LOAD MYSQL USERS TO RUNTIME;
# 持久化配置
ProxySQL Admin> SAVE MYSQL USERS TO DISK;
配置路由规则（读写分离）
由于 SELECT 语句中有一个特殊语句 SELECT ... FOR UPDATE 它会申请写锁，所以应该路由到 hostgroup_id=10 的写组
SELECT ... FOR UPDATE 规则的 rule_id 必须要小于普通的 SELECT 规则的 rule_id，因为 ProxySQL 是根据 rule_id 的顺序进行规则匹配的
# 插入两条路由规则
ProxySQL Admin> INSERT INTO mysql_query_rules (rule_id,active,username,match_digest,destination_hostgroup,apply)
VALUES \
(1,1,"proxysql",'^SELECT.*FOR UPDATE$',10,1), \
(2,1,"proxysql",'^SELECT',20,1);
# 查看当前路由规则
ProxySQL Admin> SELECT rule_id,active,username,match_digest,destination_hostgroup,apply FROM mysql_query_rules;
+---------+--------+----------+----------------------+-----------------------+-------+
| rule_id | active | username | match_digest | destination_hostgroup | apply |
+---------+--------+----------+----------------------+-----------------------+-------+
| 1 | 1 | root | ^SELECT.*FOR UPDATE$ | 10 | 1 |
| 2 | 1 | root | ^SELECT | 20 | 1 |
+---------+--------+----------+----------------------+-----------------------+-------+
现在，路由将按如下方式工作：
- 所有
SELECT ... FOR UPDATE语句都将发送给写主机组（hostgroup_id = 10） - 所有其他
SELECT语句将发送给读主机组（hostgroup_id = 20） - 其他所有内容都将发送到写主机组（
hostgroup_id = 10，用户的default_hostgroup）
加载、持久化 MySQL 查询语句的路由规则配置
# 加载配置
ProxySQL Admin> LOAD MYSQL QUERY RULES TO RUNTIME;
# 持久化配置
ProxySQL Admin> SAVE MYSQL QUERY RULES TO DISK;
开启 WEB 统计功能
# 修改管理变量，开启 web 统计功能
ProxySQL Admin> UPDATE global_variables SET variable_value='true' WHERE variable_name='admin-web_enabled';
# 查看 WEB 统计相关变量的配置
ProxySQL Admin> SELECT * FROM global_variables WHERE variable_name LIKE 'admin-web%' OR variable_name LIKE 'admin-stats%';
+----------------------------------------+----------------+
| variable_name | variable_value |
+----------------------------------------+----------------+
| admin-stats_credentials | stats:stats | # web 界面用户名密码
| admin-stats_mysql_connections | 60 |
| admin-stats_mysql_connection_pool | 60 |
| admin-stats_mysql_query_cache | 60 |
| admin-stats_mysql_query_digest_to_disk | 0 |
| admin-stats_system_cpu | 60 |
| admin-stats_system_memory | 60 |
| admin-web_enabled | true |
| admin-web_port | 6080 | # web 界面端口
| admin-web_verbosity | 0 |
+----------------------------------------+----------------+
10 rows in set (0.00 sec)
加载、持久化 ProxySQL 管理变量配置
# 加载配置
ProxySQL Admin> LOAD ADMIN VARIABLES TO RUNTIME;
# 持久化配置
ProxySQL Admin> SAVE ADMIN VARIABLES TO DISK;
检查
连接和 ping 监控是基于配置的
mysql_servers表完成的（在加载到运行时之前就已生效）
基础检查
检查 ProxySQL 与 MySQL 的连接情况
ProxySQL Admin> SELECT * FROM monitor.mysql_server_connect_log ORDER BY time_start_us DESC LIMIT 3;
检查 ProxySQL 与 MySQL 的 ping 情况
ProxySQL Admin> SELECT * FROM monitor.mysql_server_ping_log ORDER BY time_start_us DESC LIMIT 3;
检查 read_only 情况
ProxySQL Admin> SELECT * FROM monitor.mysql_server_read_only_log ORDER BY time_start_us DESC LIMIT 3;
检查主从复制情况
ProxySQL Admin> SELECT * FROM monitor.mysql_server_replication_lag_log ORDER BY time_start_us DESC LIMIT 3;
验证读写分离
查询 Service 的 ClusterIP
$ kubectl get svc -n mysql-xtra
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hs-mysql ClusterIP None <none> 3306/TCP 3d3h
svc-mysql-read ClusterIP 10.68.205.239 <none> 3306/TCP 3d3h
svc-proxysql ClusterIP 10.68.120.89 <none> 6032/TCP,6033/TCP,6080/TCP 25h
连接并创建测试数据表
# 连接到 ProxySQL 代理
$ mysql -u'proxysql' -p'qwert123..' -h 10.68.120.89 -P6033
# 创建数据库
mysql> CREATE DATABASE test_db CHARACTER SET utf8mb4;
# 进入数据库
mysql> USE test_db;
Database changed
# 创建测试用数据表
mysql> CREATE TABLE IF NOT EXISTS t_user ( \
id INT(10) PRIMARY KEY AUTO_INCREMENT, \
name VARCHAR(50) NOT NULL \
);
测试写入数据
# 创建插入测试数据的脚本（插入10条数据）
$ vi ./insert_data.sh
#!/bin/bash
for i in `seq 10`;do
mysql -u'proxysql' -p'qwert123..' -h 10.68.120.89 -P6033 -e "INSERT INTO test_db.t_user(name) VALUES('name_$i')" > /dev/null;
done
# 执行脚本
$ source ./insert_data.sh
测试读取数据
$ mysql -u'proxysql' -p'qwert123..' -h 10.68.120.89 -P6033 -e "SELECT * FROM test_db.t_user WHERE id < 3"
+----+--------+
| id | name |
+----+--------+
| 1 | name_1 |
| 2 | name_2 |
+----+--------+
查看 ProxySQL 路由情况
在 ProxySQL 管理界面查询统计表 stats_mysql_query_digest，可以看到 INSERT 语句被路由到写主机组（hostgroup_id = 10），SELECT 语句被路由到读主机组（hostgroup_id = 20）
ProxySQL Admin> SELECT * FROM stats_mysql_query_digest;
+-----------+--------------------+----------+----------------+--------------------+---------------------------------------------------------------------------------------------------+------------+------------+------------+----------+----------+----------+-------------------+---------------+
| hostgroup | schemaname | username | client_address | digest | digest_text | count_star | first_seen | last_seen | sum_time | min_time | max_time | sum_rows_affected | sum_rows_sent |
+-----------+--------------------+----------+----------------+--------------------+---------------------------------------------------------------------------------------------------+------------+------------+------------+----------+----------+----------+-------------------+---------------+
| 20 | information_schema | proxysql | | 0x8487E1AA680BB825 | SELECT * FROM test_db.t_user WHERE id < ? | 1 | 1681710384 | 1681710384 | 3511 | 3511 | 3511 | 0 | 2 |
| 10 | information_schema | proxysql | | 0xF20AA3A6EDF83D0F | INSERT INTO test_db.t_user(name) VALUES(?) | 10 | 1681710342 | 1681710342 | 57671 | 2319 | 18092 | 10 | 0 |
| 10 | test_db | proxysql | | 0xE1F1AF52FD5D2D00 | CREATE TABLE IF NOT EXISTS t_user (id INT(?) PRIMARY KEY AUTO_INCREMENT,name VARCHAR(?) NOT NULL) | 1 | 1681710307 | 1681710307 | 34333 | 34333 | 34333 | 0 | 0 |
| 10 | test_db | proxysql | | 0x99531AEFF718C501 | show tables | 1 | 1681710303 | 1681710303 | 2144 | 2144 | 2144 | 0 | 0 |
| 10 | information_schema | proxysql | | 0xF4B04587B7695EC6 | CREATE DATABASE test_db CHARACTER SET utf8mb4 | 1 | 1681710300 | 1681710300 | 7863 | 7863 | 7863 | 1 | 0 |
| 10 | information_schema | proxysql | | 0x02033E45904D3DF0 | show databases | 1 | 1681710256 | 1681710256 | 14884 | 14884 | 14884 | 0 | 5 |
| 10 | test_db | proxysql | | 0x02033E45904D3DF0 | show databases | 1 | 1681710303 | 1681710303 | 1552 | 1552 | 1552 | 0 | 6 |
| 20 | information_schema | proxysql | | 0x620B328FE9D6D71A | SELECT DATABASE() | 1 | 1681710303 | 1681710303 | 2470 | 2470 | 2470 | 0 | 1 |
| 10 | information_schema | proxysql | | 0x226CD90D52A2BA0B | select @@version_comment limit ? | 12 | 1681710248 | 1681710384 | 0 | 0 | 0 | 0 | 0 |
+-----------+--------------------+----------+----------------+--------------------+---------------------------------------------------------------------------------------------------+------------+------------+------------+----------+----------+----------+-------------------+---------------+
# 如果需要清空统计表，可以查询 stats_mysql_query_digest_reset 表
# SELECT * FROM stats_mysql_query_digest_reset
验证读组内负载情况
proxysql 的负载方式目前只支持加权轮询，通过查询主机名可以验证读组内负载情况，两台从主机均可以查询到即正常
$ mysql -u'proxysql' -p'qwert123..' -h 10.68.120.89 -P6033 -e "SELECT @@hostname";
+--------------+
| @@hostname |
+--------------+
| mysql-xtra-2 |
+--------------+
# 间隔一段时间（超过连接保持时间，即 ping 超时时间）后再次查询
$ mysql -u'proxysql' -p'qwert123..' -h 10.68.120.89 -P6033 -e "SELECT @@hostname";
+--------------+
| @@hostname |
+--------------+
| mysql-xtra-1 |
+--------------+