Kubernetes 完美部署MySQL主从集群

在Kubernetes中可以快速MySQL主从集群，但需要解决以下问题：

1. 主从节点关系自动建立；
2. 实现一主一从或一主多从，并且读写分离；
3. 从节点server-id自动生成，并且不能重复；
4. 自愈功能，主节点或从节点Pod重启后，主从复制状态自动恢复，且数据不丢失；
5. 数据库数据持久化存储。

 

1. 简化MySQL主从配置步骤

在Kubernetes中部署MySQL主从集群，需要使用GTID代替传统复制技术classic，简化主从配置。

全局事物标识：Global Transaction Identifieds。

1. GTID事物是全局唯一性的，且一个事务对应一个GTID。
2. 一个GTID在一个服务器上只执行一次，避免重复执行导致数据混乱或者主从不一致。
3. GTID用来代替classic的复制方法，不在使用binlog+pos开启复制。而是使用master_auto_postion=1的方式自动匹配GTID断点进行复制。
4. 建议在MySQL 5.7或以上版本中使用。

GTID比传统复制classic的优势：

1. 更简单的实现failover，不用以前那样在需要找log_file和log_pos。
2. 更简单的搭建主从复制，比传统复制更加安全。
3. Slave保障节点Pod重启后，主从复制状态自动恢复，且数据不丢失。

开启GTID:

    开启GTID非常简单，在MySQL配置文件中增加：  gtid_mode=on

 

2. MySQL主从节点关系自动建立

MySQL Docker镜像中有一个文件夹 /docker-entrypoint-initdb.d/，该文件夹中的.sql .sh文件都会被自动执行，但是需要注意该文件夹中的.sql .sh文件只在MySQL容器第一次初始化数据库时才会自动执行。

因此我们可以将MySQL Master和Slave初始化主从关系的SQL语句放在 /docker-entrypoint-initdb.d/文件夹下。

MySQL Master节点 - 初始化主从命令

-- create repl user
CREATE USER 'xxxx'@'%' IDENTIFIED WITH mysql_native_password BY 'xxxx';

-- grant repl
GRANT REPLICATION SLAVE ON *.* TO 'xxxx'@'%';

MySQL Slave节点 - 初始化主从命令

CHANGE MASTER TO MASTER_HOST='xxxx', MASTER_USER='xxxx', MASTER_PASSWORD='xxxx', master_auto_position=1;

 

3. 从节点server-id自动生成

从节点需要生成唯一的server-id，可以通过InitContainer为Slave节点自动生成server-id

initContainers:  # 用于生成slave server-id,确保多个slave server-id不重复
- name: init-mysql
  image: mysql:latest
  imagePullPolicy: IfNotPresent
  command:
    - bash
    - "-c"
    - |
      set -ex

      # 从config-map中复制配置文件到emptyDir，因为config-map目录为只读
      cp /etc/mysql-slave-conf/mysql-slave.cnf /etc/conf.d/mysql-slave.cnf

      # 根据Pod序号生成server-id, =~ 判断字符串包含关系，BASH_REMATCH变量存储匹配结果
      [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
      pod_number=${BASH_REMATCH[1]}

      # 将生成的server-id写入mysql-slave.cnf
      echo -e "\nserver-id=$((100 + pod_number))" >> /etc/conf.d/mysql-slave.cnf

 

4. MySQL主从自愈

MySQL数据库数据会存储在/var/lib/mysql文件夹下，因此只需要使用PVC对该文件夹做持久化存储即可。

MySQL Master节点在/var/lib/mysql做持久化存储后，重启或重建Pod都没有任何影响。

MySQL Slave节点在/var/lib/mysql做持久化存储后，重启或重建Pod都会丢失和Master节点的复制关系，需要重置复制关系。

可以在MySQL Slave节点在重启或重建Pod时执行以下Shell命令即可实现MySQL Slave节点自愈：

# wait mysqld running
until mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e "SELECT 1"; do sleep 2; done

# start slave sql script
mysql -uroot -p${MYSQL_ROOT_PASSWORD} < stop slave;
reset slave;
CHANGE MASTER TO MASTER_HOST='xxxx', MASTER_USER='repl', MASTER_PASSWORD='xxxx', master_auto_position=1;
start slave;
EOF

 

5. MySQL配置文件

可以用ConfigMap存储MySQL Master和Slave节点需要用到的配置文件、SQL语句、Shell脚本。

---
# MySQL Conf
apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-conf
data:
  mysql-master.cnf: |
    [mysqld]
    log-bin=mysql-bin
    server-id=1
    gtid_mode=on
    enforce_gtid_consistency=1
    skip-host-cache
    skip-name-resolve

  mysql-slave.cnf: |
    [mysqld]
    log-bin=mysql-bin
    # server-id=100 通过initContainer自动生成
    gtid_mode=on
    enforce_gtid_consistency=1
    read_only=on
    skip-host-cache
    skip-name-resolve

  init-master.sql: |
    CREATE USER 'repl'@'%' IDENTIFIED WITH mysql_native_password BY '123456';
    GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%';
    flush privileges;

  init-slave.sql: |
    CHANGE MASTER TO MASTER_HOST='mysql-master', MASTER_USER='repl', MASTER_PASSWORD='123456', master_auto_position=1;
    start slave;

  reset-slave.sh: |
    #!/bin/sh

    # wait mysqld startup
    until mysql -uroot -p${MYSQL_ROOT_PASSWORD} -e "SELECT 1"; do sleep 1; done

    # reset slave
    mysql -uroot -p${MYSQL_ROOT_PASSWORD} < 
  

 

6. MySQL StatefulSet

为了简化MySQL主从配置，将MySQL Master和MySQL Slave节点分离为两个单独的StatefulSet

• MySQL Master节点始终将replicas设置为1
• MySQL Slave节点replicas可以设置为1或更多

mysql-master.yaml

---
# MySQL Master Service
apiVersion: v1
kind: Service
metadata:
  name: mysql-master
spec:
  type: ClusterIP  # 可以提供MySQL读写服务
  ports:
  - name: mysql
    port: 3306
    protocol: TCP
    targetPort: 3306
  selector:
    app: mysql-master

---
# MySQL Master StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-master
spec:
  replicas: 1  # 必须是1
  selector:
    matchLabels:
      app: mysql-master
  serviceName: mysql-master
  template:
    metadata:
      labels:
        app: mysql-master
    spec:
      containers:
        - name: mysql
          image: mysql:latest
          imagePullPolicy: IfNotPresent
          livenessProbe:
            exec:
              command: ["mysqladmin", "-uroot", "-p$(MYSQL_ROOT_PASSWORD)", "ping"]
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 3
          readinessProbe:
            exec:
              command: ["mysql", "-uroot", "-p$(MYSQL_ROOT_PASSWORD)", "-e", "SELECT 1"]
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 3
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: "root"
          ports:
            - containerPort: 3306
              name: mysql
              protocol: TCP
          volumeMounts:
            - name: mysql-master-data
              mountPath: /var/lib/mysql  # 持久化数据库
            - name: mysql-conf
              mountPath: /etc/mysql/conf.d/  # mysql启动配置文件
            - name: mysql-init
              mountPath: /docker-entrypoint-initdb.d/  # master初始化sql语句
      volumes:
        - name: mysql-conf
          configMap:
            name: mysql-conf
            items:
              - key: mysql-master.cnf
                path: mysql-master.cnf
        - name: mysql-init
          configMap:
            name: mysql-conf
            items:
              - key: init-master.sql
                path: init.sql
 
  volumeClaimTemplates:  # NFS提供持久化存储
    - metadata:
        name: mysql-master-data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: NFS

mysql-salve.yaml

---
# MySQL Slave Service
apiVersion: v1
kind: Service
metadata:
  name: mysql-slave
spec:
  type: ClusterIP   # 提供MySQL只读服务
  ports:
  - name: mysql
    port: 3306
    protocol: TCP
    targetPort: 3306
  selector:
    app: mysql-slave

---
# MySQL StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-slave
spec:
  replicas: 3  # 可以是1个或更多
  selector:
    matchLabels:
      app: mysql-slave
  serviceName: mysql-slave  # 必须指定service
  template:
    metadata:
      labels:
        app: mysql-slave
    spec:
      initContainers:  # 用于生成slave server-id,确保多个slave server-id不重复
        - name: init-mysql
          image: mysql:latest
          imagePullPolicy: IfNotPresent
          command:
            - bash
            - "-c"
            - |
              set -ex

              # 从config-map中复制配置文件到emptyDir，因为config-map目录为只读
              cp /etc/mysql-slave-conf/mysql-slave.cnf /etc/conf.d/mysql-slave.cnf

              # 从Pod序号生成server-id, =~ 判断字符串包含关系，BASH_REMATCH变量存储匹配结果
              [[ `hostname` =~ -([0-9]+)$ ]] || exit 1
              pod_number=${BASH_REMATCH[1]}
              echo -e "\nserver-id=$((100 + pod_number))" >> /etc/conf.d/mysql-slave.cnf

          volumeMounts:
            - name: server-conf
              mountPath: /etc/conf.d/
            - name: mysql-slave-conf
              mountPath: /etc/mysql-slave-conf/
      containers:
        - name: mysql
          image: mysql:latest
          imagePullPolicy: IfNotPresent
          env:
            - name: MYSQL_ROOT_PASSWORD
              value: "root"
            - name: REPL_USERNAME
              value: "repl"
            - name: REPL_PASSWORD
              value: "123456"
            - name: mysql-slave-reset
              mountPath: /reset-slave.sh
              subPath: reset-slave.sh
          lifecycle:
            postStart:
              exec:
                command: ["/bin/sh","-c","if [ -f '/var/lib/mysql/mysql-bin.index' ]; then sh /reset-slave.sh; fi"]  # 通过判断mysql-bin.index文件是否存在，来确定Pod不是初次启动，从而执行reset-slave.sh脚本，否则不执行。
          livenessProbe:
            exec:
              command: ["mysqladmin", "-uroot", "-p$(MYSQL_ROOT_PASSWORD)", "ping"]
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 3
          readinessProbe:
            exec:
              command: ["mysql", "-uroot", "-p$(MYSQL_ROOT_PASSWORD)", "-e", "SELECT 1"]
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 3
          ports:
            - containerPort: 3306
              name: mysql
              protocol: TCP
          volumeMounts:
            - name: mysql-slave-data
              mountPath: /var/lib/mysql
            - name: server-conf
              mountPath: /etc/mysql/conf.d/
            - name: mysql-slave-init
              mountPath: /docker-entrypoint-initdb.d/
            - name: mysql-slave-reset
              mountPath: /reset-slave.sh
              subPath: reset-slave.sh   # 以单个文件形式挂载
      volumes:
        - name: server-conf
          emptyDir: {}
        - name: mysql-slave-conf
          configMap:
            name: mysql-conf
            items:
              - key: mysql-slave.cnf
                path: mysql-slave.cnf
        - name: mysql-slave-init
          configMap:
            name: mysql-conf
            items:
              - key: init-slave.sql
                path: init-slave.sql
        - name: mysql-slave-reset
          configMap:
            name: mysql-conf
            defaultMode: 0777    # 默认权限为0644，需要增加执行权限
            items:
              - key: reset-slave.sh
                path: reset-slave.sh

  volumeClaimTemplates:
    - metadata:
        name: mysql-slave-data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: NFS