【实战】使用Helm在K8S集群安装MySQL主从

前言

K8S对于云原生部署有着至关重要的作用，几乎所有的应用和中间件都可以部署在K8S，让其帮助我们进行管理。一般情况下我们推荐数据库不使用容器化部署，但是在某些场合为了方便管理也可以采用该种方式。今天我们就用在K8S集群部署MySQL，仅当作学习学习，在生产环境谨慎使用。

技术积累

什么是Helm

Helm是K8S中包管理工具，包管理器类似于我们在 Ubuntu 中使用的apt、Centos中使用的yum一样，能快速查找、下载和安装软件包， 能够将一组K8S资源打包统一管理, 是查找、共享和使用为Kubernetes构建的软件的最佳方式。具体介绍和安装使用参见之前博文： Kubernetes包管理工具Helm简介及使用

StorageClass

由于数据库需要使用存储资源，K8S集群中我们可手动创建pv、pvc,StorageClass工具则可以自动车床件pvc。具体简介和使用参见之前博文：【实战】Kubernetes安装持久化工具NFS-StorageClass

使用的工具版本

软件	版本
chart	9.12.0
mysql	8.0.34
kubernetes	version v1.27.2
helm	version v3.12.1

helm 安装 MySQL 1主2从

1. 添加 bitnami 的仓库

$ helm repo add bitnami https://charts.bitnami.com/bitnami

2. 查询 MySQL 资源

$ helm repo update

[root@master k8s]# helm search repo mysql
NAME CHART VERSION APP VERSION DESCRIPTION
bitnami/mysql 9.12.1 8.0.34 MySQL is a fast, reliable, scalable, and easy t…
bitnami/phpmyadmin 12.1.0 5.2.1 phpMyAdmin is a free software tool written in P…
bitnami/mariadb 13.1.2 11.0.3 MariaDB is an open source, community-developed …
bitnami/mariadb-galera 9.1.1 11.0.3 MariaDB Galera is a multi-primary database clus…

3. 拉取 MySQL chart 到本地

$ mkdir -p /k8s/mysql && cd /k8s/mysql
#拉取 chart 到本地 /root/mysql 目录
$ helm pull bitnami/mysql --version 9.12.1
$ tar -xvf mysql-9.12.1.tgz
$ cp mysql/values.yaml ./values-test.yaml

#查看当前目录层级
[root@master mysql]# tree -L 2
.
├── mysql
│ ├── Chart.lock
│ ├── charts
│ ├── Chart.yaml
│ ├── README.md
│ ├── templates
│ ├── values.schema.json
│ └── values.yaml
├── mysql-9.12.1.tgz
└── values-test.yaml

3 directories, 7 files

4. 对chart 本地 values-test.yaml 修改

查看集群 storageclasses
[root@master mysql]# kubectl get storageclass

NAME                    PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-storage (default)   nfs-provisioner   Delete          Immediate           false                  10d

修改配置
$ vim values-test.yaml

#镜像修改
image:
  registry: registry.cn-hangzhou.aliyuncs.com
  repository: senfel/mysql
  tag: 8.0.34-debian-11-r31
#架构改为集群复制
architecture: replication
#root密码 主从复制密码
auth:
  rootPassword: "123456root"

#主节点
primary:
  configuration: |-
    [mysqld]
    default_authentication_plugin=caching_sha2_password   ## 加密方式

  persistence: ## 储存方式
    storageClass: "nfs-storage"
#从节点副本数
secondary:
  replicaCount: 2
#从节点
secondary:
  configuration: |-
    [mysqld]
    default_authentication_plugin=caching_sha2_password  ## 加密方式
    read_only=1 ##限定普通用户只读
    super_read_only=on ##限定root只读

  persistence: ##储存方式
    storageClass: "nfs-storage"

5. 对本地 templates 模板 修改

mysql5会有以下问题，mysql8无需处理
[root@master mysql]# kubectl api-versions

# k8s v1.27.2 版本中，statefulsets 版本为：apps/v1  而不是模板中 apps/v1beta1 的版本
# 因此需要替换 master-statefulset.yaml 和 slave-statefulset.yaml 模板
$ sed -i "s#apps/v1beta1#apps/v1#" mysql/templates/master-statefulset.yaml
$ sed -i "s#apps/v1beta1#apps/v1#" mysql/templates/slave-statefulset.yaml

未替换模板中 statefulsets 版本，直接安装服务会报错
$ helm install mysql-cluster mysql -f values-test.yaml 
Error: INSTALLATION FAILED: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "StatefulSet" in version "apps/v1beta1"

6. 安装 MySQL 集群

#创建 mysql-cluster 名称空间
$ kubectl create ns mysql-cluster

#安装 MySQL 集群
$ helm install mysql-cluster -n mysql-cluster mysql -f values-test.yaml

##helm -n NAMESAPCE install SERVER_NAME FILE_NAME -f CONFIG_FILE
-n 指定 kubernetes 集群名称空间
-f 指定使用的配置文件，文件中定义的配置可以覆盖 mysql/values.yaml 文件中配置

[root@master mysql]# helm install mysql-cluster -n mysql-cluster mysql -f values-test.yaml

NAME: mysql-cluster
LAST DEPLOYED: Mon Sep  4 15:09:28 2023
NAMESPACE: mysql-cluster
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: mysql
CHART VERSION: 9.12.1
APP VERSION: 8.0.34

** Please be patient while the chart is being deployed **

Tip:

  Watch the deployment status using the command: kubectl get pods -w --namespace mysql-cluster

Services:

  echo Primary: mysql-cluster-primary.mysql-cluster.svc.cluster.local:3306
  echo Secondary: mysql-cluster-secondary.mysql-cluster.svc.cluster.local:3306

Execute the following to get the administrator credentials:

  echo Username: root
  MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace mysql-cluster mysql-cluster -o jsonpath="{.data.mysql-root-password}" | base64 -d)

To connect to your database:

  1. Run a pod that you can use as a client:

      kubectl run mysql-cluster-client --rm --tty -i --restart='Never' --image  registry.cn-hangzhou.aliyuncs.com/senfel/mysql:8.0.34-debian-11-r31 --namespace mysql-cluster --env MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD --command -- bash

  2. To connect to primary service (read/write):

      mysql -h mysql-cluster-primary.mysql-cluster.svc.cluster.local -uroot -p"$MYSQL_ROOT_PASSWORD"

  3. To connect to secondary service (read-only):

      mysql -h mysql-cluster-secondary.mysql-cluster.svc.cluster.local -uroot -p"$MYSQL_ROOT_PASSWORD"

7. 查看部署的 MySQL 集群

[root@master mysql]# kubectl get svc,pods -n mysql-cluster

NAME                                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/mysql-cluster-primary              ClusterIP   10.102.253.38   <none>        3306/TCP   2m2s
service/mysql-cluster-primary-headless     ClusterIP   None            <none>        3306/TCP   2m2s
service/mysql-cluster-secondary            ClusterIP   10.102.8.228    <none>        3306/TCP   2m2s
service/mysql-cluster-secondary-headless   ClusterIP   None            <none>        3306/TCP   2m2s

NAME                            READY   STATUS    RESTARTS   AGE
pod/mysql-cluster-primary-0     1/1     Running   0          2m2s
pod/mysql-cluster-secondary-0   1/1     Running   0          2m2s
pod/mysql-cluster-secondary-1   1/1     Running   0          80s

#查看 pvc
[root@master mysql]# kubectl get pvc -n mysql-cluster

NAME                             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-mysql-cluster-primary-0     Bound    pvc-1c32a6b6-4b22-4ce0-b45b-82c38fd0d107   8Gi        RWO            nfs-storage    4m12s
data-mysql-cluster-secondary-0   Bound    pvc-5d263bc4-c9d4-4f3d-951c-fd578cad6e32   8Gi        RWO            nfs-storage    4m12s
data-mysql-cluster-secondary-1   Bound    pvc-90072614-6cbb-48f2-8cdf-a8816557e46b   8Gi        RWO            nfs-storage    3m30s

#查看 pv
[root@master mysql]# kubectl get pv | grep mysql

pvc-1c32a6b6-4b22-4ce0-b45b-82c38fd0d107   8Gi        RWO            Delete           Bound         mysql-cluster/data-mysql-cluster-primary-0        nfs-storage             4m45s
pvc-5d263bc4-c9d4-4f3d-951c-fd578cad6e32   8Gi        RWO            Delete           Bound         mysql-cluster/data-mysql-cluster-secondary-0      nfs-storage             4m45s
pvc-90072614-6cbb-48f2-8cdf-a8816557e46b   8Gi        RWO            Delete           Bound         mysql-cluster/data-mysql-cluster-secondary-1      nfs-storage             4m3s
pvc-a5673e1b-e3b2-415f-b2bb-c649eb4cd33d   8Gi        RWO            Delete           Bound         default/data-mysql-cluster-secondary-0            nfs-storage             32m
pvc-a683be89-289c-4088-b46f-f2af4d322a1a   8Gi        RWO            Delete           Bound         default/data-mysql-cluster-primary-0              nfs-storage             32m

8. 连接 MySQL 集群 验证服务

#启动一个临时容器
$ kubectl run mysql-cluster-client --rm --tty -i --restart=‘Never’ --image registry.cn-hangzhou.aliyuncs.com/senfel/mysql:8.0.34-debian-11-r31 --namespace mysql-cluster --env MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD --command – bash

##登陆 MySQL Master节点
$ mysql -h mysql-cluster-primary.mysql-cluster.svc.cluster.local -uroot -p

Enter password: 123456root

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| my_database        |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.01 sec)

#查看主从状态
#查看File和Position的值，在从库配置中会显示。

mysql> show master status\G

*************************** 1. row ***************************
             File: mysql-bin.000003
         Position: 157
     Binlog_Do_DB:
 Binlog_Ignore_DB:
Executed_Gtid_Set:
1 row in set (0.00 sec)

##登陆从库，查看主从同步状态
$ mysql -h mysql-cluster-secondary.mysql-cluster.svc.cluster.local -uroot -p
Enter password: 123456root

mysql> show slave status\G;

*************************** 1. row ***************************
               Slave_IO_State: Waiting for source to send event
                  Master_Host: mysql-cluster-primary
                  Master_User: replicator
                  Master_Port: 3306
                Connect_Retry: 10
              Master_Log_File: mysql-bin.000003
          Read_Master_Log_Pos: 157
               Relay_Log_File: mysql-relay-bin.000006
                Relay_Log_Pos: 373
        Relay_Master_Log_File: mysql-bin.000003
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 157
              Relay_Log_Space: 799
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 313
                  Master_UUID: ff169bc1-4af1-11ee-b795-f6df04eaa47a
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
       Master_public_key_path:
        Get_master_public_key: 0
            Network_Namespace:
1 row in set, 1 warning (0.00 sec)

至此，使用Helm在K8S集群安装MySQL主从实战演示完成。

ingress-nginx开放主从端口

ingress-nginx是K8S开放端口的工具，可以直接将端口映射到集群外部。
#修改ingress-nginx tcp配置
$ vim values.yaml

#格式 "ns":"svc:port"
tcp: {
  "56379":"redis-cluster/redis-cluster:6379",
  "53306": "mysql-cluster/mysql-cluster-primary:3306",
  "54406": "mysql-cluster/mysql-cluster-secondary:3306"
}

#升级releas
$ helm upgrade ingress-nginx -n ingress-nginx .

#验证服务器端口是否开启
[root@node2 ~]# netstat -nplt | grep 06

tcp        0      0 0.0.0.0:53306           0.0.0.0:*               LISTEN      16944/nginx: master
tcp        0      0 0.0.0.0:54406           0.0.0.0:*               LISTEN      16944/nginx: master
tcp6       0      0 :::53306                :::*                    LISTEN      16944/nginx: master
tcp6       0      0 :::54406                :::*                    LISTEN      16944/nginx: master

查看最终的nginx.conf
#获取ingress-nginx pods
[root@node2 ~]# kubectl get pods -n ingress-nginx

NAME                             READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-75x86   1/1     Running   0          8m51s
ingress-nginx-controller-jgzks   1/1     Running   0          8m29s

#查看最终的nginx.conf
[root@master ingress-nginx]# kubectl exec ingress-nginx-controller-75x86 -n ingress-nginx – cat /etc/nginx/nginx.conf

至此TCP端口已暴露完成，对于UDP也是一样的修改value.yaml配置即可

外网navicat连接验证

两台ingress-nginx服务物理机都开放了端口，都可以连接！！！