helm安装mysql

导航：

一、前提
1.版本信息-2.查看方法-3.查看k8s集群节点状态-4.配置docker仓库下载加速
二、安装部署helm
1.下载软件包-2.安装helm-3.初始化
三、使用helm安装mysql
1.搜索mysql的helm包-2.准备后端存储-3.准备docker image-4.配置集群内的kube-dns-5.安装mysql的helm包-6.查看mysql的有状态应用、pod、svc、pvc等k8s资源-7.客户端验证
四、故障排除
五、参考资料

一、前提

使用helm安装有状态应用mysql主备集群，首先确保你已经搭建好一个3节点的k8s集群(如何搭建k8s集群、k8s集群网络配置flannel请参考百度)。

1.版本信息

k8s: v1.5.2
docker: 1.13.1
flannel:0.7.1
helm: v3.0.0

2.查看方法

kubectl version
docker version
flanneld --version
helm version

3.查看k8s集群节点状态

[root@master ~]# kubectl get node
NAME      STATUS    AGE
master    Ready     4d
node2     Ready     18h
node3     Ready     4m
[root@master ~]# 

4.配置docker仓库下载加速

使用163的国内docker仓库，提高docker image的下载速度。

[root@master ~]# cat /etc/docker/daemon.json 
{"registry-mirrors": ["http://hub-mirror.c.163.com"]}
[root@master ~]# systemctl restart docker   #重启docker服务生效
[root@master ~]# 

二、安装部署helm

使用helm的v3版本，v3版本不用配置服务端组件 Tiller。

1.下载软件包

下载helm的二进制包。

下载地址：
https://space.dingtalk.com/s/gwHOAF91cwLOABfU0gPaACBkNjc5YmMwMDUzZmQ0YWQwYjg2MmNhZWRiMGU3YmU1ZA 密码: 5kEK

2.安装helm

使用xshell或者winscp将安装包传输到linux中。

[root@master ~]# tar -tvf helm-v3.0.0-alpha.1-linux-amd64.tar.gz     #查看压缩包中的内容
drwxr-xr-x circleci/circleci 0 2019-05-16 05:44 linux-amd64/
-rwxr-xr-x circleci/circleci 39771936 2019-05-16 05:42 linux-amd64/helm
-rw-r--r-- circleci/circleci    11373 2019-05-16 05:44 linux-amd64/LICENSE
-rw-r--r-- circleci/circleci     3159 2019-05-16 05:44 linux-amd64/README.md
[root@master ~]# 

将二进制文件复制到$PATH路径下即可使用。

[root@master ~]# which helm      #查看helm二进制文件所在的路径，将helm复制到$PATH中即可直接使用，例如将helm放在/usr/bin/helm的路径
/usr/bin/helm
[root@master ~]# helm --help
The Kubernetes package manager

To begin working with Helm, run the 'helm init' command:

    $ helm init

This will set up any necessary local configuration.

Common actions from this point include:

- helm search:    search for charts
- helm fetch:     download a chart to your local directory to view
- helm install:   upload the chart to Kubernetes
- helm list:      list releases of charts

Environment:
  $HELM_HOME          set an alternative location for Helm files. By default, these are stored in ~/.helm
  $HELM_DRIVER        set the backend storage driver. Values are: configmap, secret, memory
  $HELM_NO_PLUGINS    disable plugins. Set HELM_NO_PLUGINS=1 to disable plugins.
  $KUBECONFIG         set an alternative Kubernetes configuration file (default "~/.kube/config")

Usage:
  helm [command]

Available Commands:
  chart       push, pull, tag, or remove Helm charts
  completion  Generate autocompletions script for the specified shell (bash or zsh)
  create      create a new chart with the given name
  dependency  manage a chart's dependencies
  get         download a named release
  help        Help about any command
  history     fetch release history
  home        displays the location of HELM_HOME
  init        initialize Helm client
  install     install a chart
  lint        examines a chart for possible issues
  list        list releases
  package     package a chart directory into a chart archive
  plugin      add, list, or remove Helm plugins
  pull        download a chart from a repository and (optionally) unpack it in local directory
  registry    login to or logout from a registry
  repo        add, list, remove, update, and index chart repositories
  rollback    roll back a release to a previous revision
  search      search for a keyword in charts
  show        inspect a chart
  status      displays the status of the named release
  template    locally render templates
  test        test a release or cleanup test artifacts
  uninstall   uninstall a release
  upgrade     upgrade a release
  verify      verify that a chart at the given path has been signed and is valid
  version     print the client version information

Flags:
      --debug                 enable verbose output
  -h, --help                  help for helm
      --home string           location of your Helm config. Overrides $HELM_HOME (default "/root/.helm")
      --kube-context string   name of the kubeconfig context to use
      --kubeconfig string     path to the kubeconfig file
  -n, --namespace string      namespace scope for this request

Use "helm [command] --help" for more information about a command.
[root@master ~]# 

3.初始化

使用helm前，需要初始化。

[root@docker repository]# helm init --stable-repo-url=https://apphub.aliyuncs.com     #配置helm使用的仓库为国内阿里云地址，国外的访问不了，而且比较慢
Creating /root/.helm/repository/repositories.yaml 
Adding stable repo with URL: https://apphub.aliyuncs.com 
$HELM_HOME has been configured at /root/.helm.
Happy Helming!
[root@docker repository]# ls
cache  local  repositories.yaml
[root@docker repository]# cat repositories.yaml     #helm初始化完后会有一个配置文件生成，下面是repositories.yaml的内容
apiVersion: v1
generated: "2019-08-25T07:08:38.917953914+08:00"
repositories:
- caFile: ""
  cache: /root/.helm/repository/cache/stable-index.yaml
  certFile: ""
  keyFile: ""
  name: stable
  password: ""
  url: https://apphub.aliyuncs.com
  username: ""

三、使用helm安装mysql

使用helm安装有状态应用mysql，使用到的docker镜像有：

docker.io/mysql:latest
docker.io/ist0ne/xtrabackup:latest

构建mysql的1主2从，主备集群，使用xtrabackup工具进行数据同步。

1.搜索mysql的helm包

[root@docker ~]# helm search mysql     #搜索mysql的helm包
NAME                                CHART VERSION   APP VERSION DESCRIPTION                                       
apphub/mysql                        6.3.1           8.0.17      Chart to create a Highly available MySQL cluster  
apphub/mysqldump                    2.6.0           2.4.1       A Helm chart to help backup MySQL databases usi...
apphub/mysqlha                      1.0.0           5.7.13      MySQL cluster with a single master and zero or ...
apphub/prometheus-mysql-exporter    0.5.1           v0.11.0     A Helm chart for prometheus mysql exporter with...
stable/mysql                        6.3.1           8.0.17      Chart to create a Highly available MySQL cluster  
stable/mysqldump                    2.6.0           2.4.1       A Helm chart to help backup MySQL databases usi...
stable/mysqlha                      1.0.0           5.7.13      MySQL cluster with a single master and zero or ...
stable/prometheus-mysql-exporter    0.5.1           v0.11.0     A Helm chart for prometheus mysql exporter with...
apphub/percona                      1.1.0           5.7.17      free, fully compatible, enhanced, open source d...
apphub/percona-xtradb-cluster       1.0.1           5.7.19      free, fully compatible, enhanced, open source d...
apphub/phpmyadmin                   3.0.1           4.9.0-1     phpMyAdmin is an mysql administration frontend    
stable/percona                      1.1.0           5.7.17      free, fully compatible, enhanced, open source d...
stable/percona-xtradb-cluster       1.0.1           5.7.19      free, fully compatible, enhanced, open source d...
stable/phpmyadmin                   3.0.1           4.9.0-1     phpMyAdmin is an mysql administration frontend    
apphub/mariadb                      6.8.1           10.3.17     Fast, reliable, scalable, and easy to use open-...
apphub/mariadb-galera               0.2.1           10.3.17     MariaDB Galera is a multi-master database clust...
stable/mariadb                      6.8.1           10.3.17     Fast, reliable, scalable, and easy to use open-...
stable/mariadb-galera               0.2.1           10.3.17     MariaDB Galera is a multi-master database clust...

2.准备后端存储

由于mysql的helm包需要使用pv，而开源的k8s集群默认没有配置pv后端存储，因此需要手动添加nfs服务器(如何搭建nfs服务器请参考百度)为后端存储。
nfs配置：

[root@master ~]# cat /etc/exports
/nfsdata/pv001  *(rw,no_root_squash)   #添加no_root_squash不压缩root权限，因为容器里会使用root权限写入数据，否则后面会有权限不足的报错
/nfsdata/pv002  *(rw,no_root_squash)
/nfsdata/pv003  *(rw,no_root_squash)
[root@master ~]# ls -ld /nfsdata/      #配置目录的权限为nfsnobody
drwxr-xr-x 5 nfsnobody nfsnobody 45 Aug 26 01:18 /nfsdata/
[root@master ~]# ls -ll /nfsdata/
total 0
drwxr-xr-x 2 nfsnobody nfsnobody 6 Aug 26 01:18 pv001
drwxr-xr-x 2 nfsnobody nfsnobody 6 Aug 26 01:18 pv002
drwxr-xr-x 2 nfsnobody nfsnobody 6 Aug 26 01:18 pv003
[root@master ~]# 

创建pv的yaml文件：

[root@master hrr]# cat pv.yml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv003
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce   #accessModes需要和pvc的模式一样
  nfs:
    path: /nfsdata/pv003
    server: 192.168.222.129
[root@master hrr]# 

创建pv:

[root@docker hrr]# kubectl create -f pv.yml 
persistentvolume "pv003" created
[root@docker hrr]#

查看pv：

[root@master hrr]# kubectl get pv    #Available表示没有被PVC使用
NAME      CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS      CLAIM     REASON    AGE
pv001     10Gi       RWO           Retain          Available                       26s
pv002     10Gi       RWO           Retain          Available                       14s
pv003     10Gi       RWO           Retain          Available                       3s
[root@master hrr]#
[root@master hrr]# kubectl get pv     #Bound表示被PVC使用
NAME      CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                          REASON    AGE
pv001     10Gi       RWO           Retain          Bound     default/data-mysql-mysqlha-0             32s
pv002     10Gi       RWO           Retain          Bound     default/data-mysql-mysqlha-1             17s
pv003     10Gi       RWO           Retain          Bound     default/data-mysql-mysqlha-2             4s
[root@master hrr]#

3.准备docker image

在安装mysql的helm包时需要mysql与xtrabackup的docker镜像。

[root@docker docker]# docker pull docker.io/ist0ne/xtrabackup   #下载xtrabackup镜像
Using default tag: latest
Trying to pull repository docker.io/ist0ne/xtrabackup ... 
latest: Pulling from docker.io/ist0ne/xtrabackup
386a066cd84a: Pull complete 
40f175e652e1: Pull complete 
5eb6b5905d55: Pull complete 
Digest: sha256:e5e5519b874363f1bf35541cef2caed5803645937e29834817be51b30aabc479
Status: Downloaded newer image for docker.io/ist0ne/xtrabackup:latest
[root@docker docker]# 
[root@docker docker]# docker pull docker.io/mysql   #下载mysql镜像

查看本地的镜像：

[root@master hrr]# docker images   
REPOSITORY                                                  TAG                 IMAGE ID            CREATED             SIZE
192.168.222.129:5000/nginx                                  latest              5a3221f0137b        9 days ago          126 MB
docker.io/nginx                                             latest              5a3221f0137b        9 days ago          126 MB
docker.io/mysql                                             latest              62a9f311b99c        11 days ago         445 MB
mysql                                                       5.7.13              62a9f311b99c        11 days ago         445 MB
docker.io/ist0ne/xtrabackup                                 latest              c415dbd7af07        2 years ago         265 MB
registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup   1.0                 c415dbd7af07        2 years ago         265 MB
192.168.222.129:5000/pod-infrastructure                     latest              f9d5de079539        5 years ago         240 kB
docker.io/kubernetes/pause                                  latest              f9d5de079539        5 years ago         240 kB
[root@master hrr]#

将镜像打tag，用于后面创建mysql的helm包：

[root@docker hrr]# docker tag docker.io/mysql:latest mysql:5.7.13
[root@docker docker]# docker tag docker.io/ist0ne/xtrabackup:latest registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0

将docker镜像导出，用于导入到其它k8s节点中：

[root@master ~]# docker save mysql:5.7.13 -o mysql5713.tgz  #将docker镜像导出，名为mysql5713.tgz
[root@master ~]# docker save registry.cn-hangzhou.aliyuncs.com/kubeapps/gcr-xtrabackup:1.0 -o xtrabackup10.tgz 
[root@master ~]# scp mysql5713.tgz xtrabackup10.tgz [email protected]:
[email protected]'s password: 
mysql5713.tgz                                                                                      100%  429MB  35.7MB/s   00:12    
xtrabackup10.tgz                                                                                   100%  260MB  43.4MB/s   00:06    
[root@master ~]#

在其它k8s节点中导入docker镜像：

[root@node2 ~]# docker load -i  mysql5713.tgz   #将导出的docker镜像mysql5713.tgz导入
8fa655db5360: Loading layer [==================================================>] 58.48 MB/58.48 MB
f5741d086b76: Loading layer [============&#