mysql+MHA+keepalived高可用方案

关于基础介绍及原理略过，直接上操作步骤吧。
mysql：5.7
操作系统：centos7.4

地址规划：
MHA manage：10.21.20.89
mysql主：10.21.20.71
mysql从1：10.21.20.131
mysql从2：10.21.20.132
vip：10.21.20.77

1. 安装好mysql及主从同步建立以后，先在主库上创建监听用户。

     create user 'admin'@'%' identified by 'admin';
     grant all on *.* to 'admin'@'10.21.20.%' identified by 'admin';
   

   在各个主机上建立互信，为了方便直接一路回车到底。然后将私钥文件及authorized_keys文件复制给其他主机。四台服务器都要执行。

    # ssh-keygen -t rsa
       Generating public/private rsa key pair.
        Enter file in which to save the key (/root/.ssh/id_rsa): 
        Enter passphrase (empty for no passphrase): 
        Enter same passphrase again: 
        Your identification has been saved in /root/.ssh/id_rsa.
        Your public key has been saved in /root/.ssh/id_rsa.pub.
        The key fingerprint is:
        SHA256:wyMb8WZ6yJ0jxXN1FgWOwbVAPwW/GJRaKHfaefvXak0                   
        The key's randomart image is:
        +---[RSA 2048]----+
         |           o=o*+o|
         |         . ooX = |
         |      .   o O.O .|
         |       =   + =oo.|
         |      o S .  ....|
         |     . @ *     .E|
         |      * =      oo|
         |       o .    . =|
         |             ....|
        +----[SHA256]-----+
    # ssh-copy-id -i .ssh/id_rsa.pub [email protected]
    # ssh-copy-id -i .ssh/id_rsa.pub [email protected]
    # ssh-copy-id -i .ssh/id_rsa.pub [email protected]
   

2. 安装MHA
   首先下载mha4mysql-manager、mha4mysql-node。

   # wget https://github.com/yoshinorim/mha4mysql-manager/releases/download/v0.58/mha4mysql-manager-0.58-0.el7.centos.noarch.rpm
   # wget https://github.com/yoshinorim/mha4mysql-node/releases/download/v0.58/mha4mysql-node-0.58-0.el7.centos.noarch.rpm
   

   由于我的安装环境没有公网，因此我将下载好的MHA安装包及依赖包做成了本地yum源。具体方法如下：
   （1）找到一台可以连接公网的服务器，利用yum工具将依赖包下载到服务器。做成yum源以后，打包上传到安装的服务器。

   # yum install yum-utils -y
   # yumdownloader mha4mysql-manager mha4mysql-node --resolve --destdir=/home/mysql/MHA/mharepo
   # yum install -y createrepo
   

   (2) 将mha4mysql-manager、mha4mysql-node复制到/home/mysql/MHA/mharepo目录下，然后创建yum源。

   #  createrepo -pdo /home/mysql/MHA/mharepo
   #  cd /home/mysql/MHA/
   #  tar -czvf mharepo.tar.gz mharepo

（3）将mharepo.tar.gz上传四台需要安装的服务器解压。然后新建一个yum文件，指向解压的目录。

    # cat /etc/yum.repos.d/abc.repo 
       [abc]
       name=test
       baseurl=file:///root/mharepo
       gpgcheck=0
       enabled=1

(4) 安装MHA，在3台mysql上安装node，在管理节点安装manager。

 # yum install mha4mysql-manager -y
 # yum install mha4mysql-node -y

3. 在管理节点编辑MHA的配置文件，并启动MHA manager

     # cat conf/mha.cnf 
        [server default]
        manager_workdir=/home/mysql/MHA
        manager_log=/home/mysql/MHA/log/manager.log
        user=admin
        password=admin
        ssh_user=mysql
        ssh_port=22
        repl_user=slave
        repl_password=abc123
        ping_interval=1
        shutdown_script=""
        master_ip_online_change_script=""
        report_script=""
        master_ip_failover_script="/home/mysql/MHA/scripts/master_ip_failover"
   
        [server1]
        hostname=10.21.20.71
        port=13306
        candidate_master=1
        master_binlog_dir="/home/mysql/mysql/logs/"
   
   
        [server2]
        hostname=10.21.20.131
        port=13306
        candidate_master=1
        master_binlog_dir="/home/mysql/mysql/logs/"
   
        [server3]
        hostname=10.21.20.132
        port=13306
   

   测试ssh及集群状态

    # masterha_check_ssh --conf=/home/mysql/MHA/conf/mha.cnf
    # masterha_check_repl --conf=/home/mysql/MHA/conf/mha.cnf
                      忽略部分内容
       10.21.20.71(10.21.20.71:13306) (current master)
       +--10.21.20.131(10.21.20.131:13306)
       +--10.21.20.132(10.21.20.132:13306)
   
       Sat Apr 11 11:45:12 2020 - [info] Checking replication health on      10.21.20.131..
       Sat Apr 11 11:45:12 2020 - [info]  ok.
       Sat Apr 11 11:45:12 2020 - [info] Checking replication health on     10.21.20.132..
       Sat Apr 11 11:45:12 2020 - [info]  ok.
       Sat Apr 11 11:45:12 2020 - [warning] master_ip_failover_script is not defined.
       Sat Apr 11 11:45:12 2020 - [warning] shutdown_script is not defined.
       Sat Apr 11 11:45:12 2020 - [info] Got exit code 0 (Not master dead).
   
       MySQL Replication Health is OK.
       
   

         # ln -s /home/migumysql/mysql/bin/mysql /usr/local/bin/mysql
         # ln -s /home/migumysql/mysql/bin/mysqlbinlog /usr/local/bin/mysqlbinlog
   

   直接nohup masterha_manager -conf=conf/mha.cnf > log/mha-0318.log &启动，当crt关闭时MHA manager可能会自动停止，网上找了下资料，可能是crt的bug，将启动命令写到shell里可以避免此情况。有待验证。

   # cat start_mah.sh 
     #!/bin/bash
     cd `dirname $0`
     nohup masterha_manager  -conf=conf/mha.cnf > log/mha-0410.log &
   
   # ./start_mah.sh
   

4. 在数据库节点安装keepalived（本例中为10.21.20.71、10.21.20.131）。主库的priority要比从库的高。

     # yum install keepalived -y 
     # cat /etc/keepalived/keepalived.conf
         
         global_defs {
             notification_email {
             123456.qq.com
             }
         notification_email_from 123456.qq.com
         smtp_server 127.0.0.1
         smtp_connect_timeout 30
         router_id MYSQL-HA
         script_user root
         enable_script_security
         }
   
         vrrp_script check_mysql_alived {
             script "/etc/keepalived/mysql_monitor.sh" 
             interval 5 
         }
   
         vrrp_instance VI_1 {
            state BACKUP       
            interface eth0
            virtual_router_id 191
            priority 120 
            nopreempt
            advert_int 2
            authentication {
            auth_type PASS
            auth_pass Cmcc
          }
   
          track_script {
            check_mysql_alived
          }
   
          virtual_ipaddress {
               10.21.20.77/24
           }
          }
   
     #
     # cat /etc/keepalived/mysql_monitor.sh 
       #!/bin/bash
       check(){
       /home/migumysql/mysql/bin/mysql -u'root' -p'Migu123!' -S       /home/migumysql/mysql/logs/mysql.sock  -e "select user()" > /dev/null 2>&1
        r1=$?
        r2=$(/usr/sbin/ss -nltp | grep 3306 | wc -l)
        if [[ $r1 -ne 0 ]] || [[ $r2 -eq 0 ]];then
              echo "error"
        else
              echo "ok"
        fi
        }
   
        if [ $(check) = "ok" ];then
             echo "MySQL OK!"
             exit 0
        else
             sleep 2
             if [ $(check) = "error" ];then
                   systemctl  stop keepalived
                   pkill -9 keepalived
                   echo "MySQL Error! Keepalived Stop."
              fi
         fi
   
      #
      #  chmod a+x /etc/keepalived/mysql_monitor.sh 
      #  systemctl start keepalived
   

5. 此时查看vip是否在主库上，从库没有vip。验证正确后，停止主库查看主从是否自动切换，vip是否会漂移到从库。

    #  tailf log/manager.log 
        忽略部分内容
        ----- Failover Report -----
   
        mha: MySQL Master failover 10.21.20.71(10.21.20.71:13306) to         10.21.20.131(10.21.20.131:13306) succeeded
   
        Master 10.21.20.71(10.21.20.71:13306) is down!
   
        Check MHA Manager logs at localhost:/home/migumysql/MHA/log/manager.log for details.
   
        Started automated(non-interactive) failover.
        The latest slave 10.21.20.131(10.21.20.131:13306) has all relay logs for recovery.
        Selected 10.21.20.131(10.21.20.131:13306) as a new master.
        10.21.20.131(10.21.20.131:13306): OK: Applying all logs succeeded.
        10.21.20.132(10.21.20.132:13306): This host has the latest relay log events.
        Generating relay diff files from the latest slave succeeded.
        10.21.20.132(10.21.20.132:13306): OK: Applying all logs succeeded.   Slave started, replicating from 10.21.20.131(10.21.20.131:13306)
        10.21.20.131(10.21.20.131:13306): Resetting slave info succeeded.
        Master failover to 10.21.20.131(10.21.20.131:13306) completed successfully.
   

在主库和从库上观察ip发现，vip已经自动漂移到从库上。至此高可用完成。