MySQL高可用实战(MHA)

MySQL高可用实战(MHA)

1、环境准备(所有节点)

IP	主机名	角色
192.168.54.200	master	mysql master、MHA manager、MHA node
192.168.54.201	slave1	mysql slave1、MHA node
192.168.54.202	slave2	mysql slave2、MHA node

【温馨提示】MHA manager最好是单独一台机器部署，尽量不要混部。

1.1 配置hosts

192.168.54.200 master
192.168.54.201 slave1
192.168.54.202 slave2

1.2 关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

1.3 禁用SELinux

# 永久关闭
sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config
# 临时关闭
setenforce 0

1.4 关闭swap

# 关闭swap主要是为了性能考虑
# 临时关闭
swapoff -a
# 可以通过这个命令查看swap是否关闭了
free
# 永久关闭        
sed -ri 's/.*swap.*/#&/' /etc/fstab

1.5 配置互信

ssh-keygen
ssh-copy-id master
ssh-copy-id slave1
ssh-copy-id slave2

2、mysql主从部署

2.1 安装mysql(MySQL5.7.24)(所有节点)

# 添加mysql-server源
mkdir -p /opt/mysql-mha; cd /opt/mysql-mha
wget https://dev.mysql.com/get/mysql80-community-release-el7-3.noarch.rpm
rpm -ivh mysql80-community-release-el7-3.noarch.rpm
# 解决报错如，Check that the correct key URLs are configured for this repository.
rpm --import http://repo.mysql.com/RPM-GPG-KEY-mysql-2022  

# 更新yum缓存
yum makecache

# 使用yum查看MySQL的仓库，查看MySQL的版本
yum repolist all | grep mysql
# 安装yum-config-manager
yum -y install yum-utils
# 修改为需要的版本，即禁用yum存储库中mysql不需要的版本和开启需要的版本
yum-config-manager --disable mysql80-community
yum-config-manager --enable mysql57-community

# 先禁用本地的 MySQL 模块，要不然找不到mysql-community-server，默认mysql-server是8.0的版本
yum module disable mysql

# 开始安装mysql server和mysql client
yum install mysql-community-server mysql -y

# 可以先下载好安装包然后再进行安装
curl -O https://cdn.mysql.com//Downloads/MySQL-5.7/mysql-5.7.40-1.el7.x86_64.rpm-bundle.tar
# 解压
tar -xvf mysql-5.7.40-1.el7.x86_64.rpm-bundle.tar

# 主要关注以下5个rpm，因为有依赖关系，所以后续安装顺序不能乱：
# 3个依赖包
mysql-community-common-5.7.40-1.el7.x86_64.rpm
mysql-community-libs-5.7.40-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.40-1.el7.x86_64.rpm
# 客户端和服务端
mysql-community-client-5.7.40-1.el7.x86_64.rpm
mysql-community-server-5.7.40-1.el7.x86_64.rpm

# 安装
rpm -ivh mysql-community-common-5.7.40-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-5.7.40-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-compat-5.7.40-1.el7.x86_64.rpm
rpm -ivh mysql-community-client-5.7.40-1.el7.x86_64.rpm
rpm -ivh mysql-community-server-5.7.40-1.el7.x86_64.rpm

# 如果安装过程中报如下错误
file /usr/share/mysql/charsets/swe7.xml from install of mysql-community-common-5.7.40-1.el7.x86_64 conflicts          with file from package mariadb-libs-1:5.5.68-1.el7.x86_64
# 解决方法 
yum -y remove mariadb-libs-1:5.5.68-1.el7.x86_64

2.2 mysql 节点配置(所有节点)

2.2.1 修改配置文件

修改mysql的所有节点mysql的主配置文件 (/etc/my.cnf)

Master节点：

server-id = 1
log-bin=mysql-bin
binlog_format=mixed
log-slave-updates=true

Slave1,Slave2节点：

# 三台节点server-id不可重复
server-id = 2
log_bin=mysql-bin
relay-log=relay-log-bin
relay-log-index=slave-relay-bin.index

server-id = 3
log_bin=mysql-bin
relay-log=relay-log-bin
relay-log-index=slave-relay-bin.index

2.2.2 启动服务

systemctl start mysqld

2.2.3 设置root密码

# 默认密码
grep 'temporary password' /var/log/mysqld.log
mysql -uroot -p

# 重置root密码
set global validate_password_policy=0;
set global validate_password_length=1;
ALTER user 'root'@'localhost' IDENTIFIED BY '123456';

[root@master mysql]# grep 'temporary password' /var/log/mysqld.log
2022-12-21T02:15:57.631409Z 1 [Note] A temporary password is generated for root@localhost: n/S>X#/Za8=u
[root@master mysql]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.40-log

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> set global validate_password_policy=0;
Query OK, 0 rows affected (0.00 sec)

mysql> set global validate_password_length=1;
Query OK, 0 rows affected (0.00 sec)

mysql> ALTER user 'root'@'localhost' IDENTIFIED BY '123456';
Query OK, 0 rows affected (0.01 sec)

mysql> exit
Bye

2.3 配置mysql 一主两从

2.3.1 所有数据库节点进行mysql授权(所有节点)

# 登录客户端
mysql -uroot -p
123456

set global validate_password_policy=0;
set global validate_password_length=1;

# 从库进行同步使用的用户
grant replication slave on *.* to 'myslave'@'192.168.54.%' identified by '123456';

# MHA-manager使用
grant all  on *.* to 'mha'@'192.168.54.%' identified by '123456';

#防止从库通过主机名连接不上主库
grant all on *.* to 'mha'@'master' identified by '123456';
grant all on *.* to 'mha'@'slave1' identified by '123456';
grant all on *.* to 'mha'@'slave2' identified by '123456';

FLUSH PRIVILEGES;

2.3.2 在主库查看二进制文件和偏移量(master节点)

mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000002 |     1989 |              |                  |                   |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

2.3.3 slave1,slave2 执行同步操作(slave1和slave2执行)

mysql> change master to master_host='192.168.54.200', master_user='myslave',master_password='123456', master_log_file='mysql-bin.000002', master_log_pos=1989;
Query OK, 0 rows affected, 2 warnings (0.01 sec)

mysql> start slave;
Query OK, 0 rows affected (0.01 sec)

# 两个slave节点都需要IO线程和SQL线程为yes状态
mysql> show slave status \G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.54.200
                  Master_User: myslave
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000002
          Read_Master_Log_Pos: 1989
               Relay_Log_File: relay-log-bin.000002
                Relay_Log_Pos: 320
        Relay_Master_Log_File: mysql-bin.000002
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 1989
              Relay_Log_Space: 525
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1
                  Master_UUID: 6769ee5f-80d5-11ed-9a11-00505623e2fb
             Master_Info_File: /var/lib/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.01 sec)

mysql>

2.3.4 两个从库都设置为只读模式(slave1和slave2执行)

# 通过全局变量read_only设置,设置值为1,或者on,表示开启;设置值为0或者off,表示关闭
mysql> set global read_only=1;
Query OK, 0 rows affected (0.00 sec)

mysql> show global variables like 'read_only';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| read_only     | ON    |
+---------------+-------+
1 row in set (0.00 sec)

# 【注意】得退出客户端再登录才会生效

3.4 测试主从

# 插入数据测试主从同步
# 在master节点添加数据
create database test01;
create table test01.test(id int);
insert into test01.test values (1);

# 在slave节点查数据,slave1和slave2都能查询到
select * from test01.test;

3、MHA概述

MHA（Master High Availability）是由日本人yoshinorim开发的一款成熟且开源的MySQL高可用程序，它实现了

MySQL主从环境下MASTER宕机后能够自动进行单次故障转移的功能，其本身由perl语言编写，安装方便使用简

单。

MHA官网：https://code.google.com/archive/p/mysql-master-ha/

GitHub地址：https://github.com/yoshinorim/mha4mysql-manager

文档：https://github.com/yoshinorim/mha4mysql-manager/wiki

MHA架构：当一个 master 崩溃时，MHA 会恢复下面的 rest slave。

MHA 组件：MHA 由 MHA Manager 和 MHA Node 组成。

• MHA Manager有监控MySQL master、控制master故障转移等管理程序。

• MHA 节点具有故障转移辅助脚本，例如解析 MySQL 二进制/中继日志，识别中继日志位置，中继日志应从哪

  个位置应用到其他从站，将事件应用到目标从站等。MHA 节点在每个 MySQL 服务器上运行。

• 当 MHA Manager 进行故障转移时，MHA Manager 通过 SSH 连接 MHA Node 并在需要时执行 MHA Node

  命令。

4、安装MHA软件

下载地址：https://github.com/yoshinorim/mha4mysql-manager/wiki/Downloads

4.1 所有节点安装MHA node软件(所有节点)

下载地址：https://github.com/yoshinorim/mha4mysql-node/releases/tag/v0.58

cd /opt/mysql-mha
# 注意,所有节点都需要安装MHA node
# 1、先安装相关依赖
yum install perl-DBD-MySQL -y
# 下载
wget https://github.com/yoshinorim/mha4mysql-node/releases/download/v0.58/mha4mysql-node-0.58-0.el7.centos.noarch.rpm
# 2、安装mha
rpm -ivh mha4mysql-node-0.58-0.el7.centos.noarch.rpm
# 也可以使用下面方式安装
yum install -y mha4mysql-node-0.58-0.el7.centos.noarch.rpm

4.2 安装mha manager(master节点上)

4.2.1 安装mha manager

# 先安装好依赖
yum -y install epel-release
yum -y install perl-Config-Tiny perl-Time-HiRes perl-Parallel-ForkManager perl-Log-Dispatch perl-DBD-MySQL ncftp

# https://github.com/yoshinorim/mha4mysql-manager/releases/tag/v0.58
wget https://github.com/yoshinorim/mha4mysql-manager/releases/download/v0.58/mha4mysql-manager-0.58-0.el7.centos.noarch.rpm

# 安装
rpm -ivh mha4mysql-manager-0.58-0.el7.centos.noarch.rpm

4.2.2 编写master_ip_failover脚本

/opt/mysql-mha/master_ip_failover，下面配置文件中会用到

#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
 
use Getopt::Long;
 
my (
    $command, $orig_master_host, $orig_master_ip,$ssh_user,
    $orig_master_port, $new_master_host, $new_master_ip,$new_master_port,
    $orig_master_ssh_port,$new_master_ssh_port,$new_master_user,$new_master_password
);
 
# 这里定义的虚拟IP配置要注意，这个ip必须要与你自己的集群在同一个网段，否则无效
my $vip = '192.168.54.205/24';
my $key = '1';
# 这里的网卡名称 “ens33” 需要根据你机器的网卡名称进行修改
# 如果多台机器直接的网卡名称不统一，有两种方式，一个是改脚本，二是把网卡名称修改成统一
# 我这边实际情况是修改成统一的网卡名称
my $ssh_start_vip = "sudo /sbin/ifconfig ens33:$key $vip";
my $ssh_stop_vip = "sudo /sbin/ifconfig ens33:$key down";
my $ssh_Bcast_arp= "sudo /sbin/arping -I bond0 -c 3 -A $vip";

GetOptions(
    'command=s'          => \$command,
    'ssh_user=s'         => \$ssh_user,
    'orig_master_host=s' => \$orig_master_host,
    'orig_master_ip=s'   => \$orig_master_ip,
    'orig_master_port=i' => \$orig_master_port,
    'orig_master_ssh_port=i' => \$orig_master_ssh_port,
    'new_master_host=s'  => \$new_master_host,
    'new_master_ip=s'    => \$new_master_ip,
    'new_master_port=i'  => \$new_master_port,
    'new_master_ssh_port' => \$new_master_ssh_port,
    'new_master_user' => \$new_master_user,
    'new_master_password' => \$new_master_password
 
);
 
exit &main();
 
sub main {
    $ssh_user = defined $ssh_user ? $ssh_user : 'root';
    print "\n\nIN SCRIPT TEST====$ssh_user|$ssh_stop_vip==$ssh_user|$ssh_start_vip===\n\n";
 
    if ( $command eq "stop" || $command eq "stopssh" ) {
 
        my $exit_code = 1;
        eval {
            print "Disabling the VIP on old master: $orig_master_host \n";
            &stop_vip();
            $exit_code = 0;
        };
        if ($@) {
            warn "Got Error: $@\n";
            exit $exit_code;
        }
        exit $exit_code;
    }
    elsif ( $command eq "start" ) {
 
        my $exit_code = 10;
        eval {
            print "Enabling the VIP - $vip on the new master - $new_master_host \n";
            &start_vip();
        &start_arp();
            $exit_code = 0;
        };
        if ($@) {
            warn $@;
            exit $exit_code;
        }
        exit $exit_code;
    }
    elsif ( $command eq "status" ) {
        print "Checking the Status of the script.. OK \n";
        exit 0;
    }
    else {
        &usage();
        exit 1;
    }
}
 
sub start_vip() {
    `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
sub stop_vip() {
    `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
 
sub start_arp() {
    `ssh $ssh_user\@$new_master_host \" $ssh_Bcast_arp \"`;
}
sub usage {
    print
    "Usage: master_ip_failover --command=start|stop|stopssh|status --ssh_user=user --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}

给该脚本添加可执行权限：

chmod a+x /opt/mysql-mha/master_ip_failover

4.2.3 配置(manager节点)

#创建相关目录（所有节点）
mkdir -p /opt/mysql-mha/mha-node
# manager节点
mkdir -p /opt/mysql-mha/mha
#编写配置文件
vim /opt/mysql-mha/mysql_mha.cnf
#内容如下:
------------------------------------------------------------------------
[server default]
#mha访问数据库的账号与密码
user=mha
password=123456
port=3306
#指定mha的工作目录
manager_workdir=/opt/mysql-mha/mha
#指定管理日志路径
manager_log=/opt/mysql-mha/manager.log
#指定master节点存放binlog的日志文件的目录 log_bin=mysql_bin默认是在/var/lib/mysql
master_binlog_dir=/var/lib/mysql
#指定mha在远程节点上的工作目录
remote_workdir=/opt/mysql-mha/mha-node
#指定主从复制的mysq用户和密码
repl_user=myslave
repl_password=123456
#指定检测间隔时间
ping_interval=1
#指定一个脚本，该脚本实现了在主从切换之后，将虚拟ip漂移到新的master上
master_ip_failover_script=/opt/mysql-mha/master_ip_failover
#指定检查的从服务器IP地址.有几个，就用-s选项加几个
secondary_check_script=/usr/bin/masterha_secondary_check -s 192.168.54.201 -s 192.168.54.202
#用于故障切换的时候发送邮件提醒
#report_script=/data1/mysql-mha/send_mail
[server1]
hostname=192.168.54.200
port=3306
ssh_user=root
candidate_master=1
check_repl_delay=0
[server2]
hostname=192.168.54.201
port=3306
ssh_user=root
candidate_master=1
check_repl_delay=0
[server3]
hostname=192.168.54.202
port=3306
ssh_user=root
candidate_master=1
check_repl_delay=0

candidate_master=1

设置为候选master，设置该参数以后，发生主从切换以后将会将此从库提升为主库，即使这个从库不是集群

中最新的slave，no_master=1正好相反。

check_repl_delay=0

默认情况下如果一个slave落后master 超过100M的relay logs的话，MHA将不会选择该slave作为一个新的

master， 因为对于这个slave的恢复需要花费很长时间；通过设置check_repl_delay=0，MHA触发切换在选

择一个新的master的时候将会忽略复制延时，这个参数对于设置了candidate_master=1的主机非常有用，因

为这个候选主在切换的过程中一定是新的master。

4.3 在master上手动启动虚拟iP

第一次配置需要在master节点上手动启动虚拟IP，标签要和master_ip_faioverl配置文件中my $key = ‘1’; 一样

/sbin/ifconfig ens33:1 192.168.54.205/24

4.4 在manager 节点测试ssh无密认证

[root@master ~]# masterha_check_ssh  -conf=/opt/mysql-mha/mysql_mha.cnf
Wed Dec 21 12:25:24 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed Dec 21 12:25:24 2022 - [info] Reading application default configuration from /opt/mysql-mha/mysql_mha.cnf..
Wed Dec 21 12:25:24 2022 - [info] Reading server configuration from /opt/mysql-mha/mysql_mha.cnf..
Wed Dec 21 12:25:24 2022 - [info] Starting SSH connection tests..
Wed Dec 21 12:25:26 2022 - [debug]
Wed Dec 21 12:25:24 2022 - [debug]  Connecting via SSH from [email protected](192.168.54.200:22) to [email protected](192.168.54.201:22)..
Wed Dec 21 12:25:25 2022 - [debug]   ok.
Wed Dec 21 12:25:25 2022 - [debug]  Connecting via SSH from [email protected](192.168.54.200:22) to [email protected](192.168.54.202:22)..
Wed Dec 21 12:25:26 2022 - [debug]   ok.
Wed Dec 21 12:25:27 2022 - [debug]
Wed Dec 21 12:25:25 2022 - [debug]  Connecting via SSH from [email protected](192.168.54.201:22) to [email protected](192.168.54.200:22)..
Wed Dec 21 12:25:26 2022 - [debug]   ok.
Wed Dec 21 12:25:26 2022 - [debug]  Connecting via SSH from [email protected](192.168.54.201:22) to [email protected](192.168.54.202:22)..
Wed Dec 21 12:25:26 2022 - [debug]   ok.
Wed Dec 21 12:25:27 2022 - [debug]
Wed Dec 21 12:25:25 2022 - [debug]  Connecting via SSH from [email protected](192.168.54.202:22) to [email protected](192.168.54.200:22)..
Wed Dec 21 12:25:26 2022 - [debug]   ok.
Wed Dec 21 12:25:26 2022 - [debug]  Connecting via SSH from [email protected](192.168.54.202:22) to [email protected](192.168.54.201:22)..
Wed Dec 21 12:25:27 2022 - [debug]   ok.
Wed Dec 21 12:25:27 2022 - [info] All SSH connection tests passed successfully.

4.5 在manager 节点上测试mysql主从情况

[root@master ~]# masterha_check_repl -conf=/opt/mysql-mha/mysql_mha.cnf
Wed Dec 21 12:26:15 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed Dec 21 12:26:15 2022 - [info] Reading application default configuration from /opt/mysql-mha/mysql_mha.cnf..
Wed Dec 21 12:26:15 2022 - [info] Reading server configuration from /opt/mysql-mha/mysql_mha.cnf..
Wed Dec 21 12:26:15 2022 - [info] MHA::MasterMonitor version 0.58.
Wed Dec 21 12:26:16 2022 - [info] GTID failover mode = 0
Wed Dec 21 12:26:16 2022 - [info] Dead Servers:
Wed Dec 21 12:26:16 2022 - [info] Alive Servers:
Wed Dec 21 12:26:16 2022 - [info]   192.168.54.200(192.168.54.200:3306)
Wed Dec 21 12:26:16 2022 - [info]   192.168.54.201(192.168.54.201:3306)
Wed Dec 21 12:26:16 2022 - [info]   192.168.54.202(192.168.54.202:3306)
Wed Dec 21 12:26:16 2022 - [info] Alive Slaves:
Wed Dec 21 12:26:16 2022 - [info]   192.168.54.201(192.168.54.201:3306)  Version=5.7.40-log (oldest major version between slaves) log-bin:enabled
Wed Dec 21 12:26:16 2022 - [info]     Replicating from 192.168.54.200(192.168.54.200:3306)
Wed Dec 21 12:26:16 2022 - [info]     Primary candidate for the new Master (candidate_master is set)
Wed Dec 21 12:26:16 2022 - [info]   192.168.54.202(192.168.54.202:3306)  Version=5.7.40-log (oldest major version between slaves) log-bin:enabled
Wed Dec 21 12:26:16 2022 - [info]     Replicating from 192.168.54.200(192.168.54.200:3306)
Wed Dec 21 12:26:16 2022 - [info]     Primary candidate for the new Master (candidate_master is set)
Wed Dec 21 12:26:16 2022 - [info] Current Alive Master: 192.168.54.200(192.168.54.200:3306)
Wed Dec 21 12:26:16 2022 - [info] Checking slave configurations..
Wed Dec 21 12:26:16 2022 - [warning]  relay_log_purge=0 is not set on slave 192.168.54.201(192.168.54.201:3306).
Wed Dec 21 12:26:16 2022 - [warning]  relay_log_purge=0 is not set on slave 192.168.54.202(192.168.54.202:3306).
Wed Dec 21 12:26:16 2022 - [info] Checking replication filtering settings..
Wed Dec 21 12:26:16 2022 - [info]  binlog_do_db= , binlog_ignore_db=
Wed Dec 21 12:26:16 2022 - [info]  Replication filtering check ok.
Wed Dec 21 12:26:16 2022 - [info] GTID (with auto-pos) is not supported
Wed Dec 21 12:26:16 2022 - [info] Starting SSH connection tests..
Wed Dec 21 12:26:19 2022 - [info] All SSH connection tests passed successfully.
Wed Dec 21 12:26:19 2022 - [info] Checking MHA Node version..
Wed Dec 21 12:26:19 2022 - [info]  Version check ok.
Wed Dec 21 12:26:19 2022 - [info] Checking SSH publickey authentication settings on the current master..
Wed Dec 21 12:26:20 2022 - [info] HealthCheck: SSH to 192.168.54.200 is reachable.
Wed Dec 21 12:26:20 2022 - [info] Master MHA Node version is 0.58.
Wed Dec 21 12:26:20 2022 - [info] Checking recovery script configurations on 192.168.54.200(192.168.54.200:3306)..
Wed Dec 21 12:26:20 2022 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/opt/mysql-mha/mha-node/save_binary_logs_test --manager_version=0.58 --start_file=mysql-bin.000002
Wed Dec 21 12:26:20 2022 - [info]   Connecting to [email protected](192.168.54.200:22)..
  Creating /opt/mysql-mha/mha-node if not exists..    ok.
  Checking output directory is accessible or not..
   ok.
  Binlog found at /var/lib/mysql, up to mysql-bin.000002
Wed Dec 21 12:26:20 2022 - [info] Binlog setting check done.
Wed Dec 21 12:26:20 2022 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Wed Dec 21 12:26:20 2022 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='mha' --slave_host=192.168.54.201 --slave_ip=192.168.54.201 --slave_port=3306 --workdir=/opt/mysql-mha/mha-node --target_version=5.7.40-log --manager_version=0.58 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxx
Wed Dec 21 12:26:20 2022 - [info]   Connecting to [email protected](192.168.54.201:22)..
Creating directory /opt/mysql-mha/mha-node.. done.
  Checking slave recovery environment settings..
    Opening /var/lib/mysql/relay-log.info ... ok.
    Relay log found at /var/lib/mysql, up to relay-log-bin.000002
    Temporary relay log file is /var/lib/mysql/relay-log-bin.000002
    Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
    Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Wed Dec 21 12:26:21 2022 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='mha' --slave_host=192.168.54.202 --slave_ip=192.168.54.202 --slave_port=3306 --workdir=/opt/mysql-mha/mha-node --target_version=5.7.40-log --manager_version=0.58 --relay_log_info=/var/lib/mysql/relay-log.info  --relay_dir=/var/lib/mysql/  --slave_pass=xxx
Wed Dec 21 12:26:21 2022 - [info]   Connecting to [email protected](192.168.54.202:22)..
Creating directory /opt/mysql-mha/mha-node.. done.
  Checking slave recovery environment settings..
    Opening /var/lib/mysql/relay-log.info ... ok.
    Relay log found at /var/lib/mysql, up to relay-log-bin.000002
    Temporary relay log file is /var/lib/mysql/relay-log-bin.000002
    Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
    Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Wed Dec 21 12:26:21 2022 - [info] Slaves settings check done.
Wed Dec 21 12:26:21 2022 - [info]
192.168.54.200(192.168.54.200:3306) (current master)
 +--192.168.54.201(192.168.54.201:3306)
 +--192.168.54.202(192.168.54.202:3306)

Wed Dec 21 12:26:21 2022 - [info] Checking replication health on 192.168.54.201..
Wed Dec 21 12:26:21 2022 - [info]  ok.
Wed Dec 21 12:26:21 2022 - [info] Checking replication health on 192.168.54.202..
Wed Dec 21 12:26:21 2022 - [info]  ok.
Wed Dec 21 12:26:21 2022 - [info] Checking master_ip_failover_script status:
Wed Dec 21 12:26:21 2022 - [info]   /opt/mysql-mha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.54.200 --orig_master_ip=192.168.54.200 --orig_master_port=3306


IN SCRIPT TEST====root|sudo /sbin/ifconfig ens33:1 down==root|sudo /sbin/ifconfig ens33:1 192.168.54.205/24===

Checking the Status of the script.. OK
Wed Dec 21 12:26:21 2022 - [info]  OK.
Wed Dec 21 12:26:21 2022 - [warning] shutdown_script is not defined.
Wed Dec 21 12:26:21 2022 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.

4.6 在manage上启动mha

nohup masterha_manager  \
--conf=/opt/mysql-mha/mysql_mha.cnf \
--remove_dead_master_conf \
--ignore_last_failover < /dev/null > /var/log/mha_manager.log 2>&1 &

• --remove_dead_master_conf：该参数代表当发生主从切换后，老的主库的 ip 将会从配置文件中移除。

• --manger_log：日志存放位置。

• --ignore_last_failover：在缺省情况下，如果 MHA 检测到连续发生宕机，且两次宕机间隔不足 8 小时

  的话，则不会进行 Failover， 之所以这样限制是为了避免 ping-pong 效应。该参数代表忽略上次 MHA 触发

  切换产生的文件，默认情况下，MHA 发生切换后会在日志记目录，也就是上面设置的日志

  app1.failover.complete文件，下次再次切换的时候如果发现该目录下存在该文件将不允许触发切换，除非在

  第一次切换后收到删除该文件，为了方便，这里设置为–ignore_last_failover。

[root@master mysql-mha]# nohup masterha_manager  \
> --conf=/opt/mysql-mha/mysql_mha.cnf \
> --remove_dead_master_conf \
> --ignore_last_failover < /dev/null > /var/log/mha_manager.log 2>&1 &
[1] 21751

4.7 查看MHA状态

[root@master mysql-mha]# masterha_check_status --conf=/opt/mysql-mha/mysql_mha.cnf
mysql_mha (pid:21751) is running(0:PING_OK), master:192.168.54.200

4.8 查看MHA日志文件

[root@master mysql-mha]# cat /opt/mysql-mha/manager.log | grep "current master"
Wed Dec 21 12:29:43 2022 - [info] Checking SSH publickey authentication settings on the current master..
192.168.54.200(192.168.54.200:3306) (current master)

4.9 manager节点关闭manager服务

[root@master mysql-mha]# masterha_stop --conf=/opt/mysql-mha/mysql_mha.cnf
Stopped mysql_mha successfully.
[1]+  退出 1                nohup masterha_manager --conf=/opt/mysql-mha/mysql_mha.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/mha_manager.log 2>&1

5、故障模拟与恢复

5.1 停掉mysql master

在master节点关闭mysql服务，模拟主节点数据崩溃。

[root@master mysql-mha]# systemctl stop mysqld

稍等片刻，可以看到剩下的从库中有一台已经变成了主库。

注意，故障转移完成后， manager将会自动停止， 此时使用 masterha_check_status 命令检测将会遇到错误提

示， 如下所示：

[root@master mysql-mha]# masterha_check_status --conf=/opt/mysql-mha/mysql_mha.cnf
mysql_mha is stopped(2:NOT_RUNNING).

5.2 查看主备节点

# 查看虚拟ip
[root@master mysql-mha]# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens33:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:50:56:23:e2:fb brd ff:ff:ff:ff:ff:ff
    inet 192.168.54.200/24 brd 192.168.54.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet6 2409:8903:5404:1389:2073:9f4d:f2db:5c9e/64 scope global noprefixroute dynamic
       valid_lft 3368sec preferred_lft 3368sec
    inet6 fe80::c8e0:482b:7618:82bb/64 scope link noprefixroute
       valid_lft forever preferred_lft forever


# 192.168.54.205/24飘移到slave1机器上
[root@slave1 ~]# ip a
1: lo:  mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens33:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:50:56:3a:32:32 brd ff:ff:ff:ff:ff:ff
    inet 192.168.54.201/24 brd 192.168.54.255 scope global noprefixroute ens33
       valid_lft forever preferred_lft forever
    inet 192.168.54.205/24 brd 192.168.54.255 scope global secondary ens33:1
       valid_lft forever preferred_lft forever
    inet6 2409:8903:5404:1389:81e7:77c:71d5:78a3/64 scope global noprefixroute dynamic
       valid_lft 3353sec preferred_lft 3353sec
    inet6 fe80::7fc4:f931:d96e:1776/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

# 登录mysql查看主从关系
# 192.168.54.201已经变成主节点了
[root@slave1 ~]# mysql -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 33
Server version: 5.7.40-log MySQL Community Server (GPL)

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show master status \G
*************************** 1. row ***************************
             File: mysql-bin.000003
         Position: 317
     Binlog_Do_DB:
 Binlog_Ignore_DB:
Executed_Gtid_Set:
1 row in set (0.00 sec)

# 登录mysql查看主从关系
# 192.168.54.202已经变成从节点了
[root@slave2 ~]# mysql -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 91
Server version: 5.7.40-log MySQL Community Server (GPL)

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show slave status \G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.54.201
                  Master_User: myslave
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000003
          Read_Master_Log_Pos: 317
               Relay_Log_File: relay-log-bin.000002
                Relay_Log_Pos: 320
        Relay_Master_Log_File: mysql-bin.000003
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:

5.3 故障恢复

先在当前的主库服务器slave1上查看二进制日志和同步点：

[root@slave1 ~]# mysql -uroot -p123456
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 33
Server version: 5.7.40-log MySQL Community Server (GPL)

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000003 |      317 |              |                  |                   |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)

mysql>

再在原master服务器上执行同步操作：

# 先恢复mysql服务
systemctl start mysqld

[root@master mysql-mha]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.40-log MySQL Community Server (GPL)

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> change master to
    ->   master_host='192.168.182.201',
    ->   master_user='myslave',
    ->   master_password='123456',
    ->   master_log_file='mysql-bin.000003',
    ->   master_log_pos=317;^C
mysql> change master to
    ->   master_host='192.168.54.201',
    ->   master_user='myslave',
    ->   master_password='123456',
    ->   master_log_file='mysql-bin.000003',
    ->   master_log_pos=317;
Query OK, 0 rows affected, 2 warnings (0.00 sec)

mysql> start slave;
Query OK, 0 rows affected (0.00 sec)

# 这里需要过一段时间再看同步状态
mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.54.201
                  Master_User: myslave
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000003
          Read_Master_Log_Pos: 317
               Relay_Log_File: master-relay-bin.000002
                Relay_Log_Pos: 320
        Relay_Master_Log_File: mysql-bin.000003
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes

MySQL高可用实战就到这了。