MySQL-5.7高可用MHA架构搭建部署

一、MHA

1.1.简介：

MHA（Master High Availability）在MySQL故障切换过程中，MHA能做到在0~30秒之内自动完成数据库的故障切换操作，并且在进行故障切换的过程中，MHA能在最大程度上保证数据的一致性，以达到真正意义上的高可用. 该软件由两部分组成：MHA Manager（管理节点）和MHA Node（数据节点）。MHA Manager可以单独部署在一台独立的机器上管理多个master-slave集群，也可以部署在一台slave节点上。MHA Node运行在每台MySQL服务器上，MHA Manager会定时探测集群中的master节点，当master出现故障时，它可以自动将最新数据的slave提升为新的master，然后将所有其他的slave重新指向新的master。整个故障转移过程对应用程序完全透明。

1.2.MHA优点

1：主从切换非常迅速，通常10-30s

2：最大程度上解决数据一致性的问题

3：不需要修改当前已有的MySQL架构和配置

4：不需要另外多余的服务器

5：没有性能损耗

6：没有存储引擎限制

7：默认有异步复制和半同步复制

1.3MHA故障切换流程：

（1）从宕机崩溃的master保存二进制日志事件（binlog events）;

（2）识别含有最新更新的slave；

（3）应用差异的中继日志（relay log）到其他的slave；

（4）应用从master保存的二进制日志事件（binlog events）；

（5）提升一个slave为新的master；

（6）使其他的slave连接新的master进行复制；

1.4.MHA组成

(1)Manager工具：

– masterha_check_ssh : 检查MHA的SSH配置。

– masterha_check_repl : 检查MySQL复制。

– masterha_manager : 启动MHA。

– masterha_check_status : 检测当前MHA运行状态。

– masterha_master_monitor : 监测master是否宕机。

– masterha_master_switch : 控制故障转移(自动或手动)。

– masterha_conf_host : 添加或删除配置的server信息。

(2）Node工具(这些工具通常由MHAManager的脚本触发,无需人手操作)。

– save_binary_logs : 保存和复制master的二进制日志。

– apply_diffff_relay_logs : 识别差异的中继日志事件并应用于其它slave。

– fifilter_mysqlbinlog : 去除不必要的ROLLBACK事件(MHA已不再使用这个工具)。

– purge_relay_logs : 清除中继日志(不会阻塞SQL线程)。

(3)自定义扩展：

-secondary_check_script：通过多条网络路由检测master的可用性；

-master_ip_failover_script：更新application使用的masterip；（需要修改）

-shutdown_script：强制关闭master节点；

-report_script：发送报告；

-init_conf_load_script：加载初始配置参数；

-master_ip_online_change：更新master节点ip地址；（需要修改）

二、MySQL&MHA环境准备及步骤

主机名	ip	server_id	角色
mha-master	192.168.200.101	1	mha-node&mysql-master
mha-slave1	192.168.200.102	2	mha-node&mysql-slave
mha-slave2	192.168.200.103	3	mha-node&mysql-slave
mha-manager	192.168.200.104	无	mha-manager&mha-node

1.配置基础环境

//四台环境一致
cat /etc/redhat-release
ping -c 4 baidu.com

[root@localhost ~]# sestatus
SELinux status: disabled
iptables -nvL

[root@localhost ~]# ulimit -n
65535
[root@localhost ~]# hostname --all-ip
192.168.200.101
[root@localhost ~]# free -h
total used free shared buff/cache available
Mem: 1.8G 113M 1.5G 9.6M 217M 1.5G
Swap: 3.9G 0B 3.9G
#关闭防火墙&安全机制
[root@localhost ~]# systemctl stop firewalld
[root@localhost ~]# systemctl disable firewalld
[root@localhost ~]# setenforce 0
[root@localhost ~]# sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config

2.修改主机名&生成密钥

//host1-host4
hostnamectl set-hostname mha-master && bash
hostnamectl set-hostname mha-slave1 && bash
hostnamectl set-hostname mha-slave2 && bash
hostnamectl set-hostname mha-manager && bash

ssh-keygen -t rsa
ssh-copy-id mha-master
ssh-copy-id mha-slave1
ssh-copy-id mha-slave2
ssh-copy-id mha-manager

3.修改四台主机hosts同步

 [root@mha-manager ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.200.101 mha-master
192.168.200.102 mha-slave1
192.168.200.103 mha-slave2
192.168.200.104 mha-manager

//ping测试
[root@mha-manager ~]# for i in {master,slave1,slave2,manager}; do ping -c 1 mha-$i; done

4.安装MySQL host1~host3

yum install -y http://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm

yum -y install --nogpgcheck mysql-community-server

5.修改配置文件启动 host1~host3

[root@mha-master ~]# cp /etc/my.cnf{,.back}
[root@mha-master ~]# mkdir -pv /data/mysql/logs
mkdir: 已创建目录 "/data"
mkdir: 已创建目录 "/data/mysql"
mkdir: 已创建目录 "/data/mysql/logs"
[root@mha-master ~]# chown -R mysql:mysql /data/mysql/

//这条命令很长，注意到最后 修改server-id=1 主从顺序
[root@mha-master ~]# cat >> /etc/my.cnf << EOF
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html
[mysql]
port=3306
bind-address=0.0.0.0
[mysqld]
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
character_set_server=utf8
init_connect='SET NAMES utf8'
user = mysql
datadir=/data/mysql/data
socket=/var/lib/mysql/mysql.sock
max_connections = 2000
relay-log=/data/mysql/data/relay-log
log-bin=/data/mysql/data/mysql-bin
max_binlog_size = 500M
binlog_cache_size = 128K
binlog-ignore-db = mysql
expire_logs_days = 7
binlog_format="ROW"
server-id=1
skip-external-locking
skip-name-resolve
slave-skip-errors=1
max_allowed_packet = 100M
lower_case_table_names=1
slow_query_log=on
slow_query_log_file=/data/mysql/logs/slow_query_log.log
long_query_time=2
symbolic-links=0
log-error=/data/mysql/logs/mysql.err
sql_mode=STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
max_connect_errors = 100
query_cache_type=1
query_cache_size=128M
innodb_buffer_pool_size=4G
innodb_buffer_pool_instances=16
innodb_open_files=800
innodb_log_file_size=128M
innodb_log_files_in_group=3
innodb_log_buffer_size=8M
innodb_flush_log_at_trx_commit=2

gtid_mode = on
binlog-ignore-db = mysql
master-info-repository=table
relay-log-info-repository=table
relay-log-recovery=1
sync_binlog=1
read_only = on
enforce_gtid_consistency = 1
log_slave_updates = 1
plugin_load="rpl_semi_sync_master=semisync_master.so;rpl_semi_sync_slave=semisync_slave.so"
loose_rpl_semi_sync_master_enabled = 1
loose_rpl_semi_sync_slave_enabled = 1
loose_rpl_semi_sync_master_timeout = 5000

[mysqld_safe]
pid-file=/var/run/mysqld/mysqld.pid
log-error=/data/mysql/logs/mysql.err
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[root@mha-master ~]# systemctl start mysqld
[root@mha-master ~]# systemctl enable mysqld
[root@mha-master ~]# grep password /data/mysql/logs/mysql.err
2022-01-18T03:11:42.545587Z 1 [Note] A temporary password is generated for
root@localhost: cyw9tkQO4l-V
//修改root密码
[root@mha-master ~]# mysql -uroot -p'cyw9tkQO4l-V'

mysql> set global validate_password_policy=0;
mysql> set global validate_password_length=1;
mysql> alter user 'root'@'localhost' identified by '123123';
mysql> flush privileges;
mysql> exit

6.host1~host3配置主从复制及监控用户

[root@mha-master ~]# mysql -uroot -p123123
//创建复制用户
mysql> grant replication slave on *.* to repl@'192.168.200.%' identified by 'repl_user';
Query OK, 0 rows affected, 1 warning (0.12 sec)
//创建监控用户
mysql> grant all on *.* to 'mha'@'192.168.200.%' identified by 'mha_user';
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.04 sec)

//如果遇到如此报错修改密码长度即可
mysql> grant replication slave on *.* to repl@'192.168.200.%' identified by 'repl_user';
ERROR 1819 (HY000): Your password does not satisfy the current policy requirements

mysql> set global validate_password_policy=0;
mysql> show variables like 'validate_password%';

7.完成MySQL主从

//mha-master
[root@mha-master ~]# mysql -uroot -p123123
mysql> show master status;
+------------------+----------+--------------+------------------+------------------------------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                        |
+------------------+----------+--------------+------------------+------------------------------------------+
| mysql-bin.000002 |     1724 |              | mysql,mysql      | 1b89731a-c236-11ed-a636-000c2973f3a3:1-6 |
+------------------+----------+--------------+------------------+------------------------------------------+
1 row in set (0.00 sec)

//mha-slave1完成从库复制
[root@mha-slave1 ~]# mysql -uroot -p123123
mysql> change master to master_host='192.168.200.101',master_user='repl',master_password='repl_user',master_log_file='mysql-bin.000002',master_log_pos=1724;
#注意master_log_file='mysql-bin.000002',master_log_pos=1724;修改这俩个为主库查到的

mysql> commit;
mysql> start slave;
mysql> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.200.101
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000002
          Read_Master_Log_Pos: 1724
               Relay_Log_File: relay-log.000002
                Relay_Log_Pos: 320
        Relay_Master_Log_File: mysql-bin.000002
             Slave_IO_Running: Yes  #yes及成功
            Slave_SQL_Running: Yes  #yes及成功
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 1724
              Relay_Log_Space: 521
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 1
                  Master_UUID: 1b89731a-c236-11ed-a636-000c2973f3a3
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: f1fee7ed-c235-11ed-8400-000c2945efcc:1-5
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
1 row in set (0.00 sec)

ERROR: 
No query specified

8.配置ssh互信

//上面部分操作中已经配置了主机互信，如遇报错向下执行即可
mha-master
[root@mha-master ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
[root@mha-master ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
[root@mha-master ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
[root@mha-master ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]

mha-slave1
[root@mha-slave1 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
[root@mha-slave1 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
[root@mha-slave1 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
[root@mha-slave1 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]

mha-slave2
[root@mha-slave2 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
[root@mha-slave2 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
[root@mha-slave2 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
[root@mha-slave2 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]

mha-manager
[root@mha-manager ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
[root@mha-manager ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
[root@mha-manager ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
[root@mha-manager ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]

9.MHA部署

MHA-manager&node百度网盘地址
链接：https://pan.baidu.com/s/1xqZtQeXXqAiewlvECiQnBw
提取码：xw91
--来自百度网盘超级会员V5的分享

//host1~host3
[root@mha-master ~]# mkdir -pv /data/software
mkdir: 已创建目录 "/data/software"
[root@mha-master ~]# cd /data/software/
[root@mha-master software]# wget https://github.com/yoshinorim/mha4mysqlnode/releases/download/v0.58/mha4mysql-node-0.58-0.el7.centos.noarch.rp
[root@mha-master software]# yum install -y mha4mysql-node-0.58-0.el7.centos.noarch.rpm

10.安装配置mha-manager

yum install -y perl-Parallel-ForkManager perl-Log-Dispatch perl-Config-Tiny perl-DBD-MySQL

[root@mha-manager ~]# mkdir -pv /data/software
[root@mha-manager ~]# cd /data/software
[root@mha-slave2 software]# yum -y install mha4mysql-node-0.58-0.el7.centos.noarch.rpm
[root@mha-slave2 software]# yum -y install mha4mysql-manager-0.58-0.el7.centos.noarch.rpm

[root@mha-slave2 software]# mkdir -pv /data/mha/{conf,logs,tools,tmp}
//配置切换脚本
[root@mha-slave2 ~]# vim /data/mha/tools/master_ip_failover_script


[root@mha-slave2 tools]# chmod 777 /data/mha/tools/master_ip_failover_script 

//配置mha配置文件
[root@mha-slave2 ~]# vim /data/mha/conf/mha192.168.200.101-104.conf

11.测试通信ssh通信&replication状态

[root@bogon conf]# masterha_check_ssh --conf=/data/mha/conf/mha192.168.200.102-104.conf 
Tue Mar 14 17:05:41 2023 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Mar 14 17:05:41 2023 - [info] Reading application default configuration from /data/mha/conf/mha192.168.200.102-104.conf..
Tue Mar 14 17:05:41 2023 - [info] Reading server configuration from /data/mha/conf/mha192.168.200.102-104.conf..
Tue Mar 14 17:05:41 2023 - [info] Starting SSH connection tests..
Tue Mar 14 17:05:42 2023 - [debug] 
Tue Mar 14 17:05:41 2023 - [debug]  Connecting via SSH from [email protected](192.168.200.101:22) to [email protected](192.168.200.102:22)..
Tue Mar 14 17:05:41 2023 - [debug]   ok.
Tue Mar 14 17:05:41 2023 - [debug]  Connecting via SSH from [email protected](192.168.200.101:22) to [email protected](192.168.200.103:22)..
Tue Mar 14 17:05:42 2023 - [debug]   ok.
Tue Mar 14 17:05:42 2023 - [debug] 
Tue Mar 14 17:05:42 2023 - [debug]  Connecting via SSH from [email protected](192.168.200.102:22) to [email protected](192.168.200.101:22)..
Tue Mar 14 17:05:42 2023 - [debug]   ok.
Tue Mar 14 17:05:42 2023 - [debug]  Connecting via SSH from [email protected](192.168.200.102:22) to [email protected](192.168.200.103:22)..
Tue Mar 14 17:05:42 2023 - [debug]   ok.
Tue Mar 14 17:05:43 2023 - [debug] 
Tue Mar 14 17:05:42 2023 - [debug]  Connecting via SSH from [email protected](192.168.200.103:22) to [email protected](192.168.200.101:22)..
Tue Mar 14 17:05:42 2023 - [debug]   ok.
Tue Mar 14 17:05:42 2023 - [debug]  Connecting via SSH from [email protected](192.168.200.103:22) to [email protected](192.168.200.102:22)..
Tue Mar 14 17:05:43 2023 - [debug]   ok.
Tue Mar 14 17:05:43 2023 - [info] All SSH connection tests passed successfully.

//测试replication状态
[root@mha-manager conf]# masterha_check_repl --conf=/data/mha/conf/mha192.168.200.101-104.conf 
Tue Mar 14 20:41:21 2023 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Mar 14 20:41:21 2023 - [info] Reading application default configuration from /data/mha/conf/mha192.168.200.101-104.conf..
Tue Mar 14 20:41:21 2023 - [info] Reading server configuration from /data/mha/conf/mha192.168.200.101-104.conf..
Tue Mar 14 20:41:21 2023 - [info] MHA::MasterMonitor version 0.58.
Tue Mar 14 20:41:22 2023 - [info] GTID failover mode = 1
Tue Mar 14 20:41:22 2023 - [info] Dead Servers:
Tue Mar 14 20:41:22 2023 - [info] Alive Servers:
Tue Mar 14 20:41:22 2023 - [info]   192.168.200.101(192.168.200.101:3306)
Tue Mar 14 20:41:22 2023 - [info]   192.168.200.102(192.168.200.102:3306)
Tue Mar 14 20:41:22 2023 - [info]   192.168.200.103(192.168.200.103:3306)
Tue Mar 14 20:41:22 2023 - [info] Alive Slaves:
Tue Mar 14 20:41:22 2023 - [info]   192.168.200.102(192.168.200.102:3306)  Version=5.7.41-log (oldest major version between slaves) log-bin:enabled
Tue Mar 14 20:41:22 2023 - [info]     GTID ON
Tue Mar 14 20:41:22 2023 - [info]     Replicating from 192.168.200.101(192.168.200.101:3306)
Tue Mar 14 20:41:22 2023 - [info]     Primary candidate for the new Master (candidate_master is set)
Tue Mar 14 20:41:22 2023 - [info]   192.168.200.103(192.168.200.103:3306)  Version=5.7.41-log (oldest major version between slaves) log-bin:enabled
Tue Mar 14 20:41:22 2023 - [info]     GTID ON
Tue Mar 14 20:41:22 2023 - [info]     Replicating from 192.168.200.101(192.168.200.101:3306)
Tue Mar 14 20:41:22 2023 - [info] Current Alive Master: 192.168.200.101(192.168.200.101:3306)
Tue Mar 14 20:41:22 2023 - [info] Checking slave configurations..
Tue Mar 14 20:41:22 2023 - [info] Checking replication filtering settings..
Tue Mar 14 20:41:22 2023 - [info]  binlog_do_db= , binlog_ignore_db= mysql
Tue Mar 14 20:41:22 2023 - [info]  Replication filtering check ok.
Tue Mar 14 20:41:22 2023 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Tue Mar 14 20:41:22 2023 - [info] Checking SSH publickey authentication settings on the current master..
Tue Mar 14 20:41:22 2023 - [info] HealthCheck: SSH to 192.168.200.101 is reachable.
Tue Mar 14 20:41:22 2023 - [info] 
192.168.200.101(192.168.200.101:3306) (current master)
 +--192.168.200.102(192.168.200.102:3306)
 +--192.168.200.103(192.168.200.103:3306)

Tue Mar 14 20:41:22 2023 - [info] Checking replication health on 192.168.200.102..
Tue Mar 14 20:41:22 2023 - [info]  ok.
Tue Mar 14 20:41:22 2023 - [info] Checking replication health on 192.168.200.103..
Tue Mar 14 20:41:22 2023 - [info]  ok.
Tue Mar 14 20:41:22 2023 - [info] Checking master_ip_failover_script status:
Tue Mar 14 20:41:22 2023 - [info]   /data/mha/tools/master_ip_failover_script --command=status --ssh_user=root --orig_master_host=192.168.200.101 --orig_master_ip=192.168.200.101 --orig_master_port=3306 


IN SCRIPT TEST====/sbin/ifconfig ens33:1 down==/sbin/ifconfig ens33:1 192.168.200.200/24===

Checking the Status of the script.. OK 
Tue Mar 14 20:41:22 2023 - [info]  OK.
Tue Mar 14 20:41:22 2023 - [warning] shutdown_script is not defined.
Tue Mar 14 20:41:22 2023 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK. 

最后测试结果为:MySQL Replication Health is OK表示测试正常

12.启动mha状态

[root@mha-manager conf]# nohup masterha_manager --conf=/data/mha/conf/mha192.168.200.101-104.conf &> /data/mha/logs/mha_manager.log &
[1] 16545
//检查mha状态
[root@mha-manager conf]# masterha_check_status --conf=/data/mha/conf/mha192.168.200.101-104.conf
mha192.168.200.101-104 (pid:16953) is running(0:PING_OK), master:192.168.200.101

13.配置mha-master-IP信息

[root@mha-manager conf]# ssh mha-master ifconfig ens33:1 192.168.200.200/24
[root@mha-manager conf]# ssh mha-master ifconfig