MySQL高可用之MHA
目录
一、环境准备
二、架构原理
三、安装mysql软件及初始化环境
1、安装Mysql软件
2、初始化环境
四、配置mysql环境
1、配置mysql半同步复制
2、配置mysql同步
五、安装和配置MHA
1、安装MHA相关依赖软件
2、安装MHA-manager和MHA-node软件
3、配置MHA-manager服务
六、设置VIP
1、在mysql-master01服务器上设置VIP
2、配置mysql-manager服务
3、测试服务
MHA是Mysql故障切换和主从提升的高可用软件，该软件由manager和node两部分组成，manager可以单独部署在一台独立的服务器中管理多个master-slave集群，当master宕机后，该软件会自动将最新binlog数据的slave提升为master，然后将其它的所有slave指针指向新的master，整个故障转移过程对应用程序完全透明。
一、环境准备
CentOS Linux release 7.9.2009 (Core)
mysql-community-server-5.7.35-1.el7.x86_64
mysql57-community-release-el7-9.noarch
mysql-community-libs-5.7.35-1.el7.x86_64
mysql-community-common-5.7.35-1.el7.x86_64
mysql-community-libs-compat-5.7.35-1.el7.x86_64
mysql-community-client-5.7.35-1.el7.x86_64
MHA软件：https://github.com/yoshinorim
mha4mysql-manager-master.zip
mha4mysql-node-master.zip
二、架构原理
#这个MHA管理软件可以同时管理好几套主从架构
| 角色 | 主机名 | IP | VIP | 备注 |
| manager | mysql-manager | 192.168.10.74 | 管理节点 | |
| master | mysql-master01 | 192.168.10.71 | 192.168.10.230 | 写入 |
| candicatemaster | mysql-master02 | 192.168.10.72 | 读 | |
| slave | mysql-slave02 | 192.168.10.73 | 读 |
目前MHA主要支持一主多从的架构，要搭建MHA,要求一个复制集群中必须最少有三台数据库服务器，一主二 从，即一台充当master，一台充当备用master，另外一台充当从库，因为至少需要三台服务器。
其中master对外提供写服务，备选master（实际的mysql-slave01）提供读服务，slave也提供相关的读 服务，一旦master宕机，将会把备选master提升为新的master，slave指向新的master，manager作为管理服务 器。
三、安装mysql软件及初始化环境
1、安装Mysql软件
注：这里master和2台slave都需要安装，还有4台的时间要同步，保持一致
1.1、下载mysql安装包
cd /root
wget https://downloads.mysql.com/archives/get/p/23/file/mysql-5.7.35-1.el7.x86_64.rpm-bundle.tar
1.2、为了避免冲突，需要卸载系统中自带的mariadb-libs
yum remove -y mariadb-libs1.3、安装软件
cd /root
unzip mysql-5.7.35-1.el7.x86_64.rpm-bundle.zip
cd mysql-5.7.35-1.el7.x86_64.rpm-bundle
rpm -ivh mysql-community-server-5.7.35-1.el7.x86_64.rpm mysql-community-client-5.7.35-1.el7.x86_64.rpm mysql-community-common-5.7.35-1.el7.x86_64.rpm mysql-community-libs-compat-5.7.35-1.el7.x86_64.rpm mysql-community-libs-5.7.35-1.el7.x86_64.rpm1.4、设置启动参数
systemctl enable mysqld
systemctl start mysqld
1.5、 设置密码等级
mysql5.5之后启动后默认登录需要密码，但是默认密码是随机的，需要从启动的log里找
cat /var/log/mysqld.log | grep password
2022-03-01T15:46:11.528729Z 1 [Note] A temporary password is generated for root@localhost: .eh0Bs0fa<.;#修改密码等级
(root@localhost) [(none)]>set global validate_password_length=4;
Query OK, 0 rows affected (0.00 sec)
(root@localhost) [(none)]>set global validate_password_policy=0;
Query OK, 0 rows affected (0.00 sec)
(root@localhost) [(none)]>set global validate_password_policy=LOW;
Query OK, 0 rows affected (0.00 sec)#修改密码
#本地账号密码
(root@localhost) [(none)]> alter user 'root'@'localhost' identified with mysql_native_password BY 'test@7777';
#如果有需要远程登录，需要添加远程账号
(root@localhost) [(none)]> grant all privileges on *.* to 'root'@'%' identified by 'test@7777' with grant option;
(root@localhost) [(none)]>flush privileges;
2、初始化环境
2.1、先在每台机器上加入hosts，4台机器都需要加
echo "192.168.10.74 mysql-manager" >>/etc/hosts
echo "192.168.10.71 mysql-master01" >>/etc/hosts
echo "192.168.10.72 mysql-master02" >>/etc/hosts
echo "192.168.10.73 mysql-slave01" >>/etc/hosts
2.2、建立4台服务器相互之间ssh免密关系
#mysql-manager服务器
ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
1c:cb:2d:4f:b1:80:ea:80:35:3b:89:48:5f:09:eb:2e root@mysql-manager
The key's randomart image is:
+--[ RSA 2048]----+
| . |
| o .. |
| .o. o. o . |
|o+o+.. o = o |
|+ =o. S + |
| .+ + |
| E .. . |
| . |
| |
+-----------------+
#分到其它主机
for i in mysql-manager mysql-master01 mysql-master02 mysql-slave01;do ssh-copy-id -i $i;done#测试连通性
for i in mysql-manager mysql-master01 mysql-master02 mysql-slave01;do ssh $i hostname;done
mysql-manager
mysql-master01
mysql-master02
mysql-slave01
#manager-master01机器
ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
1c:cb:2d:4f:b1:80:ea:80:35:3b:89:48:5f:09:eb:2e root@mysql-master01
The key's randomart image is:
+--[ RSA 2048]----+
| . |
| o .. |
| .o. o. o . |
|o+o+.. o = o |
|+ =o. S + |
| .+ + |
| E .. . |
| . |
| |
+-----------------+
#分到其它主机
for i in mysql-manager mysql-master01 mysql-master02 mysql-slave01;do ssh-copy-id -i $i;done#测试连通性
for i in mysql-manager mysql-master01 mysql-master02 mysql-slave01;do ssh $i hostname;done
mysql-manager
mysql-master01
mysql-master02
mysql-slave01#manager-master02机器
ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
1c:cb:2d:4f:b1:80:ea:80:35:3b:89:48:5f:09:eb:2e root@mysql-master02
The key's randomart image is:
+--[ RSA 2048]----+
| . |
| o .. |
| .o. o. o . |
|o+o+.. o = o |
|+ =o. S + |
| .+ + |
| E .. . |
| . |
| |
+-----------------+
#分到其它主机
for i in mysql-manager mysql-master01 mysql-master02 mysql-slave01;do ssh-copy-id -i $i;done#测试连通性
for i in mysql-manager mysql-master01 mysql-master02 mysql-slave01;do ssh $i hostname;done
mysql-manager
mysql-master01
mysql-master02
mysql-slave01#manager-slave01机器
ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
1c:cb:2d:4f:b1:80:ea:80:35:3b:89:48:5f:09:eb:2e root@mysql-slave01
The key's randomart image is:
+--[ RSA 2048]----+
| . |
| o .. |
| .o. o. o . |
|o+o+.. o = o |
|+ =o. S + |
| .+ + |
| E .. . |
| . |
| |
+-----------------+
#分到其它主机
for i in mysql-manager mysql-master01 mysql-master02 mysql-slave01;do ssh-copy-id -i $i;done#测试连通性
for i in mysql-manager mysql-master01 mysql-master02 mysql-slave01;do ssh $i hostname;done
mysql-manager
mysql-master01
mysql-master02
mysql-slave01四、配置mysql环境
1、配置mysql半同步复制
为了尽可能的减少主库硬件损坏宕机造成的数据丢失，因此在配置MHA的同时建议 配置成MySQL的半同步复制
1.1、检查半同步插件
注：mysql半同步插件是由谷歌提供，具体位置/usr/lib64/mysql/plugin/下：
# ll /usr/lib64/mysql/plugin/semisync_*
-rwxr-xr-x 1 root root 937544 Jun 7 2021 /usr/lib64/mysql/plugin/semisync_master.so
-rwxr-xr-x 1 root root 160568 Jun 7 2021 /usr/lib64/mysql/plugin/semisync_slave.so
1.2、分别在三个mysql节点上安装半同步插件
#检测数据库是否支持动态载入
mysql>show variables like '%have_dynamic%';
+----------------------+-------+
| Variable_name | Value |
+----------------------+-------+
| have_dynamic_loading | YES | # 显示yes表示支持
+----------------------+-------+
1 row in set (0.00 sec)
#所有mysql数据库服务器，安装半同步插件
mysql>INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so';
Query OK, 0 rows affected (0.01 sec)
mysql>INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';
Query OK, 0 rows affected (0.00 sec)
#检查Plugin是否已正确安装
mysql> show plugins; # 可以再最后两行看到如下内容
| rpl_semi_sync_master | ACTIVE | REPLICATION | semisync_master.so | GPL |
| rpl_semi_sync_slave | ACTIVE | REPLICATION | semisync_slave.so | GPL |
#也可以使用如下命令检查
(root@localhost) [(none)]> select * from information_schema.plugins order by PLUGIN_NAME desc limit 4;
+----------------------+----------------+---------------+-------------------+---------------------+----------------------+------------------------+--------------------+-------------------------------------+----------------+-------------+
| PLUGIN_NAME | PLUGIN_VERSION | PLUGIN_STATUS | PLUGIN_TYPE | PLUGIN_TYPE_VERSION | PLUGIN_LIBRARY | PLUGIN_LIBRARY_VERSION | PLUGIN_AUTHOR | PLUGIN_DESCRIPTION | PLUGIN_LICENSE | LOAD_OPTION |
+----------------------+----------------+---------------+-------------------+---------------------+----------------------+------------------------+--------------------+-------------------------------------+----------------+-------------+
| validate_password | 1.1 | ACTIVE | VALIDATE PASSWORD | 1.0 | validate_password.so | 1.7 | Oracle Corporation | check password strength | GPL | ON |
| sha256_password | 1.1 | ACTIVE | AUTHENTICATION | 1.1 | NULL | NULL | Oracle | SHA256 password authentication | GPL | FORCE |
| rpl_semi_sync_slave | 1.0 | ACTIVE | REPLICATION | 4.0 | semisync_slave.so | 1.7 | He Zhenxing | Semi-synchronous replication slave | GPL | ON |
| rpl_semi_sync_master | 1.0 | ACTIVE | REPLICATION | 4.0 | semisync_master.so | 1.7 | He Zhenxing | Semi-synchronous replication master | GPL | ON |
+----------------------+----------------+---------------+-------------------+---------------------+----------------------+------------------------+--------------------+-------------------------------------+----------------+-------------+
4 rows in set (0.00 sec)
#查看半同步相关信息
mysql>show variables like '%rpl_semi_sync%';
+-------------------------------------------+------------+
| Variable_name | Value |
+-------------------------------------------+------------+
| rpl_semi_sync_master_enabled | OFF |
| rpl_semi_sync_master_timeout | 10000 |
| rpl_semi_sync_master_trace_level | 32 |
| rpl_semi_sync_master_wait_for_slave_count | 1 |
| rpl_semi_sync_master_wait_no_slave | ON |
| rpl_semi_sync_master_wait_point | AFTER_SYNC |
| rpl_semi_sync_slave_enabled | OFF |
| rpl_semi_sync_slave_trace_level | 32 |
+-------------------------------------------+------------+
8 rows in set (0.00 sec)
#上方所示内容可以看到半同复制插件已经成功安装，只是还没有启用，所以是off
2、配置mysql同步
2.1、mysql-master01
vim /etc/my.cnf
# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html
[mysqld]
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
#symbolic-links=0
#log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
!includedir /etc/my.cnf.d
vim /etc/my.cnf.d/mysqld.cnf
[mysqld]
symbolic-links=0
transaction-isolation=READ-COMMITTED
default_password_lifetime=0
skip-external-locking
skip-name-resolve
user = mysql
socket = /var/lib/mysql/mysql.sock
pid-file = /var/lib/mysql/mysql.pid
transaction_isolation = READ-COMMITTED
explicit_defaults_for_timestamp = 1
lower_case_table_names = 1
max_allowed_packet = 48M
max_connect_errors = 1000000
character-set-server = utf8
collation-server = utf8_bin
default-storage-engine = INNODB
interactive_timeout = 300
wait_timeout = 300
datadir = /var/lib/mysql
tmpdir = /var/lib/mysql
max_connections = 4096
max_user_connections = 8000
back_log = 1024
tmp_table_size = 128M
max_heap_table_size = 128M
table_definition_cache = 4096
table_open_cache = 400
open_files_limit = 65535
read_buffer_size = 20M
read_rnd_buffer_size = 20M
sort_buffer_size = 40M
join_buffer_size = 80M
log_error = /var/lib/mysql/mysql-error.log
slow_query_log = 1
slow_query_log_file = /var/lib/mysql/mysql-slow.log
log_slow_slave_statements = 1
expire_logs_days = 5
long_query_time = 1
log_error_verbosity = 1
innodb_buffer_pool_size = 2G
innodb_buffer_pool_instances = 8
innodb_buffer_pool_load_at_startup = 1
innodb_buffer_pool_dump_at_shutdown = 1
innodb_lru_scan_depth = 2000
innodb_lock_wait_timeout = 100
innodb_flush_method = O_DIRECT
innodb_log_file_size = 4G
innodb_log_buffer_size = 16777216
innodb_purge_threads = 4
innodb_large_prefix = 1
innodb_thread_concurrency = 64
innodb_print_all_deadlocks = 1
innodb_sort_buffer_size = 67108864
sql_mode = "NO_ENGINE_SUBSTITUTION"
server-id = 1 #每台mysql的ID不能一样
log-bin = mysql-bin
binlog_format = mixed
log-bin-index = mysql-bin.index
rpl_semi_sync_master_enabled = 1 # 1表示启用，0表示关闭，slave同样
rpl_semi_sync_master_timeout = 1000 # 毫秒单位，主服务器等待确认消息10秒后，不在等待，变为异步方式
rpl_semi_sync_slave_enabled = 1
relay_log_purge = 0
relay-log = relay-bin
relay-log-index = slave-relay-bin.index
[mysqldump]
quick
max_allowed_packet = 256M
default-character-set = utf8
[mysql]
prompt = (\u@\h) [\d]>\_
no-auto-rehash
default-character-set = utf8
[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 8M
write_buffer = 8M
[mysqlhotcopy]
interactive-timeout
[mysqld_safe]
open-files-limit = 8192
vim /etc/systemd/system/multi-user.target.wants/mysqld.service
# Sets open_files_limit
LimitNOFILE = 1000
#改为65535，不然启动会报错
# Sets open_files_limit
LimitNOFILE = 655352.2、mysql-master02
#vim /etc/my.cnf与master01一样
vim /etc/my.cnf.d/mysqld.cnf
[mysqld]
symbolic-links=0
transaction-isolation=READ-COMMITTED
default_password_lifetime=0
skip-external-locking
skip-name-resolve
user = mysql
socket = /var/lib/mysql/mysql.sock
pid-file = /var/lib/mysql/mysql.pid
transaction_isolation = READ-COMMITTED
explicit_defaults_for_timestamp = 1
lower_case_table_names = 1
max_allowed_packet = 48M
max_connect_errors = 1000000
character-set-server = utf8
collation-server = utf8_bin
default-storage-engine = INNODB
interactive_timeout = 300
wait_timeout = 300
datadir = /var/lib/mysql
tmpdir = /var/lib/mysql
max_connections = 4096
max_user_connections = 8000
back_log = 1024
tmp_table_size = 128M
max_heap_table_size = 128M
table_definition_cache = 4096
table_open_cache = 400
open_files_limit = 65535
read_buffer_size = 20M
read_rnd_buffer_size = 20M
sort_buffer_size = 40M
join_buffer_size = 80M
log_error = /var/lib/mysql/mysql-error.log
slow_query_log = 1
slow_query_log_file = /var/lib/mysql/mysql-slow.log
log_slow_slave_statements = 1
expire_logs_days = 5
long_query_time = 1
log_error_verbosity = 1
innodb_buffer_pool_size = 2G
innodb_buffer_pool_instances = 8
innodb_buffer_pool_load_at_startup = 1
innodb_buffer_pool_dump_at_shutdown = 1
innodb_lru_scan_depth = 2000
innodb_lock_wait_timeout = 100
innodb_flush_method = O_DIRECT
innodb_log_file_size = 4G
innodb_log_buffer_size = 16777216
innodb_purge_threads = 4
innodb_large_prefix = 1
innodb_thread_concurrency = 64
innodb_print_all_deadlocks = 1
innodb_sort_buffer_size = 67108864
sql_mode = "NO_ENGINE_SUBSTITUTION"
server-id = 2 #这里的server-id与master01不一样
log-bin = mysql-bin
binlog_format = mixed
log-bin-index = mysql-bin.index
rpl_semi_sync_master_enabled = 1 # 1表示启用，0表示关闭，slave同样
rpl_semi_sync_master_timeout = 10000 # 毫秒单位，主服务器等待确认消息10秒后，不在等待，变为异步方式
rpl_semi_sync_slave_enabled = 1
relay_log_purge = 0
relay-log = relay-bin
relay-log-index = slave-relay-bin.index
[mysqldump]
quick
max_allowed_packet = 256M
default-character-set = utf8
[mysql]
prompt = (\u@\h) [\d]>\_
no-auto-rehash
default-character-set = utf8
[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 8M
write_buffer = 8M
[mysqlhotcopy]
interactive-timeout
[mysqld_safe]
open-files-limit = 8192
#vim /etc/systemd/system/multi-user.target.wants/mysqld.service 这个与master01一样修改一下
2.3、mysql-slave01
#vim /etc/my.cnf与master01一样
vim /etc/my.cnf.d/mysqld.cnf
[mysqld]
symbolic-links=0
transaction-isolation=READ-COMMITTED
default_password_lifetime=0
skip-external-locking
skip-name-resolve
user = mysql
socket = /var/lib/mysql/mysql.sock
pid-file = /var/lib/mysql/mysql.pid
transaction_isolation = READ-COMMITTED
explicit_defaults_for_timestamp = 1
lower_case_table_names = 1
max_allowed_packet = 48M
max_connect_errors = 1000000
character-set-server = utf8
collation-server = utf8_bin
default-storage-engine = INNODB
interactive_timeout = 300
wait_timeout = 300
datadir = /var/lib/mysql
tmpdir = /var/lib/mysql
max_connections = 4096
max_user_connections = 8000
back_log = 1024
tmp_table_size = 128M
max_heap_table_size = 128M
table_definition_cache = 4096
table_open_cache = 400
open_files_limit = 65535
read_buffer_size = 20M
read_rnd_buffer_size = 20M
sort_buffer_size = 40M
join_buffer_size = 80M
log_error = /var/lib/mysql/mysql-error.log
slow_query_log = 1
slow_query_log_file = /var/lib/mysql/mysql-slow.log
log_slow_slave_statements = 1
expire_logs_days = 5
long_query_time = 1
log_error_verbosity = 1
innodb_buffer_pool_size = 2G
innodb_buffer_pool_instances = 8
innodb_buffer_pool_load_at_startup = 1
innodb_buffer_pool_dump_at_shutdown = 1
innodb_lru_scan_depth = 2000
innodb_lock_wait_timeout = 100
innodb_flush_method = O_DIRECT
innodb_log_file_size = 4G
innodb_log_buffer_size = 16777216
innodb_purge_threads = 4
innodb_large_prefix = 1
innodb_thread_concurrency = 64
innodb_print_all_deadlocks = 1
innodb_sort_buffer_size = 67108864
sql_mode = "NO_ENGINE_SUBSTITUTION"
####################与master区别部分#######################
server-id = 3 #server-id与前面两台不一样
log-bin = mysql-bin
read_only = 1 #
rpl_semi_sync_slave_enabled = 1
relay-log = relay-bin
relay-log-index = slave-relay-bin.index
###########################################
[mysqldump]
quick
max_allowed_packet = 256M
default-character-set = utf8
[mysql]
prompt = (\u@\h) [\d]>\_
no-auto-rehash
default-character-set = utf8
[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 8M
write_buffer = 8M
[mysqlhotcopy]
interactive-timeout
[mysqld_safe]
open-files-limit = 8192
#vim /etc/systemd/system/multi-user.target.wants/mysqld.service与前面一样需要修改
#在mysql-master01添加同步账号和manager账号
#创建一个用于主从复制的帐号，在master和candicate master的主机上创建即可
mysql>grant replication slave on *.* to mharep@'192.168.10.%' identified by 'test@123';
#创建MHA管理账号，所有mysql服务器上都需要
mysql>grant all privileges on *.* to manager@'192.168.10.%' identified by 'test@123';(root@localhost) [(none)]> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000015 | 154 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
#在mysql-master02机器上创建同步账号和manager账号
#创建一个用于主从复制的帐号，在master和candicate master的主机上创建即可
mysql>grant replication slave on *.* to mharep@'192.168.10.%' identified by 'test@123';
#创建MHA管理账号，所有mysql服务器上都需要
mysql>grant all privileges on *.* to manager@'192.168.10.%' identified by 'test@123';
mysql>CHANGE MASTER TO MASTER_HOST='192.168.10.71', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000015', MASTER_LOG_POS=154, MASTER_USER='mharep', MASTER_PASSWORD='test@123';
mysql> start slave;
#在mysql-master01机器上创建manager账号
#创建MHA管理账号，所有mysql服务器上都需要
mysql>grant all privileges on *.* to manager@'192.168.10.%' identified by 'test@123';
mysql>CHANGE MASTER TO MASTER_HOST='192.168.10.71', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000015', MASTER_LOG_POS=154, MASTER_USER='mharep', MASTER_PASSWORD='test@123';
mysql> start slave;
##查看两台slave的状态，看到 Slave_IO_Running: Yes 和 Slave_SQL_Running: Yes，就代表同步正常
(root@localhost) [(none)]> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.10.71
Master_User: mharep
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000015
Read_Master_Log_Pos: 154
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 320
Relay_Master_Log_File: mysql-bin.000015
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
#三台mysql配置文件修改完成后，需要重启一下mysql服务
2.4、查看三台的半同步相关信息和半同步状态
#查看master01的半同步相关信息和状态
(root@localhost) [(none)]> show variables like '%rpl_semi_sync%';
+-------------------------------------------+------------+
| Variable_name | Value |
+-------------------------------------------+------------+
| rpl_semi_sync_master_enabled | ON |
| rpl_semi_sync_master_timeout | 1000 |
| rpl_semi_sync_master_trace_level | 32 |
| rpl_semi_sync_master_wait_for_slave_count | 1 |
| rpl_semi_sync_master_wait_no_slave | ON |
| rpl_semi_sync_master_wait_point | AFTER_SYNC |
| rpl_semi_sync_slave_enabled | ON |
| rpl_semi_sync_slave_trace_level | 32 |
+-------------------------------------------+------------+
(root@localhost) [(none)]> show status like '%rpl_semi_sync%';
+--------------------------------------------+-------+
| Variable_name | Value |
+--------------------------------------------+-------+
| Rpl_semi_sync_master_clients | 2 |
| Rpl_semi_sync_master_net_avg_wait_time | 0 |
| Rpl_semi_sync_master_net_wait_time | 0 |
| Rpl_semi_sync_master_net_waits | 0 |
| Rpl_semi_sync_master_no_times | 0 |
| Rpl_semi_sync_master_no_tx | 0 |
| Rpl_semi_sync_master_status | ON |
| Rpl_semi_sync_master_timefunc_failures | 0 |
| Rpl_semi_sync_master_tx_avg_wait_time | 0 |
| Rpl_semi_sync_master_tx_wait_time | 0 |
| Rpl_semi_sync_master_tx_waits | 0 |
| Rpl_semi_sync_master_wait_pos_backtraverse | 0 |
| Rpl_semi_sync_master_wait_sessions | 0 |
| Rpl_semi_sync_master_yes_tx | 0 |
| Rpl_semi_sync_slave_status | OFF |
+--------------------------------------------+-------+
#查看master02的半同步相关信息和状态
(root@localhost) [(none)]> show variables like '%rpl_semi_sync%';
+-------------------------------------------+------------+
| Variable_name | Value |
+-------------------------------------------+------------+
| rpl_semi_sync_master_enabled | ON |
| rpl_semi_sync_master_timeout | 10000 |
| rpl_semi_sync_master_trace_level | 32 |
| rpl_semi_sync_master_wait_for_slave_count | 1 |
| rpl_semi_sync_master_wait_no_slave | ON |
| rpl_semi_sync_master_wait_point | AFTER_SYNC |
| rpl_semi_sync_slave_enabled | ON |
| rpl_semi_sync_slave_trace_level | 32 |
+-------------------------------------------+------------+
8 rows in set (0.01 sec)
(root@localhost) [(none)]> show status like '%rpl_semi_sync%';
+--------------------------------------------+-------+
| Variable_name | Value |
+--------------------------------------------+-------+
| Rpl_semi_sync_master_clients | 0 |
| Rpl_semi_sync_master_net_avg_wait_time | 0 |
| Rpl_semi_sync_master_net_wait_time | 0 |
| Rpl_semi_sync_master_net_waits | 0 |
| Rpl_semi_sync_master_no_times | 0 |
| Rpl_semi_sync_master_no_tx | 0 |
| Rpl_semi_sync_master_status | ON |
| Rpl_semi_sync_master_timefunc_failures | 0 |
| Rpl_semi_sync_master_tx_avg_wait_time | 0 |
| Rpl_semi_sync_master_tx_wait_time | 0 |
| Rpl_semi_sync_master_tx_waits | 0 |
| Rpl_semi_sync_master_wait_pos_backtraverse | 0 |
| Rpl_semi_sync_master_wait_sessions | 0 |
| Rpl_semi_sync_master_yes_tx | 0 |
| Rpl_semi_sync_slave_status | ON |
+--------------------------------------------+-------+
#查看slave01的半同步相关信息和状态
(root@localhost) [(none)]> show variables like '%rpl_semi_sync%';
+-------------------------------------------+------------+
| Variable_name | Value |
+-------------------------------------------+------------+
| rpl_semi_sync_master_enabled | OFF |
| rpl_semi_sync_master_timeout | 10000 |
| rpl_semi_sync_master_trace_level | 32 |
| rpl_semi_sync_master_wait_for_slave_count | 1 |
| rpl_semi_sync_master_wait_no_slave | ON |
| rpl_semi_sync_master_wait_point | AFTER_SYNC |
| rpl_semi_sync_slave_enabled | ON |
| rpl_semi_sync_slave_trace_level | 32 |
+-------------------------------------------+------------+
(root@localhost) [(none)]> show status like '%rpl_semi_sync%';
+--------------------------------------------+-------+
| Variable_name | Value |
+--------------------------------------------+-------+
| Rpl_semi_sync_master_clients | 0 |
| Rpl_semi_sync_master_net_avg_wait_time | 0 |
| Rpl_semi_sync_master_net_wait_time | 0 |
| Rpl_semi_sync_master_net_waits | 0 |
| Rpl_semi_sync_master_no_times | 0 |
| Rpl_semi_sync_master_no_tx | 0 |
| Rpl_semi_sync_master_status | OFF |
| Rpl_semi_sync_master_timefunc_failures | 0 |
| Rpl_semi_sync_master_tx_avg_wait_time | 0 |
| Rpl_semi_sync_master_tx_wait_time | 0 |
| Rpl_semi_sync_master_tx_waits | 0 |
| Rpl_semi_sync_master_wait_pos_backtraverse | 0 |
| Rpl_semi_sync_master_wait_sessions | 0 |
| Rpl_semi_sync_master_yes_tx | 0 |
| Rpl_semi_sync_slave_status | ON |
+--------------------------------------------+-------+
#有几个状态参数值得关注的：
rpl_semi_sync_master_status ：显示主服务是异步复制模式还是半同步复制模式
rpl_semi_sync_master_clients ：显示有多少个从服务器配置为半同步复制模式
rpl_semi_sync_master_yes_tx ：显示从服务器确认成功提交的数量
rpl_semi_sync_master_no_tx ：显示从服务器确认不成功提交的数量
rpl_semi_sync_master_tx_avg_wait_time ：事务因开启 semi_sync ，平均需要额外等待的时间
rpl_semi_sync_master_net_avg_wait_time ：事务进入等待队列后，到网络平均等待时间
五、安装和配置MHA
1、安装MHA相关依赖软件
#以下4台服务器都需安装相关软件
yum -y install perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-ParallelForkManager perl-Config-IniFiles ncftp perl-Params-Validate perl-CPAN perl-TestMock-LWP.noarch perl-LWP-Authen-Negotiate.noarch perl-devel perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker perl-ExtUtils-Embed#这个安装时间会有点长
perl -MCPAN -e shellcpan -i "inc::Module::Install"2、安装MHA-manager和MHA-node软件
#软件下载地址https://github.com/yoshinorim，如果没有外网的可以在我这边下载(https://download.csdn.net/download/m0_48898914/83934238)
2.1、安装MHA-node，三台mysql都需要安装MHA-node软件
[root@mysql-master01] unzip mha4mysql-node-master.zip
[root@mysql-master01] cd mha4mysql-node-master
[root@mysql-master01] perl Makefile.PL
*** Module::AutoInstall version 1.06
*** Checking for Perl dependencies...
[Core Features]
- DBI ...loaded. (1.627)
- DBD::mysql ...loaded. (4.023)
*** Module::AutoInstall configuration finished.
Checking if your kit is complete...
Looks good
Writing Makefile for mha4mysql::node
[root@mysql-master01] make && make install
2.2、mysql-manager机器上安装MHA-master和MHA-node软件
[root@mysql-manager] unzip mha4mysql-manager-master.zip
[root@mysql-manager] cd mha4mysql-manager-master
[root@mysql-manager] perl Makefile.PL
*** Module::AutoInstall version 1.06
*** Checking for Perl dependencies...
[Core Features]
- DBI ...loaded. (1.627)
- DBD::mysql ...loaded. (4.023)
- Time::HiRes ...loaded. (1.9725)
- Config::Tiny ...loaded. (2.14)
- Log::Dispatch ...loaded. (2.41)
- Parallel::ForkManager ...missing.
- MHA::NodeConst ...loaded. (0.58)
==> Auto-install the 1 mandatory module(s) from CPAN? [y] y
*** Dependencies will be installed the next time you type 'make'.
*** Module::AutoInstall configuration finished.
Checking if your kit is complete...
Looks good
Warning: prerequisite Parallel::ForkManager 0 not found.
Writing Makefile for mha4mysql::manager
[root@mysql-manager] make && make install
[root@mysql-master01] unzip mha4mysql-node-master.zip
[root@mysql-master01] cd mha4mysql-node-master
[root@mysql-master01] perl Makefile.PL
*** Module::AutoInstall version 1.06
*** Checking for Perl dependencies...
[Core Features]
- DBI ...loaded. (1.627)
- DBD::mysql ...loaded. (4.023)
*** Module::AutoInstall configuration finished.
Checking if your kit is complete...
Looks good
Writing Makefile for mha4mysql::node
[root@mysql-master01] make && make install
3、配置MHA-manager服务
3.1、创建目录，copy配置文件到相应的目录下
mkdir -p /etc/mha/
mkdir -p /masterha/app1/
mkdir /script/masterha
cd /root/mha4mysql-manager-master/
cp samples/conf/* /etc/mha
cp samples/scrip/* /script/masterha
3.2、修改/etc/mha/app1.cnf配置文件
[server default]
manager_workdir=/masterha/app1
manager_log=/masterha/app1/manager.log
#mysql用户名和密码
user=manager
password=test@123
#系统ssh用户
ssh_user=root
#mysql同步复制用户和密码
repl_user=mharep
repl_password=test@123
#监控
ping_interval=1
#用防止master故障时，切换时slave有延迟，卡在那里切不过来
check_repl_delay=0
#切换调用的脚本
master_ip_failover_script=/usr/local/bin/master_ip_failover
master_ip_online_change_script=/usr/local/bin/master_ip_online_change
#secondary_check_script=masterha_secondary_check -s 192.168.10.73 -s 192.168.10.72
secondary_check_script=masterha_secondary_check -s 192.168.10.73
[server1]
hostname=192.168.10.71
port=3306
master_binlog_dir=/var/lib/mysql
candidate_master=1
[server2]
hostname=192.168.10.72
port=3306
master_binlog_dir=/var/lib/mysql
candidate_master=1
[server3]
hostname=192.168.10.73
port=3306
master_binlog_dir=/var/lib/mysql
no_master=1
3.3、验证ssh有效性
# masterha_check_ssh --global_conf=/etc/mha/masterha_default.cnf --conf=/etc/mha/app1.cnf
Wed Mar 9 10:10:28 2022 - [info] Reading default configuration from /etc/mha/masterha_default.cnf..
Wed Mar 9 10:10:28 2022 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Wed Mar 9 10:10:28 2022 - [info] Reading server configuration from /etc/mha/app1.cnf..
Wed Mar 9 10:10:28 2022 - [info] Starting SSH connection tests..
Wed Mar 9 10:10:29 2022 - [debug]
Wed Mar 9 10:10:28 2022 - [debug] Connecting via SSH from [email protected](192.168.10.71:22) to [email protected](192.168.10.72:22)..
Wed Mar 9 10:10:28 2022 - [debug] ok.
Wed Mar 9 10:10:28 2022 - [debug] Connecting via SSH from [email protected](192.168.10.71:22) to [email protected](192.168.10.73:22)..
Wed Mar 9 10:10:29 2022 - [debug] ok.
Wed Mar 9 10:10:29 2022 - [debug]
Wed Mar 9 10:10:28 2022 - [debug] Connecting via SSH from [email protected](192.168.10.72:22) to [email protected](192.168.10.71:22)..
Wed Mar 9 10:10:29 2022 - [debug] ok.
Wed Mar 9 10:10:29 2022 - [debug] Connecting via SSH from [email protected](192.168.10.72:22) to [email protected](192.168.10.73:22)..
Wed Mar 9 10:10:29 2022 - [debug] ok.
Wed Mar 9 10:10:30 2022 - [debug]
Wed Mar 9 10:10:29 2022 - [debug] Connecting via SSH from [email protected](192.168.10.73:22) to [email protected](192.168.10.71:22)..
Wed Mar 9 10:10:29 2022 - [debug] ok.
Wed Mar 9 10:10:29 2022 - [debug] Connecting via SSH from [email protected](192.168.10.73:22) to [email protected](192.168.10.72:22)..
Wed Mar 9 10:10:30 2022 - [debug] ok.
Wed Mar 9 10:10:30 2022 - [info] All SSH connection tests passed successfully.
3.4、集群复制的有效性验证（mysql必须都启动）
# masterha_check_repl --global_conf=/etc/mha/masterha_default.cnf --conf=/etc/mha/app1.cnf
Wed Mar 9 10:29:30 2022 - [info] Reading default configuration from /etc/mha/masterha_default.cnf..
Wed Mar 9 10:29:30 2022 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Wed Mar 9 10:29:30 2022 - [info] Reading server configuration from /etc/mha/app1.cnf..
Wed Mar 9 10:29:30 2022 - [info] MHA::MasterMonitor version 0.58.
Wed Mar 9 10:29:31 2022 - [info] GTID failover mode = 0
Wed Mar 9 10:29:31 2022 - [info] Dead Servers:
Wed Mar 9 10:29:31 2022 - [info] Alive Servers:
Wed Mar 9 10:29:31 2022 - [info] 192.168.10.71(192.168.10.71:3306)
Wed Mar 9 10:29:31 2022 - [info] 192.168.10.72(192.168.10.72:3306)
Wed Mar 9 10:29:31 2022 - [info] 192.168.10.73(192.168.10.73:3306)
Wed Mar 9 10:29:31 2022 - [info] Alive Slaves:
Wed Mar 9 10:29:31 2022 - [info] 192.168.10.72(192.168.10.72:3306) Version=5.7.35-log (oldest major version between slaves) log-bin:enabled
Wed Mar 9 10:29:31 2022 - [info] Replicating from 192.168.10.71(192.168.10.71:3306)
Wed Mar 9 10:29:31 2022 - [info] Primary candidate for the new Master (candidate_master is set)
Wed Mar 9 10:29:31 2022 - [info] 192.168.10.73(192.168.10.73:3306) Version=5.7.35-log (oldest major version between slaves) log-bin:enabled
Wed Mar 9 10:29:31 2022 - [info] Replicating from 192.168.10.71(192.168.10.71:3306)
Wed Mar 9 10:29:31 2022 - [info] Not candidate for the new Master (no_master is set)
Wed Mar 9 10:29:31 2022 - [info] Current Alive Master: 192.168.10.71(192.168.10.71:3306)
Wed Mar 9 10:29:31 2022 - [info] Checking slave configurations..
Wed Mar 9 10:29:31 2022 - [info] read_only=1 is not set on slave 192.168.10.72(192.168.10.72:3306).
Wed Mar 9 10:29:31 2022 - [warning] relay_log_purge=0 is not set on slave 192.168.10.73(192.168.10.73:3306).
Wed Mar 9 10:29:31 2022 - [info] Checking replication filtering settings..
Wed Mar 9 10:29:31 2022 - [info] binlog_do_db= , binlog_ignore_db=
Wed Mar 9 10:29:31 2022 - [info] Replication filtering check ok.
Wed Mar 9 10:29:31 2022 - [info] GTID (with auto-pos) is not supported
Wed Mar 9 10:29:31 2022 - [info] Starting SSH connection tests..
Wed Mar 9 10:29:33 2022 - [info] All SSH connection tests passed successfully.
Wed Mar 9 10:29:33 2022 - [info] Checking MHA Node version..
Wed Mar 9 10:29:34 2022 - [info] Version check ok.
Wed Mar 9 10:29:34 2022 - [info] Checking SSH publickey authentication settings on the current master..
Wed Mar 9 10:29:34 2022 - [info] HealthCheck: SSH to 192.168.10.71 is reachable.
Wed Mar 9 10:29:35 2022 - [info] Master MHA Node version is 0.58.
Wed Mar 9 10:29:35 2022 - [info] Checking recovery script configurations on 192.168.10.71(192.168.10.71:3306)..
Wed Mar 9 10:29:35 2022 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/data/log/masterha/save_binary_logs_test --manager_version=0.58 --start_file=mysql-bin.000015
Wed Mar 9 10:29:35 2022 - [info] Connecting to [email protected](192.168.10.71:22)..
Creating /data/log/masterha if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /var/lib/mysql, up to mysql-bin.000015
Wed Mar 9 10:29:35 2022 - [info] Binlog setting check done.
Wed Mar 9 10:29:35 2022 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Wed Mar 9 10:29:35 2022 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='manager' --slave_host=192.168.10.72 --slave_ip=192.168.10.72 --slave_port=3306 --workdir=/data/log/masterha --target_version=5.7.35-log --manager_version=0.58 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx
Wed Mar 9 10:29:35 2022 - [info] Connecting to [email protected](192.168.10.72:22)..
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to relay-bin.000002
Temporary relay log file is /var/lib/mysql/relay-bin.000002
Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Wed Mar 9 10:29:36 2022 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='manager' --slave_host=192.168.10.73 --slave_ip=192.168.10.73 --slave_port=3306 --workdir=/data/log/masterha --target_version=5.7.35-log --manager_version=0.58 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx
Wed Mar 9 10:29:36 2022 - [info] Connecting to [email protected](192.168.10.73:22)..
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to relay-bin.000002
Temporary relay log file is /var/lib/mysql/relay-bin.000002
Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Wed Mar 9 10:29:36 2022 - [info] Slaves settings check done.
Wed Mar 9 10:29:36 2022 - [info]
192.168.10.71(192.168.10.71:3306) (current master)
+--192.168.10.72(192.168.10.72:3306)
+--192.168.10.73(192.168.10.73:3306)
Wed Mar 9 10:29:36 2022 - [info] Checking replication health on 192.168.10.72..
Wed Mar 9 10:29:36 2022 - [info] ok.
Wed Mar 9 10:29:36 2022 - [info] Checking replication health on 192.168.10.73..
Wed Mar 9 10:29:36 2022 - [info] ok.
Wed Mar 9 10:29:36 2022 - [info] Checking master_ip_failover_script status:
Wed Mar 9 10:29:36 2022 - [info] /script/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.10.71 --orig_master_ip=192.168.10.71 --orig_master_port=3306
IN SCRIPT TEST====/sbin/ifconfig ens192:230 down==/sbin/ifconfig ens192:230 192.168.10.230/24===
Checking the Status of the script.. OK
Wed Mar 9 10:29:36 2022 - [info] OK.
Wed Mar 9 10:29:36 2022 - [warning] shutdown_script is not defined.
Wed Mar 9 10:29:36 2022 - [info] Got exit code 0 (Not master dead).
MySQL Replication Health is OK.
#验证成功的话，从日志就可以看出，会自动识别出所有服务器和主从状况
3.5、启动MHA-manager服务
由于这个manager服务，不是守护进程，需要通过nohup或者screen协助运行
# masterha_manager --conf=/etc/mha/app1.cnf --global_conf=/etc/mha/masterha_default.cnf
Wed Mar 9 11:00:40 2022 - [info] Reading default configuration from /etc/mha/masterha_default.cnf..
Wed Mar 9 11:00:40 2022 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Wed Mar 9 11:00:40 2022 - [info] Reading server configuration from /etc/mha/app1.cnf..
3.6、状态检查
# masterha_check_status --global_conf=/etc/mha/masterha_default.cnf --conf=/etc/mha/app1.cnf
app1 (pid:22189) is running(0:PING_OK), master:192.168.10.71
#看查日志
#tail -f /masterha/app1/manager.log -n200
ed Mar 9 11:44:15 2022 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='manager' --slave_host=192.168.10.73 --slave_ip=192.168.10.73 --slave_port=3306 --workdir=/data/log/masterha --target_version=5.7.35-log --manager_version=0.58 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx
Wed Mar 9 11:44:15 2022 - [info] Connecting to [email protected](192.168.10.73:22)..
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to relay-bin.000006
Temporary relay log file is /var/lib/mysql/relay-bin.000006
Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Wed Mar 9 11:44:16 2022 - [info] Slaves settings check done.
Wed Mar 9 11:44:16 2022 - [info]
192.168.10.71(192.168.10.71:3306) (current master)
+--192.168.10.72(192.168.10.72:3306)
+--192.168.10.73(192.168.10.73:3306)
Checking the Status of the script.. OK
Wed Mar 9 11:44:16 2022 - [info] OK.
Wed Mar 9 11:44:16 2022 - [warning] shutdown_script is not defined.
Wed Mar 9 11:44:16 2022 - [info] Set master ping interval 1 seconds.
Wed Mar 9 11:44:16 2022 - [info] Set secondary check script: masterha_secondary_check -s 192.168.10.73
Wed Mar 9 11:44:16 2022 - [info] Starting ping health check on 192.168.10.71(192.168.10.71:3306)..
Wed Mar 9 11:44:16 2022 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
六、设置VIP
1、在mysql-master01服务器上设置VIP
为什么要设置vip呢？应用服务一开始的写入IP就是VIP，当主库切换后，应用不需要切换主IP，让VIP从原来mysql-master01的服务器飘移到mysql-master02服务器上。
#ens192:230，ens192:代表网卡名称，230，我这里用vip的最后一组数字命名
#ifconfig ens192:230 192.168.10.230/24
#ifconfig
ens192: flags=4163 mtu 1500
inet 192.168.10.71 netmask 255.255.255.0 broadcast 192.168.10.255
inet6 fe80::20c:29ff:fe0b:ffe5 prefixlen 64 scopeid 0x20
ether 00:0c:29:0b:ff:e5 txqueuelen 1000 (Ethernet)
RX packets 5396314 bytes 1409615265 (1.3 GiB)
RX errors 0 dropped 141478 overruns 0 frame 0
TX packets 1030157 bytes 1197361685 (1.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens192:230: flags=4163 mtu 1500
inet 192.168.10.230 netmask 255.255.255.0 broadcast 192.168.10.255
ether 00:0c:29:0b:ff:e5 txqueuelen 1000 (Ethernet)
2、配置mysql-manager服务
2.1、配置脚本
vim /usr/local/bin/master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);
my $vip = '192.168.10.230/24'; # Virtual IP
my $key = "230";
my $ssh_start_vip = "/sbin/ifconfig ens192:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens192:$key down";
#my $ssh_Bcast_arp = "arping -c 3 -A 192.168.10.230"; #ARP回复模式，更新邻居。要是不加则服务器会自动等到vip缓存失效，期间VIP会有一定时间的不可用。
my $ssh_Bcast_arp = "arping -c 3 -A 192.168.10.230 -I ens192"; #ARP回复模式，更新邻居。要是不加则服务器会自动等到vip缓存失效，期间VIP会有一定时间的不可用
$ssh_user = "root";
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);
exit &main();
sub main {
print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
&start_arp();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
exit 0;
}
else {
&usage();
exit 1;
}
}
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
sub start_arp() {
`ssh $ssh_user\@$new_master_host \" $ssh_Bcast_arp \"`;
}
sub stop_vip() {
return 0 unless ($ssh_user);
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip -
-orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}
vim /usr/local/bin/master_ip_online_change
#!/usr/bin/env perl
use strict;
use warnings FATAL =>'all';
use Getopt::Long;
my $vip = '192.168.10.230/24'; # Virtual IP
my $key = "1";
my $ssh_start_vip = "/sbin/ifconfig ens192:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens192:$key down";
my $exit_code = 0;
my $ssh_Bcast_arp = "arping -c 3 -A 192.168.10.230 -I ens192"; #ARP回复模式，更新邻居。要是不加则服务器会自动等到vip缓存失效，期间VIP会有一定时间的不可用。
my (
$command, $orig_master_is_new_slave, $orig_master_host,
$orig_master_ip, $orig_master_port, $orig_master_user,
$orig_master_password, $orig_master_ssh_user, $new_master_host,
$new_master_ip, $new_master_port, $new_master_user,
$new_master_password, $new_master_ssh_user,
);
GetOptions(
'command=s' => \$command,
'orig_master_is_new_slave' => \$orig_master_is_new_slave,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'orig_master_user=s' => \$orig_master_user,
'orig_master_password=s' => \$orig_master_password,
'orig_master_ssh_user=s' => \$orig_master_ssh_user,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
'new_master_user=s' => \$new_master_user,
'new_master_password=s' => \$new_master_password,
'new_master_ssh_user=s' => \$new_master_ssh_user,
);
#my $ssh_Bcast_arp = "arping -c 3 -A 192.168.10.230"; #ARP回复模式，更新邻居。要是不加则服务器会自动等到vip缓存失效，期间VIP会有一定时间的不可用。
exit &main();
sub main {
#print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
# $orig_master_host, $orig_master_ip, $orig_master_port are passed.
# If you manage master ip address at global catalog database,
# invalidate orig_master_ip here.
my $exit_code = 1;
eval {
print "\n\n\n***************************************************************\n";
print "Disabling the VIP - $vip on old master: $orig_master_host\n";
print "***************************************************************\n\n\n\n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
# all arguments are passed.
# If you manage master ip address at global catalog database,
# activate new_master_ip here.
# You can also grant write access (create user, set read_only=0, etc) here.
my $exit_code = 10;
eval {
print "\n\n\n***************************************************************\n";
print "Enabling the VIP - $vip on new master: $new_master_host \n";
print "***************************************************************\n\n\n\n";
&start_vip();
&start_arp();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
`ssh $orig_master_ssh_user\@$orig_master_host \" $ssh_start_vip \"`;
exit 0;
}
else {
&usage();
exit 1;
}
}
# A simple system call that enable the VIP on the new master
sub start_vip() {
`ssh $new_master_ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
sub start_arp() {
`ssh $new_master_ssh_user\@$new_master_host \" $ssh_Bcast_arp \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $orig_master_ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --ori
g_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}3、测试服务
3.1、停掉mysql-master01的mysql
systemctl stop mysqld3.2、观察mysql-manager上的MHA-manager日志
#从日志看，mysql主库已经从mysql-master01切换到mysql-master02服务器上
Thu Mar 10 10:43:51 2022 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
Thu Mar 10 10:43:51 2022 - [info] Executing secondary network check script: masterha_secondary_check -s 192.168.10.73 --user=root --master_host=192.168.10.71 --master_ip=192.168.10.71 --master_port=3306 --master_user=manager --master_password=test@123 --ping_type=SELECT
Thu Mar 10 10:43:51 2022 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql --output_file=/data/log/masterha/save_binary_logs_test --manager_version=0.58 --binlog_prefix=mysql-bin
Monitoring server 192.168.10.73 is reachable, Master is not reachable from 192.168.10.73. OK.
Thu Mar 10 10:43:51 2022 - [info] Master is not reachable from all other monitoring servers. Failover should start.
Thu Mar 10 10:43:51 2022 - [info] HealthCheck: SSH to 192.168.10.71 is reachable.
Thu Mar 10 10:43:52 2022 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.10.71' (111))
Thu Mar 10 10:43:52 2022 - [warning] Connection failed 2 time(s)..
Thu Mar 10 10:43:53 2022 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.10.71' (111))
Thu Mar 10 10:43:53 2022 - [warning] Connection failed 3 time(s)..
Thu Mar 10 10:43:54 2022 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.10.71' (111))
Thu Mar 10 10:43:54 2022 - [warning] Connection failed 4 time(s)..
Thu Mar 10 10:43:54 2022 - [warning] Master is not reachable from health checker!
Thu Mar 10 10:43:54 2022 - [warning] Master 192.168.10.71(192.168.10.71:3306) is not reachable!
Thu Mar 10 10:43:54 2022 - [warning] SSH is reachable.
Thu Mar 10 10:43:54 2022 - [info] Connecting to a master server failed. Reading configuration file /etc/mha/masterha_default.cnf and /etc/mha/app1.cnf again, and trying to connect to all servers to check server status..
Thu Mar 10 10:43:54 2022 - [info] Reading default configuration from /etc/mha/masterha_default.cnf..
Thu Mar 10 10:43:54 2022 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Thu Mar 10 10:43:54 2022 - [info] Reading server configuration from /etc/mha/app1.cnf..
Thu Mar 10 10:43:55 2022 - [info] GTID failover mode = 0
Thu Mar 10 10:43:55 2022 - [info] Dead Servers:
Thu Mar 10 10:43:55 2022 - [info] 192.168.10.71(192.168.10.71:3306)
Thu Mar 10 10:43:55 2022 - [info] Alive Servers:
Thu Mar 10 10:43:55 2022 - [info] 192.168.10.72(192.168.10.72:3306)
Thu Mar 10 10:43:55 2022 - [info] 192.168.10.73(192.168.10.73:3306)
Thu Mar 10 10:43:55 2022 - [info] Alive Slaves:
Thu Mar 10 10:43:55 2022 - [info] 192.168.10.72(192.168.10.72:3306) Version=5.7.35-log (oldest major version between slaves) log-bin:enabled
Thu Mar 10 10:43:55 2022 - [info] Replicating from 192.168.10.71(192.168.10.71:3306)
Thu Mar 10 10:43:55 2022 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Mar 10 10:43:55 2022 - [info] 192.168.10.73(192.168.10.73:3306) Version=5.7.35-log (oldest major version between slaves) log-bin:enabled
Thu Mar 10 10:43:55 2022 - [info] Replicating from 192.168.10.71(192.168.10.71:3306)
Thu Mar 10 10:43:55 2022 - [info] Not candidate for the new Master (no_master is set)
Thu Mar 10 10:43:55 2022 - [info] Checking slave configurations..
Thu Mar 10 10:43:55 2022 - [info] read_only=1 is not set on slave 192.168.10.72(192.168.10.72:3306).
Thu Mar 10 10:43:55 2022 - [warning] relay_log_purge=0 is not set on slave 192.168.10.73(192.168.10.73:3306).
Thu Mar 10 10:43:55 2022 - [info] Checking replication filtering settings..
Thu Mar 10 10:43:55 2022 - [info] Replication filtering check ok.
Thu Mar 10 10:43:55 2022 - [info] Master is down!
Thu Mar 10 10:43:55 2022 - [info] Terminating monitoring script.
Thu Mar 10 10:43:55 2022 - [info] Got exit code 20 (Master dead).
Thu Mar 10 10:43:55 2022 - [info] MHA::MasterFailover version 0.58.
Thu Mar 10 10:43:55 2022 - [info] Starting master failover.
Thu Mar 10 10:43:55 2022 - [info]
Thu Mar 10 10:43:55 2022 - [info] * Phase 1: Configuration Check Phase..
Thu Mar 10 10:43:55 2022 - [info]
Thu Mar 10 10:43:56 2022 - [info] GTID failover mode = 0
Thu Mar 10 10:43:56 2022 - [info] Dead Servers:
Thu Mar 10 10:43:56 2022 - [info] 192.168.10.71(192.168.10.71:3306)
Thu Mar 10 10:43:56 2022 - [info] Checking master reachability via MySQL(double check)...
Thu Mar 10 10:43:56 2022 - [info] ok.
Thu Mar 10 10:43:56 2022 - [info] Alive Servers:
Thu Mar 10 10:43:56 2022 - [info] 192.168.10.72(192.168.10.72:3306)
Thu Mar 10 10:43:56 2022 - [info] 192.168.10.73(192.168.10.73:3306)
Thu Mar 10 10:43:56 2022 - [info] Alive Slaves:
Thu Mar 10 10:43:56 2022 - [info] 192.168.10.72(192.168.10.72:3306) Version=5.7.35-log (oldest major version between slaves) log-bin:enabled
Thu Mar 10 10:43:56 2022 - [info] Replicating from 192.168.10.71(192.168.10.71:3306)
Thu Mar 10 10:43:56 2022 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Mar 10 10:43:56 2022 - [info] 192.168.10.73(192.168.10.73:3306) Version=5.7.35-log (oldest major version between slaves) log-bin:enabled
Thu Mar 10 10:43:56 2022 - [info] Replicating from 192.168.10.71(192.168.10.71:3306)
Thu Mar 10 10:43:56 2022 - [info] Not candidate for the new Master (no_master is set)
Thu Mar 10 10:43:56 2022 - [info] Starting Non-GTID based failover.
Thu Mar 10 10:43:56 2022 - [info]
Thu Mar 10 10:43:56 2022 - [info] ** Phase 1: Configuration Check Phase completed.
Thu Mar 10 10:43:56 2022 - [info]
Thu Mar 10 10:43:56 2022 - [info] * Phase 2: Dead Master Shutdown Phase..
Thu Mar 10 10:43:56 2022 - [info]
Thu Mar 10 10:43:56 2022 - [info] Forcing shutdown so that applications never connect to the current master..
Thu Mar 10 10:43:56 2022 - [info] Executing master IP deactivation script:
Thu Mar 10 10:43:56 2022 - [info] /usr/local/bin/master_ip_failover --orig_master_host=192.168.10.71 --orig_master_ip=192.168.10.71 --orig_master_port=3306 --command=stopssh --ssh_user=root
IN SCRIPT TEST====/sbin/ifconfig ens192:230 down==/sbin/ifconfig ens192:230 192.168.10.230/24===
Disabling the VIP on old master: 192.168.10.71
Thu Mar 10 10:43:56 2022 - [info] done.
Thu Mar 10 10:43:56 2022 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Thu Mar 10 10:43:56 2022 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Thu Mar 10 10:43:56 2022 - [info]
Thu Mar 10 10:43:56 2022 - [info] * Phase 3: Master Recovery Phase..
Thu Mar 10 10:43:56 2022 - [info]
Thu Mar 10 10:43:56 2022 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Thu Mar 10 10:43:56 2022 - [info]
Thu Mar 10 10:43:56 2022 - [info] The latest binary log file/position on all slaves is mysql-bin.000001:154
Thu Mar 10 10:43:56 2022 - [info] Latest slaves (Slaves that received relay log files to the latest):
Thu Mar 10 10:43:56 2022 - [info] 192.168.10.72(192.168.10.72:3306) Version=5.7.35-log (oldest major version between slaves) log-bin:enabled
Thu Mar 10 10:43:56 2022 - [info] Replicating from 192.168.10.71(192.168.10.71:3306)
Thu Mar 10 10:43:56 2022 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Mar 10 10:43:56 2022 - [info] 192.168.10.73(192.168.10.73:3306) Version=5.7.35-log (oldest major version between slaves) log-bin:enabled
Thu Mar 10 10:43:56 2022 - [info] Replicating from 192.168.10.71(192.168.10.71:3306)
Thu Mar 10 10:43:56 2022 - [info] Not candidate for the new Master (no_master is set)
Thu Mar 10 10:43:56 2022 - [info] The oldest binary log file/position on all slaves is mysql-bin.000001:154
Thu Mar 10 10:43:56 2022 - [info] Oldest slaves:
Thu Mar 10 10:43:56 2022 - [info] 192.168.10.72(192.168.10.72:3306) Version=5.7.35-log (oldest major version between slaves) log-bin:enabled
Thu Mar 10 10:43:56 2022 - [info] Replicating from 192.168.10.71(192.168.10.71:3306)
Thu Mar 10 10:43:56 2022 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Mar 10 10:43:56 2022 - [info] 192.168.10.73(192.168.10.73:3306) Version=5.7.35-log (oldest major version between slaves) log-bin:enabled
Thu Mar 10 10:43:56 2022 - [info] Replicating from 192.168.10.71(192.168.10.71:3306)
Thu Mar 10 10:43:56 2022 - [info] Not candidate for the new Master (no_master is set)
Thu Mar 10 10:43:56 2022 - [info]
Thu Mar 10 10:43:56 2022 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Thu Mar 10 10:43:56 2022 - [info]
Thu Mar 10 10:43:57 2022 - [info] Fetching dead master's binary logs..
Thu Mar 10 10:43:57 2022 - [info] Executing command on the dead master 192.168.10.71(192.168.10.71:3306): save_binary_logs --command=save --start_file=mysql-bin.000001 --start_pos=154 --binlog_dir=/var/lib/mysql --output_file=/data/log/masterha/saved_master_binlog_from_192.168.10.71_3306_20220310104355.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.58
Creating /data/log/masterha if not exists.. ok.
Concat binary/relay logs from mysql-bin.000001 pos 154 to mysql-bin.000001 EOF into /data/log/masterha/saved_master_binlog_from_192.168.10.71_3306_20220310104355.binlog ..
Binlog Checksum enabled
Dumping binlog format description event, from position 0 to 154.. ok.
No need to dump effective binlog data from /var/lib/mysql/mysql-bin.000001 (pos starts 154, filesize 154). Skipping.
Binlog Checksum enabled
/data/log/masterha/saved_master_binlog_from_192.168.10.71_3306_20220310104355.binlog has no effective data events.
Event not exists.
Thu Mar 10 10:43:57 2022 - [info] Additional events were not found from the orig master. No need to save.
Thu Mar 10 10:43:57 2022 - [info]
Thu Mar 10 10:43:57 2022 - [info] * Phase 3.3: Determining New Master Phase..
Thu Mar 10 10:43:57 2022 - [info]
Thu Mar 10 10:43:57 2022 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Thu Mar 10 10:43:57 2022 - [info] All slaves received relay logs to the same position. No need to resync each other.
Thu Mar 10 10:43:57 2022 - [info] Searching new master from slaves..
Thu Mar 10 10:43:57 2022 - [info] Candidate masters from the configuration file:
Thu Mar 10 10:43:57 2022 - [info] 192.168.10.72(192.168.10.72:3306) Version=5.7.35-log (oldest major version between slaves) log-bin:enabled
Thu Mar 10 10:43:57 2022 - [info] Replicating from 192.168.10.71(192.168.10.71:3306)
Thu Mar 10 10:43:57 2022 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Mar 10 10:43:57 2022 - [info] Non-candidate masters:
Thu Mar 10 10:43:57 2022 - [info] 192.168.10.73(192.168.10.73:3306) Version=5.7.35-log (oldest major version between slaves) log-bin:enabled
Thu Mar 10 10:43:57 2022 - [info] Replicating from 192.168.10.71(192.168.10.71:3306)
Thu Mar 10 10:43:57 2022 - [info] Not candidate for the new Master (no_master is set)
Thu Mar 10 10:43:57 2022 - [info] Searching from candidate_master slaves which have received the latest relay log events..
Thu Mar 10 10:43:57 2022 - [info] New master is 192.168.10.72(192.168.10.72:3306)
Thu Mar 10 10:43:57 2022 - [info] Starting master failover..
Thu Mar 10 10:43:57 2022 - [info]
From:
192.168.10.71(192.168.10.71:3306) (current master)
+--192.168.10.72(192.168.10.72:3306)
+--192.168.10.73(192.168.10.73:3306)
To:
192.168.10.72(192.168.10.72:3306) (new master)
+--192.168.10.73(192.168.10.73:3306)
Thu Mar 10 10:43:57 2022 - [info]
Thu Mar 10 10:43:57 2022 - [info] * Phase 3.4: New Master Diff Log Generation Phase..
Thu Mar 10 10:43:57 2022 - [info]
Thu Mar 10 10:43:57 2022 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Thu Mar 10 10:43:57 2022 - [info]
Thu Mar 10 10:43:57 2022 - [info] * Phase 3.5: Master Log Apply Phase..
Thu Mar 10 10:43:57 2022 - [info]
Thu Mar 10 10:43:57 2022 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Thu Mar 10 10:43:57 2022 - [info] Starting recovery on 192.168.10.72(192.168.10.72:3306)..
Thu Mar 10 10:43:57 2022 - [info] This server has all relay logs. Waiting all logs to be applied..
Thu Mar 10 10:43:57 2022 - [info] done.
Thu Mar 10 10:43:57 2022 - [info] All relay logs were successfully applied.
Thu Mar 10 10:43:57 2022 - [info] Getting new master's binlog name and position..
Thu Mar 10 10:43:57 2022 - [info] mysql-bin.000001:154
Thu Mar 10 10:43:57 2022 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.10.72', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=154, MASTER_USER='mharep', MASTER_PASSWORD='xxx';
Thu Mar 10 10:43:57 2022 - [info] Executing master IP activate script:
Thu Mar 10 10:43:57 2022 - [info] /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.10.71 --orig_master_ip=192.168.10.71 --orig_master_port=3306 --new_master_host=192.168.10.72 --new_master_ip=192.168.10.72 --new_master_port=3306 --new_master_user='manager' --new_master_password=xxx
Unknown option: new_master_user
Unknown option: new_master_password
IN SCRIPT TEST====/sbin/ifconfig ens192:230 down==/sbin/ifconfig ens192:230 192.168.10.230/24===
Enabling the VIP - 192.168.10.230/24 on the new master - 192.168.10.72
Thu Mar 10 10:44:00 2022 - [info] OK.
Thu Mar 10 10:44:00 2022 - [info] ** Finished master recovery successfully.
Thu Mar 10 10:44:00 2022 - [info] * Phase 3: Master Recovery Phase completed.
Thu Mar 10 10:44:00 2022 - [info]
Thu Mar 10 10:44:00 2022 - [info] * Phase 4: Slaves Recovery Phase..
Thu Mar 10 10:44:00 2022 - [info]
Thu Mar 10 10:44:00 2022 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Thu Mar 10 10:44:00 2022 - [info]
Thu Mar 10 10:44:00 2022 - [info] -- Slave diff file generation on host 192.168.10.73(192.168.10.73:3306) started, pid: 6773. Check tmp log /masterha/app1/192.168.10.73_3306_20220310104355.log if it takes time..
Thu Mar 10 10:44:01 2022 - [info]
Thu Mar 10 10:44:01 2022 - [info] Log messages from 192.168.10.73 ...
Thu Mar 10 10:44:01 2022 - [info]
Thu Mar 10 10:44:00 2022 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Thu Mar 10 10:44:01 2022 - [info] End of log messages from 192.168.10.73.
Thu Mar 10 10:44:01 2022 - [info] -- 192.168.10.73(192.168.10.73:3306) has the latest relay log events.
Thu Mar 10 10:44:01 2022 - [info] Generating relay diff files from the latest slave succeeded.
Thu Mar 10 10:44:01 2022 - [info]
Thu Mar 10 10:44:01 2022 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Thu Mar 10 10:44:01 2022 - [info]
Thu Mar 10 10:44:01 2022 - [info] -- Slave recovery on host 192.168.10.73(192.168.10.73:3306) started, pid: 6775. Check tmp log /masterha/app1/192.168.10.73_3306_20220310104355.log if it takes time..
Thu Mar 10 10:44:02 2022 - [info]
Thu Mar 10 10:44:02 2022 - [info] Log messages from 192.168.10.73 ...
Thu Mar 10 10:44:02 2022 - [info]
Thu Mar 10 10:44:01 2022 - [info] Starting recovery on 192.168.10.73(192.168.10.73:3306)..
Thu Mar 10 10:44:01 2022 - [info] This server has all relay logs. Waiting all logs to be applied..
Thu Mar 10 10:44:01 2022 - [info] done.
Thu Mar 10 10:44:01 2022 - [info] All relay logs were successfully applied.
Thu Mar 10 10:44:01 2022 - [info] Resetting slave 192.168.10.73(192.168.10.73:3306) and starting replication from the new master 192.168.10.72(192.168.10.72:3306)..
Thu Mar 10 10:44:01 2022 - [info] Executed CHANGE MASTER.
Thu Mar 10 10:44:01 2022 - [info] Slave started.
Thu Mar 10 10:44:02 2022 - [info] End of log messages from 192.168.10.73.
Thu Mar 10 10:44:02 2022 - [info] -- Slave recovery on host 192.168.10.73(192.168.10.73:3306) succeeded.
Thu Mar 10 10:44:02 2022 - [info] All new slave servers recovered successfully.
Thu Mar 10 10:44:02 2022 - [info]
Thu Mar 10 10:44:02 2022 - [info] * Phase 5: New master cleanup phase..
Thu Mar 10 10:44:02 2022 - [info]
Thu Mar 10 10:44:02 2022 - [info] Resetting slave info on the new master..
Thu Mar 10 10:44:02 2022 - [info] 192.168.10.72: Resetting slave info succeeded.
Thu Mar 10 10:44:02 2022 - [info] Master failover to 192.168.10.72(192.168.10.72:3306) completed successfully.
Thu Mar 10 10:44:02 2022 - [info]
----- Failover Report -----
app1: MySQL Master failover 192.168.10.71(192.168.10.71:3306) to 192.168.10.72(192.168.10.72:3306) succeeded
Master 192.168.10.71(192.168.10.71:3306) is down!
Check MHA Manager logs at mysql-manager:/masterha/app1/manager.log for details.
Started automated(non-interactive) failover.
Invalidated master IP address on 192.168.10.71(192.168.10.71:3306)
The latest slave 192.168.10.72(192.168.10.72:3306) has all relay logs for recovery.
Selected 192.168.10.72(192.168.10.72:3306) as a new master.
192.168.10.72(192.168.10.72:3306): OK: Applying all logs succeeded.
192.168.10.72(192.168.10.72:3306): OK: Activated master IP address.
192.168.10.73(192.168.10.73:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
192.168.10.73(192.168.10.73:3306): OK: Applying all logs succeeded. Slave started, replicating from 192.168.10.72(192.168.10.72:3306)
192.168.10.72(192.168.10.72:3306): Resetting slave info succeeded.
Master failover to 192.168.10.72(192.168.10.72:3306) completed successfully.#查看vip是否切换成功，mysql-master01服务器已没有VIP
10:46 [root@mysql-master01]5
# ifconfig
ens192: flags=4163 mtu 1500
inet 192.168.10.71 netmask 255.255.255.0 broadcast 192.168.10.255
inet6 fe80::20c:29ff:fe0b:ffe5 prefixlen 64 scopeid 0x20
ether 00:0c:29:0b:ff:e5 txqueuelen 1000 (Ethernet)
RX packets 5890706 bytes 1450802446 (1.3 GiB)
RX errors 0 dropped 155727 overruns 0 frame 0
TX packets 1118525 bytes 1207641877 (1.1 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73 mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 1200 bytes 163457 (159.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1200 bytes 163457 (159.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
#mysql-master02服务器上已启动vip
10:41 [root@mysql-master02]1
# ifconfig
ens192: flags=4163 mtu 1500
inet 192.168.10.72 netmask 255.255.255.0 broadcast 192.168.10.255
inet6 fe80::20c:29ff:feba:3f45 prefixlen 64 scopeid 0x20
ether 00:0c:29:ba:3f:45 txqueuelen 1000 (Ethernet)
RX packets 305534 bytes 26926127 (25.6 MiB)
RX errors 0 dropped 12001 overruns 0 frame 0
TX packets 9028 bytes 1195112 (1.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens192:230: flags=4163 mtu 1500
inet 192.168.10.230 netmask 255.255.255.0 broadcast 192.168.10.255
ether 00:0c:29:ba:3f:45 txqueuelen 1000 (Ethernet)
lo: flags=73 mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 330 bytes 43699 (42.6 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 330 bytes 43699 (42.6 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 #登录mysql-slave01服务器，查看mysql同步状态，已切找到mysql-master02上
(root@localhost) [(none)]> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.10.72
Master_User: mharep
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 154
Relay_Log_File: relay-bin.000002
Relay_Log_Pos: 320
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 154
Relay_Log_Space: 521
#当主库切换后,masterha_manager的服务会自动退出，这时候需要把故障的主库修好，把它当成slave库连接到新的master库，masterha_manager服务才可以启动，不然启动masterha_manager服务会报错，还有，当要再次启动masterha_manager服务前，需要把app1.failover.complete这个文件删除，不然再次切换时会失败