MHA是什么？
MHA是由日本Mysql专家用Perl写的一套Mysql故障切换方案，来保障数据库的高可用性，它的功能是能在0-30s之内实现主Mysql故障转移（failover），MHA故障转移可以很好的帮我们解决从库数据的一致性问题，同时最大化挽回故障发生后数据的一致性。
MHA里有两个角色一个是node节点 一个是manager节点，要实现这个MHA，必须最少要三台数据库服务器，一主多备，即一台充当master，一台充当master的备份机，另外一台是从属机，这里实验为了实现更好的效果使用四台机器，需要说明的是一旦主服务器宕机，备份机即开始充当master提供服务，如果主服务器上线也不会再成为master了，因为如果这样数据库的一致性就被改变了。MHA有两个重要的角色，一个是manager，另外一个是node
192.168.10.94 manager 管理节点
192.168.10.91 master 主库
192.168.10.92 slave01 从库 + 备库
192.168.10.93 slave02 从库
一、环境初始化
1、在master、slave01及slave02上部署mysql（参考文档《【MySQL5.6.27安装规范】-【运维部】-张伟科》）
2、修改主机名
主机: manager执行命令
# sed -i 's/HOSTNAME=.*/HOSTNAME=manager/g' /etc/sysconfig/network && hostname manager
主机: master执行命令
# sed -i 's/HOSTNAME=.*/HOSTNAME=master/g' /etc/sysconfig/network && hostname master
主机: slave01执行命令
# sed -i 's/HOSTNAME=.*/HOSTNAME=slave01/g' /etc/sysconfig/network && hostname slave01
主机: slave02执行命令
# sed -i 's/HOSTNAME=.*/HOSTNAME=slave02/g' /etc/sysconfig/network && hostname slave02
2、主机名解析
在manager上执行如下命令
[root@manager ~]# cat >> /etc/hosts << EOF
192.168.10.94 manager
192.168.10.91 master
192.168.10.92 slave01
192.168.10.93 slave02
> EOF
[root@manager ~]# scp -o StrictHostKeyChecking=no /etc/hosts root@master:/etc/
[root@manager ~]# scp -o StrictHostKeyChecking=no /etc/hosts root@slave01:/etc/
[root@manager ~]# scp -o StrictHostKeyChecking=no /etc/hosts root@slave02:/etc/
3、ssh无密码登录
主机: manager执行命令
[root@manager ~]# ssh-keygen -t rsa
[root@manager ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master
[root@manager ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave01
[root@manager ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave02
主机: master执行命令
[root@master ~]# ssh-keygen -t rsa
[root@master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@manager
[root@master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave01
[root@master ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave02
主机: slave01执行命令
[root@slave01 ~]# ssh-keygen -t rsa
[root@slave01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@manager
[root@slave01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master
[root@slave01 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave02
主机: slave02执行命令
[root@slave02 ~]# ssh-keygen -t rsa
[root@slave02 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@manager
[root@slave02 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master
[root@slave02 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave01
二、规划mysql
1、配置master、slave01和slave02之间的主从复制
在MySQL5.6 的Replication配置中，master端同样要开启两个重要的选项，server-id和log-bin,并且选项server-id在全局架构中并且唯一，不能被其它主机使用，这里采用主机ip地址的最后一位充当server-id的值；slave端要开启relay-log;然后重启mysql。
主机: master执行命令
[root@master ~]# egrep "log_bin|server_id|relay_log_purge" /app/mysql/my.cnf
server-id = 91
log-bin = master-bin
relay-log-purge=0
[root@master ~]# /sbin/iptables -I INPUT -p tcp --dport 3306 -j ACCEPT
[root@master ~]# /etc/rc.d/init.d/iptables save
主机: slave01执行命令
[root@slave01 ~]# egrep "log_bin|server_id|relay_log_purge" /app/mysql/my.cnf
server-id = 92
log-bin = master-bin
relay-log-purge=0
[root@master ~]# /sbin/iptables -I INPUT -p tcp --dport 3306 -j ACCEPT
[root@master ~]# /etc/rc.d/init.d/iptables save
主机: slave02执行命令
[root@slave02 ~]# egrep "log_bin|server_id|relay_log_purge" /app/mysql/my.cnf
server-id = 93
log-bin = master-bin
read-only=1
relay-log-purge=0
[root@master ~]# /sbin/iptables -I INPUT -p tcp --dport 3306 -j ACCEPT
[root@master ~]# /etc/rc.d/init.d/iptables save
2、在master、slave01上创建主从同步的账号。slave01是备用master，这个也需要建立授权用户
[root@master ~]# mysql -e "grant all privileges on *.* to 'rep'@'%' identified by '20151012';flush privileges"
[root@slave01 ~]# mysql -e "grant all privileges on *.* to 'rep'@'%' identified by '20151012';flush privileges"
4、在master上执行命令，查看master状态信息
[root@master ~]# mysql -e 'show master status;'
+-------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+-------------------+----------+--------------+------------------+-------------------+
| master-bin.000001 | 120 | | | |
+-------------------+----------+--------------+------------------+-------------------+
5、在slave01和slave02上执行主从同步
[root@slave01 ~]# mysql
mysql>CHANGE MASTER TO
MASTER_HOST='192.168.10.91',
MASTER_USER='rep',
MASTER_PASSWORD='20151012',
MASTER_PORT=3306,
MASTER_LOG_FILE='mysql_bin.000001',
MASTER_LOG_POS=120;
Query OK, 0 rows affected, 2 warnings (0.02 sec)
mysql> start slave;
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.10.91
Master_User: rep
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql_bin.000001
Read_Master_Log_Pos: 120
Relay_Log_File: slave01-relay-bin.000002
Relay_Log_Pos: 283
Relay_Master_Log_File: mysql_bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 120
Relay_Log_Space: 458
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 91
Master_UUID: 9b1eb4a5-00f3-11e8-a3ba-ce006127c972
Master_Info_File: /app/mysql/data/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
1 row in set (0.00 sec)
实验到这里表示主从已经配置完成！接下来我们就开始规划mha
三、规划mha
1、创建mha管理用的复制账号，每台数据库上都要创建4个账号，在这里以其中master为例
[root@master ~]# mysql
mysql> grant all privileges on *.* to 'mha_rep'@'%' identified by '20151012'; flush privileges;
mysql> select user,host,password from mysql.user where user='mha_rep';
+---------+------+-------------------------------------------+
| user | host | password |
+---------+------+-------------------------------------------+
| mha_rep | % | *F9B93BD42F62D26FD094239C15535EE045F7BB22 |
+---------+------+-------------------------------------------+
2、在3台主机上(master、slave01和slave02)上分别安装mha4mysql-node包，这里以master为例，其它主机同理。
准确的来讲的话，应该是所有的节点包括manager和node的所有节点都要安装mha4mysql-node包，只不过等会manager要安装node节点也要安装manager节点，所以把manager单独在下面安装了。
[root@master ~]# yum install perl-DBD-MySQL -y
[root@master ~]# cd /usr/local/src/
[root@master ~]# wget https://downloads.mariadb.com/files/MHA/mha4mysql-node-0.54-0.el6.noarch.rpm
[root@master ~]# rpm -ivh mha4mysql-node-0.54-0.el6.noarch.rpm
/usr/bin/apply_diff_relay_logs //识别差异的中继日志事件并将其差异的事件应用于其他的slave
/usr/bin/filter_mysqlbinlog //去除不必要的ROLLBACK事件（MHA已不再使用这个工具）
/usr/bin/purge_relay_logs //清除中继日志（不会阻塞SQL线程）
/usr/bin/save_binary_logs//保存和复制master的二进制日志
3、在manager上安装mha4mysql-manager和mha4mysql-node包
[root@manager ~]# yum install perl cpan perl-DBD-MySQL perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager perl-Net-Telnet -y
注释：由于yum源里没有这四个安装包，因此我们需要单独下载来安装。
[root@master ~]# cd /usr/local/src/
[root@manager ~]# wget http://rpmfind.net/linux/dag/redhat/el6/en/x86_64/dag/RPMS/perl-Log-Dispatch-2.26-1.el6.rf.noarch.rpm
[root@manager ~]# wget ftp://rpmfind.net/linux/dag/redhat/el6/en/x86_64/dag/RPMS/perl-Parallel-ForkManager-0.7.5-2.2.el6.rf.noarch.rpm
[root@manager ~]# yum localinstall *.rpm -y
安装manager和node包
[root@manager ~]#cd /usr/local/src/
[root@manager ~]# wget https://downloads.mariadb.com/MHA/mha4mysql-manager-0.55-0.el6.noarch.rpm
[root@manager ~]# wget https://downloads.mariadb.com/MHA/mha4mysql-node-0.54-0.el6.noarch.rpm
[root@manager ~]# yum localinstall mha4mysql-node-0.54-0.el6.noarch.rpm -y
[root@manager ~]# yum localinstall mha4mysql-manager-0.55-0.el6.noarch.rpm -y
4、查看mha4mysql-manager安装了哪些工具
[root@manager ~]# rpm -ql mha4mysql-manager |grep bin
/usr/bin/masterha_check_repl //检查MySQL复制状况
/usr/bin/masterha_check_ssh //检查MHA的SSH配置状况
/usr/bin/masterha_check_status //检测当前MHA运行状态
/usr/bin/masterha_conf_host //添加或删除配置的server信息
/usr/bin/masterha_manager //启动MHA
/usr/bin/masterha_master_monitor //检测master是否宕机
/usr/bin/masterha_master_switch //控制故障转移(自动或者手动)
/usr/bin/masterha_secondary_check
/usr/bin/masterha_stop
5、修改脚本/usr/bin/masterha_secondary_check配置的ssh端口
6、在manager主机上下载mha4mysql-manager的源码包
# wget https://downloads.mariadb.com/MHA/mha4mysql-manager-0.56.tar.gz
7、在manager主机上从mha4mysql-manager的源码包中提取mha的配置配置文件和脚本
[root@manager ~]# tar xf mha4mysql-manager-0.56.tar.gz
[root@manager ~]# mkdir -p /app/mha/scripts
[root@manager ~]# cp mha4mysql-manager-0.56/samples/scripts/* /app/mha/scripts/
[root@manager ~]# cp mha4mysql-manager-0.56/samples/conf/app1.cnf /app/mha/mha.cnf
[root@manager ~]# tree /app/mha/
/app/mha/
├── mha.cnf
└── scripts
├── master_ip_failover //故障自动切换时对vip管理的脚本，不是必须。如果我们使用keepalived的，我们可以自己编写脚本完成对vip的管理，比如监控mysql，如果mysql异常，我们停止keepalived就行，这样vip就会自动漂移
├── master_ip_online_change //在线切换时对vip的管理，不是必须，同样可以自行编写简单的shell完成。
├── power_manager //故障发生后关闭主机的脚本，不是必须
└── send_report //因故障切换后发送报警的脚本，不是必须，可自行编写简单的shell完成
8、修改manager端mha的配置文件，如下
[root@manager ~]# cat /app/mha/mha.cnf
[server default]
#监控用户
user=mha_rep
#监控用户的密码
password=20151012
#ssh登录用户名
ssh_user=root
#复制用户名
repl_user=rep
#复制用户的密码
repl_password=20151012
#设置监控主库,发送ping包的时间间隔,默认是3秒,尝试三次没有回应的时候自动进行failover
ping_interval=1
#设置manager的工作目录和日志目录
manager_workdir=/app/mha
manager_log=/app/mha/manager.log
# monitor mysql
#一旦MHA到master的监控之间出现问题,MHA Manager将会尝试从mysql01,mysql02登录到master
secondary_check_script= masterha_secondary_check -s 192.168.10.91 -s 192.168.10.92 -s 192.168.10.93
#设置发生切换后发送的报警的脚本
report_script= /app/mha/scripts/send_report
#设置手动切换时候的切换脚本（脚本有瑕疵,需要自行修改）
master_ip_online_change_script= /app/mha/scripts/master_ip_online_change
#设置自动failover时候的切换脚本(脚本有瑕疵,需要自行修改)
master_ip_failover_script=/app/mha/scripts/master_ip_failover
#设置故障发生后关闭故障主机脚本（该脚本的主要作用是关闭主机发生脑裂,这里没有使用）
#shutdown_script= /app/mha/scripts/power_manager
#check_repl_delay=0
[server1]
hostname=master
ssh_port=60022
candidate_master=1
check_repl_delay=0
master_binlog_dir=/app/mysql/data
[server2]
hostname=slave01
ssh_port=60022
candidate_master=1
check_repl_delay=0
master_binlog_dir=/app/mysql/data
[server3]
hostname=slave02
ssh_port=60022
no_master=1
master_binlog_dir=/app/mysql/data
9、检查ssh是否畅通
[root@manager ~]#masterha_check_ssh --conf=/app/mha/mha.cnf
Thu Jan 25 11:50:58 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Thu Jan 25 11:50:58 2018 - [info] Reading application default configurations from /app/mha/mha.cnf..
Thu Jan 25 11:50:58 2018 - [info] Reading server configurations from /app/mha/mha.cnf..
Thu Jan 25 11:50:58 2018 - [info] Starting SSH connection tests..
Thu Jan 25 11:51:01 2018 - [debug]
Thu Jan 25 11:50:59 2018 - [debug] Connecting via SSH from root@slave02(192.168.10.93:60022) to root@master(192.168.10.91:60022)..
Warning: Permanently added '[192.168.10.93]:60022' (ECDSA) to the list of known hosts.
Warning: Permanently added '[192.168.10.91]:60022' (ECDSA) to the list of known hosts.
Thu Jan 25 11:51:00 2018 - [debug] ok.
Thu Jan 25 11:51:00 2018 - [debug] Connecting via SSH from root@slave02(192.168.10.93:60022) to root@slave01(192.168.10.92:60022)..
Thu Jan 25 11:51:01 2018 - [debug] ok.
Thu Jan 25 11:51:01 2018 - [debug]
Thu Jan 25 11:50:59 2018 - [debug] Connecting via SSH from root@slave01(192.168.10.92:60022) to root@master(192.168.10.91:60022)..
Warning: Permanently added '[192.168.10.91]:60022' (ECDSA) to the list of known hosts.
Thu Jan 25 11:51:00 2018 - [debug] ok.
Thu Jan 25 11:51:00 2018 - [debug] Connecting via SSH from root@slave01(192.168.10.92:60022) to root@slave02(192.168.10.93:60022)..
Warning: Permanently added '[192.168.10.93]:60022' (ECDSA) to the list of known hosts.
Thu Jan 25 11:51:00 2018 - [debug] ok.
Thu Jan 25 11:51:01 2018 - [debug]
Thu Jan 25 11:50:58 2018 - [debug] Connecting via SSH from root@master(192.168.10.91:60022) to root@slave01(192.168.10.92:60022)..
Warning: Permanently added '[192.168.10.91]:60022' (ECDSA) to the list of known hosts.
Warning: Permanently added '[192.168.10.92]:60022' (ECDSA) to the list of known hosts.
Thu Jan 25 11:51:00 2018 - [debug] ok.
Thu Jan 25 11:51:00 2018 - [debug] Connecting via SSH from root@master(192.168.10.91:60022) to root@slave02(192.168.10.93:60022)..
Warning: Permanently added '[192.168.10.93]:60022' (ECDSA) to the list of known hosts.
Thu Jan 25 11:51:00 2018 - [debug] ok.
Thu Jan 25 11:51:01 2018 - [info] All SSH connection tests passed successfully.
如果得到以上结果，表明主机之间ssh互信是畅通的
10、检查主从复制是否正常
执行主从复制检查的时候，这个由于我是用源码编译的mysql会出现路径找不到的问题；比如
(1) Can't exec "mysqlbinlog": No such file or directory at /usr/local/perl5/MHA/BinlogManager.pm line 99.
解决办法：
在master、slave01和slave02上分别执行如下命令
# ln -s /app/mysql/bin/mysqlbinlog /usr/bin/mysqlbinlog
(2)mysqlbinlog: unknown variable 'default-character-set=utf8'
解决办法：
在master、slave01和slave02上分别执行注释client部分的default-character-set=utf8选项，并重启mysqld服务
(3)Testing mysql connection and privileges..sh: mysql: command not found
解决办法：
在master、slave01和slave02上分别执行如下命令
# ln -s /app/mysql/bin/mysql /usr/bin/mysql
[root@manager ~]# masterha_check_repl --conf=/app/mha/mha.cnf
Thu Jan 25 12:11:16 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Thu Jan 25 12:11:16 2018 - [info] Reading application default configurations from /app/mha/mha.cnf..
Thu Jan 25 12:11:16 2018 - [info] Reading server configurations from /app/mha/mha.cnf..
Thu Jan 25 12:11:16 2018 - [info] MHA::MasterMonitor version 0.55.
Thu Jan 25 12:11:17 2018 - [info] Dead Servers:
Thu Jan 25 12:11:17 2018 - [info] Alive Servers:
Thu Jan 25 12:11:17 2018 - [info] master(192.168.10.91:3306)
Thu Jan 25 12:11:17 2018 - [info] slave01(192.168.10.92:3306)
Thu Jan 25 12:11:17 2018 - [info] slave02(192.168.10.93:3306)
Thu Jan 25 12:11:17 2018 - [info] Alive Slaves:
Thu Jan 25 12:11:17 2018 - [info] slave01(192.168.10.92:3306) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled
Thu Jan 25 12:11:17 2018 - [info] Replicating from 192.168.10.91(192.168.10.91:3306)
Thu Jan 25 12:11:17 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Jan 25 12:11:17 2018 - [info] slave02(192.168.10.93:3306) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled
Thu Jan 25 12:11:17 2018 - [info] Replicating from 192.168.10.91(192.168.10.91:3306)
Thu Jan 25 12:11:17 2018 - [info] Not candidate for the new Master (no_master is set)
Thu Jan 25 12:11:17 2018 - [info] Current Alive Master: master(192.168.10.91:3306)
Thu Jan 25 12:11:17 2018 - [info] Checking slave configurations..
Thu Jan 25 12:11:17 2018 - [info] read_only=1 is not set on slave slave01(192.168.10.92:3306).
Thu Jan 25 12:11:17 2018 - [info] Checking replication filtering settings..
Thu Jan 25 12:11:17 2018 - [info] binlog_do_db= , binlog_ignore_db=
Thu Jan 25 12:11:17 2018 - [info] Replication filtering check ok.
Thu Jan 25 12:11:17 2018 - [info] Starting SSH connection tests..
Thu Jan 25 12:11:20 2018 - [info] All SSH connection tests passed successfully.
Thu Jan 25 12:11:20 2018 - [info] Checking MHA Node version..
Thu Jan 25 12:11:20 2018 - [info] Version check ok.
Thu Jan 25 12:11:20 2018 - [info] Checking SSH publickey authentication settings on the current master..
Thu Jan 25 12:11:21 2018 - [info] HealthCheck: SSH to master is reachable.
Thu Jan 25 12:11:21 2018 - [info] Master MHA Node version is 0.54.
Thu Jan 25 12:11:21 2018 - [info] Checking recovery script configurations on the current master..
Thu Jan 25 12:11:21 2018 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/app/mysql/data --output_file=/var/tmp/save_binary_logs_test --manager_version=0.55 --start_file=mysql_bin.000001
Thu Jan 25 12:11:21 2018 - [info] Connecting to root@master(master)..
Creating /var/tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /app/mysql/data, up to mysql_bin.000001
Thu Jan 25 12:11:22 2018 - [info] Master setting check done.
Thu Jan 25 12:11:22 2018 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Thu Jan 25 12:11:22 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mha_rep' --slave_host=slave01 --slave_ip=192.168.10.92 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.27-log --manager_version=0.55 --relay_log_info=/app/mysql/data/relay-log.info --relay_dir=/app/mysql/data/ --slave_pass=xxx
Thu Jan 25 12:11:22 2018 - [info] Connecting to [email protected](slave01:60022)..
Checking slave recovery environment settings..
Opening /app/mysql/data/relay-log.info ... ok.
Relay log found at /app/mysql/data, up to slave01-relay-bin.000002
Temporary relay log file is /app/mysql/data/slave01-relay-bin.000002
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Thu Jan 25 12:11:22 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mha_rep' --slave_host=slave02 --slave_ip=192.168.10.93 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.27-log --manager_version=0.55 --relay_log_info=/app/mysql/data/relay-log.info --relay_dir=/app/mysql/data/ --slave_pass=xxx
Thu Jan 25 12:11:22 2018 - [info] Connecting to [email protected](slave02:60022)..
Checking slave recovery environment settings..
Opening /app/mysql/data/relay-log.info ... ok.
Relay log found at /app/mysql/data, up to slave02-relay-bin.000002
Temporary relay log file is /app/mysql/data/slave02-relay-bin.000002
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Thu Jan 25 12:11:22 2018 - [info] Slaves settings check done.
Thu Jan 25 12:11:22 2018 - [info]
master (current master)
+--slave01
+--slave02
Thu Jan 25 12:11:22 2018 - [info] Checking replication health on slave01..
Thu Jan 25 12:11:22 2018 - [info] ok.
Thu Jan 25 12:11:22 2018 - [info] Checking replication health on slave02..
Thu Jan 25 12:11:22 2018 - [info] ok.
Thu Jan 25 12:11:22 2018 - [warning] master_ip_failover_script is not defined.
Thu Jan 25 12:11:22 2018 - [warning] shutdown_script is not defined.
Thu Jan 25 12:11:22 2018 - [info] Got exit code 0 (Not master dead).
MySQL Replication Health is OK.
或者在命令执行后大家会看到有警告信息；比如
Tue Sep 15 23:45:35 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
在命令执行后输出结果的第一行有这样的警告信息，找不到masterha_default.cnf，其实这个文件是mha的全局默认配置文件，由于我们没有使用全局，所以就跳过了这项，不过不妨碍真个环境。如果大家想用，其实也是可以的，在源码包里就有这个默认的模板配置文件，大家只需要稍作修改就可以排查这个警告信息
Tue Sep 15 23:45:38 2015 - [warning] master_ip_failover_script is not defined.
Tue Sep 15 23:45:38 2015 - [warning] shutdown_script is not defined.
在命令执行后输出结果的最后几行中，提示未定义，大家看看/app/mha/mha.cnf文件中，我们正好注释了这两行代码，其中master_ip_failover_script是后期做vip的时候才用到的。
四、mha实验模拟
1、在每次做mha实验的时候，我们都最好先执行如下命令做检测
[root@manager ~]# masterha_check_ssh --conf=/app/mha/mha.cnf
[root@manager ~]# masterha_check_repl --conf=/app/mha/mha.cnf
确定两条命令的返回结果都是无异常的，然后启动mha服务
2、在manager端启动mha服务并时刻监控日志文件的输出变化
[root@manager ~]# nohup masterha_manager --conf=/app/mha/mha.cnf > /app/mha/log/mha_manager.log 2>&1 &
[root@manager ~]# ps -ef |grep masterha |grep -v 'grep'
root 799234 788691 1 14:09 pts/1 00:00:00 perl /bin/masterha_manager --conf=/app/mha/mha.cnf
3、实验流程第一阶段
准备，先来检查主从是否都均已正常
首先，停止master端的mysqld服务进程，然后查看备库也就是slave01是否已经提升到主库
其次，登录slave02端查看主从是否正常，是否更新到新的master的ip上也就是是否执行slave01的ip地址
最后，启动master端的mysqld服务进程，并将其加入到主从模式中
准备，实验开始
在slave01和slave02上执行，检查主从同步是否都正常，这里以slave01为例，slave02同理
[root@slave01 ~]# mysql -e 'show slave status\G' |egrep 'Slave_IO_Running:|Slave_SQL_Running:'
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
首先，实验开始
(1)在master端上执行命令来停止mysqld服务进程
[root@master ~]# /etc/init.d/mysql stop
Shutting down MySQL.... SUCCESS!
(2)查看manager端的mha输出日志，在这里只截取了一部分日志信息
[root@manager ~]# tail -f /app/mha/manager.log
----- Failover Report -----
mha: MySQL Master failover master to slave01 succeeded
Master master is down!
Check MHA Manager logs at manager:/app/mha/manager.log for details.
Started automated(non-interactive) failover.
The latest slave slave01(192.168.10.92:3306) has all relay logs for recovery.
Selected slave01 as a new master.
slave01: OK: Applying all logs succeeded.
slave02: This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
slave02: OK: Applying all logs succeeded. Slave started, replicating from slave01.
slave01: Resetting slave info succeeded.
Master failover to slave01(192.168.10.92:3306) completed successfully.
Thu Jan 25 13:52:52 2018 - [info] Sending mail..
Unknown option: conf
其次，实验开始
登录slave02查看主从同步是否正常，查看是否已经转移到新的master的ip上
[root@slave02 ~]# mysql -e 'show slave status\G' |egrep 'Master_Host|Slave_IO_Running:|Slave_SQL_Running:'
Master_Host: 192.168.10.92
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
最后，实验开始
(1)在master端启动mysqld服务
[root@master ~]# /etc/init.d/mysql start
Starting MySQL. SUCCESS!
(2)在manager端的mha日志文件中找到主从同步的sql语句，这条语句只需要修改密码即可使用
[root@manager ~]# grep 'MASTER_HOST' /app/mha/manager.log |tail -n 1
Thu Jan 25 13:52:50 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='slave01 or 192.168.10.92', MASTER_PORT=3306, MASTER_LOG_FILE='mysql_bin.000001', MASTER_LOG_POS=120, MASTER_USER='rep', MASTER_PASSWORD='xxx';
注意：
MASTER_HOST='slave01 or 192.168.10.92' 这个位置需要注意一下，最好只写一个并建议写ip地址
(3)在master上启动主从同步，密码为20151012
[root@master ~]# mysql -e "CHANGE MASTER TO MASTER_HOST='192.168.10.92', MASTER_PORT=3306, MASTER_LOG_FILE='mysql_bin.000001', MASTER_LOG_POS=120, MASTER_USER='rep', MASTER_PASSWORD='20151012';start slave;"
[root@master ~]# mysql -e "show slave status\G"
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.10.92
Master_User: rep
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql_bin.000001
Read_Master_Log_Pos: 120
Relay_Log_File: master-relay-bin.000002
Relay_Log_Pos: 283
Relay_Master_Log_File: mysql_bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 120
Relay_Log_Space: 457
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 92
Master_UUID: 996b4343-00f3-11e8-a3ba-b6c824ce1080
Master_Info_File: /app/mysql/data/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
4、实验流程第二阶段
准备，先来检查主从是否都均已正常
首先，停止slave01端的mysqld服务进程，然后查看master是否已经提升到新的主库
其次，登录slave02端查看主从是否正常，是否更新到新的master的ip上也就是是否执行master的ip地址
最后，启动slave01端的mysqld服务进程，并将其加入到主从模式中
这里强调下，默认情况下每次主备库切换后，mha服务都会停止。
在这里我们需要重新启动mha服务
[root@manager ~]# rm -rf /app/mha/mha.failover.complete
[root@manager ~]# nohup masterha_manager --conf=/app/mha/mha.cnf > /app/mha/log/mha_manager.log 2>&1 &
[1] 3606
[root@manager ~]# ps -ef |grep masterha |grep -v 'grep'
root 799234 788691 1 14:09 pts/1 00:00:00 perl /bin/masterha_manager --conf=/app/mha/mha.cnf
[root@manager ~]# masterha_check_status --conf=/app/mha/mha.cnf
mha (pid:799234) is running(0:PING_OK), master:slave01 #表明现在的master是slave01主机
准备，实验开始
在master和slave02上执行，检查主从同步是否都正常，这里以master为例，slave02同理
[root@master ~]# mysql -e 'show slave status\G' |egrep 'Slave_IO_Running:|Slave_SQL_Running:'
Master_Host: 192.168.10.92
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
首先，实验开始
(1)在slave01端上执行命令来停止mysqld服务进程
[root@slave01 ~]# /etc/init.d/mysql stop
Shutting down MySQL.... SUCCESS!
(2)查看manager端的mha输出日志，在这里只截取了一部分日志信息
[root@manager ~]# tail -f /app/mha/manager.log
----- Failover Report -----
mha: MySQL Master failover slave01 to master succeeded
Master slave01 is down!
Check MHA Manager logs at manager:/app/mha/manager.log for details.
Started automated(non-interactive) failover.
The latest slave master(192.168.10.91:3306) has all relay logs for recovery.
Selected master as a new master.
master: OK: Applying all logs succeeded.
slave02: This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
slave02: OK: Applying all logs succeeded. Slave started, replicating from master.
master: Resetting slave info succeeded.
Master failover to master(192.168.10.91:3306) completed successfully.
Thu Jan 25 14:25:48 2018 - [info] Sending mail..
Unknown option: conf
其次，实验开始
登录slave02查看主从同步是否正常，查看是否已经转移到新的master的ip上
[root@slave02 ~]#mysql -e 'show slave status\G' |egrep 'Master_Host|Slave_IO_Running:|Slave_SQL_Running:'
Master_Host: 192.168.10.91
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
最后，实验开始
(1)在slave01端启动mysqld服务
[root@slave01 ~]# /etc/init.d/mysql start
Starting MySQL. SUCCESS!
(2)在manager端的mha日志文件中找到主从同步的sql语句，这条语句只需要修改密码即可使用
[root@manager ~]# grep 'MASTER_HOST' /app/mha/manager.log | tail -n 1
Thu Jan 25 14:25:46 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='master or 192.168.10.91', MASTER_PORT=3306, MASTER_LOG_FILE='mysql_bin.000002', MASTER_LOG_POS=120, MASTER_USER='rep', MASTER_PASSWORD='xxx';
注意：
MASTER_HOST='master or 192.168.10.91' 这个位置需要注意一下，最好只写一个并建议写ip地址
(3)在slave01上启动主从同步，密码为20151012
[root@slave01 ~]#mysql -e "CHANGE MASTER TO MASTER_HOST='192.168.10.91', MASTER_PORT=3306, MASTER_LOG_FILE='mysql_bin.000002', MASTER_LOG_POS=120, MASTER_USER='rep', MASTER_PASSWORD='20151012'; start slave;"
[root@slave01 ~]# mysql -e "show slave status\G"
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.10.91
Master_User: rep
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql_bin.000002
Read_Master_Log_Pos: 120
Relay_Log_File: slave01-relay-bin.000002
Relay_Log_Pos: 283
Relay_Master_Log_File: mysql_bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 120
Relay_Log_Space: 458
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 91
Master_UUID: 9b1eb4a5-00f3-11e8-a3ba-ce006127c972
Master_Info_File: /app/mysql/data/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
实验到这一步，已经完成了主库down后，备用的从库会自动提升到主库，并且其它从库也会重新指向新的master的ip地址。但是这里却存在一个问题，就是主备库确实实现了切换，但是对外提供的ip总不能是两个吧！为了整合keepalived/heartbeat的功能，这里也引入了vip，实现无透明切换
至于如何实现vip的故障转移，网上也有很多组合，有的是用keepalived实现的故障转移，也有实现这篇文章中将要提供的脚本检测功能。
说到这里，实验过程中，大家会注意执行命令的输出结果中的警告信息[warning],下面就来说说这个吧
首先，我们看下这个脚本，我们如果想用这个vip的功能，需要打开这个选项
[root@manager ~]# grep '^#master_ip_failover_script' /app/mha/mha.cnf
master_ip_failover_script=/app/mha/scripts/master_ip_failover
其次，修改里面几处配置
[root@manager ~]# mv /app/mha/scripts/{master_ip_failover,master_ip_failover_bak}
[root@manager ~]# cat /app/mha/scripts/master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);
my $vip = '192.168.10.90'; # Virtual IP #可修改
my $gateway = '192.168.10.254';#Gateway IP #可修改
my $interface = 'eth0'; #可修改
my $key = "1";
my $ssh_start_vip = "/sbin/ifconfig $interface:$key $vip;/sbin/arping -I $interface -c 3 -s $vip $gateway >/dev/null 2>&1";
my $ssh_stop_vip = "/sbin/ifconfig $interface:$key down";
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);
exit &main();
sub main {
print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
# $orig_master_host, $orig_master_ip, $orig_master_port are passed.
# If you manage master ip address at global catalog database,
# invalidate orig_master_ip here.
my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
# all arguments are passed.
# If you manage master ip address at global catalog database,
# activate new_master_ip here.
# You can also grant write access (create user, set read_only=0, etc) here.
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
`ssh $ssh_user\@$orig_master_host \" $ssh_start_vip \"`;
exit 0;
}
else {
&usage();
exit 1;
}
}
# A simple system call that enable the VIP on the new master
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}
[root@manager ~]# chmod +x /app/mha/scripts/master_ip_failover
进行ssh检查
Thu Jan 25 15:56:56 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Thu Jan 25 15:56:56 2018 - [info] Reading application default configurations from /app/mha/mha.cnf..
Thu Jan 25 15:56:56 2018 - [info] Reading server configurations from /app/mha/mha.cnf..
Thu Jan 25 15:56:56 2018 - [info] Starting SSH connection tests..
Thu Jan 25 15:56:58 2018 - [debug]
Thu Jan 25 15:56:56 2018 - [debug] Connecting via SSH from root@master(192.168.10.91:60022) to root@slave01(192.168.10.92:60022)..
Thu Jan 25 15:56:57 2018 - [debug] ok.
Thu Jan 25 15:56:57 2018 - [debug] Connecting via SSH from root@master(192.168.10.91:60022) to root@slave02(192.168.10.93:60022)..
Thu Jan 25 15:56:58 2018 - [debug] ok.
Thu Jan 25 15:56:59 2018 - [debug]
Thu Jan 25 15:56:57 2018 - [debug] Connecting via SSH from root@slave02(192.168.10.93:60022) to root@master(192.168.10.91:60022)..
Thu Jan 25 15:56:58 2018 - [debug] ok.
Thu Jan 25 15:56:58 2018 - [debug] Connecting via SSH from root@slave02(192.168.10.93:60022) to root@slave01(192.168.10.92:60022)..
Thu Jan 25 15:56:58 2018 - [debug] ok.
Thu Jan 25 15:56:59 2018 - [debug]
Thu Jan 25 15:56:57 2018 - [debug] Connecting via SSH from root@slave01(192.168.10.92:60022) to root@master(192.168.10.91:60022)..
Thu Jan 25 15:56:58 2018 - [debug] ok.
Thu Jan 25 15:56:58 2018 - [debug] Connecting via SSH from root@slave01(192.168.10.92:60022) to root@slave02(192.168.10.93:60022)..
Thu Jan 25 15:56:58 2018 - [debug] ok.
Thu Jan 25 15:56:59 2018 - [info] All SSH connection tests passed successfully.
进行主从复制检查
[root@manager ~]# masterha_check_repl --conf=/app/mha/mha.cnf
Thu Jan 25 15:55:56 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Thu Jan 25 15:55:56 2018 - [info] Reading application default configurations from /app/mha/mha.cnf..
Thu Jan 25 15:55:56 2018 - [info] Reading server configurations from /app/mha/mha.cnf..
Thu Jan 25 15:55:56 2018 - [info] MHA::MasterMonitor version 0.55.
Thu Jan 25 15:55:57 2018 - [info] Dead Servers:
Thu Jan 25 15:55:57 2018 - [info] Alive Servers:
Thu Jan 25 15:55:57 2018 - [info] master(192.168.10.91:3306)
Thu Jan 25 15:55:57 2018 - [info] slave01(192.168.10.92:3306)
Thu Jan 25 15:55:57 2018 - [info] slave02(192.168.10.93:3306)
Thu Jan 25 15:55:57 2018 - [info] Alive Slaves:
Thu Jan 25 15:55:57 2018 - [info] slave01(192.168.10.92:3306) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled
Thu Jan 25 15:55:57 2018 - [info] Replicating from 192.168.10.91(192.168.10.91:3306)
Thu Jan 25 15:55:57 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Jan 25 15:55:57 2018 - [info] slave02(192.168.10.93:3306) Version=5.6.27-log (oldest major version between slaves) log-bin:enabled
Thu Jan 25 15:55:57 2018 - [info] Replicating from 192.168.10.91(192.168.10.91:3306)
Thu Jan 25 15:55:57 2018 - [info] Not candidate for the new Master (no_master is set)
Thu Jan 25 15:55:57 2018 - [info] Current Alive Master: master(192.168.10.91:3306)
Thu Jan 25 15:55:57 2018 - [info] Checking slave configurations..
Thu Jan 25 15:55:57 2018 - [info] read_only=1 is not set on slave slave01(192.168.10.92:3306).
Thu Jan 25 15:55:57 2018 - [info] Checking replication filtering settings..
Thu Jan 25 15:55:57 2018 - [info] binlog_do_db= , binlog_ignore_db=
Thu Jan 25 15:55:57 2018 - [info] Replication filtering check ok.
Thu Jan 25 15:55:57 2018 - [info] Starting SSH connection tests..
Thu Jan 25 15:55:59 2018 - [info] All SSH connection tests passed successfully.
Thu Jan 25 15:55:59 2018 - [info] Checking MHA Node version..
Thu Jan 25 15:56:00 2018 - [info] Version check ok.
Thu Jan 25 15:56:00 2018 - [info] Checking SSH publickey authentication settings on the current master..
Thu Jan 25 15:56:00 2018 - [info] HealthCheck: SSH to master is reachable.
Thu Jan 25 15:56:01 2018 - [info] Master MHA Node version is 0.54.
Thu Jan 25 15:56:01 2018 - [info] Checking recovery script configurations on the current master..
Thu Jan 25 15:56:01 2018 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/app/mysql/data --output_file=/var/tmp/save_binary_logs_test --manager_version=0.55 --start_file=mysql_bin.000004
Thu Jan 25 15:56:01 2018 - [info] Connecting to root@master(master)..
Creating /var/tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /app/mysql/data, up to mysql_bin.000004
Thu Jan 25 15:56:02 2018 - [info] Master setting check done.
Thu Jan 25 15:56:02 2018 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Thu Jan 25 15:56:02 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mha_rep' --slave_host=slave01 --slave_ip=192.168.10.92 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.27-log --manager_version=0.55 --relay_log_info=/app/mysql/data/relay-log.info --relay_dir=/app/mysql/data/ --slave_pass=xxx
Thu Jan 25 15:56:02 2018 - [info] Connecting to [email protected](slave01:60022)..
Checking slave recovery environment settings..
Opening /app/mysql/data/relay-log.info ... ok.
Relay log found at /app/mysql/data, up to slave01-relay-bin.000006
Temporary relay log file is /app/mysql/data/slave01-relay-bin.000006
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Thu Jan 25 15:56:02 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mha_rep' --slave_host=slave02 --slave_ip=192.168.10.93 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.27-log --manager_version=0.55 --relay_log_info=/app/mysql/data/relay-log.info --relay_dir=/app/mysql/data/ --slave_pass=xxx
Thu Jan 25 15:56:02 2018 - [info] Connecting to [email protected](slave02:60022)..
Checking slave recovery environment settings..
Opening /app/mysql/data/relay-log.info ... ok.
Relay log found at /app/mysql/data, up to slave02-relay-bin.000006
Temporary relay log file is /app/mysql/data/slave02-relay-bin.000006
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Thu Jan 25 15:56:02 2018 - [info] Slaves settings check done.
Thu Jan 25 15:56:02 2018 - [info]
master (current master)
+--slave01
+--slave02
Thu Jan 25 15:56:02 2018 - [info] Checking replication health on slave01..
Thu Jan 25 15:56:02 2018 - [info] ok.
Thu Jan 25 15:56:02 2018 - [info] Checking replication health on slave02..
Thu Jan 25 15:56:02 2018 - [info] ok.
Thu Jan 25 15:56:02 2018 - [info] Checking master_ip_failover_script status:
Thu Jan 25 15:56:02 2018 - [info] /app/mha/scripts/master_ip_failover --command=status --ssh_user=root --orig_master_host=master --orig_master_ip=192.168.10.91 --orig_master_port=3306 --orig_master_ssh_port=60022
Unknown option: orig_master_ssh_port
IN SCRIPT TEST====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 192.168.10.90;/sbin/arping -I eth0 -c 3 -s 192.168.10.90 192.168.10.254 >/dev/null 2>&1===
Checking the Status of the script.. OK
Thu Jan 25 15:56:06 2018 - [info] OK.
Thu Jan 25 15:56:06 2018 - [warning] shutdown_script is not defined.
Thu Jan 25 15:56:06 2018 - [info] Got exit code 0 (Not master dead).
MySQL Replication Health is OK.
如果在命令执行后的输出结果中找不到[warning] master_ip_failover_script is not defined.表示已经启动此功能
接下来，我们来启动mha服务
接下来的流程大致可以这样来做
准备，启动mha服务
首先，停止master端的mysqld进程，让slave01提供到主库并获取vip地址
其次，查看其它从库slave02上主从同步是否正常，是否重新指向新的master的地址
最后，启动master端的mysqld进程，重新加入到主从模式中
准备，实验开始
[root@manager ~]# rm -rf /app/mha/mha.failover.complete
[root@manager ~]# nohup masterha_manager --conf=/app/mha/mha.cnf > /app/mha/log/mha_manager.log 2>&1 &
[1] 4066
[root@manager ~]# ps -ef |grep masterha |grep -v 'grep'
root 805559 805262 0 15:52 pts/2 00:00:00 perl /bin/masterha_check_repl --conf=/app/mha/mha.cnf
root 806133 805710 0 15:58 pts/3 00:00:00 perl /bin/masterha_manager --conf=/app/mha/mha.cnf
[root@manager ~]# masterha_check_status --conf=/app/mha/mha.cnf
ha (pid:806133) is running(0:PING_OK), master:master
首先，实验开始
(1)在master端上执行命令来停止mysqld服务进程
[root@master ~]# /etc/init.d/mysql stop
Shutting down MySQL.... SUCCESS!
(2)查看manager端的mha输出日志，在这里只截取了一部分日志信息
[root@manager ~]# tail -f /app/mha/manager.log
Enabling the VIP - 192.168.10.90 on the new master - slave01
#表示vip的地址是192.168.10.90已经在新的master上开启，新的master是slave01
----- Failover Report -----
mha: MySQL Master failover master to slave01 succeeded
#表示Master由master转移到slave01
Master master is down!
#表示master已经down机
Check MHA Manager logs at manager:/app/mha/manager.log for details.
Started automated(non-interactive) failover.
Invalidated master IP address on master.
The latest slave slave01(192.168.10.92:3306) has all relay logs for recovery.
Selected slave01 as a new master.
slave01: OK: Applying all logs succeeded.
slave01: OK: Activated master IP address.
slave02: This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
slave02: OK: Applying all logs succeeded. Slave started, replicating from slave01.
slave01: Resetting slave info succeeded.
Master failover to slave01(192.168.10.92:3306) completed successfully.
Thu Jan 25 16:00:26 2018 - [info] Sending mail..
Unknown option: conf
(3)登录slave01查看是否获取到vip地址
[root@slave01 ~]# ip addr list
1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: mtu 1500 qdisc mq state UP qlen 1000
link/ether b6:c8:24:ce:10:80 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.92/24 brd 192.168.10.255 scope global eth0
valid_lft forever preferred_lft forever
inet 192.168.10.90/24 brd 192.168.10.255 scope global secondary eth0:1
valid_lft forever preferred_lft forever
其次，实验开始
登录slave02查看主从同步是否正常，查看是否已经转移到新的master的ip上
[root@slave02 ~]# mysql -e 'show slave status\G' |egrep 'Master_Host|Slave_IO_Running:|Slave_SQL_Running:'
Master_Host: 192.168.10.92
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
最后，实验开始
(1)在master端启动mysqld服务
[root@master ~]# /etc/init.d/mysql start
Starting MySQL. SUCCESS!
(2)在manager端的mha日志文件中找到主从同步的sql语句，这条语句只需要修改密码即可使用
[root@manager ~]# grep 'MASTER_HOST' /app/mha/manager.log | tail -n 1
Thu Jan 25 16:00:21 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='slave01 or 192.168.10.92', MASTER_PORT=3306, MASTER_LOG_FILE='mysql_bin.000002', MASTER_LOG_POS=120, MASTER_USER='rep', MASTER_PASSWORD='xxx';
(3)在master上启动主从同步，密码为20151012
[root@master ~]#mysql -e "CHANGE MASTER TO MASTER_HOST='192.168.10.92', MASTER_PORT=3306, MASTER_LOG_FILE='mysql_bin.000002', MASTER_LOG_POS=120, MASTER_USER='rep', MASTER_PASSWORD='20151012'; start slave;"
[root@master ~]# mysql -e "show slave status\G"
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.10.92
Master_User: rep
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql_bin.000002
Read_Master_Log_Pos: 120
Relay_Log_File: c0a80a5b-relay-bin.000002
Relay_Log_Pos: 283
Relay_Master_Log_File: mysql_bin.000002
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 120
Relay_Log_Space: 459
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 92
Master_UUID: 996b4343-00f3-11e8-a3ba-b6c824ce1080
Master_Info_File: /app/mysql/data/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0