实战MHA部署、高可用测试
简介：
MHA（Master High Availability）目前在MySQL高可用方面是一个相对成熟的解决方案，它由日本DeNA公司youshimaton（现就职于Facebook公司）开发，是一套优秀的作为MySQL高可用性环境下故障切换和主从提升的高可用软件。在MySQL故障切换过程中，MHA能做到在0~30秒之内自动完成数据库的故障切换操作，并且在进行故障切换的过程中，MHA能在最大程度上保证数据的一致性，以达到真正意义上的高可用。
该软件由两部分组成：MHA Manager（管理节点）和MHA Node（数据节点）。MHA Manager可以单独部署在一台独立的机器上管理多个master-slave集群，也可以部署在一台slave节点上。MHA Node运行在每台MySQL服务器上，MHA Manager会定时探测集群中的master节点，当master出现故障时，它可以自动将最新数据的slave提升为新的master，然后将所有其他的slave重新指向新的master。整个故障转移过程对应用程序完全透明。
在MHA自动故障切换过程中，MHA试图从宕机的主服务器上保存二进制日志，最大程度的保证数据的不丢失，但这并不总是可行的。例如，如果主服务器硬件故障或无法通过ssh访问，MHA没法保存二进制日志，只进行故障转移而丢失了最新的数据。使用MySQL 5.5的半同步复制，可以大大降低数据丢失的风险。MHA可以与半同步复制结合起来。如果只有一个slave已经收到了最新的二进制日志，MHA可以将最新的二进制日志应用于其他所有的slave服务器上，因此可以保证所有节点的数据一致性。
目前MHA主要支持一主多从的架构，要搭建MHA,要求一个复制集群中必须最少有三台数据库服务器，一主二从，即一台充当master，一台充当备用master，另外一台充当从库，因为至少需要三台服务器，出于机器成本的考虑，淘宝也在该基础上进行了改造，目前淘宝TMHA已经支持一主一从。
我们自己使用其实也可以使用1主1从，但是master主机宕机后无法切换，以及无法补全binlog。master的mysqld进程crash后，还是可以切换成功，以及补全binlog的。
官方介绍：https://code.google.com/p/mysql-master-ha/
MHA的切换过程
（1）从宕机崩溃的master保存二进制日志事件（binlog events）;
（2）识别含有最新更新的slave；
（3）应用差异的中继日志（relay log）到其他的slave；
（4）应用从master保存的二进制日志事件（binlog events）；
（5）提升一个slave为新的master；
（6）使其他的slave连接新的master进行复制；
部署：
IP设置：
192.168.1.2 主库
192.168.1.4 备用主库
192.168.1.5 从库
192.168.1.5 管理主机
在所有主机hosts文件添加以下内容
192.168.1.2 db1
192.168.1.4 db3
192.168.1.5 db4
192.168.1.6 db5
主从配置
在所有节点上安装依赖包，脚本如下
cat install.sh
#!/bin/bash
wget http://xrl.us/cpanm --no-check-certificate
mv cpanm /usr/bin
chmod 755 /usr/bin/cpanm
cat > /root/list << EOF
install DBD::mysql
EOF
for package in `cat /root/list`
do
cpanm $package
done
192.168.1.2为主，192.168.1.4，192.168.1.5为从
称192.168.1.4为备主更合适点
主从配置使用传统的指定主库、日志、事务点来配置，gtid mha 5.3是不支持的，这点要注意，
5.6支持，以下是官方的描述
Starting from MHA 0.56, MHA supported both GTID based failover and traditional relay log based failover. MHA automatically distinguishes which failover to choose. To do GTID based failover, all of the following is needed.Using MySQL 5.6 (or later)
All MySQL instances use gtid_mode=1
At least one of the instances has Auto_Position enabled
在192.168.1.2和192.168.1.4上执行
grant replication slave,replication client on *.* to slave@'%'identified by 'itvgd123';
flush privileges
在192.168.1.4和192.168.1.5上执行
CHANGE MASTER TO MASTER_HOST='192.168.1.2',MASTER_USER='slave', MASTER_PASSWORD='itvgd123',MASTER_LOG_FILE='mysql-bin.000005',MASTER_LOG_POS=120;
start slave;
wget http://mysql-master-ha.googlecode.com/files/mha4mysql-manager-0.53.tar.gz
wget http://mysql-master-ha.googlecode.com/files/mha4mysql-node-0.53.tar.gz
把mha4mysql-node-0.53.tar.gz上传到三个数据库节点和管理节点
在4台主机上安装依赖包
tar -zvxf mha4mysql-node-0.53.tar.gz
cd mha4mysql-node-0.53
perl Makefile.PL
make
make install
把mha4mysql-manager-0.53.tar.gz上传到管理节点
安装
tar -zvxf mha4mysql-manager-0.53.tar.gz
cd mha4mysql-manager-0.53
perl Makefile.PL
make
make install
配置管理主机到数据库节点间的免密钥
192.168.1.2到192.168.1.4
192.168.1.2到192.168.1.5
192.168.1.4到192.168.1.2
192.168.1.4到192.168.1.5
192.168.1.5到192.168.1.2
192.168.1.5到192.168.1.4
方法如下
ssh-key-get -t rsa
cd /root/.ssh
ssh-copy-id -i id_rsa.pub [email protected]
在两台从库上执行以下命令
mysql -e 'set global read_only=1'
mysql -e 'set global relay_log_purge=0'
在主库（包括主库和备用主库）上创建监控用户
grant all privileges on *.* to 'root'@'%' identified by '123456';
flush privileges;
配置MHA（在管理主机上操作）
mkdir -p /etc/masterha
cd /tmp/mha4mysql-manager-0.53/samples/conf
cp -rp app1.cnf /etc/masterha/
执行
masterha_check_ssh --conf=/etc/masterha/app1.cnf
报错
Base class package "Log::Dispatch::Base" is empty.
(Perhaps you need to 'use' the module which defines that package first,
or make that module available in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .).
at /usr/local/share/perl5/Log/Dispatch.pm line 10
BEGIN failed--compilation aborted at /usr/local/share/perl5/Log/Dispatch.pm line 10.
Compilation failed in require at /usr/local/share/perl5/MHA/SSHCheck.pm line 29.
BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/SSHCheck.pm line 29.
Compilation failed in require at /usr/local/bin/masterha_check_ssh line 25.
BEGIN failed--compilation aborted at /usr/local/bin/masterha_check_ssh line 25.
执行
perl -MCPAN -e "install Log::Dispatch"
后报错消失，估计是在安装依赖包的时候网络出了问题，少装了这个包
[root@db5 mha4mysql-manager-0.53]# masterha_check_repl --conf=/etc/masterha/app1.cnf
Can't locate MHA/NodeConst.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/local/share/perl5/MHA/ManagerConst.pm line 25.
BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/ManagerConst.pm line 25.
Compilation failed in require at /usr/local/share/perl5/MHA/Server.pm line 27.
BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/Server.pm line 27.
Compilation failed in require at /usr/local/share/perl5/MHA/Config.pm line 29.
BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/Config.pm line 29.
Compilation failed in require at /usr/local/share/perl5/MHA/MasterMonitor.pm line 30.
BEGIN failed--compilation aborted at /usr/local/share/perl5/MHA/MasterMonitor.pm line 30.
Compilation failed in require at /usr/local/bin/masterha_check_repl line 25.
BEGIN failed--compilation aborted at /usr/local/bin/masterha_check_repl line 25.
这个报错是因为在管理主机上没有安装mha4mysql-node-0.53.tar.gz所致
在管理节点安装mha4mysql-node-0.53.tar.gz 后报错消失
mha-manager配置文件
cat /etc/masterha/app1.cnf
[server default]
manager_log=/var/log/masterha/app1/manager.log
manager_workdir=/var/log/masterha/app1.log
master_binlog_dir=/home/mysql/data
master_ip_failover_script=/usr/local/bin/master_ip_failover
master_ip_online_change_script=/usr/local/bin/master_ip_online_change
password=123456
ping_interval=1
remote_workdir=/tmp
repl_password=123456
repl_user=repl
report_script=/usr/local/send_report
secondary_check_script=/usr/local/bin/masterha_secondary_check -s db1 -s db3 --user=root --master_host=db1 --master_ip=192.168.1.2 --master_port=3306
shutdown_script=""
ssh_user=root
user=root
[server1]
hostname=192.168.1.2
port=3306
[server2]
candidate_master=1
check_repl_delay=0
hostname=192.168.1.4
port=3306
[server3]
hostname=192.168.1.5
port=3306
检查ssh连接
[root@db5 mha4mysql-manager-0.53]# masterha_check_ssh --conf=/etc/masterha/app1.cnf
Fri Sep 25 21:16:29 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Sep 25 21:16:29 2015 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Fri Sep 25 21:16:29 2015 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Fri Sep 25 21:16:29 2015 - [info] Starting SSH connection tests..
Fri Sep 25 21:16:31 2015 - [debug]
Fri Sep 25 21:16:29 2015 - [debug] Connecting via SSH from [email protected](192.168.1.2:22) to [email protected](192.168.1.4:22)..
Fri Sep 25 21:16:30 2015 - [debug] ok.
Fri Sep 25 21:16:30 2015 - [debug] Connecting via SSH from [email protected](192.168.1.2:22) to [email protected](192.168.1.5:22)..
Fri Sep 25 21:16:31 2015 - [debug] ok.
Fri Sep 25 21:16:51 2015 - [debug]
Fri Sep 25 21:16:30 2015 - [debug] Connecting via SSH from [email protected](192.168.1.4:22) to [email protected](192.168.1.2:22)..
Fri Sep 25 21:16:41 2015 - [debug] ok.
Fri Sep 25 21:16:41 2015 - [debug] Connecting via SSH from [email protected](192.168.1.4:22) to [email protected](192.168.1.5:22)..
Fri Sep 25 21:16:51 2015 - [debug] ok.
Fri Sep 25 21:16:52 2015 - [debug]
Fri Sep 25 21:16:30 2015 - [debug] Connecting via SSH from [email protected](192.168.1.5:22) to [email protected](192.168.1.2:22)..
Fri Sep 25 21:16:41 2015 - [debug] ok.
Fri Sep 25 21:16:41 2015 - [debug] Connecting via SSH from [email protected](192.168.1.5:22) to [email protected](192.168.1.4:22)..
Fri Sep 25 21:16:52 2015 - [debug] ok.
Fri Sep 25 21:16:52 2015 - [info] All SSH connection tests passed successfully.
检查整个复制情况
[ mha4mysql-manager-0.53]# masterha_check_repl --conf=/etc/masterha/app1.cnf
Fri Sep 25 21:29:14 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Sep 25 21:29:14 2015 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Fri Sep 25 21:29:14 2015 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Fri Sep 25 21:29:14 2015 - [info] MHA::MasterMonitor version 0.53.
Fri Sep 25 21:29:14 2015 - [info] Dead Servers:
Fri Sep 25 21:29:14 2015 - [info] Alive Servers:
Fri Sep 25 21:29:14 2015 - [info] 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 21:29:14 2015 - [info] 192.168.1.4(192.168.1.4:3306)
Fri Sep 25 21:29:14 2015 - [info] 192.168.1.5(192.168.1.5:3306)
Fri Sep 25 21:29:14 2015 - [info] Alive Slaves:
Fri Sep 25 21:29:14 2015 - [info] 192.168.1.4(192.168.1.4:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Fri Sep 25 21:29:14 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 21:29:14 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Fri Sep 25 21:29:14 2015 - [info] 192.168.1.5(192.168.1.5:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Fri Sep 25 21:29:14 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 21:29:14 2015 - [info] Not candidate for the new Master (no_master is set)
Fri Sep 25 21:29:14 2015 - [info] Current Alive Master: 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 21:29:14 2015 - [info] Checking slave configurations..
Fri Sep 25 21:29:14 2015 - [info] Checking replication filtering settings..
Fri Sep 25 21:29:14 2015 - [info] binlog_do_db= , binlog_ignore_db=
Fri Sep 25 21:29:14 2015 - [info] Replication filtering check ok.
Fri Sep 25 21:29:14 2015 - [info] Starting SSH connection tests..
Fri Sep 25 21:29:42 2015 - [info] All SSH connection tests passed successfully.
Fri Sep 25 21:29:42 2015 - [info] Checking MHA Node version..
Fri Sep 25 21:30:03 2015 - [info] Version check ok.
Fri Sep 25 21:30:03 2015 - [info] Checking SSH publickey authentication settings on the current master..
Fri Sep 25 21:30:03 2015 - [info] HealthCheck: SSH to 192.168.1.2 is reachable.
Fri Sep 25 21:30:04 2015 - [info] Master MHA Node version is 0.53.
Fri Sep 25 21:30:04 2015 - [info] Checking recovery script configurations on the current master..
Fri Sep 25 21:30:04 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/mysql/data --output_file=/usr/local/mha/save_binary_logs_test --manager_version=0.53 --start_file=mysql-bin.000015
Fri Sep 25 21:30:04 2015 - [info] Connecting to (192.168.1.2)..
Creating /usr/local/mha if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /home/mysql/data, up to mysql-bin.000015
Fri Sep 25 21:30:05 2015 - [info] Master setting check done.
Fri Sep 25 21:30:05 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Fri Sep 25 21:30:05 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=root --slave_host=192.168.1.4 --slave_ip=192.168.1.4 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.6.25-73.0-log --manager_version=0.53 --relay_log_info=/home/mysql/data/relay-log.info --relay_dir=/home/mysql/data/ --slave_pass=xxx
Fri Sep 25 21:30:05 2015 - [info] Connecting to (192.168.1.4:22)..
Creating directory /usr/local/mha.. done.
Checking slave recovery environment settings..
Opening /home/mysql/data/relay-log.info ... ok.
Relay log found at /home/mysql/data, up to db3-relay-bin.000006
Temporary relay log file is /home/mysql/data/db3-relay-bin.000006
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Sep 25 21:30:16 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=root --slave_host=192.168.1.5 --slave_ip=192.168.1.5 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.6.25-73.0-log --manager_version=0.53 --relay_log_info=/home/mysql/data/relay-log.info --relay_dir=/home/mysql/data/ --slave_pass=xxx
Fri Sep 25 21:30:16 2015 - [info] Connecting to (192.168.1.5:22)..
Creating directory /usr/local/mha.. done.
Checking slave recovery environment settings..
Opening /home/mysql/data/relay-log.info ... ok.
Relay log found at /home/mysql/data, up to db4-relay-bin.000006
Temporary relay log file is /home/mysql/data/db4-relay-bin.000006
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Sep 25 21:30:27 2015 - [info] Slaves settings check done.
Fri Sep 25 21:30:27 2015 - [info]
192.168.1.2 (current master)
+--192.168.1.4
+--192.168.1.5
Fri Sep 25 21:30:27 2015 - [info] Checking replication health on 192.168.1.4..
Fri Sep 25 21:30:27 2015 - [info] ok.
Fri Sep 25 21:30:27 2015 - [info] Checking replication health on 192.168.1.5..
Fri Sep 25 21:30:27 2015 - [info] ok.
Fri Sep 25 21:30:27 2015 - [info] Checking master_ip_failover_script status:
Fri Sep 25 21:30:27 2015 - [info] /usr/local/scripts/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.2 --orig_master_ip=192.168.1.2 --orig_master_port=3306
Fri Sep 25 21:30:27 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln383] Error happend on checking configurations. Can't exec "/usr/local/scripts/master_ip_failover": 没有那个文件或目录 at /usr/local/share/perl5/MHA/ManagerUtil.pm line 68.
Fri Sep 25 21:30:27 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln478] Error happened on monitoring servers.
Fri Sep 25 21:30:27 2015 - [info] Got exit code 1 (Not master dead).
MySQL Replication Health is NOT OK!
Fri Sep 25 21:30:27 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln214] Failed to get master_ip_failover_script status with return code 1:0.
Fri Sep 25 21:30:27 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln383] Error happend on checking configurations. at /usr/local/bin/masterha_check_repl line 48
Fri Sep 25 21:30:27 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln478] Error happened on monitoring servers.
Fri Sep 25 21:30:27 2015 - [info] Got exit code 1 (Not master dead).
以上报错先不管
检查MHA Maneger的状态
[ mha4mysql-manager-0.53]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 is stopped(2:NOT_RUNNING).
开启MHA Manager监控
nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &
[ mha4mysql-manager-0.53]# masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover
Fri Sep 25 21:48:09 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Sep 25 21:48:09 2015 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Fri Sep 25 21:48:09 2015 - [info] Reading server configurations from /etc/masterha/app1.cnf..
一直卡住
查看后台日志
[ ~]# tailf /usr/local/mha/manager.log
Fri Sep 25 21:48:53 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/mysql/data --output_file=/usr/local/mha/save_binary_logs_test --manager_version=0.53 --start_file=mysql-bin.000015
Fri Sep 25 21:48:53 2015 - [info] Connecting to (192.168.1.2)..
Creating /usr/local/mha if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /home/mysql/data, up to mysql-bin.000015
Fri Sep 25 21:48:54 2015 - [info] Master setting check done.
Fri Sep 25 21:48:54 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Fri Sep 25 21:48:54 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=root --slave_host=192.168.1.4 --slave_ip=192.168.1.4 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.6.25-73.0-log --manager_version=0.53 --relay_log_info=/home/mysql/data/relay-log.info --relay_dir=/home/mysql/data/ --slave_pass=xxx
Fri Sep 25 21:48:54 2015 - [info] Connecting to (192.168.1.4:22)..
Checking slave recovery environment settings..
Opening /home/mysql/data/relay-log.info ... ok.
Relay log found at /home/mysql/data, up to db3-relay-bin.000006
Temporary relay log file is /home/mysql/data/db3-relay-bin.000006
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Sep 25 21:49:09 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=root --slave_host=192.168.1.5 --slave_ip=192.168.1.5 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.6.25-73.0-log --manager_version=0.53 --relay_log_info=/home/mysql/data/relay-log.info --relay_dir=/home/mysql/data/ --slave_pass=xxx
Fri Sep 25 21:49:09 2015 - [info] Connecting to (192.168.1.5:22)..
Checking slave recovery environment settings..
Opening /home/mysql/data/relay-log.info ... ok.
Relay log found at /home/mysql/data, up to db4-relay-bin.000006
Temporary relay log file is /home/mysql/data/db4-relay-bin.000006
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Sep 25 21:49:20 2015 - [info] Slaves settings check done.
Fri Sep 25 21:49:20 2015 - [info]
192.168.1.2 (current master)
+--192.168.1.4
+--192.168.1.5
Fri Sep 25 21:49:20 2015 - [info] Checking master_ip_failover_script status:
Fri Sep 25 21:49:20 2015 - [info] /usr/local/scripts/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.2 --orig_master_ip=192.168.1.2 --orig_master_port=3306
Fri Sep 25 21:49:20 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln214] Failed to get master_ip_failover_script status with return code 127:0.
Fri Sep 25 21:49:20 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln383] Error happend on checking configurations. at /usr/local/bin/masterha_manager line 50
Fri Sep 25 21:49:20 2015 - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln478] Error happened on monitoring servers.
Fri Sep 25 21:49:20 2015 - [info] Got exit code 1 (Not master dead).
原来是sh: /usr/local/scripts/master_ip_failover: 没有那个文件或目录
把配置文件的/usr/local/scripts/master_ip_failover注释掉
执行
nohup masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/masterha/app1/manager.log 2>&1 &
查看日志
Fri Sep 25 22:02:09 2015 - [info] MHA::MasterMonitor version 0.53.
Fri Sep 25 22:02:09 2015 - [warning] /usr/local/mha/app1.master_status.health already exists. You might have killed manager with SIGKILL(-9), may run two or more monitoring process for the same application, or use the same working directory. Check for details, and consider setting --workdir separately.
Fri Sep 25 22:02:09 2015 - [info] Dead Servers:
Fri Sep 25 22:02:09 2015 - [info] Alive Servers:
Fri Sep 25 22:02:09 2015 - [info] 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 22:02:09 2015 - [info] 192.168.1.4(192.168.1.4:3306)
Fri Sep 25 22:02:09 2015 - [info] 192.168.1.5(192.168.1.5:3306)
Fri Sep 25 22:02:09 2015 - [info] Alive Slaves:
Fri Sep 25 22:02:09 2015 - [info] 192.168.1.4(192.168.1.4:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Fri Sep 25 22:02:09 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 22:02:09 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Fri Sep 25 22:02:09 2015 - [info] 192.168.1.5(192.168.1.5:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Fri Sep 25 22:02:09 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 22:02:09 2015 - [info] Not candidate for the new Master (no_master is set)
Fri Sep 25 22:02:09 2015 - [info] Current Alive Master: 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 22:02:09 2015 - [info] Checking slave configurations..
Fri Sep 25 22:02:09 2015 - [info] Checking replication filtering settings..
Fri Sep 25 22:02:09 2015 - [info] binlog_do_db= , binlog_ignore_db=
Fri Sep 25 22:02:09 2015 - [info] Replication filtering check ok.
Fri Sep 25 22:02:09 2015 - [info] Starting SSH connection tests..
Fri Sep 25 22:02:32 2015 - [info] All SSH connection tests passed successfully.
Fri Sep 25 22:02:32 2015 - [info] Checking MHA Node version..
Fri Sep 25 22:02:54 2015 - [info] Version check ok.
Fri Sep 25 22:02:54 2015 - [info] Checking SSH publickey authentication settings on the current master..
Fri Sep 25 22:02:55 2015 - [info] HealthCheck: SSH to 192.168.1.2 is reachable.
Fri Sep 25 22:02:56 2015 - [info] Master MHA Node version is 0.53.
Fri Sep 25 22:02:56 2015 - [info] Checking recovery script configurations on the current master..
Fri Sep 25 22:02:56 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/mysql/data --output_file=/usr/local/mha/save_binary_logs_test --manager_version=0.53 --start_file=mysql-bin.000015
Fri Sep 25 22:02:56 2015 - [info] Connecting to [email protected](192.168.1.2)..
Creating /usr/local/mha if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /home/mysql/data, up to mysql-bin.000015
Fri Sep 25 22:02:56 2015 - [info] Master setting check done.
Fri Sep 25 22:02:56 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Fri Sep 25 22:02:56 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=root --slave_host=192.168.1.4 --slave_ip=192.168.1.4 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.6.25-73.0-log --manager_version=0.53 --relay_log_info=/home/mysql/data/relay-log.info --relay_dir=/home/mysql/data/ --slave_pass=xxx
Fri Sep 25 22:02:56 2015 - [info] Connecting to [email protected](192.168.1.4:22)..
Checking slave recovery environment settings..
Opening /home/mysql/data/relay-log.info ... ok.
Relay log found at /home/mysql/data, up to db3-relay-bin.000006
Temporary relay log file is /home/mysql/data/db3-relay-bin.000006
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Sep 25 22:03:07 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=root --slave_host=192.168.1.5 --slave_ip=192.168.1.5 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.6.25-73.0-log --manager_version=0.53 --relay_log_info=/home/mysql/data/relay-log.info --relay_dir=/home/mysql/data/ --slave_pass=xxx
Fri Sep 25 22:03:07 2015 - [info] Connecting to [email protected](192.168.1.5:22)..
Checking slave recovery environment settings..
Opening /home/mysql/data/relay-log.info ... ok.
Relay log found at /home/mysql/data, up to db4-relay-bin.000006
Temporary relay log file is /home/mysql/data/db4-relay-bin.000006
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Sep 25 22:03:18 2015 - [info] Slaves settings check done.
Fri Sep 25 22:03:18 2015 - [info]
192.168.1.2 (current master)
+--192.168.1.4
+--192.168.1.5
Fri Sep 25 22:03:18 2015 - [warning] master_ip_failover_script is not defined.
Fri Sep 25 22:03:18 2015 - [warning] shutdown_script is not defined.
Fri Sep 25 22:03:18 2015 - [info] Set master ping interval 1 seconds.
Fri Sep 25 22:03:18 2015 - [info] Set secondary check script: masterha_secondary_check -s 192.168.1.226 -s 192.168.1.227
Fri Sep 25 22:03:18 2015 - [info] Starting ping health check on 192.168.1.2(192.168.1.2:3306)..
Fri Sep 25 22:03:18 2015 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
查看mha manager状态
[root@db5 ~]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 is stopped(2:NOT_RUNNING).
[root@db5 ~]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 monitoring program is now on initialization phase(10:INITIALIZING_MONITOR). Wait for a while and try checking again.[root@db5 ~]# masterha_check_status --conf=/etc/masterha/app1.cnf
app1 (pid:5394) is running(0:PING_OK), master:192.168.1.2
现在修改配置文件
vi /etc/masterha/app1.cnf
把注释掉的
/usr/local/scripts/master_ip_failover改回来
整个配置文件就这样了
# mkdir -p /usr/local/mha
# mkdir -p /etc/mha
# cat /etc/mha/mha.conf
[server default]
user=root
password=123456
manager_workdir=/usr/local/mha
manager_log=/usr/local/mha/manager.log
remote_workdir=/usr/local/mha
ssh_user=root
repl_user=slave
repl_password=itvgd123
ping_interval=1
secondary_check_script= masterha_secondary_check -s 192.168.1.4 -s 192.168.1.5
master_ip_failover_script=/usr/local/scripts/master_ip_failover
[server1]
hostname=192.168.1.2
ssh_port=22
master_binlog_dir=/home/mysql/data
candidate_master=1
[server2]
hostname=192.168.1.4
ssh_port=22
master_binlog_dir=/home/mysql/data
candidate_master=1
[server3]
hostname=192.168.1.5
ssh_port=22
master_binlog_dir=/home/mysql/data
no_master=1
以下给出
master_ip_failover脚本的内容
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';
use Getopt::Long;
my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);
my $vip = '192.168.1.88';
my $ssh_start_vip = "/etc/init.d/keepalived start";
my $ssh_stop_vip = "/etc/init.d/keepalived stop";
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);
exit &main();
sub main {
print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {
my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
exit 0;
}
else {
&usage();
exit 1;
}
}
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}
以上脚本需要使用keepalived提供虚拟IP，现在我们安装和配置keepalivded
wget http://www.keepalived.org/software/keepalived-1.2.12.tar.gz
tar xf keepalived-1.2.12.tar.gz
cd keepalived-1.2.12
./configure --prefix=/usr/local/keepalived
make && make install
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
mkdir /etc/keepalived
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
主库
/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id MySQL-HA
}
vrrp_instance VI_1 {
state BACKUP
interface eth1
virtual_router_id 51
priority 150
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.1.88
}
}
备用主库
! Configuration File for keepalived
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id MySQL-HA
}
vrrp_instance VI_1 {
state BACKUP
interface eth1
virtual_router_id 51
priority 120
advert_int 1
nopreempt
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress { 192.168.0.88
}
}
主库和备用主库的配置文件一样，只是两者的priority值不一样
在主库上先开启keepalived，备用主库上后启动
主库
/etc/init.d/keepalived start
tailf /var/log/messages
Sep 27 04:41:06 db1 Keepalived[1628]: Starting Keepalived v1.2.12 (09/26,2015)
Sep 27 04:41:06 db1 Keepalived[1629]: Starting Healthcheck child process, pid=1631
Sep 27 04:41:06 db1 Keepalived[1629]: Starting VRRP child process, pid=1632
Sep 27 04:41:06 db1 Keepalived_healthcheckers[1631]: Netlink reflector reports IP 192.168.1.2 added
Sep 27 04:41:06 db1 Keepalived_healthcheckers[1631]: Netlink reflector reports IP fe80::20c:29ff:fe39:7429 added
Sep 27 04:41:06 db1 Keepalived_healthcheckers[1631]: Registering Kernel netlink reflector
Sep 27 04:41:06 db1 Keepalived_healthcheckers[1631]: Registering Kernel netlink command channel
Sep 27 04:41:06 db1 Keepalived_healthcheckers[1631]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 27 04:41:06 db1 Keepalived_healthcheckers[1631]: Configuration is using : 7093 Bytes
Sep 27 04:41:06 db1 Keepalived_vrrp[1632]: Netlink reflector reports IP 192.168.1.2 added
Sep 27 04:41:06 db1 Keepalived_vrrp[1632]: Netlink reflector reports IP fe80::20c:29ff:fe39:7429 added
Sep 27 04:41:06 db1 Keepalived_vrrp[1632]: Registering Kernel netlink reflector
Sep 27 04:41:06 db1 Keepalived_vrrp[1632]: Registering Kernel netlink command channel
Sep 27 04:41:06 db1 Keepalived_vrrp[1632]: Registering gratuitous ARP shared channel
Sep 27 04:41:06 db1 Keepalived_vrrp[1632]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 27 04:41:06 db1 Keepalived_vrrp[1632]: Configuration is using : 62838 Bytes
Sep 27 04:41:06 db1 Keepalived_vrrp[1632]: Using LinkWatch kernel netlink reflector...
Sep 27 04:41:06 db1 Keepalived_healthcheckers[1631]: Using LinkWatch kernel netlink reflector...
Sep 27 04:41:06 db1 Keepalived_vrrp[1632]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 27 04:41:06 db1 Keepalived_vrrp[1632]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Sep 27 04:41:10 db1 Keepalived_vrrp[1632]: VRRP_Instance(VI_1) Transition to MASTER STATE
Sep 27 04:41:11 db1 Keepalived_vrrp[1632]: VRRP_Instance(VI_1) Entering MASTER STATE
Sep 27 04:41:11 db1 Keepalived_vrrp[1632]: VRRP_Instance(VI_1) setting protocol VIPs.
Sep 27 04:41:11 db1 Keepalived_healthcheckers[1631]: Netlink reflector reports IP 192.168.1.88 added
Sep 27 04:41:11 db1 Keepalived_vrrp[1632]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.1.88
Sep 27 04:41:16 db1 Keepalived_vrrp[1632]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.1.88
执行命令
ip addr | grep eth1
[root@db1 ~]# ip addr | grep eth1
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
inet 192.168.1.2/24 brd 192.168.1.255 scope global eth1
inet 192.168.1.88/32 scope global eth1
我们看到主库已经获得虚拟Ip192.168.1.88
在备用主库启动keepalivied
/etc/init.d/keepalived start
tailf /var/log/messages
Sep 27 09:06:27 db3 Keepalived[2005]: Starting Keepalived v1.2.12 (09/26,2015)
Sep 27 09:06:27 db3 Keepalived[2006]: Starting Healthcheck child process, pid=2008
Sep 27 09:06:27 db3 Keepalived[2006]: Starting VRRP child process, pid=2009
Sep 27 09:06:27 db3 Keepalived_vrrp[2009]: Netlink reflector reports IP 192.168.1.4 added
Sep 27 09:06:27 db3 Keepalived_vrrp[2009]: Netlink reflector reports IP fe80::20c:29ff:fe94:fb1a added
Sep 27 09:06:27 db3 Keepalived_vrrp[2009]: Registering Kernel netlink reflector
Sep 27 09:06:27 db3 Keepalived_vrrp[2009]: Registering Kernel netlink command channel
Sep 27 09:06:27 db3 Keepalived_vrrp[2009]: Registering gratuitous ARP shared channel
Sep 27 09:06:27 db3 Keepalived_vrrp[2009]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 27 09:06:27 db3 Keepalived_vrrp[2009]: Configuration is using : 62838 Bytes
Sep 27 09:06:27 db3 Keepalived_vrrp[2009]: Using LinkWatch kernel netlink reflector...
Sep 27 09:06:27 db3 Keepalived_vrrp[2009]: VRRP_Instance(VI_1) Entering BACKUP STATE
Sep 27 09:06:27 db3 Keepalived_vrrp[2009]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Sep 27 09:06:27 db3 kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
Sep 27 09:06:27 db3 kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
Sep 27 09:06:27 db3 kernel: IPVS: ipvs loaded.
Sep 27 09:06:27 db3 Keepalived_healthcheckers[2008]: Netlink reflector reports IP 192.168.1.4 added
Sep 27 09:06:27 db3 Keepalived_healthcheckers[2008]: Netlink reflector reports IP fe80::20c:29ff:fe94:fb1a added
Sep 27 09:06:27 db3 Keepalived_healthcheckers[2008]: Registering Kernel netlink reflector
Sep 27 09:06:27 db3 Keepalived_healthcheckers[2008]: Registering Kernel netlink command channel
Sep 27 09:06:27 db3 Keepalived_healthcheckers[2008]: Opening file '/etc/keepalived/keepalived.conf'.
Sep 27 09:06:27 db3 Keepalived_healthcheckers[2008]: Configuration is using : 7093 Bytes
Sep 27 09:06:27 db3 Keepalived_healthcheckers[2008]: Using LinkWatch kernel netlink reflector...
chmod +x /usr/local/scripts/master_ip_failover
启动MHA Manager
masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover
查看日志
tailf /usr/local/mha/manager.log
Fri Sep 25 22:38:08 2015 - [info] MHA::MasterMonitor version 0.53.
Fri Sep 25 22:38:08 2015 - [info] Dead Servers:
Fri Sep 25 22:38:08 2015 - [info] Alive Servers:
Fri Sep 25 22:38:08 2015 - [info] 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 22:38:08 2015 - [info] 192.168.1.4(192.168.1.4:3306)
Fri Sep 25 22:38:08 2015 - [info] 192.168.1.5(192.168.1.5:3306)
Fri Sep 25 22:38:08 2015 - [info] Alive Slaves:
Fri Sep 25 22:38:08 2015 - [info] 192.168.1.4(192.168.1.4:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Fri Sep 25 22:38:08 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 22:38:08 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Fri Sep 25 22:38:08 2015 - [info] 192.168.1.5(192.168.1.5:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Fri Sep 25 22:38:08 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 22:38:08 2015 - [info] Not candidate for the new Master (no_master is set)
Fri Sep 25 22:38:08 2015 - [info] Current Alive Master: 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 22:38:08 2015 - [info] Checking slave configurations..
Fri Sep 25 22:38:08 2015 - [info] Checking replication filtering settings..
Fri Sep 25 22:38:08 2015 - [info] binlog_do_db= , binlog_ignore_db=
Fri Sep 25 22:38:08 2015 - [info] Replication filtering check ok.
Fri Sep 25 22:38:08 2015 - [info] Starting SSH connection tests..
Fri Sep 25 22:38:30 2015 - [info] All SSH connection tests passed successfully.
Fri Sep 25 22:38:30 2015 - [info] Checking MHA Node version..
Fri Sep 25 22:38:51 2015 - [info] Version check ok.
Fri Sep 25 22:38:51 2015 - [info] Checking SSH publickey authentication settings on the current master..
Fri Sep 25 22:38:52 2015 - [info] HealthCheck: SSH to 192.168.1.2 is reachable.
Fri Sep 25 22:38:52 2015 - [info] Master MHA Node version is 0.53.
Fri Sep 25 22:38:52 2015 - [info] Checking recovery script configurations on the current master..
Fri Sep 25 22:38:52 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/mysql/data --output_file=/usr/local/mha/save_binary_logs_test --manager_version=0.53 --start_file=mysql-bin.000015
Fri Sep 25 22:38:52 2015 - [info] Connecting to [email protected](192.168.1.2)..
Creating /usr/local/mha if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /home/mysql/data, up to mysql-bin.000015
Fri Sep 25 22:38:53 2015 - [info] Master setting check done.
Fri Sep 25 22:38:53 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Fri Sep 25 22:38:53 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=root --slave_host=192.168.1.4 --slave_ip=192.168.1.4 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.6.25-73.0-log --manager_version=0.53 --relay_log_info=/home/mysql/data/relay-log.info --relay_dir=/home/mysql/data/ --slave_pass=xxx
Fri Sep 25 22:38:53 2015 - [info] Connecting to [email protected](192.168.1.4:22)..
Checking slave recovery environment settings..
Opening /home/mysql/data/relay-log.info ... ok.
Relay log found at /home/mysql/data, up to db3-relay-bin.000006
Temporary relay log file is /home/mysql/data/db3-relay-bin.000006
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Sep 25 22:39:03 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=root --slave_host=192.168.1.5 --slave_ip=192.168.1.5 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.6.25-73.0-log --manager_version=0.53 --relay_log_info=/home/mysql/data/relay-log.info --relay_dir=/home/mysql/data/ --slave_pass=xxx
Fri Sep 25 22:39:03 2015 - [info] Connecting to [email protected](192.168.1.5:22)..
Checking slave recovery environment settings..
Opening /home/mysql/data/relay-log.info ... ok.
Relay log found at /home/mysql/data, up to db4-relay-bin.000006
Temporary relay log file is /home/mysql/data/db4-relay-bin.000006
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Sep 25 22:39:14 2015 - [info] Slaves settings check done.
Fri Sep 25 22:39:14 2015 - [info]
192.168.1.2 (current master)
+--192.168.1.4
+--192.168.1.5
Fri Sep 25 22:39:14 2015 - [info] Checking master_ip_failover_script status:
Fri Sep 25 22:39:14 2015 - [info] /usr/local/scripts/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.2 --orig_master_ip=192.168.1.2 --orig_master_port=3306
IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===
Checking the Status of the script.. OK
Fri Sep 25 22:39:14 2015 - [info] OK.
Fri Sep 25 22:39:14 2015 - [warning] shutdown_script is not defined.
Fri Sep 25 22:39:14 2015 - [info] Set master ping interval 1 seconds.
Fri Sep 25 22:39:14 2015 - [info] Set secondary check script: masterha_secondary_check -s 192.168.1.4 -s 192.168.1.5
Fri Sep 25 22:39:14 2015 - [info] Starting ping health check on 192.168.1.2(192.168.1.2:3306)..
Fri Sep 25 22:39:14 2015 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
切换测试
停掉主库，按照原理的话，MHA会尽量应用主库的日志到备用主库，之后备用主库会获得虚拟IP并提升为主库
在管理主机启动MHA―Manager
masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover
在主库上执行，也就是192.168.1.2上执行
service mysqld stop
查看MHA日志
tailf /usr/local/mha/manager.log
Fri Sep 25 23:36:40 2015 - [info] MHA::MasterMonitor version 0.53.
Fri Sep 25 23:36:41 2015 - [info] Dead Servers:
Fri Sep 25 23:36:41 2015 - [info] 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 23:36:41 2015 - [info] Alive Servers:
Fri Sep 25 23:36:41 2015 - [info] 192.168.1.4(192.168.1.4:3306)
Fri Sep 25 23:36:41 2015 - [info] 192.168.1.5(192.168.1.5:3306)
Fri Sep 25 23:36:41 2015 - [info] Alive Slaves:
Fri Sep 25 23:36:41 2015 - [info] 192.168.1.4(192.168.1.4:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Fri Sep 25 23:36:41 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 23:36:41 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Fri Sep 25 23:36:41 2015 - [info] 192.168.1.5(192.168.1.5:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Fri Sep 25 23:36:41 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 23:36:41 2015 - [info] Not candidate for the new Master (no_master is set)
Fri Sep 25 23:36:41 2015 - [warning] MySQL master is not currently alive!
Fri Sep 25 23:36:41 2015 - [info] Checking slave configurations..
Fri Sep 25 23:36:41 2015 - [info] Checking replication filtering settings..
Fri Sep 25 23:36:41 2015 - [info] Replication filtering check ok.
Fri Sep 25 23:36:41 2015 - [info] Starting SSH connection tests..
Fri Sep 25 23:36:52 2015 - [info] All SSH connection tests passed successfully.
Fri Sep 25 23:36:52 2015 - [info] Checking MHA Node version..
Fri Sep 25 23:37:13 2015 - [info] Version check ok.
Fri Sep 25 23:37:13 2015 - [info] Getting current master (maybe dead) info ..
Fri Sep 25 23:37:13 2015 - [info] Identified master is 192.168.1.2(192.168.1.2:3306).
Fri Sep 25 23:37:13 2015 - [info] Checking SSH publickey authentication settings on the current master..
Fri Sep 25 23:37:16 2015 - [info] HealthCheck: SSH to 192.168.1.2 is reachable.
Fri Sep 25 23:37:18 2015 - [info] Master MHA Node version is 0.53.
Fri Sep 25 23:37:18 2015 - [info] Checking recovery script configurations on the current master..
Fri Sep 25 23:37:18 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/mysql/data --output_file=/usr/local/mha/save_binary_logs_test --manager_version=0.53 --start_file=mysql-bin.000015
Fri Sep 25 23:37:18 2015 - [info] Connecting to [email protected](192.168.1.2)..
Creating /usr/local/mha if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /home/mysql/data, up to mysql-bin.000016
Fri Sep 25 23:37:20 2015 - [info] Master setting check done.
Fri Sep 25 23:37:20 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Fri Sep 25 23:37:20 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=root --slave_host=192.168.1.4 --slave_ip=192.168.1.4 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.6.25-73.0-log --manager_version=0.53 --relay_log_info=/home/mysql/data/relay-log.info --relay_dir=/home/mysql/data/ --slave_pass=xxx
Fri Sep 25 23:37:20 2015 - [info] Connecting to [email protected](192.168.1.4:22)..
Checking slave recovery environment settings..
Opening /home/mysql/data/relay-log.info ... ok.
Relay log found at /home/mysql/data, up to db3-relay-bin.000007
Temporary relay log file is /home/mysql/data/db3-relay-bin.000007
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Sep 25 23:37:36 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=root --slave_host=192.168.1.5 --slave_ip=192.168.1.5 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.6.25-73.0-log --manager_version=0.53 --relay_log_info=/home/mysql/data/relay-log.info --relay_dir=/home/mysql/data/ --slave_pass=xxx
Fri Sep 25 23:37:36 2015 - [info] Connecting to [email protected](192.168.1.5:22)..
Checking slave recovery environment settings..
Opening /home/mysql/data/relay-log.info ... ok.
Relay log found at /home/mysql/data, up to db4-relay-bin.000007
Temporary relay log file is /home/mysql/data/db4-relay-bin.000007
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Sep 25 23:37:47 2015 - [info] Slaves settings check done.
Fri Sep 25 23:37:47 2015 - [info]
192.168.1.2 (current master)
+--192.168.1.4
+--192.168.1.5
Fri Sep 25 23:37:47 2015 - [info] Checking master_ip_failover_script status:
Fri Sep 25 23:37:47 2015 - [info] /usr/local/scripts/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.2 --orig_master_ip=192.168.1.2 --orig_master_port=3306
IN SCRIPT TEST====/sbin/ifconfig eth1:1 down==/sbin/ifconfig eth1:1 192.168.1.88/24===
Checking the Status of the script.. OK
Fri Sep 25 23:37:47 2015 - [info] OK.
Fri Sep 25 23:37:47 2015 - [warning] shutdown_script is not defined.
Fri Sep 25 23:37:47 2015 - [error][/usr/local/share/perl5/MHA/Server.pm, ln445] Checking slave status failed on 192.168.1.4(192.168.1.4:3306). err=Got error when executing SHOW SLAVE STATUS. MySQL server has gone away
Fri Sep 25 23:37:47 2015 - [info] Set master ping interval 1 seconds.
Fri Sep 25 23:37:47 2015 - [info] Set secondary check script: masterha_secondary_check -s 192.168.1.4 -s 192.168.1.5
Fri Sep 25 23:37:47 2015 - [info] Starting ping health check on 192.168.1.2(192.168.1.2:3306)..
Fri Sep 25 23:37:47 2015 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
Fri Sep 25 23:38:34 2015 - [info] Got terminate signal. Exit.
Fri Sep 25 23:38:35 2015 - [info] MHA::MasterMonitor version 0.53.
Fri Sep 25 23:38:35 2015 - [info] Dead Servers:
Fri Sep 25 23:38:35 2015 - [info] Alive Servers:
Fri Sep 25 23:38:35 2015 - [info] 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 23:38:35 2015 - [info] 192.168.1.4(192.168.1.4:3306)
Fri Sep 25 23:38:35 2015 - [info] 192.168.1.5(192.168.1.5:3306)
Fri Sep 25 23:38:35 2015 - [info] Alive Slaves:
Fri Sep 25 23:38:35 2015 - [info] 192.168.1.4(192.168.1.4:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Fri Sep 25 23:38:35 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 23:38:35 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Fri Sep 25 23:38:35 2015 - [info] 192.168.1.5(192.168.1.5:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Fri Sep 25 23:38:35 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 23:38:35 2015 - [info] Not candidate for the new Master (no_master is set)
Fri Sep 25 23:38:35 2015 - [info] Current Alive Master: 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 23:38:35 2015 - [info] Checking slave configurations..
Fri Sep 25 23:38:35 2015 - [info] Checking replication filtering settings..
Fri Sep 25 23:38:35 2015 - [info] binlog_do_db= , binlog_ignore_db=
Fri Sep 25 23:38:35 2015 - [info] Replication filtering check ok.
Fri Sep 25 23:38:35 2015 - [info] Starting SSH connection tests..
Fri Sep 25 23:38:49 2015 - [info] Got terminate signal. Exit.
Fri Sep 25 23:38:54 2015 - [info] MHA::MasterMonitor version 0.53.
Fri Sep 25 23:38:55 2015 - [info] Dead Servers:
Fri Sep 25 23:38:55 2015 - [info] Alive Servers:
Fri Sep 25 23:38:55 2015 - [info] 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 23:38:55 2015 - [info] 192.168.1.4(192.168.1.4:3306)
Fri Sep 25 23:38:55 2015 - [info] 192.168.1.5(192.168.1.5:3306)
Fri Sep 25 23:38:55 2015 - [info] Alive Slaves:
Fri Sep 25 23:38:55 2015 - [info] 192.168.1.4(192.168.1.4:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Fri Sep 25 23:38:55 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 23:38:55 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Fri Sep 25 23:38:55 2015 - [info] 192.168.1.5(192.168.1.5:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Fri Sep 25 23:38:55 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 23:38:55 2015 - [info] Not candidate for the new Master (no_master is set)
Fri Sep 25 23:38:55 2015 - [info] Current Alive Master: 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 23:38:55 2015 - [info] Checking slave configurations..
Fri Sep 25 23:38:55 2015 - [info] Checking replication filtering settings..
Fri Sep 25 23:38:55 2015 - [info] binlog_do_db= , binlog_ignore_db=
Fri Sep 25 23:38:55 2015 - [info] Replication filtering check ok.
Fri Sep 25 23:38:55 2015 - [info] Starting SSH connection tests..
Fri Sep 25 23:39:17 2015 - [info] All SSH connection tests passed successfully.
Fri Sep 25 23:39:17 2015 - [info] Checking MHA Node version..
Fri Sep 25 23:39:38 2015 - [info] Version check ok.
Fri Sep 25 23:39:38 2015 - [info] Checking SSH publickey authentication settings on the current master..
Fri Sep 25 23:39:39 2015 - [info] HealthCheck: SSH to 192.168.1.2 is reachable.
Fri Sep 25 23:39:40 2015 - [info] Master MHA Node version is 0.53.
Fri Sep 25 23:39:40 2015 - [info] Checking recovery script configurations on the current master..
Fri Sep 25 23:39:40 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/mysql/data --output_file=/usr/local/mha/save_binary_logs_test --manager_version=0.53 --start_file=mysql-bin.000016
Fri Sep 25 23:39:40 2015 - [info] Connecting to [email protected](192.168.1.2)..
Creating /usr/local/mha if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /home/mysql/data, up to mysql-bin.000016
Fri Sep 25 23:39:40 2015 - [info] Master setting check done.
Fri Sep 25 23:39:40 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Fri Sep 25 23:39:40 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=root --slave_host=192.168.1.4 --slave_ip=192.168.1.4 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.6.25-73.0-log --manager_version=0.53 --relay_log_info=/home/mysql/data/relay-log.info --relay_dir=/home/mysql/data/ --slave_pass=xxx
Fri Sep 25 23:39:40 2015 - [info] Connecting to [email protected](192.168.1.4:22)..
Checking slave recovery environment settings..
Opening /home/mysql/data/relay-log.info ... ok.
Relay log found at /home/mysql/data, up to db3-relay-bin.000007
Temporary relay log file is /home/mysql/data/db3-relay-bin.000007
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Sep 25 23:39:51 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=root --slave_host=192.168.1.5 --slave_ip=192.168.1.5 --slave_port=3306 --workdir=/usr/local/mha --target_version=5.6.25-73.0-log --manager_version=0.53 --relay_log_info=/home/mysql/data/relay-log.info --relay_dir=/home/mysql/data/ --slave_pass=xxx
Fri Sep 25 23:39:51 2015 - [info] Connecting to [email protected](192.168.1.5:22)..
Checking slave recovery environment settings..
Opening /home/mysql/data/relay-log.info ... ok.
Relay log found at /home/mysql/data, up to db4-relay-bin.000007
Temporary relay log file is /home/mysql/data/db4-relay-bin.000007
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Sep 25 23:40:02 2015 - [info] Slaves settings check done.
Fri Sep 25 23:40:02 2015 - [info]
192.168.1.2 (current master)
+--192.168.1.4
+--192.168.1.5
Fri Sep 25 23:40:02 2015 - [info] Checking master_ip_failover_script status:
Fri Sep 25 23:40:02 2015 - [info] /usr/local/scripts/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.2 --orig_master_ip=192.168.1.2 --orig_master_port=3306
IN SCRIPT TEST====/sbin/ifconfig eth1:1 down==/sbin/ifconfig eth1:1 192.168.1.88/24===
Checking the Status of the script.. OK
Fri Sep 25 23:40:02 2015 - [info] OK.
Fri Sep 25 23:40:02 2015 - [warning] shutdown_script is not defined.
Fri Sep 25 23:40:02 2015 - [info] Set master ping interval 1 seconds.
Fri Sep 25 23:40:02 2015 - [info] Set secondary check script: masterha_secondary_check -s 192.168.1.4 -s 192.168.1.5
Fri Sep 25 23:40:02 2015 - [info] Starting ping health check on 192.168.1.2(192.168.1.2:3306)..
Fri Sep 25 23:40:02 2015 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
Fri Sep 25 23:40:21 2015 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
Fri Sep 25 23:40:21 2015 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/mysql/data --output_file=/usr/local/mha/save_binary_logs_test --manager_version=0.53 --binlog_prefix=mysql-bin
Fri Sep 25 23:40:21 2015 - [info] Executing seconary network check script: masterha_secondary_check -s 192.168.1.4 -s 192.168.1.5 --user=root --master_host=192.168.1.2 --master_ip=192.168.1.2 --master_port=3306
Fri Sep 25 23:40:21 2015 - [info] HealthCheck: SSH to 192.168.1.2 is reachable.
Fri Sep 25 23:40:22 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Fri Sep 25 23:40:22 2015 - [warning] Connection failed 1 time(s)..
Fri Sep 25 23:40:23 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Fri Sep 25 23:40:23 2015 - [warning] Connection failed 2 time(s)..
Fri Sep 25 23:40:24 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Fri Sep 25 23:40:24 2015 - [warning] Connection failed 3 time(s)..
Monitoring server 192.168.1.4 is reachable, Master is not reachable from 192.168.1.4. OK.
Monitoring server 192.168.1.5 is reachable, Master is not reachable from 192.168.1.5. OK.
Fri Sep 25 23:40:42 2015 - [info] Master is not reachable from all other monitoring servers. Failover should start.
Fri Sep 25 23:40:42 2015 - [warning] Master is not reachable from health checker!
Fri Sep 25 23:40:42 2015 - [warning] Master 192.168.1.2(192.168.1.2:3306) is not reachable!
Fri Sep 25 23:40:42 2015 - [warning] SSH is reachable.
Fri Sep 25 23:40:42 2015 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.cnf again, and trying to connect to all servers to check server status..
Fri Sep 25 23:40:42 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Sep 25 23:40:42 2015 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Fri Sep 25 23:40:42 2015 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Fri Sep 25 23:40:42 2015 - [info] Dead Servers:
Fri Sep 25 23:40:42 2015 - [info] 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 23:40:42 2015 - [info] Alive Servers:
Fri Sep 25 23:40:42 2015 - [info] 192.168.1.4(192.168.1.4:3306)
Fri Sep 25 23:40:42 2015 - [info] 192.168.1.5(192.168.1.5:3306)
Fri Sep 25 23:40:42 2015 - [info] Alive Slaves:
Fri Sep 25 23:40:42 2015 - [info] 192.168.1.4(192.168.1.4:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Fri Sep 25 23:40:42 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 23:40:42 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Fri Sep 25 23:40:42 2015 - [info] 192.168.1.5(192.168.1.5:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Fri Sep 25 23:40:42 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Fri Sep 25 23:40:42 2015 - [info] Not candidate for the new Master (no_master is set)
Fri Sep 25 23:40:42 2015 - [info] Checking slave configurations..
Fri Sep 25 23:40:42 2015 - [info] Checking replication filtering settings..
Fri Sep 25 23:40:42 2015 - [info] Replication filtering check ok.
Fri Sep 25 23:40:42 2015 - [info] Master is down!
Fri Sep 25 23:40:42 2015 - [info] Terminating monitoring script.
Fri Sep 25 23:40:42 2015 - [info] Got exit code 20 (Master dead).
Fri Sep 25 23:40:42 2015 - [info] MHA::MasterFailover version 0.53.
Fri Sep 25 23:40:42 2015 - [info] Starting master failover.
Fri Sep 25 23:40:42 2015 - [error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln178] Got ERROR: Use of uninitialized value $msg in scalar chomp at /usr/local/share/perl5/MHA/ManagerConst.pm line 90.
发现MHA-manager抛出一个报错
Fri Sep 25 23:40:42 2015 - [error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln178] Got ERROR: Use of uninitialized value $msg in scalar chomp at /usr/local/share/perl5/MHA/ManagerConst.pm line 90.
随后
manager进程退出
试了多次都是这样，在绝望的时候一个朋友提醒我说这是5.3的一个bug，他当时测试的时候踩过这个坑，建议换成5.6，听他这么一说果断换成5.6
老样子，上传5.6 node的包到各个节点，manger包上传到管理节点
安装方法省略
重新测试
在管理节点上启动manager
masterha_manager --conf=/etc/masterha/app1.cnf --remove_dead_master_conf --ignore_last_failover
查看后台日志
tailf /usr/local/mha/manager.log
Sun Sep 27 09:57:26 2015 - [info] MHA::MasterMonitor version 0.56.
Sun Sep 27 09:57:27 2015 - [info] GTID failover mode = 0
Sun Sep 27 09:57:27 2015 - [info] Dead Servers:
Sun Sep 27 09:57:27 2015 - [info] Alive Servers:
Sun Sep 27 09:57:27 2015 - [info] 192.168.1.4(192.168.1.4:3306)
Sun Sep 27 09:57:27 2015 - [info] 192.168.1.5(192.168.1.5:3306)
Sun Sep 27 09:57:27 2015 - [info] Alive Slaves:
Sun Sep 27 09:57:27 2015 - [info] 192.168.1.5(192.168.1.5:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Sun Sep 27 09:57:27 2015 - [info] Replicating from 192.168.1.4(192.168.1.4:3306)
Sun Sep 27 09:57:27 2015 - [info] Current Alive Master: 192.168.1.4(192.168.1.4:3306)
Sun Sep 27 09:57:27 2015 - [info] Checking slave configurations..
Sun Sep 27 09:57:27 2015 - [info] read_only=1 is not set on slave 192.168.1.5(192.168.1.5:3306).
Sun Sep 27 09:57:27 2015 - [warning] relay_log_purge=0 is not set on slave 192.168.1.5(192.168.1.5:3306).
Sun Sep 27 09:57:27 2015 - [info] Checking replication filtering settings..
Sun Sep 27 09:57:27 2015 - [info] binlog_do_db= , binlog_ignore_db=
Sun Sep 27 09:57:27 2015 - [info] Replication filtering check ok.
Sun Sep 27 09:57:27 2015 - [info] GTID (with auto-pos) is not supported
Sun Sep 27 09:57:27 2015 - [info] Starting SSH connection tests..
Sun Sep 27 09:57:28 2015 - [info] All SSH connection tests passed successfully.
Sun Sep 27 09:57:28 2015 - [info] Checking MHA Node version..
Sun Sep 27 09:57:28 2015 - [info] Version check ok.
Sun Sep 27 09:57:28 2015 - [info] Checking SSH publickey authentication settings on the current master..
Sun Sep 27 09:57:28 2015 - [info] HealthCheck: SSH to 192.168.1.4 is reachable.
Sun Sep 27 09:57:29 2015 - [info] Master MHA Node version is 0.56.
Sun Sep 27 09:57:29 2015 - [info] Checking recovery script configurations on 192.168.1.4(192.168.1.4:3306)..
Sun Sep 27 09:57:29 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/mysql/data --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.000012
Sun Sep 27 09:57:29 2015 - [info] Connecting to [email protected](192.168.1.4:22)..
Creating /tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /home/mysql/data, up to mysql-bin.000012
Sun Sep 27 09:57:29 2015 - [info] Binlog setting check done.
Sun Sep 27 09:57:29 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Sun Sep 27 09:57:29 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=192.168.1.5 --slave_ip=192.168.1.5 --slave_port=3306 --workdir=/tmp --target_version=5.6.25-73.0-log --manager_version=0.56 --relay_log_info=/home/mysql/data/relay-log.info --relay_dir=/home/mysql/data/ --slave_pass=xxx
Sun Sep 27 09:57:29 2015 - [info] Connecting to [email protected](192.168.1.5:22)..
Checking slave recovery environment settings..
Opening /home/mysql/data/relay-log.info ... ok.
Relay log found at /home/mysql/data, up to db4-relay-bin.000005
Temporary relay log file is /home/mysql/data/db4-relay-bin.000005
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Sun Sep 27 09:57:30 2015 - [info] Slaves settings check done.
Sun Sep 27 09:57:30 2015 - [info]
192.168.1.4(192.168.1.4:3306) (current master)
+--192.168.1.5(192.168.1.5:3306)
Sun Sep 27 09:57:30 2015 - [info] Checking master_ip_failover_script status:
Sun Sep 27 09:57:30 2015 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.4 --orig_master_ip=192.168.1.4 --orig_master_port=3306
IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===
Checking the Status of the script.. OK
Sun Sep 27 09:57:30 2015 - [info] OK.
Sun Sep 27 09:57:30 2015 - [warning] shutdown_script is not defined.
Sun Sep 27 09:57:30 2015 - [info] Set master ping interval 1 seconds.
Sun Sep 27 09:57:30 2015 - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s db1 -s db3 --user=root --master_host=db1 --master_ip=192.168.1.2 --master_port=3306
Sun Sep 27 09:57:30 2015 - [info] Starting ping health check on 192.168.1.4(192.168.1.4:3306)..
Sun Sep 27 09:57:30 2015 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
现在把主库停掉
tailf /usr/local/mha/manager.log
Sun Sep 27 10:12:45 2015 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
Sun Sep 27 10:12:45 2015 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/mysql/data --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --binlog_prefix=mysql-bin
Sun Sep 27 10:12:45 2015 - [info] Executing secondary network check script: /usr/local/bin/masterha_secondary_check -s db1 -s db3 --user=root --master_host=db1 --master_ip=192.168.1.2 --master_port=3306 --user=root --master_host=192.168.1.2 --master_ip=192.168.1.2 --master_port=3306 --master_user=root --master_password=123456 --ping_type=SELECT
Sun Sep 27 10:12:46 2015 - [info] HealthCheck: SSH to 192.168.1.2 is reachable.
Monitoring server db1 is reachable, Master is not reachable from db1. OK.
Sun Sep 27 10:12:46 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Sun Sep 27 10:12:46 2015 - [warning] Connection failed 2 time(s)..
Monitoring server db3 is reachable, Master is not reachable from db3. OK.
Sun Sep 27 10:12:46 2015 - [info] Master is not reachable from all other monitoring servers. Failover should start.
Sun Sep 27 10:12:47 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Sun Sep 27 10:12:47 2015 - [warning] Connection failed 3 time(s)..
Sun Sep 27 10:12:48 2015 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Sun Sep 27 10:12:48 2015 - [warning] Connection failed 4 time(s)..
Sun Sep 27 10:12:48 2015 - [warning] Master is not reachable from health checker!
Sun Sep 27 10:12:48 2015 - [warning] Master 192.168.1.2(192.168.1.2:3306) is not reachable!
Sun Sep 27 10:12:48 2015 - [warning] SSH is reachable.
Sun Sep 27 10:12:48 2015 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.cnf again, and trying to connect to all servers to check server status..
Sun Sep 27 10:12:48 2015 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun Sep 27 10:12:48 2015 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Sun Sep 27 10:12:48 2015 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Sun Sep 27 10:12:48 2015 - [info] GTID failover mode = 0
Sun Sep 27 10:12:48 2015 - [info] Dead Servers:
Sun Sep 27 10:12:48 2015 - [info] 192.168.1.2(192.168.1.2:3306)
Sun Sep 27 10:12:48 2015 - [info] Alive Servers:
Sun Sep 27 10:12:48 2015 - [info] 192.168.1.4(192.168.1.4:3306)
Sun Sep 27 10:12:48 2015 - [info] 192.168.1.5(192.168.1.5:3306)
Sun Sep 27 10:12:48 2015 - [info] Alive Slaves:
Sun Sep 27 10:12:48 2015 - [info] 192.168.1.4(192.168.1.4:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Sun Sep 27 10:12:48 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Sun Sep 27 10:12:48 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Sun Sep 27 10:12:48 2015 - [info] 192.168.1.5(192.168.1.5:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Sun Sep 27 10:12:48 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Sun Sep 27 10:12:48 2015 - [info] Checking slave configurations..
Sun Sep 27 10:12:48 2015 - [info] Checking replication filtering settings..
Sun Sep 27 10:12:48 2015 - [info] Replication filtering check ok.
Sun Sep 27 10:12:48 2015 - [info] Master is down!
Sun Sep 27 10:12:48 2015 - [info] Terminating monitoring script.
Sun Sep 27 10:12:48 2015 - [info] Got exit code 20 (Master dead).
Sun Sep 27 10:12:48 2015 - [info] MHA::MasterFailover version 0.56.
Sun Sep 27 10:12:48 2015 - [info] Starting master failover.
Sun Sep 27 10:12:48 2015 - [info]
Sun Sep 27 10:12:48 2015 - [info] * Phase 1: Configuration Check Phase..
Sun Sep 27 10:12:48 2015 - [info]
Sun Sep 27 10:12:49 2015 - [info] GTID failover mode = 0
Sun Sep 27 10:12:49 2015 - [info] Dead Servers:
Sun Sep 27 10:12:49 2015 - [info] 192.168.1.2(192.168.1.2:3306)
Sun Sep 27 10:12:49 2015 - [info] Checking master reachability via MySQL(double check)...
Sun Sep 27 10:12:49 2015 - [info] ok.
Sun Sep 27 10:12:49 2015 - [info] Alive Servers:
Sun Sep 27 10:12:49 2015 - [info] 192.168.1.4(192.168.1.4:3306)
Sun Sep 27 10:12:49 2015 - [info] 192.168.1.5(192.168.1.5:3306)
Sun Sep 27 10:12:49 2015 - [info] Alive Slaves:
Sun Sep 27 10:12:49 2015 - [info] 192.168.1.4(192.168.1.4:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Sun Sep 27 10:12:49 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Sun Sep 27 10:12:49 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Sun Sep 27 10:12:49 2015 - [info] 192.168.1.5(192.168.1.5:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Sun Sep 27 10:12:49 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Sun Sep 27 10:12:49 2015 - [info] Starting Non-GTID based failover.
Sun Sep 27 10:12:49 2015 - [info]
Sun Sep 27 10:12:49 2015 - [info] ** Phase 1: Configuration Check Phase completed.
Sun Sep 27 10:12:49 2015 - [info]
Sun Sep 27 10:12:49 2015 - [info] * Phase 2: Dead Master Shutdown Phase..
Sun Sep 27 10:12:49 2015 - [info]
Sun Sep 27 10:12:49 2015 - [info] Forcing shutdown so that applications never connect to the current master..
Sun Sep 27 10:12:49 2015 - [info] Executing master IP deactivation script:
Sun Sep 27 10:12:49 2015 - [info] /usr/local/bin/master_ip_failover --orig_master_host=192.168.1.2 --orig_master_ip=192.168.1.2 --orig_master_port=3306 --command=stopssh --ssh_user=root
IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===
Disabling the VIP on old master: 192.168.1.2
Sun Sep 27 10:12:50 2015 - [info] done.
Sun Sep 27 10:12:50 2015 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Sun Sep 27 10:12:50 2015 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Sun Sep 27 10:12:50 2015 - [info]
Sun Sep 27 10:12:50 2015 - [info] * Phase 3: Master Recovery Phase..
Sun Sep 27 10:12:50 2015 - [info]
Sun Sep 27 10:12:50 2015 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Sun Sep 27 10:12:50 2015 - [info]
Sun Sep 27 10:12:50 2015 - [info] The latest binary log file/position on all slaves is mysql-bin.000006:120
Sun Sep 27 10:12:50 2015 - [info] Latest slaves (Slaves that received relay log files to the latest):
Sun Sep 27 10:12:50 2015 - [info] 192.168.1.4(192.168.1.4:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Sun Sep 27 10:12:50 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Sun Sep 27 10:12:50 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Sun Sep 27 10:12:50 2015 - [info] 192.168.1.5(192.168.1.5:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Sun Sep 27 10:12:50 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Sun Sep 27 10:12:50 2015 - [info] The oldest binary log file/position on all slaves is mysql-bin.000006:120
Sun Sep 27 10:12:50 2015 - [info] Oldest slaves:
Sun Sep 27 10:12:50 2015 - [info] 192.168.1.4(192.168.1.4:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Sun Sep 27 10:12:50 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Sun Sep 27 10:12:50 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Sun Sep 27 10:12:50 2015 - [info] 192.168.1.5(192.168.1.5:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Sun Sep 27 10:12:50 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Sun Sep 27 10:12:50 2015 - [info]
Sun Sep 27 10:12:50 2015 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Sun Sep 27 10:12:50 2015 - [info]
Sun Sep 27 10:12:51 2015 - [info] Fetching dead master's binary logs..
Sun Sep 27 10:12:51 2015 - [info] Executing command on the dead master 192.168.1.2(192.168.1.2:3306): save_binary_logs --command=save --start_file=mysql-bin.000006 --start_pos=120 --binlog_dir=/home/mysql/data --output_file=/tmp/saved_master_binlog_from_192.168.1.2_3306_20150927101248.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56
Creating /tmp if not exists.. ok.
Concat binary/relay logs from mysql-bin.000006 pos 120 to mysql-bin.000006 EOF into /tmp/saved_master_binlog_from_192.168.1.2_3306_20150927101248.binlog ..
Binlog Checksum enabled
Dumping binlog format description event, from position 0 to 120.. ok.
Dumping effective binlog data from /home/mysql/data/mysql-bin.000006 position 120 to tail(143).. ok.
Binlog Checksum enabled
Concat succeeded.
Sun Sep 27 10:12:52 2015 - [info] scp from [email protected]:/tmp/saved_master_binlog_from_192.168.1.2_3306_20150927101248.binlog to local:/var/log/masterha/app1.log/saved_master_binlog_from_192.168.1.2_3306_20150927101248.binlog succeeded.
Sun Sep 27 10:12:52 2015 - [info] HealthCheck: SSH to 192.168.1.4 is reachable.
Sun Sep 27 10:12:53 2015 - [info] HealthCheck: SSH to 192.168.1.5 is reachable.
Sun Sep 27 10:12:53 2015 - [info]
Sun Sep 27 10:12:53 2015 - [info] * Phase 3.3: Determining New Master Phase..
Sun Sep 27 10:12:53 2015 - [info]
Sun Sep 27 10:12:53 2015 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Sun Sep 27 10:12:53 2015 - [info] All slaves received relay logs to the same position. No need to resync each other.
Sun Sep 27 10:12:53 2015 - [info] Searching new master from slaves..
Sun Sep 27 10:12:53 2015 - [info] Candidate masters from the configuration file:
Sun Sep 27 10:12:53 2015 - [info] 192.168.1.4(192.168.1.4:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Sun Sep 27 10:12:53 2015 - [info] Replicating from 192.168.1.2(192.168.1.2:3306)
Sun Sep 27 10:12:53 2015 - [info] Primary candidate for the new Master (candidate_master is set)
Sun Sep 27 10:12:53 2015 - [info] Non-candidate masters:
Sun Sep 27 10:12:53 2015 - [info] Searching from candidate_master slaves which have received the latest relay log events..
Sun Sep 27 10:12:53 2015 - [info] New master is 192.168.1.4(192.168.1.4:3306)
Sun Sep 27 10:12:53 2015 - [info] Starting master failover..
Sun Sep 27 10:12:53 2015 - [info]
From:
192.168.1.2(192.168.1.2:3306) (current master)
+--192.168.1.4(192.168.1.4:3306)
+--192.168.1.5(192.168.1.5:3306)
To:
192.168.1.4(192.168.1.4:3306) (new master)
+--192.168.1.5(192.168.1.5:3306)
Sun Sep 27 10:12:53 2015 - [info]
Sun Sep 27 10:12:53 2015 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
Sun Sep 27 10:12:53 2015 - [info]
Sun Sep 27 10:12:53 2015 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Sun Sep 27 10:12:53 2015 - [info] Sending binlog..
Sun Sep 27 10:12:53 2015 - [info] scp from local:/var/log/masterha/app1.log/saved_master_binlog_from_192.168.1.2_3306_20150927101248.binlog to [email protected]:/tmp/saved_master_binlog_from_192.168.1.2_3306_20150927101248.binlog succeeded.
Sun Sep 27 10:12:53 2015 - [info]
Sun Sep 27 10:12:53 2015 - [info] * Phase 3.4: Master Log Apply Phase..
Sun Sep 27 10:12:53 2015 - [info]
Sun Sep 27 10:12:53 2015 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Sun Sep 27 10:12:53 2015 - [info] Starting recovery on 192.168.1.4(192.168.1.4:3306)..
Sun Sep 27 10:12:53 2015 - [info] Generating diffs succeeded.
Sun Sep 27 10:12:53 2015 - [info] Waiting until all relay logs are applied.
Sun Sep 27 10:12:54 2015 - [info] done.
Sun Sep 27 10:12:54 2015 - [info] Getting slave status..
Sun Sep 27 10:12:54 2015 - [info] This slave(192.168.1.4)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000006:120). No need to recover from Exec_Master_Log_Pos.
Sun Sep 27 10:12:54 2015 - [info] Connecting to the target slave host 192.168.1.4, running recover script..
Sun Sep 27 10:12:54 2015 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='root' --slave_host=192.168.1.4 --slave_ip=192.168.1.4 --slave_port=3306 --apply_files=/tmp/saved_master_binlog_from_192.168.1.2_3306_20150927101248.binlog --workdir=/tmp --target_version=5.6.25-73.0-log --timestamp=20150927101248 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --slave_pass=xxx
Sun Sep 27 10:12:54 2015 - [info]
Applying differential binary/relay log files /tmp/saved_master_binlog_from_192.168.1.2_3306_20150927101248.binlog on 192.168.1.4:3306. This may take long time...
Applying log files succeeded.
Sun Sep 27 10:12:54 2015 - [info] All relay logs were successfully applied.
Sun Sep 27 10:12:54 2015 - [info] Getting new master's binlog name and position..
Sun Sep 27 10:12:54 2015 - [info] mysql-bin.000012:120
Sun Sep 27 10:12:54 2015 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.4', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000012', MASTER_LOG_POS=120, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Sun Sep 27 10:12:54 2015 - [info] Executing master IP activate script:
Sun Sep 27 10:12:54 2015 - [info] /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.1.2 --orig_master_ip=192.168.1.2 --orig_master_port=3306 --new_master_host=192.168.1.4 --new_master_ip=192.168.1.4 --new_master_port=3306 --new_master_user='root' --new_master_password='123456'
Unknown option: new_master_user
Unknown option: new_master_password
IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===
Enabling the VIP - 192.168.1.88 on the new master - 192.168.1.4
Sun Sep 27 10:12:55 2015 - [info] OK.
Sun Sep 27 10:12:55 2015 - [info] Setting read_only=0 on 192.168.1.4(192.168.1.4:3306)..
Sun Sep 27 10:12:55 2015 - [info] ok.
Sun Sep 27 10:12:55 2015 - [info] ** Finished master recovery successfully.
Sun Sep 27 10:12:55 2015 - [info] * Phase 3: Master Recovery Phase completed.
Sun Sep 27 10:12:55 2015 - [info]
Sun Sep 27 10:12:55 2015 - [info] * Phase 4: Slaves Recovery Phase..
Sun Sep 27 10:12:55 2015 - [info]
Sun Sep 27 10:12:55 2015 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Sun Sep 27 10:12:55 2015 - [info]
Sun Sep 27 10:12:55 2015 - [info] -- Slave diff file generation on host 192.168.1.5(192.168.1.5:3306) started, pid: 5206. Check tmp log /var/log/masterha/app1.log/192.168.1.5_3306_20150927101248.log if it takes time..
Sun Sep 27 10:12:55 2015 - [info]
Sun Sep 27 10:12:55 2015 - [info] Log messages from 192.168.1.5 ...
Sun Sep 27 10:12:55 2015 - [info]
Sun Sep 27 10:12:55 2015 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Sun Sep 27 10:12:55 2015 - [info] End of log messages from 192.168.1.5.
Sun Sep 27 10:12:55 2015 - [info] -- 192.168.1.5(192.168.1.5:3306) has the latest relay log events.
Sun Sep 27 10:12:55 2015 - [info] Generating relay diff files from the latest slave succeeded.
Sun Sep 27 10:12:55 2015 - [info]
Sun Sep 27 10:12:55 2015 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Sun Sep 27 10:12:55 2015 - [info]
Sun Sep 27 10:12:55 2015 - [info] -- Slave recovery on host 192.168.1.5(192.168.1.5:3306) started, pid: 5208. Check tmp log /var/log/masterha/app1.log/192.168.1.5_3306_20150927101248.log if it takes time..
Sun Sep 27 10:12:56 2015 - [info]
Sun Sep 27 10:12:56 2015 - [info] Log messages from 192.168.1.5 ...
Sun Sep 27 10:12:56 2015 - [info]
Sun Sep 27 10:12:55 2015 - [info] Sending binlog..
Sun Sep 27 10:12:55 2015 - [info] scp from local:/var/log/masterha/app1.log/saved_master_binlog_from_192.168.1.2_3306_20150927101248.binlog to [email protected]:/tmp/saved_master_binlog_from_192.168.1.2_3306_20150927101248.binlog succeeded.
Sun Sep 27 10:12:55 2015 - [info] Starting recovery on 192.168.1.5(192.168.1.5:3306)..
Sun Sep 27 10:12:55 2015 - [info] Generating diffs succeeded.
Sun Sep 27 10:12:55 2015 - [info] Waiting until all relay logs are applied.
Sun Sep 27 10:12:55 2015 - [info] done.
Sun Sep 27 10:12:55 2015 - [info] Getting slave status..
Sun Sep 27 10:12:55 2015 - [info] This slave(192.168.1.5)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000006:120). No need to recover from Exec_Master_Log_Pos.
Sun Sep 27 10:12:55 2015 - [info] Connecting to the target slave host 192.168.1.5, running recover script..
Sun Sep 27 10:12:55 2015 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='root' --slave_host=192.168.1.5 --slave_ip=192.168.1.5 --slave_port=3306 --apply_files=/tmp/saved_master_binlog_from_192.168.1.2_3306_20150927101248.binlog --workdir=/tmp --target_version=5.6.25-73.0-log --timestamp=20150927101248 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.56 --slave_pass=xxx
Sun Sep 27 10:12:56 2015 - [info]
Applying differential binary/relay log files /tmp/saved_master_binlog_from_192.168.1.2_3306_20150927101248.binlog on 192.168.1.5:3306. This may take long time...
Applying log files succeeded.
Sun Sep 27 10:12:56 2015 - [info] All relay logs were successfully applied.
Sun Sep 27 10:12:56 2015 - [info] Resetting slave 192.168.1.5(192.168.1.5:3306) and starting replication from the new master 192.168.1.4(192.168.1.4:3306)..
Sun Sep 27 10:12:56 2015 - [info] Executed CHANGE MASTER.
Sun Sep 27 10:12:56 2015 - [info] Slave started.
Sun Sep 27 10:12:56 2015 - [info] End of log messages from 192.168.1.5.
Sun Sep 27 10:12:56 2015 - [info] -- Slave recovery on host 192.168.1.5(192.168.1.5:3306) succeeded.
Sun Sep 27 10:12:56 2015 - [info] All new slave servers recovered successfully.
Sun Sep 27 10:12:56 2015 - [info]
Sun Sep 27 10:12:56 2015 - [info] * Phase 5: New master cleanup phase..
Sun Sep 27 10:12:56 2015 - [info]
Sun Sep 27 10:12:56 2015 - [info] Resetting slave info on the new master..
Sun Sep 27 10:12:56 2015 - [info] 192.168.1.4: Resetting slave info succeeded.
Sun Sep 27 10:12:56 2015 - [info] Master failover to 192.168.1.4(192.168.1.4:3306) completed successfully.
Sun Sep 27 10:12:56 2015 - [info] Deleted server1 entry from /etc/masterha/app1.cnf .
Sun Sep 27 10:12:56 2015 - [info]
----- Failover Report -----
app1: MySQL Master failover 192.168.1.2(192.168.1.2:3306) to 192.168.1.4(192.168.1.4:3306) succeeded
Master 192.168.1.2(192.168.1.2:3306) is down!
Check MHA Manager logs at db5:/var/log/masterha/app1/manager.log for details.
Started automated(non-interactive) failover.
Invalidated master IP address on 192.168.1.2(192.168.1.2:3306)
The latest slave 192.168.1.4(192.168.1.4:3306) has all relay logs for recovery.
Selected 192.168.1.4(192.168.1.4:3306) as a new master.
192.168.1.4(192.168.1.4:3306): OK: Applying all logs succeeded.
192.168.1.4(192.168.1.4:3306): OK: Activated master IP address.
192.168.1.5(192.168.1.5:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
192.168.1.5(192.168.1.5:3306): OK: Applying all logs succeeded. Slave started, replicating from 192.168.1.4(192.168.1.4:3306)
192.168.1.4(192.168.1.4:3306): Resetting slave info succeeded.
Master failover to 192.168.1.4(192.168.1.4:3306) completed successfully.
Sun Sep 27 10:12:56 2015 - [info] Sending mail..
Can't locate Mail/Sender.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/local/send_report line 24.
BEGIN failed--compilation aborted at /usr/local/send_report line 24.
Sun Sep 27 10:12:56 2015 - [error][/usr/local/share/perl5/MHA/MasterFailover.pm, ln2065] Failed to send mail with return code 2:0
我们可以看到主库和备主库已经发生了切换，从库的主库指向了备主库，切换完成后，MHA-manager进程退出（MHA的一个bug）
备主库获得了浮动ip
[root@db3 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:94:fb:1a brd ff:ff:ff:ff:ff:ff
inet 192.168.1.4/24 brd 192.168.1.255 scope global eth1
inet 192.168.1.88/32 scope global eth1
inet6 fe80::20c:29ff:fe94:fb1a/64 scope link
valid_lft forever preferred_lft forever
这个时候再次启动MHA-manager
tailf /usr/local/mha/manager.log
Sun Sep 27 10:17:29 2015 - [info] MHA::MasterMonitor version 0.56.
Sun Sep 27 10:17:29 2015 - [info] GTID failover mode = 0
Sun Sep 27 10:17:29 2015 - [info] Dead Servers:
Sun Sep 27 10:17:29 2015 - [info] Alive Servers:
Sun Sep 27 10:17:29 2015 - [info] 192.168.1.4(192.168.1.4:3306)
Sun Sep 27 10:17:29 2015 - [info] 192.168.1.5(192.168.1.5:3306)
Sun Sep 27 10:17:29 2015 - [info] Alive Slaves:
Sun Sep 27 10:17:29 2015 - [info] 192.168.1.5(192.168.1.5:3306) Version=5.6.25-73.0-log (oldest major version between slaves) log-bin:enabled
Sun Sep 27 10:17:29 2015 - [info] Replicating from 192.168.1.4(192.168.1.4:3306)
Sun Sep 27 10:17:29 2015 - [info] Current Alive Master: 192.168.1.4(192.168.1.4:3306)
Sun Sep 27 10:17:29 2015 - [info] Checking slave configurations..
Sun Sep 27 10:17:29 2015 - [info] Checking replication filtering settings..
Sun Sep 27 10:17:29 2015 - [info] binlog_do_db= , binlog_ignore_db=
Sun Sep 27 10:17:29 2015 - [info] Replication filtering check ok.
Sun Sep 27 10:17:29 2015 - [info] GTID (with auto-pos) is not supported
Sun Sep 27 10:17:29 2015 - [info] Starting SSH connection tests..
Sun Sep 27 10:17:30 2015 - [info] All SSH connection tests passed successfully.
Sun Sep 27 10:17:30 2015 - [info] Checking MHA Node version..
Sun Sep 27 10:17:30 2015 - [info] Version check ok.
Sun Sep 27 10:17:30 2015 - [info] Checking SSH publickey authentication settings on the current master..
Sun Sep 27 10:17:31 2015 - [info] HealthCheck: SSH to 192.168.1.4 is reachable.
Sun Sep 27 10:17:31 2015 - [info] Master MHA Node version is 0.56.
Sun Sep 27 10:17:31 2015 - [info] Checking recovery script configurations on 192.168.1.4(192.168.1.4:3306)..
Sun Sep 27 10:17:31 2015 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/mysql/data --output_file=/tmp/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.000012
Sun Sep 27 10:17:31 2015 - [info] Connecting to [email protected](192.168.1.4:22)..
Creating /tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /home/mysql/data, up to mysql-bin.000012
Sun Sep 27 10:17:31 2015 - [info] Binlog setting check done.
Sun Sep 27 10:17:31 2015 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Sun Sep 27 10:17:31 2015 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=192.168.1.5 --slave_ip=192.168.1.5 --slave_port=3306 --workdir=/tmp --target_version=5.6.25-73.0-log --manager_version=0.56 --relay_log_info=/home/mysql/data/relay-log.info --relay_dir=/home/mysql/data/ --slave_pass=xxx
Sun Sep 27 10:17:31 2015 - [info] Connecting to [email protected](192.168.1.5:22)..
Checking slave recovery environment settings..
Opening /home/mysql/data/relay-log.info ... ok.
Relay log found at /home/mysql/data, up to db4-relay-bin.000002
Temporary relay log file is /home/mysql/data/db4-relay-bin.000002
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Sun Sep 27 10:17:32 2015 - [info] Slaves settings check done.
Sun Sep 27 10:17:32 2015 - [info]
192.168.1.4(192.168.1.4:3306) (current master)
+--192.168.1.5(192.168.1.5:3306)
Sun Sep 27 10:17:32 2015 - [info] Checking master_ip_failover_script status:
Sun Sep 27 10:17:32 2015 - [info] /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.4 --orig_master_ip=192.168.1.4 --orig_master_port=3306
IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===
Checking the Status of the script.. OK
Sun Sep 27 10:17:32 2015 - [info] OK.
Sun Sep 27 10:17:32 2015 - [warning] shutdown_script is not defined.
Sun Sep 27 10:17:32 2015 - [info] Set master ping interval 1 seconds.
Sun Sep 27 10:17:32 2015 - [info] Set secondary check script: /usr/local/bin/masterha_secondary_check -s db1 -s db3 --user=root --master_host=db1 --master_ip=192.168.1.2 --master_port=3306
Sun Sep 27 10:17:32 2015 - [info] Starting ping health check on 192.168.1.4(192.168.1.4:3306)..
Sun Sep 27 10:17:32 2015 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
我们可以看到192.168.1.2已经被踢出mysql集群
此时的配置文件变为
[server default]
manager_log=/var/log/masterha/app1/manager.log
manager_workdir=/var/log/masterha/app1.log
master_binlog_dir=/home/mysql/data
master_ip_failover_script=/usr/local/bin/master_ip_failover
master_ip_online_change_script=/usr/local/bin/master_ip_online_change
password=123456
ping_interval=1
remote_workdir=/tmp
repl_password=123456
repl_user=repl
report_script=/usr/local/send_report
secondary_check_script=/usr/local/bin/masterha_secondary_check -s db1 -s db3 --user=root --master_host=db1 --master_ip=192.168.1.2 --master_port=3306
shutdown_script=""
ssh_user=root
user=root
[server2]
candidate_master=1
check_repl_delay=0
hostname=192.168.1.4
port=3306
[server3]
hostname=192.168.1.5
port=3306
测试完成，每次切换MHA-manager都会把配置文件关于某个主库的那一段配置文件移除，这种一主二从的架构可以进行两次主从测试
注意：在切换完成后，MHA-manager的进程还是会抛出报错，然后退出，这是5.6的一个bug，可以不管，我们主要确定主从切换正常即可