使用MHA发生切换后，将原主库加入集群并切换回最初状态
环境：
M：192.168.111.20
S1: 192.168.111.30 (从1，备主)
S2：192.168.111.10 (从2）
VIP: 192.168.111.80（这个VIP手工设置在MASTER上）
master配置文件：
[mysql@MYSQL-SVR1 var]$ vi /etc/masterha/app1.cnf
[server default]
manager_workdir=/masterha/app1
manager_log=/masterha/app1/manager.log
user=root
password=123
ssh_user=root
repl_user=repl
repl_password=repl
ping_interval=1
shutdown_script=""
master_ip_failover_script="/masterha/app1/script/master_ip_failover"
master_ip_online_change_script="/masterha/app1/script/master_ip_online_change"
#master_ip_online_change_script=""
#report_script=""
[server1]
hostname=192.168.111.20
master_binlog_dir="/home/mysql/mysql_5621/var"
port=5621
candidate_master=1
[server2]
hostname=192.168.111.30
master_binlog_dir="/home/mysql/mysql_5621/var"
port=5621
candidate_master=1
[server3]
hostname=192.168.111.10
master_binlog_dir="/home/mysql/mysql_5621/var"
port=5621
no_master=1
启动master:
[root@MYSQL-SVR1 ~]# nohup masterha_manager --conf=/etc/masterha/app1.cnf --ignore_fail_on_start &> /dev/null &
[1] 5358
启动日志：
IN SCRIPT====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 192.168.111.80/24===
Checking the Status of the script.. OK
ssh: cluster1: Temporary failure in name resolution
Mon May 16 23:02:19 2016 - [info] OK.
Mon May 16 23:02:19 2016 - [warning] shutdown_script is not defined.
Mon May 16 23:02:19 2016 - [info] Set master ping interval 1 seconds.
Mon May 16 23:02:19 2016 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Mon May 16 23:02:19 2016 - [info] Starting ping health check on 192.168.111.20(192.168.111.20:5621)..
Mon May 16 23:02:19 2016 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
Mon May 16 23:02:35 2016 - [info] Got terminate signal. Exit.
Mon May 16 23:10:54 2016 - [info] MHA::MasterMonitor version 0.55.
Mon May 16 23:10:55 2016 - [info] Dead Servers:
Mon May 16 23:10:55 2016 - [info] Alive Servers:
Mon May 16 23:10:55 2016 - [info] 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:10:55 2016 - [info] 192.168.111.30(192.168.111.30:5621)
Mon May 16 23:10:55 2016 - [info] 192.168.111.10(192.168.111.10:5621)
Mon May 16 23:10:55 2016 - [info] Alive Slaves:
Mon May 16 23:10:55 2016 - [info] 192.168.111.30(192.168.111.30:5621) Version=5.6.21-log (oldest major version between slaves) log-bin:enabled
Mon May 16 23:10:55 2016 - [info] Replicating from 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:10:55 2016 - [info] Primary candidate for the new Master (candidate_master is set)
Mon May 16 23:10:55 2016 - [info] 192.168.111.10(192.168.111.10:5621) Version=5.6.21-log (oldest major version between slaves) log-bin:enabled
Mon May 16 23:10:55 2016 - [info] Replicating from 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:10:55 2016 - [info] Not candidate for the new Master (no_master is set)
Mon May 16 23:10:55 2016 - [info] Current Alive Master: 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:10:55 2016 - [info] Checking slave configurations..
Mon May 16 23:10:55 2016 - [warning] relay_log_purge=0 is not set on slave 192.168.111.30(192.168.111.30:5621).
Mon May 16 23:10:55 2016 - [warning] relay_log_purge=0 is not set on slave 192.168.111.10(192.168.111.10:5621).
Mon May 16 23:10:55 2016 - [info] Checking replication filtering settings..
Mon May 16 23:10:55 2016 - [info] binlog_do_db= , binlog_ignore_db=
Mon May 16 23:10:55 2016 - [info] Replication filtering check ok.
Mon May 16 23:10:55 2016 - [info] Starting SSH connection tests..
Mon May 16 23:10:58 2016 - [info] All SSH connection tests passed successfully.
Mon May 16 23:10:58 2016 - [info] Checking MHA Node version..
Mon May 16 23:10:59 2016 - [info] Version check ok.
Mon May 16 23:10:59 2016 - [info] Checking SSH publickey authentication settings on the current master..
Mon May 16 23:11:00 2016 - [info] HealthCheck: SSH to 192.168.111.20 is reachable.
Mon May 16 23:11:00 2016 - [info] Master MHA Node version is 0.54.
Mon May 16 23:11:00 2016 - [info] Checking recovery script configurations on the current master..
Mon May 16 23:11:00 2016 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/mysql/mysql_5621/var --output_file=/var/tmp/save_binary_logs_test --manager_version=0.55 --start_file=mysql-bin.000031
Mon May 16 23:11:00 2016 - [info] Connecting to root@192.168.111.20(192.168.111.20)..
Creating /var/tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /home/mysql/mysql_5621/var, up to mysql-bin.000031
Mon May 16 23:11:00 2016 - [info] Master setting check done.
Mon May 16 23:11:00 2016 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Mon May 16 23:11:00 2016 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=192.168.111.30 --slave_ip=192.168.111.30 --slave_port=5621 --workdir=/var/tmp --target_version=5.6.21-log --manager_version=0.55 --relay_log_info=/home/mysql/mysql_5621/var/relay-log.info --relay_dir=/home/mysql/mysql_5621/var/ --slave_pass=xxx
Mon May 16 23:11:00 2016 - [info] Connecting to root@192.168.111.30(192.168.111.30:22)..
Checking slave recovery environment settings..
Opening /home/mysql/mysql_5621/var/relay-log.info ... ok.
Relay log found at /home/mysql/mysql_5621/var, up to relay-log.000015
Temporary relay log file is /home/mysql/mysql_5621/var/relay-log.000015
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Mon May 16 23:11:00 2016 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='root' --slave_host=192.168.111.10 --slave_ip=192.168.111.10 --slave_port=5621 --workdir=/var/tmp --target_version=5.6.21-log --manager_version=0.55 --relay_log_info=/home/mysql/mysql_5621/var/relay-log.info --relay_dir=/home/mysql/mysql_5621/var/ --slave_pass=xxx
Mon May 16 23:11:00 2016 - [info] Connecting to root@192.168.111.10(192.168.111.10:22)..
Checking slave recovery environment settings..
Opening /home/mysql/mysql_5621/var/relay-log.info ... ok.
Relay log found at /home/mysql/mysql_5621/var, up to relay-log.000002
Temporary relay log file is /home/mysql/mysql_5621/var/relay-log.000002
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Mon May 16 23:11:01 2016 - [info] Slaves settings check done.
Mon May 16 23:11:01 2016 - [info]
192.168.111.20 (current master)
+--192.168.111.30
+--192.168.111.10
Mon May 16 23:11:01 2016 - [info] Checking master_ip_failover_script status:
Mon May 16 23:11:01 2016 - [info] /masterha/app1/script/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.111.20 --orig_master_ip=192.168.111.20 --orig_master_port=5621
IN SCRIPT====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 192.168.111.80/24===
Checking the Status of the script.. OK
ssh: cluster1: Temporary failure in name resolution
Mon May 16 23:11:01 2016 - [info] OK.
Mon May 16 23:11:01 2016 - [warning] shutdown_script is not defined.
Mon May 16 23:11:01 2016 - [info] Set master ping interval 1 seconds.
Mon May 16 23:11:01 2016 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Mon May 16 23:11:01 2016 - [info] Starting ping health check on 192.168.111.20(192.168.111.20:5621)..
Mon May 16 23:11:01 2016 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
从启动日志可以看出当前主从环境信息，以及VIP信息。
使用sysbench产生流量，这里直接连接VIP测试：
[mysql@MYSQL-SVR1 bin]$ ./sysbench --test=oltp --mysql-user=root --mysql-password=123 --mysql-host=192.168.111.80 --mysql-port=5621 --mysql-table-engine=innodb --mysql-db=db1 --oltp-table-name=t4 --oltp-table-size=1000000 --oltp-auto-inc=on --oltp-read-only=off --oltp-skip-trx=off --oltp-test-mode=complex --num-threads=1 --max-requests=1000000 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 1
Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 1000000
Threads started!
登陆主从查看是否有数据写入：
查看主库是否有数据写入：
mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000031 | 2566521 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.01 sec)
mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000031 | 2629549 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000031 | 2708130 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000031 | 10813170 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
可以看到position一致在增大，说明有数据写入
查看从库是否有数据写入：
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.111.20
Master_User: repl
Master_Port: 5621
Connect_Retry: 10
Master_Log_File: mysql-bin.000031
Read_Master_Log_Pos: 4912141
Relay_Log_File: relay-log.000015
Relay_Log_Pos: 4909031
Relay_Master_Log_File: mysql-bin.000031
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table: mysql.%,test.%
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 4908868
Relay_Log_Space: 4912634
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 200
Master_UUID: 0ee37546-3e46-11e5-a399-000c297ada8c
Master_Info_File: /home/mysql/mysql_5621/var/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Reading event from the relay log
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
1 row in set (0.00 sec)
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.111.20
Master_User: repl
Master_Port: 5621
Connect_Retry: 10
Master_Log_File: mysql-bin.000031
Read_Master_Log_Pos: 10562647
Relay_Log_File: relay-log.000015
Relay_Log_Pos: 10562810
Relay_Master_Log_File: mysql-bin.000031
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table: mysql.%,test.%
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 10562647
Relay_Log_Space: 10563140
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 200
Master_UUID: 0ee37546-3e46-11e5-a399-000c297ada8c
Master_Info_File: /home/mysql/mysql_5621/var/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
1 row in set (0.00 sec)
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.111.20
Master_User: repl
Master_Port: 5621
Connect_Retry: 10
Master_Log_File: mysql-bin.000031
Read_Master_Log_Pos: 10566736
Relay_Log_File: relay-log.000015
Relay_Log_Pos: 10566899
Relay_Master_Log_File: mysql-bin.000031
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table: mysql.%,test.%
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 10566736
Relay_Log_Space: 10567229
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 200
Master_UUID: 0ee37546-3e46-11e5-a399-000c297ada8c
Master_Info_File: /home/mysql/mysql_5621/var/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Reading event from the relay log
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
1 row in set (0.09 sec)
可以看到
Exec_Master_Log_Pos一直在增大，说明数据已经同步过来了。
停掉主库的mysql进程。
[mysql@MYSQL-SVR2 mysql_5621]$ ./bin/mysql.server stop
Shutting down MySQL............
观察切换日志:
Mon May 16 23:12:08 2016 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
Mon May 16 23:12:08 2016 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/home/mysql/mysql_5621/var --output_file=/var/tmp/save_binary_logs_test --manager_version=0.55 --binlog_prefix=mysql-bin
Mon May 16 23:12:09 2016 - [info] HealthCheck: SSH to 192.168.111.20 is reachable.
Mon May 16 23:12:09 2016 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.111.20' (111))
Mon May 16 23:12:09 2016 - [warning] Connection failed 1 time(s)..
Mon May 16 23:12:10 2016 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.111.20' (111))
Mon May 16 23:12:10 2016 - [warning] Connection failed 2 time(s)..
Mon May 16 23:12:11 2016 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.111.20' (111))
Mon May 16 23:12:11 2016 - [warning] Connection failed 3 time(s)..
Mon May 16 23:12:11 2016 - [warning] Master is not reachable from health checker!
Mon May 16 23:12:11 2016 - [warning] Master 192.168.111.20(192.168.111.20:5621) is not reachable!
Mon May 16 23:12:11 2016 - [warning] SSH is reachable.
Mon May 16 23:12:11 2016 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/app1.cnf again, and trying to connect to all servers to check server status..
Mon May 16 23:12:11 2016 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Mon May 16 23:12:11 2016 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Mon May 16 23:12:11 2016 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Mon May 16 23:12:11 2016 - [info] Dead Servers:
Mon May 16 23:12:11 2016 - [info] 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:12:11 2016 - [info] Alive Servers:
Mon May 16 23:12:11 2016 - [info] 192.168.111.30(192.168.111.30:5621)
Mon May 16 23:12:11 2016 - [info] 192.168.111.10(192.168.111.10:5621)
Mon May 16 23:12:11 2016 - [info] Alive Slaves:
Mon May 16 23:12:11 2016 - [info] 192.168.111.30(192.168.111.30:5621) Version=5.6.21-log (oldest major version between slaves) log-bin:enabled
Mon May 16 23:12:11 2016 - [info] Replicating from 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:12:11 2016 - [info] Primary candidate for the new Master (candidate_master is set)
Mon May 16 23:12:11 2016 - [info] 192.168.111.10(192.168.111.10:5621) Version=5.6.21-log (oldest major version between slaves) log-bin:enabled
Mon May 16 23:12:11 2016 - [info] Replicating from 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:12:11 2016 - [info] Not candidate for the new Master (no_master is set)
Mon May 16 23:12:11 2016 - [info] Checking slave configurations..
Mon May 16 23:12:11 2016 - [warning] relay_log_purge=0 is not set on slave 192.168.111.30(192.168.111.30:5621).
Mon May 16 23:12:11 2016 - [warning] relay_log_purge=0 is not set on slave 192.168.111.10(192.168.111.10:5621).
Mon May 16 23:12:11 2016 - [info] Checking replication filtering settings..
Mon May 16 23:12:11 2016 - [info] Replication filtering check ok.
Mon May 16 23:12:11 2016 - [info] Master is down!
Mon May 16 23:12:11 2016 - [info] Terminating monitoring script.
Mon May 16 23:12:11 2016 - [info] Got exit code 20 (Master dead).
Mon May 16 23:12:11 2016 - [info] MHA::MasterFailover version 0.55.
Mon May 16 23:12:11 2016 - [info] Starting master failover.
Mon May 16 23:12:11 2016 - [info]
Mon May 16 23:12:11 2016 - [info] * Phase 1: Configuration Check Phase..
Mon May 16 23:12:11 2016 - [info]
Mon May 16 23:12:12 2016 - [info] Dead Servers:
Mon May 16 23:12:12 2016 - [info] 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:12:12 2016 - [info] Checking master reachability via mysql(double check)..
Mon May 16 23:12:12 2016 - [info] ok.
Mon May 16 23:12:12 2016 - [info] Alive Servers:
Mon May 16 23:12:12 2016 - [info] 192.168.111.30(192.168.111.30:5621)
Mon May 16 23:12:12 2016 - [info] 192.168.111.10(192.168.111.10:5621)
Mon May 16 23:12:12 2016 - [info] Alive Slaves:
Mon May 16 23:12:12 2016 - [info] 192.168.111.30(192.168.111.30:5621) Version=5.6.21-log (oldest major version between slaves) log-bin:enabled
Mon May 16 23:12:12 2016 - [info] Replicating from 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:12:12 2016 - [info] Primary candidate for the new Master (candidate_master is set)
Mon May 16 23:12:12 2016 - [info] 192.168.111.10(192.168.111.10:5621) Version=5.6.21-log (oldest major version between slaves) log-bin:enabled
Mon May 16 23:12:12 2016 - [info] Replicating from 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:12:12 2016 - [info] Not candidate for the new Master (no_master is set)
Mon May 16 23:12:12 2016 - [info] ** Phase 1: Configuration Check Phase completed.
Mon May 16 23:12:12 2016 - [info]
Mon May 16 23:12:12 2016 - [info] * Phase 2: Dead Master Shutdown Phase..
Mon May 16 23:12:12 2016 - [info]
Mon May 16 23:12:12 2016 - [info] Forcing shutdown so that applications never connect to the current master..
Mon May 16 23:12:12 2016 - [info] Executing master IP deactivatation script:
Mon May 16 23:12:12 2016 - [info] /masterha/app1/script/master_ip_failover --orig_master_host=192.168.111.20 --orig_master_ip=192.168.111.20 --orig_master_port=5621 --command=stopssh --ssh_user=root
IN SCRIPT====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 192.168.111.80/24===
Disabling the VIP on old master: 192.168.111.20
Mon May 16 23:12:12 2016 - [info] done.
Mon May 16 23:12:12 2016 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Mon May 16 23:12:12 2016 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Mon May 16 23:12:12 2016 - [info]
Mon May 16 23:12:12 2016 - [info] * Phase 3: Master Recovery Phase..
Mon May 16 23:12:12 2016 - [info]
Mon May 16 23:12:12 2016 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Mon May 16 23:12:12 2016 - [info]
Mon May 16 23:12:12 2016 - [info] The latest binary log file/position on all slaves is mysql-bin.000031:11751371
Mon May 16 23:12:12 2016 - [info] Latest slaves (Slaves that received relay log files to the latest):
Mon May 16 23:12:12 2016 - [info] 192.168.111.30(192.168.111.30:5621) Version=5.6.21-log (oldest major version between slaves) log-bin:enabled
Mon May 16 23:12:12 2016 - [info] Replicating from 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:12:12 2016 - [info] Primary candidate for the new Master (candidate_master is set)
Mon May 16 23:12:12 2016 - [info] 192.168.111.10(192.168.111.10:5621) Version=5.6.21-log (oldest major version between slaves) log-bin:enabled
Mon May 16 23:12:12 2016 - [info] Replicating from 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:12:12 2016 - [info] Not candidate for the new Master (no_master is set)
Mon May 16 23:12:12 2016 - [info] The oldest binary log file/position on all slaves is mysql-bin.000031:11751371
Mon May 16 23:12:12 2016 - [info] Oldest slaves:
Mon May 16 23:12:12 2016 - [info] 192.168.111.30(192.168.111.30:5621) Version=5.6.21-log (oldest major version between slaves) log-bin:enabled
Mon May 16 23:12:12 2016 - [info] Replicating from 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:12:12 2016 - [info] Primary candidate for the new Master (candidate_master is set)
Mon May 16 23:12:12 2016 - [info] 192.168.111.10(192.168.111.10:5621) Version=5.6.21-log (oldest major version between slaves) log-bin:enabled
Mon May 16 23:12:12 2016 - [info] Replicating from 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:12:12 2016 - [info] Not candidate for the new Master (no_master is set)
Mon May 16 23:12:12 2016 - [info]
Mon May 16 23:12:12 2016 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Mon May 16 23:12:12 2016 - [info]
Mon May 16 23:12:12 2016 - [info] Fetching dead master's binary logs..
Mon May 16 23:12:12 2016 - [info] Executing command on the dead master 192.168.111.20(192.168.111.20:5621): save_binary_logs --command=save --start_file=mysql-bin.000031 --start_pos=11751371 --binlog_dir=/home/mysql/mysql_5621/var --output_file=/var/tmp/saved_master_binlog_from_192.168.111.20_5621_20160516231211.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.55
Creating /var/tmp if not exists.. ok.
Concat binary/relay logs from mysql-bin.000031 pos 11751371 to mysql-bin.000031 EOF into /var/tmp/saved_master_binlog_from_192.168.111.20_5621_20160516231211.binlog ..
Dumping binlog format description event, from position 0 to 120.. ok.
No need to dump effective binlog data from /home/mysql/mysql_5621/var/mysql-bin.000031 (pos starts 11751371, filesize 11751371). Skipping.
/var/tmp/saved_master_binlog_from_192.168.111.20_5621_20160516231211.binlog has no effective data events.
Event not exists.
Mon May 16 23:12:12 2016 - [info] Additional events were not found from the orig master. No need to save.
Mon May 16 23:12:12 2016 - [info]
Mon May 16 23:12:12 2016 - [info] * Phase 3.3: Determining New Master Phase..
Mon May 16 23:12:12 2016 - [info]
Mon May 16 23:12:12 2016 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Mon May 16 23:12:12 2016 - [info] All slaves received relay logs to the same position. No need to resync each other.
Mon May 16 23:12:12 2016 - [info] Searching new master from slaves..
Mon May 16 23:12:12 2016 - [info] Candidate masters from the configuration file:
Mon May 16 23:12:12 2016 - [info] 192.168.111.30(192.168.111.30:5621) Version=5.6.21-log (oldest major version between slaves) log-bin:enabled
Mon May 16 23:12:12 2016 - [info] Replicating from 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:12:12 2016 - [info] Primary candidate for the new Master (candidate_master is set)
Mon May 16 23:12:12 2016 - [info] Non-candidate masters:
Mon May 16 23:12:12 2016 - [info] 192.168.111.10(192.168.111.10:5621) Version=5.6.21-log (oldest major version between slaves) log-bin:enabled
Mon May 16 23:12:12 2016 - [info] Replicating from 192.168.111.20(192.168.111.20:5621)
Mon May 16 23:12:12 2016 - [info] Not candidate for the new Master (no_master is set)
Mon May 16 23:12:12 2016 - [info] Searching from candidate_master slaves which have received the latest relay log events..
Mon May 16 23:12:12 2016 - [info] New master is 192.168.111.30(192.168.111.30:5621)
Mon May 16 23:12:12 2016 - [info] Starting master failover..
Mon May 16 23:12:12 2016 - [info]
From:
192.168.111.20 (current master)
+--192.168.111.30
+--192.168.111.10
To:
192.168.111.30 (new master)
+--192.168.111.10
Mon May 16 23:12:12 2016 - [info]
Mon May 16 23:12:12 2016 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
Mon May 16 23:12:12 2016 - [info]
Mon May 16 23:12:12 2016 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Mon May 16 23:12:12 2016 - [info]
Mon May 16 23:12:12 2016 - [info] * Phase 3.4: Master Log Apply Phase..
Mon May 16 23:12:12 2016 - [info]
Mon May 16 23:12:12 2016 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Mon May 16 23:12:12 2016 - [info] Starting recovery on 192.168.111.30(192.168.111.30:5621)..
Mon May 16 23:12:12 2016 - [info] This server has all relay logs. Waiting all logs to be applied..
Mon May 16 23:12:12 2016 - [info] done.
Mon May 16 23:12:12 2016 - [info] All relay logs were successfully applied.
Mon May 16 23:12:12 2016 - [info] Getting new master's binlog name and position..
Mon May 16 23:12:12 2016 - [info] mysql-bin.000025:367
Mon May 16 23:12:12 2016 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.111.30', MASTER_PORT=5621, MASTER_LOG_FILE='mysql-bin.000025', MASTER_LOG_POS=367, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Mon May 16 23:12:12 2016 - [info] Executing master IP activate script:
Mon May 16 23:12:12 2016 - [info] /masterha/app1/script/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.111.20 --orig_master_ip=192.168.111.20 --orig_master_port=5621 --new_master_host=192.168.111.30 --new_master_ip=192.168.111.30 --new_master_port=5621 --new_master_user='root' --new_master_password='123'
Unknown option: new_master_user
Unknown option: new_master_password
IN SCRIPT====/sbin/ifconfig eth0:1 down==/sbin/ifconfig eth0:1 192.168.111.80/24===
Enabling the VIP - 192.168.111.80/24 on the new master - 192.168.111.30
Mon May 16 23:12:13 2016 - [info] OK.
Mon May 16 23:12:13 2016 - [info] Setting read_only=0 on 192.168.111.30(192.168.111.30:5621)..
Mon May 16 23:12:13 2016 - [info] ok.
Mon May 16 23:12:13 2016 - [info] ** Finished master recovery successfully.
Mon May 16 23:12:13 2016 - [info] * Phase 3: Master Recovery Phase completed.
Mon May 16 23:12:13 2016 - [info]
Mon May 16 23:12:13 2016 - [info] * Phase 4: Slaves Recovery Phase..
Mon May 16 23:12:13 2016 - [info]
Mon May 16 23:12:13 2016 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Mon May 16 23:12:13 2016 - [info]
Mon May 16 23:12:13 2016 - [info] -- Slave diff file generation on host 192.168.111.10(192.168.111.10:5621) started, pid: 5711. Check tmp log /masterha/app1/192.168.111.10_5621_20160516231211.log if it takes time..
Mon May 16 23:12:13 2016 - [info]
Mon May 16 23:12:13 2016 - [info] Log messages from 192.168.111.10 ...
Mon May 16 23:12:13 2016 - [info]
Mon May 16 23:12:13 2016 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Mon May 16 23:12:13 2016 - [info] End of log messages from 192.168.111.10.
Mon May 16 23:12:13 2016 - [info] -- 192.168.111.10(192.168.111.10:5621) has the latest relay log events.
Mon May 16 23:12:13 2016 - [info] Generating relay diff files from the latest slave succeeded.
Mon May 16 23:12:13 2016 - [info]
Mon May 16 23:12:13 2016 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Mon May 16 23:12:13 2016 - [info]
Mon May 16 23:12:13 2016 - [info] -- Slave recovery on host 192.168.111.10(192.168.111.10:5621) started, pid: 5713. Check tmp log /masterha/app1/192.168.111.10_5621_20160516231211.log if it takes time..
Mon May 16 23:12:13 2016 - [info]
Mon May 16 23:12:13 2016 - [info] Log messages from 192.168.111.10 ...
Mon May 16 23:12:13 2016 - [info]
Mon May 16 23:12:13 2016 - [info] Starting recovery on 192.168.111.10(192.168.111.10:5621)..
Mon May 16 23:12:13 2016 - [info] This server has all relay logs. Waiting all logs to be applied..
Mon May 16 23:12:13 2016 - [info] done.
Mon May 16 23:12:13 2016 - [info] All relay logs were successfully applied.
Mon May 16 23:12:13 2016 - [info] Resetting slave 192.168.111.10(192.168.111.10:5621) and starting replication from the new master 192.168.111.30(192.168.111.30:5621)..
Mon May 16 23:12:13 2016 - [info] Executed CHANGE MASTER.
Mon May 16 23:12:13 2016 - [info] Slave started.
Mon May 16 23:12:13 2016 - [info] End of log messages from 192.168.111.10.
Mon May 16 23:12:13 2016 - [info] -- Slave recovery on host 192.168.111.10(192.168.111.10:5621) succeeded.
Mon May 16 23:12:13 2016 - [info] All new slave servers recovered successfully.
Mon May 16 23:12:13 2016 - [info]
Mon May 16 23:12:13 2016 - [info] * Phase 5: New master cleanup phase..
Mon May 16 23:12:13 2016 - [info]
Mon May 16 23:12:13 2016 - [info] Resetting slave info on the new master..
Mon May 16 23:12:13 2016 - [info] 192.168.111.30: Resetting slave info succeeded.
Mon May 16 23:12:13 2016 - [info] Master failover to 192.168.111.30(192.168.111.30:5621) completed successfully.
Mon May 16 23:12:13 2016 - [info]
----- Failover Report -----
app1: MySQL Master failover 192.168.111.20 to 192.168.111.30 succeeded
Master 192.168.111.20 is down!
Check MHA Manager logs at MYSQL-SVR1:/masterha/app1/manager.log for details.
Started automated(non-interactive) failover.
Invalidated master IP address on 192.168.111.20.
The latest slave 192.168.111.30(192.168.111.30:5621) has all relay logs for recovery.
Selected 192.168.111.30 as a new master.
192.168.111.30: OK: Applying all logs succeeded.
192.168.111.30: OK: Activated master IP address.
192.168.111.10: This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
192.168.111.10: OK: Applying all logs succeeded. Slave started, replicating from 192.168.111.30.
192.168.111.30: Resetting slave info succeeded.
Master failover to 192.168.111.30(192.168.111.30:5621) completed successfully.
从切换日志可以看出如何发生切换：
From:
192.168.111.20 (current master)
+--192.168.111.30
+--192.168.111.10
To:
192.168.111.30 (new master)
+--192.168.111.10
以及切换后新主库的binlog以及position，这里很重要，待会原主库重新加入集群需要用到
Mon May 16 23:12:12 2016 - [info] * Phase 3.4: Master Log Apply Phase..
Mon May 16 23:12:12 2016 - [info]
Mon May 16 23:12:12 2016 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Mon May 16 23:12:12 2016 - [info] Starting recovery on 192.168.111.30(192.168.111.30:5621)..
Mon May 16 23:12:12 2016 - [info] This server has all relay logs. Waiting all logs to be applied..
Mon May 16 23:12:12 2016 - [info] done.
Mon May 16 23:12:12 2016 - [info] All relay logs were successfully applied.
Mon May 16 23:12:12 2016 - [info] Getting new master's binlog name and position..
Mon May 16 23:12:12 2016 - [info] mysql-bin.000025:367
这里可以看出，发生切换后，新主库的起始日志和点分别是
mysql
-
bin
.
000025
:
367
原主库stop之后，sysbench中断，为了模拟流量，重起sysbench
[mysql@MYSQL-SVR1 bin]$ ./sysbench --test=oltp --mysql-user=root --mysql-password=123 --mysql-host=192.168.111.80 --mysql-port=5621 --mysql-table-engine=innodb --mysql-db=db1 --oltp-table-name=t4 --oltp-table-size=1000000 --oltp-auto-inc=on --oltp-read-only=off --oltp-skip-trx=off --oltp-test-mode=complex --num-threads=1 --max-requests=1000000 run
sysbench 0.4.12: multi-threaded system evaluation benchmark
No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 1
Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Maximum number of requests for OLTP test is limited to 1000000
Threads started!
同时，因为原主库stop之后，sysbench也中断，所以这时候从新主库（30）上也能看出效果，不过生产环境上很难找出这个点，因为生产环境的数据是不断写入的。
mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000025 | 367 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000025 | 216497 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000025 | 268057 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
mysql> show master status;
+------------------+----------+--------------+------------------+-------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000025 | 8043844 | | | |
+------------------+----------+--------------+------------------+-------------------+
1 row in set (0.00 sec)
可以看出从367不断增大
启动之后，可以看到新主库的点不断增大，我们手动切换2次日志
flush logs;
flush logs;
--------------------------我是悲催的分割线----------------------------------
下面由于电脑死机，不得不重启，所有试验记录毁于一旦
（提醒做试验的小伙伴，sysbench差不多得了，千万别给太大压力，否则你的8G内存真的吃不消）
接下来的部分试验没有记录，只能简单说明一下
现在数据在30上不断写入，并且在写入的过程中，我们手动切换了2次日志，这些日志会同步到10
我们将原主库（20）重启，重启之后使用change master 指向新主库30
CHANGE MASTER TO
MASTER_HOST='192.168.111.30',
MASTER_USER='repl',
MASTER_PASSWORD='repl',
MASTER_PORT=5621,
MASTER_LOG_FILE=
'mysql-bin.000025
',
MASTER_LOG_POS=
367,
MASTER_CONNECT_RETRY=10;
然后在20 上show slave status。 可以看到点也是不断增大的，说明30的数据也开始同步到20。
接下来思路是：
待10和30 都和20完全同步后，使用mha的在线切换命令，将主库切换成20，并且30和10都作为20的从库，这就和最初的状态一致。
下面解决死机重启造成的问题，并且恢复主从关系（恢复之后30是主，10和20是从），然后使用MHA在线切换命令是主从关系回归最初状态。
重启以后，查看10的状态：
mysql> SHOW SLave status\G
*************************** 1. row ***************************
Slave_IO_State:
Master_Host: 192.168.111.30
Master_User: repl
Master_Port: 5621
Connect_Retry: 10
Master_Log_File: mysql-bin.000027
Read_Master_Log_Pos: 68709884
Relay_Log_File: relay-log.000006
Relay_Log_Pos: 68710047
Relay_Master_Log_File: mysql-bin.000027
Slave_IO_Running: No
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table: mysql.%,test.%
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 68709884
Relay_Log_Space: 68710249
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 1236
Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from position > file size; the first event 'mysql-bin.000027' at 68709884, the last event read from './mysql-bin.000027' at 4, the last byte read from './mysql-bin.000027' at 4.'
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 300
Master_UUID: 0ee37546-3e46-11e5-a399-000c297ada9c
Master_Info_File: /home/mysql/mysql_5621/var/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp: 160517 00:21:57
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
1 row in set (0.00 sec)
可以看到执行到的position是：
68709884
查看30的状态：
mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State:
Master_Host: 192.168.111.30
Master_User: repl
Master_Port: 5621
Connect_Retry: 10
Master_Log_File: mysql-bin.000027
Read_Master_Log_Pos: 68709966
Relay_Log_File: relay-log.000006
Relay_Log_Pos: 68710047
Relay_Master_Log_File: mysql-bin.000027
Slave_IO_Running: No
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table: mysql.%,test.%
Last_Errno: 1594
Last_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.
Skip_Counter: 0
Exec_Master_Log_Pos: 68709884
Relay_Log_Space: 68710463
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 1236
Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from position > file size; the first event 'mysql-bin.000027' at 68709966, the last event read from './mysql-bin.000027' at 4, the last byte read from './mysql-bin.000027' at 4.'
Last_SQL_Errno: 1594
Last_SQL_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.
Replicate_Ignore_Server_Ids:
Master_Server_Id: 300
Master_UUID: 0ee37546-3e46-11e5-a399-000c297ada9c
Master_Info_File: /home/mysql/mysql_5621/var/master.info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp: 160517 00:20:16
Last_SQL_Error_Timestamp: 160517 00:20:16
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set:
Auto_Position: 0
1 row in set (0.00 sec)
可以看到执行到的position也是：
68709884
那么这个点在主库上是否存在呢，其实是不存在的。
[mysql@MYSQL-SVR3 var]$ mysqlbinlog mysql-bin.000027 | tail -20
UPDATE t4 set k=k+1 where id=499066
/*!*/;
# at 68698931
#160516 23:58:40 server id 300 end_log_pos 68699038 CRC32 0x4689da97 Query thread_id=2652 exec_time=0 error_code=0
SET TIMESTAMP=1463414320/*!*/;
DELETE from t4 where id=499066
/*!*/;
# at 68699038
#160516 23:58:40 server id 300 end_log_pos 68699203 CRC32 0x7979978e Query thread_id=2652 exec_time=0 error_code=0
SET TIMESTAMP=1463414320/*!*/;
INSERT INTO t4 values(499066,0,' ','aaaaaaaaaaffffffffffrrrrrrrrrreeeeeeeeeeyyyyyyyyyy')
/*!*/;
# at 68699203
#160516 23:58:40 server id 300 end_log_pos 68699234 CRC32 0x5257e2e0 Xid = 2101722
COMMIT/*!*/;
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
可以看到主库上的最后一个点是 68699203，比68709884要小。
要修复2个从库，我们可以从这个点恢复，就是不知道丢不丢数据？
百度1236错误，给出的解决方法也都大同小异，因为是测试环境，我们直接恢复。
mysql> stop slave;
Query OK, 0 rows affected, 1 warning (0.00 sec)
mysql> CHANGE MASTER TO
-> MASTER_HOST='192.168.111.30',
-> MASTER_USER='repl',
-> MASTER_PASSWORD='repl',
-> MASTER_PORT=5621,
-> MASTER_LOG_FILE='mysql-bin.000027',
-> MASTER_LOG_POS=68699203,
-> MASTER_CONNECT_RETRY=10;
Query OK, 0 rows affected, 2 warnings (0.11 sec)
查看2个从库状态OK.
接下来使用MHA在线手动切换将主库切回20
[root@MYSQL-SVR1 ~]# masterha_master_switch --conf=/etc/masterha/app1.cnf --master_state=alive --new_master_host=192.168.111.20 --new_master_port=5621 --orig_master_is_new_slave
Tue May 17 01:43:19 2016 - [info] MHA::MasterRotate version 0.55.
Tue May 17 01:43:19 2016 - [info] Starting online master switch..
Tue May 17 01:43:19 2016 - [info]
Tue May 17 01:43:19 2016 - [info] * Phase 1: Configuration Check Phase..
Tue May 17 01:43:19 2016 - [info]
Tue May 17 01:43:19 2016 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue May 17 01:43:20 2016 - [info] Reading application default configurations from /etc/masterha/app1.cnf..
Tue May 17 01:43:20 2016 - [info] Reading server configurations from /etc/masterha/app1.cnf..
Tue May 17 01:43:20 2016 - [info] Current Alive Master: 192.168.111.30(192.168.111.30:5621)
Tue May 17 01:43:20 2016 - [info] Alive Slaves:
Tue May 17 01:43:20 2016 - [info] 192.168.111.20(192.168.111.20:5621) Version=5.6.21-log (oldest major version between slaves) log-bin:enabled
Tue May 17 01:43:20 2016 - [info] Replicating from 192.168.111.30(192.168.111.30:5621)
Tue May 17 01:43:20 2016 - [info] Primary candidate for the new Master (candidate_master is set)
Tue May 17 01:43:20 2016 - [info] 192.168.111.10(192.168.111.10:5621) Version=5.6.21-log (oldest major version between slaves) log-bin:enabled
Tue May 17 01:43:20 2016 - [info] Replicating from 192.168.111.30(192.168.111.30:5621)
Tue May 17 01:43:20 2016 - [info] Not candidate for the new Master (no_master is set)
It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.111.30(192.168.111.30:5621)? (YES/no): yes
Tue May 17 01:43:26 2016 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Tue May 17 01:43:26 2016 - [info] ok.
Tue May 17 01:43:26 2016 - [info] Checking MHA is not monitoring or doing failover..
Tue May 17 01:43:26 2016 - [info] Checking replication health on 192.168.111.20..
Tue May 17 01:43:26 2016 - [info] ok.
Tue May 17 01:43:26 2016 - [info] Checking replication health on 192.168.111.10..
Tue May 17 01:43:26 2016 - [info] ok.
Tue May 17 01:43:26 2016 - [info] 192.168.111.20 can be new master.
Tue May 17 01:43:26 2016 - [info]
From:
192.168.111.30 (current master)
+--192.168.111.20
+--192.168.111.10
To:
192.168.111.20 (new master)
+--192.168.111.10
+--192.168.111.30
Starting master switch from 192.168.111.30(192.168.111.30:5621) to 192.168.111.20(192.168.111.20:5621)? (yes/NO): yes
Tue May 17 01:43:29 2016 - [info] Checking whether 192.168.111.20(192.168.111.20:5621) is ok for the new master..
Tue May 17 01:43:29 2016 - [info] ok.
Tue May 17 01:43:29 2016 - [info] 192.168.111.30(192.168.111.30:5621): SHOW SLAVE STATUS returned empty result. To check replication filtering rules, temporarily executing CHANGE MASTER to a dummy host.
Tue May 17 01:43:29 2016 - [info] 192.168.111.30(192.168.111.30:5621): Resetting slave pointing to the dummy host.
Tue May 17 01:43:29 2016 - [info] ** Phase 1: Configuration Check Phase completed.
Tue May 17 01:43:29 2016 - [info]
Tue May 17 01:43:29 2016 - [info] * Phase 2: Rejecting updates Phase..
Tue May 17 01:43:29 2016 - [info]
Tue May 17 01:43:29 2016 - [info] Executing master ip online change script to disable write on the current master:
Tue May 17 01:43:29 2016 - [info] /masterha/app1/script/master_ip_online_change --command=stop --orig_master_host=192.168.111.30 --orig_master_ip=192.168.111.30 --orig_master_port=5621 --orig_master_user='root' --orig_master_password='123' --new_master_host=192.168.111.20 --new_master_ip=192.168.111.20 --new_master_port=5621 --new_master_user='root' --new_master_password='123'
***************************************************************
Disabling the VIP - 192.168.111.80/24 on old master: 192.168.111.30
SIOCSIFFLAGS: 无法指定被请求的地址
Disabled the VIP failed
***************************************************************
Tue May 17 01:43:29 2016 - [info] ok.
Tue May 17 01:43:29 2016 - [info] Locking all tables on the orig master to reject updates from everybody (including root):
Tue May 17 01:43:29 2016 - [info] Executing FLUSH TABLES WITH READ LOCK..
Tue May 17 01:43:29 2016 - [info] ok.
Tue May 17 01:43:29 2016 - [info] Orig master binlog:pos is mysql-bin.000028:120.
Tue May 17 01:43:29 2016 - [info] Waiting to execute all relay logs on 192.168.111.20(192.168.111.20:5621)..
Tue May 17 01:43:29 2016 - [info] master_pos_wait(mysql-bin.000028:120) completed on 192.168.111.20(192.168.111.20:5621). Executed 0 events.
Tue May 17 01:43:29 2016 - [info] done.
Tue May 17 01:43:29 2016 - [info] Getting new master's binlog name and position..
Tue May 17 01:43:29 2016 - [info] mysql-bin.000033:120
Tue May 17 01:43:29 2016 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.111.20', MASTER_PORT=5621, MASTER_LOG_FILE='mysql-bin.000033', MASTER_LOG_POS=120, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Tue May 17 01:43:29 2016 - [info] Executing master ip online change script to allow write on the new master:
Tue May 17 01:43:29 2016 - [info] /masterha/app1/script/master_ip_online_change --command=start --orig_master_host=192.168.111.30 --orig_master_ip=192.168.111.30 --orig_master_port=5621 --orig_master_user='root' --orig_master_password='123' --new_master_host=192.168.111.20 --new_master_ip=192.168.111.20 --new_master_port=5621 --new_master_user='root' --new_master_password='123'
***************************************************************
Enabling the VIP - 192.168.111.80/24 on new master: 192.168.111.20
Enabled the VIP successfully
***************************************************************
Tue May 17 01:43:30 2016 - [info] ok.
Tue May 17 01:43:30 2016 - [info]
Tue May 17 01:43:30 2016 - [info] * Switching slaves in parallel..
Tue May 17 01:43:30 2016 - [info]
Tue May 17 01:43:30 2016 - [info] -- Slave switch on host 192.168.111.10(192.168.111.10:5621) started, pid: 32588
Tue May 17 01:43:30 2016 - [info]
Tue May 17 01:43:30 2016 - [info] Log messages from 192.168.111.10 ...
Tue May 17 01:43:30 2016 - [info]
Tue May 17 01:43:30 2016 - [info] Waiting to execute all relay logs on 192.168.111.10(192.168.111.10:5621)..
Tue May 17 01:43:30 2016 - [info