MySQL新一代HA架构：mysqlfailover + semisync of mysql5.7

使用mysql5.7 增强半同步，GTID复制，设置master wait永不超时，外加mysqlfailover自动切换。可实现新一代高可用架构，零事务丢失自动切换！

搭建步骤

搭建一主两从（至少两从）增强半同步架构（安装略）。
master 半同步配置用不超时：
```
rpl_semi_sync_master_wait_for_slave_count=1
rpl_semi_sync_master_wait_no_slave=ON
rpl_semi_sync_master_timeout=1000000000000000000
rpl_semi_sync_master_wait_point=AFTER_SYNC
```
salve 的my.cnf配置(主库也要配置，防止主库切换后，再次以从库的角色加入集群，无法被自动发现)：
```
report_host='172.40.0.164' # the ip of the slave
report_port=3306
master_info_repository=TABLE
```
注意从库不能配置super_read_only=ON

在一台单独的监控机上安装mysqlfailover.

wget -c https://cdn.mysql.com/archives/mysql-utilities/mysql-utilities-1.6.5-1.el7.noarch.rpm

#mysqlfailover脚本依赖依赖mysql python connector

wget -c https://dev.mysql.com/get/Downloads/Connector-Python/mysql-connector-python-2.1.7-1.el7.x86_64.rpm

#安装：
rpm -ivh mysql-connector-python-2.1.7-1.el7.x86_64.rpm
rpm -ivh mysql-utilities-1.6.5-1.el7.noarch.rpm

failover监控脚本账号创建。mysqlfailover脚本使用一个数据库账号去监控集群中各节点状态，如果主库故障，会启用自动切换，切换时脚本会执行change master 等操作,且所有节点需要此账号，因此需要在主库上创建。此账号权限较大，为安全起见，可使用login-path.
```
create user fo@'192.168.216.146' identified by 'fo';
grant SUPER, REPLICATION SLAVE, SELECT, RELOAD, DROP, CREATE, INSERT on *.* to fo@'192.168.216.146'  with grant option;
```
对于mysql5.7还需要授权：
```
grant update,delete on mysql.* to 'fo'@'192.168.216.146';
```
因为mysql 5.7中对于已经存在的账号，再次使用grant ... identified by 需要修改mysql库中的相关表。
在监控节点配置login-path

mysql_config_editor set --login-path=fo --user=fo --host=192.168.216.144 --port=3306 --password
# test login
mysql --login-path=fo

启动mysqlfailover

mysqlfailover --master=fo --discover-slaves-login=fo --log=/tmp/fo.log  --rpl-user=repl:repl  --force

mysqlfailover在第一次连接的时候，会在mysql库下面创建failover_console表，里面记录了所监控master的ip和端口。正常退出，这个表会被drop掉。但对于非正常退出，这个表会一直存在。如果下次再尝试连接，则会警告：

Multiple instances of failover console found for master 192.168.216.144:3306.
If this is an error, restart the console with --force. 
Failover mode changed to 'FAIL' for this instance. 
Console will start in 10 seconds..........starting Console.

上述信息是说mysqlfailover检查到已经有一个进程在运行了，因此这个进程的mode被修改为FAIL, FAIL mode的意思是该进程只监控，不会执行切换。如果确认只有一个mysqlfailover在运行，可以使用--force参数。这里需要说一下failover的三种模式：

auto: (默认)。自动切换模式。该模式在切换时，首先从candicate列表中选择，如果candicate列表的主机不可用，则会从其它的从节点选择一个作为新的master. 然后把新master配置为其它从节点的从库，复制缺少的事务，直到该master包含所有的事务（most update slave）

elect: 该模式和auto模式类似，只是它只从candidate列表中选择。如果列表不可用则报错退出。使用该模式必须用--candidates选项。

fail:该模式只监控，不执行切换。

测试

停止主库，等待约10s左右，可以看到切换动作

Failed to reconnect to the master after 3 attemps.
Failover starting in 'auto' mode...
# Candidate slave 192.168.216.145:3306 will become the new master.
# Checking slaves status (before failover).
# Preparing candidate for failover.
# Creating replication user if it does not exist.
# Stopping slaves.
# Performing STOP on all slaves.
# Switching slaves to new master.
# Disconnecting new master as slave.
# Starting slaves.
# Performing START on all slaves.
# Checking slaves for errors.
# Failover complete.
# Discovering slaves for master at 192.168.216.145:3306

Failover console will restart in 5 seconds.

重新启动原来的主节点，然后将其作为从库加入集群：

mysqlreplicate --master=fo:[email protected]:3306 --slave=fo:[email protected]:3306 --rpl-user=repl:repl

加入后，failover脚本很快就发现新节点，显示一主两从的架构。

重新启用原来的主节点为master

mysqlrpladmin --master=fo:[email protected]:3306 \
> --slaves=fo:[email protected]:3306,fo:[email protected]:3306  \
> --rpl-user=repl:repl --new-master=fo:[email protected]:3306 --demote-master switchover
WARNING: Using a password on the command line interface can be insecure.
# Checking privileges.
# Performing switchover from master at 192.168.216.145:3306 to slave at 192.168.216.144:3306.
# Checking candidate slave prerequisites.
# Checking slaves configuration to master.
# Waiting for slaves to catch up to old master.
# Stopping slaves.
# Performing STOP on all slaves.
# Demoting old master to be a slave to the new master.
# Switching slaves to new master.
# Starting all slaves.
# Performing START on all slaves.
# Checking slaves for errors.
# Switchover complete.
#
# Replication Topology Health:
+------------------+-------+---------+--------+------------+---------+
| host             | port  | role    | state  | gtid_mode  | health  |
+------------------+-------+---------+--------+------------+---------+
| 192.168.216.144  | 3306  | MASTER  | UP     | ON         | OK      |
| 192.168.216.145  | 3306  | SLAVE   | UP     | ON         | OK      |
| 192.168.216.147  | 3306  | SLAVE   | UP     | ON         | OK      |
+------------------+-------+---------+--------+------------+---------+
# ...done.

切换后发现，failover脚本监控失败：


Replication Health Status
+------------------+-------+---------+--------+------------+------------------------------------+
| host             | port  | role    | state  | gtid_mode  | health                             |
+------------------+-------+---------+--------+------------+------------------------------------+
| 192.168.216.145  | 3306  | MASTER  | UP     | ON         | OK                                 |
| 192.168.216.144  | 3306  | SLAVE   | WARN   |            | Slave is not connected to master.  |
| 192.168.216.147  | 3306  | SLAVE   | WARN   |            | Slave is not connected to master.  |
+------------------+-------+---------+--------+------------+------------------------------------+

看来这种情下必须重启mysqlfailover脚本！

使用sysbench 对主库压测,监控信息会显示延迟情况。

Replication Health Status
+------------------+-------+---------+--------+------------+--------------------------------------------------------------------------------------------+
| host             | port  | role    | state  | gtid_mode  | health                                                                                     |
+------------------+-------+---------+--------+------------+--------------------------------------------------------------------------------------------+
| 192.168.216.144  | 3306  | MASTER  | UP     | ON         | OK                                                                                         |
| 192.168.216.145  | 3306  | SLAVE   | UP     | ON         | Slave delay is 6 seconds behind master., No, Slave has 3916 transactions behind master.    |
| 192.168.216.147  | 3306  | SLAVE   | UP     | ON         | Slave delay is 59 seconds behind master., No, Slave has 33416 transactions behind master.  |
+------------------+-------+---------+--------+------------+--------------------------------------------------------------------------------------------+

在主从复制延迟的情况下，停掉主库，查看failover脚本切换状况。

Failed to reconnect to the master after 3 attemps.
Failover starting in 'auto' mode...
# Candidate slave 192.168.216.145:3306 will become the new master.
# Checking slaves status (before failover).
# Preparing candidate for failover.

发现切换时间明显增长，在Preparing candidate for failover.等待。查看从库状态，发现有一个尚未应用完binlog，当所有从库都应用完binglog时，才正式切换。这对保证数据一致性很重要！mysqlfailover已经做到这一点儿了。另外如果切换时主从延迟严重，切换时间会较长，这可以通过启用MTS来缓解。

问题

切换后，从库被提升为master ，但只读属性没有被取消。这会导致应用依然不能做业务。还一个问题是切换后，应该通知应用，连接新的master. (如修改DNS，或zookeeper中的masterIP,这取决于应用数据源的配置方式) 这些都可以通过脚本来实现。
mysqlfailover 提供了两个注入脚本的接口:
- --exec-before=script ,在切换开始前执行。
- --exec-after=script ，在切换完成后执行。
  这意味着我们可以在切换完成后写个脚本去修改新master的只读属性，已经通知应用修改数据源地址（使用vip来解决更为优雅，但有些云环境不支持vip）。
使用--daemon=start 启动脚本时，不能使用--daemon=stop来终止。会造成从库sql_thread报错断开。须使用kill -9 pid . 启动时，加--force参数。我已向官方递交此bug (91562)

参考资料：

https://dev.mysql.com/doc/mysql-utilities/1.6/en/mysqlfailover.html

MySQL新一代HA架构：mysqlfailover + semisync of mysql5.7

搭建步骤

测试

问题

你可能感兴趣的:(MySQL新一代HA架构：mysqlfailover + semisync of mysql5.7)