高可用
查看MHA配置文件
[root@db03 ~]# cat /etc/mha/app1.cnf
[server default]
manager_log=/var/log/mha/app1/manager
manager_workdir=/var/log/mha/app1
master_binlog_dir=/data/binlog
user=mha
password=mha
ping_interval=2
repl_password=123
repl_user=repl
ssh_user=root
[server1]
hostname=10.0.0.51
port=3306
[server2]
hostname=10.0.0.52
port=3306
[server3]
hostname=10.0.0.53
port=3306
检查主从状态
[root@db03 ~]# masterha_check_repl --conf=/etc/mha/app1.cnf
MySQL Replication Health is OK.
开启MHA(db03)
nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null> /var/log/mha/app1/manager.log 2>&1 &
主从复制架构演变介绍
基本结构
（1）一主一从
（2）一主多从
（3）多级主从
（4）双主
（5）循环复制
高级应用架构演变
高性能架构
(1) 读写分离架构(读性能较高)
代码级别
MySQL proxy (Atlas,mysql router,proxySQL(percona),maxscale)、
amoeba(taobao)
xx-dbproxy等。
(2) 分布式架构(读写性能都提高):
分库分表——cobar--->TDDL(头都大了),DRDS
Mycat--->DBLE自主研发等。
NewSQL-->TiDB
高可用架构
（3）单活:MMM架构——mysql-mmm（google）
（4）单活:MHA架构——mysql-master-ha（日本DeNa）,T-MHA
（5）多活:MGR ——5.7 新特性 MySQL Group replication(5.7.17) --->Innodb Cluster
（6）多活:MariaDB Galera Cluster架构,(PXC)Percona XtraDB Cluster、MySQL Cluster(Oracle rac)架构
高可用MHA ☆☆☆☆☆
架构工作原理
- MHA通过脚本(masterha_master_monitor)实时监控主库的状态(主机,数据库状态)
- 监控到主库宕机之后,会发生自动选主(masterha_master_switch )
(1) 权重: 参数
(2) 日志量
(3) 配置文件的顺序- 数据补偿
(1) 当主库SSH能够连接,各个从库会立即保存(save_binary_logs)缺失部分的日志到/var/tmp,进行数据补偿.
(2) 当主库SSH无法连接,通过脚本(apply_diff_relay_logs)自动进行从库的relaylog的差异处理,进行数据补偿.- 构建新的主从关系,并将故障节点移除(配置文件移除)
- MHA主进程完成工作后,自动退出.
- 应用透明 VIP(master_ip_failover_script=/usr/local/bin/master_ip_failover)
- 故障通知 (report_script=/usr/local/bin/send)
- 提供额外数据补偿的手段(binlog server)
- 自动自愈,待开发....
架构介绍:
1主2从，master：db01 slave：db02 db03 ）：
MHA 高可用方案软件构成
Manager软件：选择一个从节点安装
Node软件：所有节点都要安装
MHA软件构成
Manager工具包主要包括以下几个工具：
masterha_manger 启动MHA
masterha_check_ssh 检查MHA的SSH配置状况
masterha_check_repl 检查MySQL复制状况
masterha_master_monitor 检测master是否宕机
masterha_check_status 检测当前MHA运行状态
masterha_master_switch 控制故障转移（自动或者手动）
masterha_conf_host 添加或删除配置的server信息
Node工具包主要包括以下几个工具：
这些工具通常由MHA Manager的脚本触发，无需人为操作
save_binary_logs 保存和复制master的二进制日志
apply_diff_relay_logs 识别差异的中继日志事件并将其差异的事件应用于其他的
purge_relay_logs 清除中继日志（不会阻塞SQL线程）
MHA环境搭建
软件链接: 提取码: bc95
配置关键程序软连接
10.0.0.51/52/53:
ln -s /application/mysql/bin/mysqlbinlog /usr/bin/mysqlbinlog
ln -s /application/mysql/bin/mysql /usr/bin/mysql
配置各节点互信(SSH协议)
在db01上创建秘钥：
rm -rf /root/.ssh
ssh-keygen #生成秘钥 一直回车
cd /root/.ssh
mv id_rsa.pub authorized_keys
scp -r /root/.ssh 10.0.0.52:/root
scp -r /root/.ssh 10.0.0.53:/root
#第二种方法在各节点都分发秘钥
cat /server/scripts/ssh.sh
#!/bin/bash
rm -rf ~/.ssh/*
#make key pair
ssh-keygen -t dsa -f ~/.ssh/id_dsa -P ''
#fenfa public key
for ip in 51 52 53
do
sshpass -p123456 ssh-copy-id -o StrictHostKeyChecking=no 10.0.0.$ip;
done
#各节点验证
db01:
ssh 10.0.0.51 date
ssh 10.0.0.52 date
ssh 10.0.0.53 date
db02:
ssh 10.0.0.51 date
ssh 10.0.0.52 date
ssh 10.0.0.53 date
db03:
ssh 10.0.0.51 date
ssh 10.0.0.52 date
ssh 10.0.0.53 date
安装软件
下载mha软件
mha官网：https://code.google.com/archive/p/mysql-master-ha/
github下载地址：https://github.com/yoshinorim/mha4mysql-manager/wiki/Downloads
所有节点安装Node软件依赖包
yum install perl-DBD-MySQL -y
rpm -ivh mha4mysql-node-0.56-0.el6.noarch.rpm
在db01主库中创建mha需要的用户
grant all privileges on *.* to mha@'10.0.0.%' identified by 'mha';
Manager软件安装（db03）
yum install -y perl-Config-Tiny epel-release perl-Log-Dispatch perl-Parallel-ForkManager perl-Time-HiRes
rpm -ivh mha4mysql-manager-0.56-0.el6.noarch.rpm
配置文件准备(db03)
创建配置文件目录
mkdir -p /etc/mha
创建日志目录
mkdir -p /var/log/mha/app1
编辑mha配置文件
vim /etc/mha/app1.cnf
[server default]
manager_log=/var/log/mha/app1/manager
manager_workdir=/var/log/mha/app1
master_binlog_dir=/data/binlog
user=mha
password=mha
ping_interval=2
repl_password=123
repl_user=repl
ssh_user=root
[server1]
hostname=10.0.0.51
port=3306
[server2]
hostname=10.0.0.52
port=3306
[server3]
hostname=10.0.0.53
port=3306
状态检查
互信检查
masterha_check_ssh --conf=/etc/mha/app1.cnf
Fri Apr 19 16:39:34 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Apr 19 16:39:34 2019 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Fri Apr 19 16:39:34 2019 - [info] Reading server configuration from /etc/mha/app1.cnf..
Fri Apr 19 16:39:34 2019 - [info] Starting SSH connection tests..
Fri Apr 19 16:39:35 2019 - [debug]
Fri Apr 19 16:39:34 2019 - [debug] Connecting via SSH from [email protected](10.0.0.51:22) to [email protected](10.0.0.52:22)..
Fri Apr 19 16:39:34 2019 - [debug] ok.
Fri Apr 19 16:39:34 2019 - [debug] Connecting via SSH from [email protected](10.0.0.51:22) to [email protected](10.0.0.53:22)..
Fri Apr 19 16:39:35 2019 - [debug] ok.
Fri Apr 19 16:39:36 2019 - [debug]
Fri Apr 19 16:39:35 2019 - [debug] Connecting via SSH from [email protected](10.0.0.52:22) to [email protected](10.0.0.51:22)..
Fri Apr 19 16:39:35 2019 - [debug] ok.
Fri Apr 19 16:39:35 2019 - [debug] Connecting via SSH from [email protected](10.0.0.52:22) to [email protected](10.0.0.53:22)..
Fri Apr 19 16:39:35 2019 - [debug] ok.
Fri Apr 19 16:39:37 2019 - [debug]
Fri Apr 19 16:39:35 2019 - [debug] Connecting via SSH from [email protected](10.0.0.53:22) to [email protected](10.0.0.51:22)..
Fri Apr 19 16:39:35 2019 - [debug] ok.
Fri Apr 19 16:39:35 2019 - [debug] Connecting via SSH from [email protected](10.0.0.53:22) to [email protected](10.0.0.52:22)..
Fri Apr 19 16:39:36 2019 - [debug] ok.
Fri Apr 19 16:39:37 2019 - [info] All SSH connection tests passed successfully.
主从状态检查
[root@db03 ~]# masterha_check_ssh --conf=/etc/mha/app1.cnf
Fri Apr 19 16:39:34 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Apr 19 16:39:34 2019 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Fri Apr 19 16:39:34 2019 - [info] Reading server configuration from /etc/mha/app1.cnf..
Fri Apr 19 16:39:34 2019 - [info] Starting SSH connection tests..
Fri Apr 19 16:39:35 2019 - [debug]
Fri Apr 19 16:39:34 2019 - [debug] Connecting via SSH from [email protected](10.0.0.51:22) to [email protected](10.0.0.52:22)..
Fri Apr 19 16:39:34 2019 - [debug] ok.
Fri Apr 19 16:39:34 2019 - [debug] Connecting via SSH from [email protected](10.0.0.51:22) to [email protected](10.0.0.53:22)..
Fri Apr 19 16:39:35 2019 - [debug] ok.
Fri Apr 19 16:39:36 2019 - [debug]
Fri Apr 19 16:39:35 2019 - [debug] Connecting via SSH from [email protected](10.0.0.52:22) to [email protected](10.0.0.51:22)..
Fri Apr 19 16:39:35 2019 - [debug] ok.
Fri Apr 19 16:39:35 2019 - [debug] Connecting via SSH from [email protected](10.0.0.52:22) to [email protected](10.0.0.53:22)..
Fri Apr 19 16:39:35 2019 - [debug] ok.
Fri Apr 19 16:39:37 2019 - [debug]
Fri Apr 19 16:39:35 2019 - [debug] Connecting via SSH from [email protected](10.0.0.53:22) to [email protected](10.0.0.51:22)..
Fri Apr 19 16:39:35 2019 - [debug] ok.
Fri Apr 19 16:39:35 2019 - [debug] Connecting via SSH from [email protected](10.0.0.53:22) to [email protected](10.0.0.52:22)..
Fri Apr 19 16:39:36 2019 - [debug] ok.
Fri Apr 19 16:39:37 2019 - [info] All SSH connection tests passed successfully.
[root@db03 ~]# masterha_check_repl --conf=/etc/mha/app1.cnf
Fri Apr 19 16:40:50 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Apr 19 16:40:50 2019 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Fri Apr 19 16:40:50 2019 - [info] Reading server configuration from /etc/mha/app1.cnf..
Fri Apr 19 16:40:50 2019 - [info] MHA::MasterMonitor version 0.56.
Fri Apr 19 16:40:51 2019 - [info] GTID failover mode = 1
Fri Apr 19 16:40:51 2019 - [info] Dead Servers:
Fri Apr 19 16:40:51 2019 - [info] Alive Servers:
Fri Apr 19 16:40:51 2019 - [info] 10.0.0.51(10.0.0.51:3306)
Fri Apr 19 16:40:51 2019 - [info] 10.0.0.52(10.0.0.52:3306)
Fri Apr 19 16:40:51 2019 - [info] 10.0.0.53(10.0.0.53:3306)
Fri Apr 19 16:40:51 2019 - [info] Alive Slaves:
Fri Apr 19 16:40:51 2019 - [info] 10.0.0.52(10.0.0.52:3306) Version=5.7.20-log (oldest major version between slaves) log-bin:enabled
Fri Apr 19 16:40:51 2019 - [info] GTID ON
Fri Apr 19 16:40:51 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
Fri Apr 19 16:40:51 2019 - [info] 10.0.0.53(10.0.0.53:3306) Version=5.7.20-log (oldest major version between slaves) log-bin:enabled
Fri Apr 19 16:40:51 2019 - [info] GTID ON
Fri Apr 19 16:40:51 2019 - [info] Replicating from 10.0.0.51(10.0.0.51:3306)
Fri Apr 19 16:40:51 2019 - [info] Current Alive Master: 10.0.0.51(10.0.0.51:3306)
Fri Apr 19 16:40:51 2019 - [info] Checking slave configurations..
Fri Apr 19 16:40:51 2019 - [info] read_only=1 is not set on slave 10.0.0.52(10.0.0.52:3306).
Fri Apr 19 16:40:51 2019 - [info] read_only=1 is not set on slave 10.0.0.53(10.0.0.53:3306).
Fri Apr 19 16:40:51 2019 - [info] Checking replication filtering settings..
Fri Apr 19 16:40:51 2019 - [info] binlog_do_db= , binlog_ignore_db=
Fri Apr 19 16:40:51 2019 - [info] Replication filtering check ok.
Fri Apr 19 16:40:51 2019 - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Fri Apr 19 16:40:51 2019 - [info] Checking SSH publickey authentication settings on the current master..
Fri Apr 19 16:40:51 2019 - [info] HealthCheck: SSH to 10.0.0.51 is reachable.
Fri Apr 19 16:40:51 2019 - [info]
10.0.0.51(10.0.0.51:3306) (current master)
+--10.0.0.52(10.0.0.52:3306)
+--10.0.0.53(10.0.0.53:3306)
Fri Apr 19 16:40:51 2019 - [info] Checking replication health on 10.0.0.52..
Fri Apr 19 16:40:51 2019 - [info] ok.
Fri Apr 19 16:40:51 2019 - [info] Checking replication health on 10.0.0.53..
Fri Apr 19 16:40:51 2019 - [info] ok.
Fri Apr 19 16:40:51 2019 - [warning] master_ip_failover_script is not defined.
Fri Apr 19 16:40:51 2019 - [warning] shutdown_script is not defined.
Fri Apr 19 16:40:51 2019 - [info] Got exit code 0 (Not master dead).
MySQL Replication Health is OK.
开启MHA(db03)：
nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null> /var/log/mha/app1/manager.log 2>&1 &
查看MHA状态(db03):
[root@db03 ~]# masterha_check_status --conf=/etc/mha/app1.cnf
app1 (pid:4719) is running(0:PING_OK), master:10.0.0.51
故障模拟测试
关闭51主库（db01上）
[root@db01 ~]# /etc/init.d/mysqld stop
Shutting down MySQL............ SUCCESS!
此时主库跳到52上（db03上查看）
imageimage.png
修复主库
[root@db01 ~]# /etc/init.d/mysqld start
Starting MySQL. SUCCESS!
修复从库
[root@db01 ~]# mysql -e "change master to master_host='10.0.0.52',master_user='repl',master_password='123' ,MASTER_AUTO_POSITION=1;"
[root@db01 ~]# mysql -e "start slave;"
[root@db01 ~]# mysql -e "show slave status \G"|grep Running:
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
修改配置文件(db03)
image.png
启动MHA (db03)
[root@db03 ~]# nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null> /var/log/mha/app1/manager.log 2>&1 &
[1] 7658
[root@db03 ~]# masterha_check_status --conf=/etc/mha/app1.cnf
app1 (pid:7658) is running(0:PING_OK), master:10.0.0.52
Manager额外参数介绍
说明：
主库宕机谁来接管？
1\. 所有从节点日志都是一致的，默认会以配置文件的顺序去选择一个新主。
2\. 从节点日志不一致，自动选择最接近于主库的从库
3\. 如果对于某节点设定了权重（candidate_master=1），权重节点会优先选择。
但是此节点日志量落后主库100M日志的话，也不会被选择。可以配合check_repl_delay=0，关闭日志量的检查，强制选择候选节点。
(1) ping_interval=1
#设置监控主库，发送ping包的时间间隔，尝试三次没有回应的时候自动进行failover
(2) candidate_master=1
#设置为候选master，如果设置该参数以后，发生主从切换以后将会将此从库提升为主库，即使这个主库不是集群中事件最新的slave
(3)check_repl_delay=0
#默认情况下如果一个slave落后master 100M的relay logs的话，
MHA将不会选择该slave作为一个新的master，因为对于这个slave的恢复需要花费很长时间，
通过设置check_repl_delay=0,MHA触发切换在选择一个新的master的时候将会忽略复制延时，
这个参数对于设置了candidate_master=1的主机非常有用，因为这个候选主在切换的过程中一定是新的master
MHA的vip功能（db03）
拷贝master_ip_failover脚本到/user/local/bin下
[root@db03 /server/tools]# ll
total 9936
-rw-r--r-- 1 root root 4963681 Jun 28 11:35 Atlas-2.2.1.el6.x86_64.rpm
-rw-r--r-- 1 root root 20904 Jun 28 11:36 email_2019-最新.zip
-rw-r--r-- 1 root root 2248 Jun 28 11:36 master_ip_failover.txt
-rw-r--r-- 1 root root 5051108 Aug 19 18:16 MHA-2019-6.28.zip
-rw-r--r-- 1 root root 87119 Jun 28 11:34 mha4mysql-manager-0.56-0.el6.noarch.rpm
-rw-r--r-- 1 root root 36326 Jun 28 11:35 mha4mysql-node-0.56-0.el6.noarch.rpm
[root@db03 /server/tools]# cp master_ip_failover.txt /usr/local/bin/master_ip_failover
[root@db03 /server/tools]# cd /usr/local/bin/
[root@db03 /usr/local/bin]# yum install -y dos2unix
[root@db03 /usr/local/bin]# dos2unix master_ip_failover
dos2unix: converting file master_ip_failover to Unix format ...
[root@db03 /usr/local/bin]# chmod +x master_ip_failover
更改manager配置文件：
vim /etc/mha/app1.cnf
添加此行：
master_ip_failover_script=/usr/local/bin/master_ip_failover
修改文件内容:
[root@db03 /usr/local/bin]# vim /usr/local/bin/master_ip_failover
my $vip = '10.0.0.55/24';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig eth0:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig eth0:$key down";
image
主库上，手工生成第一个vip地址(db02)
手工在主库上绑定vip，注意一定要和配置文件中的ethN一致，我的是eth0:1(1是key指定的值)
ifconfig eth0:1 10.0.0.55/24
重启mha
masterha_stop --conf=/etc/mha/app1.cnf
nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/mha/app1/manager.log 2>&1 &
邮件提醒
添加配置选项
vi /etc/mha/app1.cnf
report_script=/usr/local/bin/send
拷贝脚本
[root@db03 /server/tools]# cp email/* /usr/local/bin
添加执行权限
[root@db03 /usr/local/bin]# chmod +x ./*
[root@db03 /usr/local/bin]# ll
total 92
-rwxr-xr-x 1 root root 2166 Aug 20 15:44 master_ip_failover
-rwxr-xr-x 1 root root 35 Aug 20 15:52 send
-rwxr-xr-x 1 root root 80213 Aug 20 15:52 sendEmail
-rwxr-xr-x 1 root root 204 Aug 20 15:53 testpl
修改脚本
[root@db03 /usr/local/bin]# vim testpl
#!/bin/bash
/usr/local/bin/sendEmail -o tls=no -f [email protected] -t [email protected] -s smt
p.163.com:25 -xu lichenxing0430 -xp -u "MHA Waring" -m "YOUR MHA MAY BE F
AILOVER ，发送成功！" &>/tmp/sendmail.log
停止MHA
masterha_stop --conf=/etc/mha/app1.cnf
开启MHA
nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/mha/app1/manager.log 2>&1 &
发送邮件
[root@db03 /usr/local/bin]# ./testpl
image.png
image.png
关闭主库,看警告邮件(db02)
[root@db02 ~]# pkill mysqld
查看vip并收到邮件
imageimage.png
故障恢复(db03)
mysql -e "change master to master_host='10.0.0.51',master_user='repl',master_password='123' ,MASTER_AUTO_POSITION=1;"
mysql -e "start slave;"
mysql -e "show slave status \G"|grep "Running:"
binlog server（db03）
添加配置文件
vim /etc/mha/app1.cnf
[binlog1]
no_master=1
hostname=10.0.0.53
master_binlog_dir=/data/mysql/binlog
创建必要目录
mkdir -p /data/mysql/binlog
chown -R mysql.mysql /data/*
拉取主库binlog日志
cd /data/mysql/binlog
mysqlbinlog -R --host=10.0.0.51 --user=mha --password=mha --raw --stop-never mysql-bin.000001 &
重启MHA
masterha_stop --conf=/etc/mha/app1.cnf
nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/mha/app1/manager.log 2>&1 &
故障处理
主库宕机，binlogserver 自动停掉，manager 也会自动停止。
处理思路：
1、重新获取新主库的binlog到binlogserver中
2、重新配置文件binlog server信息
3、最后再启动MHA3.管理员在高可用架构维护的职责
- 搭建：MHA+VIP+SendReport+BinlogServer
- 监控及故障处理
- 高可用架构的优化
核心是：尽可能降低主从的延时，让MHA花在数据补偿上的时间尽量减少。
5.7 版本，开启GTID模式，开启从库SQL并发复制。
MHA通用故障修复
检查MHA状态
[root@db03 ~]# masterha_check_status --conf=/etc/mha/app1.cnf
检查配置文件
[server default]
manager_log=/var/log/mha/app1/manager
manager_workdir=/var/log/mha/app1
master_binlog_dir=/data/binlog
master_ip_failover_script=/usr/local/bin/master_ip_failover
password=mha
ping_interval=2
repl_password=123
repl_user=repl
report_script=/usr/local/bin/send
ssh_user=root
user=mha
[server1]
hostname=10.0.0.51
port=3306
[server2]
hostname=10.0.0.52
port=3306
[server3]
hostname=10.0.0.53
port=3306
[binlog1]
no_master=1
hostname=10.0.0.53
master_binlog_dir=/data/mysql/binlog
检查ssh状态:
[root@db03 ~]# masterha_check_ssh --conf=/etc/mha/app1.cnf
检查脚本
[root@db03 /usr/local/bin]# ls -l /usr/local/bin/
检查binlogserver
[root@db03 /usr/local/bin]# ps -ef |grep mysqlbinlog
确认主库,恢复binlog_server
db01 [(none)]>show master status
+------------------+----------+--------------+------------------+------------------------------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+------------------------------------------+
| mysql-bin.000006 | 194 | | | a5260572-c262-11e9-bab1-000c2980e248:1-2 |
+------------------+----------+--------------+------------------+------------------------------------------+
1 row in set (0.00 sec)
[root@db03 /usr/local/bin]# cd /data/mysql/binlog/
[root@db03 /data/mysql/binlog]# rm -rf *
[root@db03 /data/mysql/binlog]# mysqlbinlog -R --host=10.0.0.51 --user=mha --password=mha --raw --stop-never mysql-bin.000006 &
检查主库vip
[root@db01 ~]# ifconfig eth0:1 10.0.0.55/24
检查主从状态
[root@db03 /data/mysql/binlog]# masterha_check_repl --conf=/etc/mha/app1.cnf
启动MHA
nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/mha/app1/manager.log 2>&1 &
[root@db03 ~]# masterha_check_status --conf=/etc/mha/app1.cnf
app1 (pid:8255) is running(0:PING_OK), master:10.0.0.51