DBA之路

MySQL高可用架构MHA原理与搭建

MHA简介

基本环境

MHA安装配置

配置主从复制

安装MHA node

安装MHA manager

配置节点互信

配置MHA

检查状态

启停MHA manager监控

创建自动清除relay log脚本

使用keepalived配置VIP

使用脚本引入VIP

实验验证

1 测试自动切换

2 手动failover

3 在线切换

4 修复宕机master

参考&感谢

MHA简介

MHA（Master High Availability）目前在MySQL高可用方面是一个相对成熟的解决方案，它由日本DeNA公司youshimaton（现就职于Facebook公司）开发，是一套优秀的作为MySQL高可用性环境下故障切换和主从提升的高可用软件。在MySQL故障切换过程中，MHA能做到在0~30秒之内自动完成数据库的故障切换操作，并且在进行故障切换的过程中，MHA能在最大程度上保证数据的一致性，以达到真正意义上的高可用。

该软件由两部分组成：MHA Manager（管理节点）和MHA Node（数据节点）。MHA Manager可以单独部署在一台独立的机器上管理多个master-slave集群，也可以部署在一台slave节点上。MHA Node运行在每台MySQL服务器上，MHA Manager会定时探测集群中的master节点，当master出现故障时，它可以自动将最新数据的slave提升为新的master，然后将所有其他的slave重新指向新的master。整个故障转移过程对应用程序完全透明。

在MHA自动故障切换过程中，MHA试图从宕机的主服务器上保存二进制日志，最大程度的保证数据的不丢失，但这并不总是可行的。例如，如果主服务器硬件故障或无法通过ssh访问，MHA没法保存二进制日志，只进行故障转移而丢失了最新的数据。使用MySQL 5.5的半同步复制，可以大大降低数据丢失的风险。MHA可以与半同步复制结合起来。如果只有一个slave已经收到了最新的二进制日志，MHA可以将最新的二进制日志应用于其他所有的slave服务器上，因此可以保证所有节点的数据一致性。

目前MHA主要支持一主多从的架构，要搭建MHA,要求一个复制集群中必须最少有三台数据库服务器，一主二从，即一台充当master，一台充当备用master，另外一台充当从库，因为至少需要三台服务器，出于机器成本的考虑，淘宝也在该基础上进行了改造，目前淘宝TMHA已经支持一主一从。（出自：《深入浅出MySQL(第二版)》）

MHA架构：

MHA工作原理总结为以下几条：
（1）从宕机崩溃的master保存二进制日志事件（binlog events）;
（2）识别含有最新更新的slave;
（3）应用差异的中继日志(relay log) 到其他slave;
（4）应用从master保存的二进制日志事件(binlog events);
（5）提升一个slave为新master;
（6）使用其他的slave连接新的master进行复制。

Git介绍：https://github.com/yoshinorim/mha4mysql-manager/wiki

基本环境

操作系统 CentOS Linux release 7.5.1804 (Core)
数据库版本 5.7.21-log MySQL Community Server (GPL)
MHA版本 0.58
keepalived版本 keepalived-1.2.12
虚拟ip 10.238.162.100
因资源有限，将MHA manager安装在了MySQL master节点上，生产上不可以这样做。MHA Manager可以单独部署在一台独立的机器上管理多个master-slave集群，也可以部署在一台slave节点上

服务器信息
角色	IP	主机名	server_id	网卡	功能
MHA manager	10.238.162.31	Master			监控复制组
master	10.238.162.31	Master	101	ens160	接受写请求
candidate master	10.238.162.33	Slave01	330602	ens160	接受读请求
slave	10.238.162.34	Slave02	330604	ens160	接受读请求

实验架构如下

涉及的主要配置文件和脚本所在路径

/etc/mha/app1.cnf MHA配置文件
/etc/keepalived/keepalived.conf keepalived配置文件
/usr/local/bin/master_ip_failover 自动故障转移脚本分为keepalived引入VIP和脚本引入VIP
/usr/local/bin/master_ip_online_change 手动故障转移脚本
/root/purge_relay_log.sh 自动清除relay log脚本

MHA安装配置

配置主从复制

主从环境的搭建不在具体演示，大家应该都是轻车熟路，很快就能搭建好一套一主两从的MySQL主从环境。

但需要注意的几点：

1 不要将read_only=1写进从库的配置文件，因为主库宕机时，从库要提升为主库接受写请求 mysql -e"set global read_only=1"
2 主从节点复制的过滤规则要相同，即binlog_do_db 与 binlog_ignore_db 参数主从配置需要相同
3 从节点需要修改配置参数 relay_log_purge=0 ,即关闭中继日志的清除

#采用命令方式将从库设为只读，不要将该参数写进配置文件中
mysql -e"set global read_only=1"
#关闭中继日志的清除
mysql -e"set global relay_log_purge=0"

安装MHA node

所有节点（数据库master，slave，MHA manager节点）都需要安装MHA node。因为MHA manager也需要依赖MHA node。

#下载MHA-node  https://github.com/yoshinorim/mha4mysql-node/releases
#安装依赖，这里遇到的报错大多事因为没有安装依赖环境
yum install -y perl-DBD-MySQL perl-ExtUtils-MakeMaker perl-CPAN
#解压
tar zxf mha4mysql-node-0.58.tar.gz
#移动到/usr/local/mha4mysql-node-0.58目录下
mv mha4mysql-node-0.58 /usr/local/mha4mysql-node-0.58
#编译
cd /usr/local/mha4mysql-node-0.58
perl Makefile.PL
#make并安装
make
make install

安装完成后会在/usr/local/bin/生成以下脚本文件，Node脚本说明：(这些工具通常由MHA Manager的脚本触发，无需人为操作)

save_binary_logs               //保存和复制master的二进制日志
apply_diff_relay_logs          //识别差异的中继日志事件并将其差异的事件应用于其他的slave
filter_mysqlbinlog             //去除不必要的ROLLBACK事件（MHA已不再使用这个工具）
purge_relay_logs               //清除中继日志（不会阻塞SQL线程）

安装MHA manager

只需要在10.238.162.31节点安装MHA-manager

#下载MHA-manager 
https://github.com/yoshinorim/mha4mysql-manager/releases
#安装依赖包
yum install perl-DBD-MySQL  perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager -y
#解压
tar -zxf mha4mysql-manager-0.58.tar.gz
#移动目录
mv mha4mysql-manager-0.58 /usr/local/mha4mysql-manager-0.58
# 编译安装
cd /usr/local/mha4mysql-manager-0.58/
perl Makefile.PL
make
make install

安装完成后，在/usr/local/bin会多出以下相关的命令脚本：

masterha_check_repl //检查MySQL复制健康状况
masterha_check_ssh //检查ssh健康状况
masterha_check_status //检测当前MHA运行状态
masterha_conf_host  //添加或者删除配置的server信息
masterha_manager //启动MHA
masterha_master_monitor //检测master是否宕机
masterha_master_switch //控制故障转移（自动或者手动）
masterha_secondary_check //如果从manager节点发现
masterha_stop //停止MHA

在/usr/local/mha4mysql-manager-0.58/samples/scripts 目录下会有相关脚本

master_ip_failover //自动切换时vip管理的脚本，不是必须，如果我们使用keepalived的，我们可以自己编写脚本完成对vip的管理，比如监控mysql，如果mysql异常，我们停止keepalived就行，这样vip就会自动漂移
master_ip_online_change //在线切换时vip的管理，不是必须，同样可以可以自行编写简单的shell完成
power_manager //故障发生后关闭主机的脚本，不是必须
send_report  //因故障切换后发送报警的脚本，不是必须，可自行编写简单的shell完成

安装部分参考Git文档 https://github.com/yoshinorim/mha4mysql-manager/wiki/Installation

配置节点互信

后面为方便阐述，用31代表10.238.162.33服务器。

#所有节点
mkdir ~/.ssh
cd  ~/.ssh

#生成公钥和私钥文件，一直回车使用默认即可
ssh-keygen -t rsa

#10.238.162.33服务器上，将生成的公钥复制到10.238.162.31下
scp id_rsa.pub  [email protected]:/root/.ssh/id_rsa.pub_node1

#10.238.162.34服务器上，将生成的公钥复制到10.238.162.31下
scp id_rsa.pub  [email protected]:/root/.ssh/id_rsa.pub_node2

#10.238.162.31服务器上，将31，33，34服务器的公钥追加到认证文件中
cat id_rsa.pub id_rsa.pub_node1 id_rsa.pub_node2 >> authorized_keys

#10.238.162.31服务器上，将认证文件scp到33和34服务器一份
scp authorized_keys [email protected]:/root/.ssh/
scp authorized_keys [email protected]:/root/.ssh/

#所有节点 验证ssh免密登录，第一次可能需要输入yes，后续不需要输入密码
#31服务器
ssh 10.238.162.33 date
ssh 10.238.162.34 date
#33服务器
ssh 10.238.162.31 date
ssh 10.238.162.34 date
#34服务器
ssh 10.238.162.31 date
ssh 10.238.162.33 date

配置MHA

为保证MHA正常工作，需要配置MHA的配置文件，为参数设置合理正确的值，这些参数包括服务器IP，数据库用户名密码，工作目录与日志等。 MHA manager有两个配置文件模板，在路径 $MHA_BASE/samples/conf/ 下的app1.cnf 和 masterha_default.cnf。

app1.cnf #是对某个复制组的配置文件。
masterha_default.cnf #MHA manager的全局配置文件，可以通过这一个配置文件管理多个复制组。

app1.cnf配置文件，这里先注释掉故障切换脚本，修改好脚本后再打开注释。

创建数据库监控用户：

GRANT ALL PRIVILEGES ON *.* TO 'mha'@'%' identified by '123456';
flush privileges;

cp /usr/local/mha4mysql-manager-0.58/samples/conf/app1.cnf /etc/mha/

# vim /etc/mha/app1.cnf

[server default]
#mha manager日志文件
manager_log=/server/mha/app1/manager.log
#manager工作目录
manager_workdir=/server/mha/app1
#master节点存放binlog日志路径,以便MHA找到binlog，这里就是MySQL的数据目录
master_binlog_dir=/server/mysql_data
#发生切换时slave节点binlog日志存放路径
remote_workdir=/server/mha_tmp
#自动切换脚本
master_ip_failover_script=/usr/local/bin/master_ip_failover
#手动切换脚本
master_ip_online_change_script=/usr/bin/master_ip_online_change

#一旦MHA到31的监控之间出现问题，MHA Manager将会尝试从33和34登录到31。secondary_check_script=/usr/local/bin/masterha_secondary_check -s 10.238.162.33 -s 10.238.162.34 --user=root --port=22 --master_host=10.238.162.31 --master_port=3306
#监控主节点时间间隔
ping_interval=3
#设置故障发生后关闭故障主机脚本（该脚本的主要作用是关闭主机放在发生脑裂,这里没有使用）
#shutdown_script=""
#数据库监控用户
user=mha
password=123456
#复制用户
repl_password=repl
repl_user=repl
#ssh登录用户
ssh_user=root

[server1]
hostname=10.238.162.31
port=3306

[server2]
#设置为候选master，如果设置该参数以后，发生主从切换以后将会将此从库提升为主库，即使这个主库不是集群中事件最新的slave
candidate_master=1
#默认情况下如果一个slave落后master 100M的relay logs的话，MHA将不会选择该slave作为一个新的master，因为对于这个slave的恢复需要花费很长时间，通过设置check_repl_delay=0,MHA触发切换在选择一个新的master的时候将会忽略复制延时，这个参数对于设置了candidate_master=1的主机非常有用，因为这个候选主在切换的过程中一定是新的master
check_repl_delay=0
hostname=10.238.162.33
port=3306

[server3]
hostname=10.238.162.34
port=3306

检查状态

检查ssh状态，成功！

masterha_check_ssh --conf=/etc/mha/app1.cnf

检查集群的健康状态

失败原因大该有以下几点，这个总结下：

1 写binlog和复制过滤规则主从必须相同。

2 找不到命令问题，解决方法，为命令建立软连接。

ln -s /usr/local/mysql/bin/mysqlbinlog /usr/local/bin/

n -s /usr/local/mysql/bin/mysql /usr/local/bin/

masterha_check_repl --conf=/etc/mha/app1.cnf

[root@Master ~]# masterha_check_repl --conf=/etc/mha/app1.cnf
Mon Jan 14 20:56:21 2019 - [info] Reading default configuration from /etc/masterha_default.cnf..
Mon Jan 14 20:56:21 2019 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Mon Jan 14 20:56:21 2019 - [info] Reading server configuration from /etc/mha/app1.cnf..
Mon Jan 14 20:56:21 2019 - [info] MHA::MasterMonitor version 0.58.
Mon Jan 14 20:56:23 2019 - [info] GTID failover mode = 0
Mon Jan 14 20:56:23 2019 - [info] Dead Servers:
Mon Jan 14 20:56:23 2019 - [info] Alive Servers:
Mon Jan 14 20:56:23 2019 - [info] 10.238.162.31(10.238.162.31:3306)
Mon Jan 14 20:56:23 2019 - [info] 10.238.162.33(10.238.162.33:3306)
Mon Jan 14 20:56:23 2019 - [info] 10.238.162.34(10.238.162.34:3306)
Mon Jan 14 20:56:23 2019 - [info] Alive Slaves:
Mon Jan 14 20:56:23 2019 - [info] 10.238.162.33(10.238.162.33:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
Mon Jan 14 20:56:23 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Mon Jan 14 20:56:23 2019 - [info] 10.238.162.34(10.238.162.34:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
Mon Jan 14 20:56:23 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Mon Jan 14 20:56:23 2019 - [info] Current Alive Master: 10.238.162.31(10.238.162.31:3306)
Mon Jan 14 20:56:23 2019 - [info] Checking slave configurations..
Mon Jan 14 20:56:23 2019 - [info] read_only=1 is not set on slave 10.238.162.33(10.238.162.33:3306).
Mon Jan 14 20:56:23 2019 - [info] read_only=1 is not set on slave 10.238.162.34(10.238.162.34:3306).
Mon Jan 14 20:56:23 2019 - [warning] relay_log_purge=0 is not set on slave 10.238.162.34(10.238.162.34:3306).
Mon Jan 14 20:56:23 2019 - [info] Checking replication filtering settings..
Mon Jan 14 20:56:23 2019 - [info] binlog_do_db= , binlog_ignore_db=
Mon Jan 14 20:56:23 2019 - [info] Replication filtering check ok.
Mon Jan 14 20:56:23 2019 - [info] GTID (with auto-pos) is not supported
Mon Jan 14 20:56:23 2019 - [info] Starting SSH connection tests..
Mon Jan 14 20:56:25 2019 - [info] All SSH connection tests passed successfully.
Mon Jan 14 20:56:25 2019 - [info] Checking MHA Node version..
Mon Jan 14 20:56:26 2019 - [info] Version check ok.
Mon Jan 14 20:56:26 2019 - [info] Checking SSH publickey authentication settings on the current master..
Mon Jan 14 20:56:26 2019 - [info] HealthCheck: SSH to 10.238.162.31 is reachable.
Mon Jan 14 20:56:26 2019 - [info] Master MHA Node version is 0.58.
Mon Jan 14 20:56:26 2019 - [info] Checking recovery script configurations on 10.238.162.31(10.238.162.31:3306)..
Mon Jan 14 20:56:26 2019 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/server/mysql_data --output_file=/server/mha_tmp/save_binary_logs_test --manager_version=0.58 --start_file=mysql-bin.000010
Mon Jan 14 20:56:26 2019 - [info] Connecting to [email protected](10.238.162.31:22)..
Creating /server/mha_tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /server/mysql_data, up to mysql-bin.000010
Mon Jan 14 20:56:27 2019 - [info] Binlog setting check done.
Mon Jan 14 20:56:27 2019 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Mon Jan 14 20:56:27 2019 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mha' --slave_host=10.238.162.33 --slave_ip=10.238.162.33 --slave_port=3306 --workdir=/server/mha_tmp --target_version=5.7.21-log --manager_version=0.58 --relay_dir=/server/mysql_data --current_relay_log=relay-bin.000005 --slave_pass=xxx
Mon Jan 14 20:56:27 2019 - [info] Connecting to [email protected](10.238.162.33:22)..
Checking slave recovery environment settings..
Relay log found at /server/mysql_data, up to relay-bin.000008
Temporary relay log file is /server/mysql_data/relay-bin.000008
Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Mon Jan 14 20:56:27 2019 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mha' --slave_host=10.238.162.34 --slave_ip=10.238.162.34 --slave_port=3306 --workdir=/server/mha_tmp --target_version=5.7.21-log --manager_version=0.58 --relay_log_info=/server/mysql_data/relay-log.info --relay_dir=/server/mysql_data/ --slave_pass=xxx
Mon Jan 14 20:56:27 2019 - [info] Connecting to [email protected](10.238.162.34:22)..
Checking slave recovery environment settings..
Opening /server/mysql_data/relay-log.info ... ok.
Relay log found at /server/mysql_data, up to slave02-relay-bin.000006
Temporary relay log file is /server/mysql_data/slave02-relay-bin.000006
Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Mon Jan 14 20:56:28 2019 - [info] Slaves settings check done.
Mon Jan 14 20:56:28 2019 - [info]
10.238.162.31(10.238.162.31:3306) (current master)
+--10.238.162.33(10.238.162.33:3306)
+--10.238.162.34(10.238.162.34:3306)

Mon Jan 14 20:56:28 2019 - [info] Checking replication health on 10.238.162.33..
Mon Jan 14 20:56:28 2019 - [info] ok.
Mon Jan 14 20:56:28 2019 - [info] Checking replication health on 10.238.162.34..
Mon Jan 14 20:56:28 2019 - [info] ok.
Mon Jan 14 20:56:28 2019 - [warning] master_ip_failover_script is not defined.
Mon Jan 14 20:56:28 2019 - [warning] shutdown_script is not defined.
Mon Jan 14 20:56:28 2019 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.

启停MHA manager监控

#查看MHA manager监控状态，这里没有运行
# masterha_check_status --conf=/etc/mha/app1.cnf
app1 is stopped(2:NOT_RUNNING).

#启动MHA监控 --remove_dead_master_conf --ignore_last_failover
nohup  masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover  &

# 停止MHA监控
masterha_stop --conf=/etc/mha/app1.cnf

创建自动清除relay log脚本

MHA在发生切换的过程中，从库的恢复过程中依赖于relay log的相关信息，所以这里要将relay log的自动清除设置为OFF，采用手动清除relay log的方式。在默认情况下，从服务器上的中继日志会在SQL线程执行完毕后被自动删除。但是在MHA环境中，这些中继日志在恢复其他从服务器时可能会被用到，因此需要禁用中继日志的自动删除功能。定期清除中继日志需要考虑到复制延时的问题。在ext3的文件系统下，删除大的文件需要一定的时间，会导致严重的复制延时。为了避免复制延时，需要暂时为中继日志创建硬链接，因为在linux系统中通过硬链接删除大文件速度会很快。（在mysql数据库中，删除大表时，通常也采用建立硬链接的方式）

MHA节点中包含了pure_relay_logs命令工具，它可以为中继日志创建硬链接，执行SET GLOBAL relay_log_purge=1,等待几秒钟以便SQL线程切换到新的中继日志，再执行SET GLOBAL relay_log_purge=0。

清理日志脚本

# cat purge_relay_log.sh

#!/bin/bash
#数据库用户名密码端口
user=root
passwd=root
port=3306
#脚本日志存放路径
log_dir='/server/mha/app1'
#指定创建relay log的硬链接的位置，默认是/var/tmp。由于系统不同分区创建硬链接文件会失败，故需要执行硬链接具体位置，成功执行脚本后，硬链接的中继日志文件被删除。
work_dir='/server/mha'
#删除中继日志脚本
purge='/usr/local/bin/purge_relay_logs'

if [ ! -d $log_dir ]
then
mkdir $log_dir -p
fi

#--disable_relay_log_purge ：默认情况下，如果relay_log_purge=1，脚本会什么都不清理，自动退出。通过设定这个参数，当relay_log_purge=1的情况下会将relay_log_purge设置为0。清理relay log之后，最后将参数设置为OFF。
$purge --user=$user --password=$passwd --disable_relay_log_purge --port=$port --workdir=$work_dir >> $log_dir/purge_relay_logs.log 2>&1

并授予可执行权限

chmod +x purge_relay_log.sh

添加至定时任务

00 03 * * * /bin/bash /root/purge_relay_log.sh

手动执行脚本查看输出：看大神的博客说建立硬链接删除日志，这里并没有展示出来。

# /usr/local/bin/purge_relay_logs --user=root --password=root --disable_relay_log_purge --port=3306 --workdir=/server/mha
2019-01-17 16:07:07: purge_relay_logs script started.
Found relay_log.info: /server/mysql_data/relay-log.info
Opening /server/mysql_data/slave02-relay-bin.000003 ..
Opening /server/mysql_data/slave02-relay-bin.000004 ..
Executing SET GLOBAL relay_log_purge=1; FLUSH LOGS; sleeping a few seconds so that SQL thread can delete older relay log files (if it keeps up); SET GLOBAL relay_log_purge=0; .. ok.
2019-01-17 16:07:10: All relay log purging operations succeeded.

使用keepalived配置VIP

vip配置可以采用两种方式，一种通过keepalived的方式管理虚拟ip的浮动；另外一种通过脚本方式启动虚拟ip的方式（即不需要keepalived或者heartbeat类似的软件）

安装keepalived ，安装keepalived时遇到挺多问题，这里选择1.2.12版本，安装时没有问题的。在master31 和 candidate master 33上安装

#下载源码包
wget http://www.keepalived.org/software/keepalived-1.2.12.tar.gz
#解压
tar zxf keepalived-1.2.12.tar.gz 
cd keepalived-1.2.12
#编译安装
./configure --prefix=/usr/local/keepalived
make && make install
#拷贝启动命令
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/keepalived
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
#拷贝配置文件
mkdir /etc/keepalived
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/

配置文件如下：其中router_id MySQL HA表示设定keepalived组的名称，将10.238.162.100这个虚拟ip绑定到该主机的ens160网卡上，并且设置了状态为backup模式，将keepalived的模式设置为非抢占模式（nopreempt），priority 150表示设置的优先级为150。下面的配置略有不同，但是都是一个意思。

keepalived配置文件

#31服务器

# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id MySQL-HA
}

vrrp_instance VI_1 {
state BACKUP
interface ens160
virtual_router_id 51
priority 150
advert_int 1
nopreempt

authentication {
auth_type PASS
auth_pass 1111
}

virtual_ipaddress {
10.238.162.100
}
}

#33服务器

# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id MySQL-HA
}

vrrp_instance VI_1 {
state BACKUP
interface ens160
virtual_router_id 51
priority 120
advert_int 1
nopreempt

authentication {
auth_type PASS
auth_pass 1111
}

virtual_ipaddress {
10.238.162.100
}
}

31服务器启动keepalived

# /etc/init.d/keepalived start
Starting keepalived (via systemctl):                       [  确定  ]

日志输出如下 tail -100 /var/log/messages

33服务器启动keepalived

/etc/init.d/keepalived start

日志输出部分

查看vip绑定状况：可以看到虚拟IP10.238.162.100已经绑定到了10.238.162.31服务器的ens160网卡上。

MHA引入keepalived

MySQL服务进程挂掉时通过MHA 停止keepalived:

要想把keepalived服务引入MHA，我们只需要修改切换是触发的脚本文件master_ip_failover即可，在该脚本中添加在master发生宕机时对keepalived的处理。

编辑自动故障转移脚本/usr/local/bin/master_ip_failover，修改后如下（10.238.162.31）操作：

# cat master_ip_failover
#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';

use Getopt::Long;

my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);

my $vip = '10.238.162.100';
my $ssh_start_vip = "/etc/init.d/keepalived start";
my $ssh_stop_vip = "/etc/init.d/keepalived stop";

GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);

exit &main();

sub main {

print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

if ( $command eq "stop" || $command eq "stopssh" ) {

my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {

my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
exit 0;
}
else {
&usage();
exit 1;
}
}
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}

sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}

将mha配置文件中#master_ip_failover_script= /usr/local/bin/master_ip_failover打开

[root@Master bin]# grep 'master_ip_failover_script' /etc/mha/app1.cnf
master_ip_failover_script=/usr/local/bin/master_ip_failover

再次执行检查

masterha_check_repl --conf=/etc/mha/app1.cnf

MySQL Replication Health is OK.

使用脚本引入VIP

通过脚本的方式管理VIP。这里是修改/usr/local/bin/master_ip_failover。修改完成后内容如下，而且如果使用脚本管理vip的话，需要手动在master服务器上绑定一个vip。

/sbin/ifconfig ens160:1 10.238.162.100/24

通过脚本引入VIP的实验不在演示，我自己做了自动切换的实验是可以的。大家可以自己做做实验看看日志输出理解原理。脚本如下供参考，需要根据自己环境修改VIP与网卡等信息

# cat master_ip_failover
#!/usr/bin/env perl

use strict;
use warnings FATAL => 'all';

use Getopt::Long;

my (
$command, $ssh_user, $orig_master_host, $orig_master_ip,
$orig_master_port, $new_master_host, $new_master_ip, $new_master_port
);

my $vip = '10.238.162.100/24';
my $key = '1';
my $ssh_start_vip = "/sbin/ifconfig ens160:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens160:$key down";

GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
);

exit &main();

sub main {

print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";

if ( $command eq "stop" || $command eq "stopssh" ) {

my $exit_code = 1;
eval {
print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {

my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
exit 0;
}
else {
&usage();
exit 1;
}
}

sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
sub stop_vip() {
return 0 unless ($ssh_user);
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}

sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}

实验验证

1 测试自动切换

在31服务器上停止MySQL进程

# /etc/init.d/mysqld stop
Shutting down MySQL............ SUCCESS!

查看MHA manager日志（建议阅读切换日志，明白MHA工作原理） 31上的MySQL服务挂掉，33成为新的master，34成为33的slave。

cat /server/mha/app1/manager.log

Wed Jan 16 16:41:23 2019 - [warning] Got error on MySQL select ping: 2006 (MySQL server has gone away)
Wed Jan 16 16:41:23 2019 - [info] Executing secondary network check script: /usr/local/bin/masterha_secondary_check -s 10.238
.162.33 -s 10.238.162.34 --user=root --port=22 --master_host=10.238.162.31 --master_port=3306 --user=root --master_host=10.
238.162.31 --master_ip=10.238.162.31 --master_port=3306 --master_user=mha --master_password=123456 --ping_type=SELECT
Wed Jan 16 16:41:23 2019 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/ser
ver/mysql_data --output_file=/server/mha_tmp/save_binary_logs_test --manager_version=0.58 --binlog_prefix=mysql-bin
Wed Jan 16 16:41:23 2019 - [info] HealthCheck: SSH to 10.238.162.31 is reachable.
Monitoring server 10.238.162.33 is reachable, Master is not reachable from 10.238.162.33. OK.
Monitoring server 10.238.162.34 is reachable, Master is not reachable from 10.238.162.34. OK.
Wed Jan 16 16:41:24 2019 - [info] Master is not reachable from all other monitoring servers. Failover should start.
Wed Jan 16 16:41:26 2019 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.238.162.31' (111)
)
Wed Jan 16 16:41:26 2019 - [warning] Connection failed 2 time(s)..
Wed Jan 16 16:41:29 2019 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.238.162.31' (111)
)
Wed Jan 16 16:41:29 2019 - [warning] Connection failed 3 time(s)..
Wed Jan 16 16:41:32 2019 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '10.238.162.31' (111)
)
Wed Jan 16 16:41:32 2019 - [warning] Connection failed 4 time(s)..
Wed Jan 16 16:41:32 2019 - [warning] Master is not reachable from health checker!
Wed Jan 16 16:41:32 2019 - [warning] Master 10.238.162.31(10.238.162.31:3306) is not reachable!
Wed Jan 16 16:41:32 2019 - [warning] SSH is reachable.
Wed Jan 16 16:41:32 2019 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf
and /etc/mha/app1.cnf again, and trying to connect to all servers to check server status..
Wed Jan 16 16:41:32 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed Jan 16 16:41:32 2019 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Wed Jan 16 16:41:32 2019 - [info] Reading server configuration from /etc/mha/app1.cnf..
Wed Jan 16 16:41:33 2019 - [info] GTID failover mode = 0
Wed Jan 16 16:41:33 2019 - [info] Dead Servers:
Wed Jan 16 16:41:33 2019 - [info] 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 16:41:33 2019 - [info] Alive Servers:
Wed Jan 16 16:41:33 2019 - [info] 10.238.162.33(10.238.162.33:3306)
Wed Jan 16 16:41:33 2019 - [info] 10.238.162.34(10.238.162.34:3306)
Wed Jan 16 16:41:33 2019 - [info] Alive Slaves:
Wed Jan 16 16:41:33 2019 - [info] 10.238.162.33(10.238.162.33:3306) Version=5.7.21-log (oldest major version between slave
s) log-bin:enabled
Wed Jan 16 16:41:33 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 16:41:33 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Wed Jan 16 16:41:33 2019 - [info] 10.238.162.34(10.238.162.34:3306) Version=5.7.21-log (oldest major version between slave
s) log-bin:enabled
Wed Jan 16 16:41:33 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 16:41:33 2019 - [info] Checking slave configurations..
Wed Jan 16 16:41:33 2019 - [info] read_only=1 is not set on slave 10.238.162.34(10.238.162.34:3306).
Wed Jan 16 16:41:33 2019 - [info] Checking replication filtering settings..
Wed Jan 16 16:41:33 2019 - [info] Replication filtering check ok.
Wed Jan 16 16:41:33 2019 - [info] Master is down!
Wed Jan 16 16:41:33 2019 - [info] Terminating monitoring script.
Wed Jan 16 16:41:33 2019 - [info] Got exit code 20 (Master dead).
Wed Jan 16 16:41:33 2019 - [info] MHA::MasterFailover version 0.58.
Wed Jan 16 16:41:33 2019 - [info] Starting master failover.
Wed Jan 16 16:41:33 2019 - [info]
Wed Jan 16 16:41:33 2019 - [info] * Phase 1: Configuration Check Phase..
Wed Jan 16 16:41:33 2019 - [info]
Wed Jan 16 16:41:34 2019 - [info] GTID failover mode = 0
Wed Jan 16 16:41:34 2019 - [info] Dead Servers:
Wed Jan 16 16:41:34 2019 - [info] 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 16:41:34 2019 - [info] Checking master reachability via MySQL(double check)...
Wed Jan 16 16:41:34 2019 - [info] ok.
Wed Jan 16 16:41:34 2019 - [info] Alive Servers:
Wed Jan 16 16:41:34 2019 - [info] 10.238.162.33(10.238.162.33:3306)
Wed Jan 16 16:41:34 2019 - [info] 10.238.162.34(10.238.162.34:3306)
Wed Jan 16 16:41:34 2019 - [info] Alive Slaves:
Wed Jan 16 16:41:34 2019 - [info] 10.238.162.33(10.238.162.33:3306) Version=5.7.21-log (oldest major version between slave
s) log-bin:enabled
Wed Jan 16 16:41:34 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 16:41:34 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Wed Jan 16 16:41:34 2019 - [info] 10.238.162.34(10.238.162.34:3306) Version=5.7.21-log (oldest major version between slave
s) log-bin:enabled
Wed Jan 16 16:41:34 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 16:41:34 2019 - [info] Starting Non-GTID based failover.
Wed Jan 16 16:41:34 2019 - [info]
Wed Jan 16 16:41:34 2019 - [info] ** Phase 1: Configuration Check Phase completed.
Wed Jan 16 16:41:34 2019 - [info]
Wed Jan 16 16:41:34 2019 - [info] * Phase 2: Dead Master Shutdown Phase..
Wed Jan 16 16:41:34 2019 - [info]
Wed Jan 16 16:41:34 2019 - [info] Forcing shutdown so that applications never connect to the current master..
Wed Jan 16 16:41:34 2019 - [info] Executing master IP deactivation script:
Wed Jan 16 16:41:34 2019 - [info] /usr/local/bin/master_ip_failover --orig_master_host=10.238.162.31 --orig_master_ip=10.23
8.162.31 --orig_master_port=3306 --command=stopssh --ssh_user=root

IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===

Disabling the VIP on old master: 10.238.162.31
Wed Jan 16 16:41:35 2019 - [info] done.
Wed Jan 16 16:41:35 2019 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Wed Jan 16 16:41:35 2019 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Wed Jan 16 16:41:35 2019 - [info]
Wed Jan 16 16:41:35 2019 - [info] * Phase 3: Master Recovery Phase..
Wed Jan 16 16:41:35 2019 - [info]
Wed Jan 16 16:41:35 2019 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Wed Jan 16 16:41:35 2019 - [info]
Wed Jan 16 16:41:35 2019 - [info] The latest binary log file/position on all slaves is mysql-bin.000010:306
Wed Jan 16 16:41:35 2019 - [info] Latest slaves (Slaves that received relay log files to the latest):
Wed Jan 16 16:41:35 2019 - [info] 10.238.162.33(10.238.162.33:3306) Version=5.7.21-log (oldest major version between slave
s) log-bin:enabled
Wed Jan 16 16:41:35 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 16:41:35 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Wed Jan 16 16:41:35 2019 - [info] 10.238.162.34(10.238.162.34:3306) Version=5.7.21-log (oldest major version between slave
s) log-bin:enabled
Wed Jan 16 16:41:35 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 16:41:35 2019 - [info] The oldest binary log file/position on all slaves is mysql-bin.000010:306
Wed Jan 16 16:41:35 2019 - [info] Oldest slaves:
Wed Jan 16 16:41:35 2019 - [info] 10.238.162.33(10.238.162.33:3306) Version=5.7.21-log (oldest major version between slave
s) log-bin:enabled
Wed Jan 16 16:41:35 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 16:41:35 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Wed Jan 16 16:41:35 2019 - [info] 10.238.162.34(10.238.162.34:3306) Version=5.7.21-log (oldest major version between slave
s) log-bin:enabled
Wed Jan 16 16:41:35 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 16:41:35 2019 - [info]
Wed Jan 16 16:41:35 2019 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Wed Jan 16 16:41:35 2019 - [info]
Wed Jan 16 16:41:35 2019 - [info] Fetching dead master's binary logs..
Wed Jan 16 16:41:35 2019 - [info] Executing command on the dead master 10.238.162.31(10.238.162.31:3306): save_binary_logs --
command=save --start_file=mysql-bin.000010 --start_pos=306 --binlog_dir=/server/mysql_data --output_file=/server/mha_tmp/sav
ed_master_binlog_from_10.238.162.31_3306_20190116164133.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.
58
Creating /server/mha_tmp if not exists.. ok.
Concat binary/relay logs from mysql-bin.000010 pos 306 to mysql-bin.000010 EOF into /server/mha_tmp/saved_master_binlog_from
_10.238.162.31_3306_20190116164133.binlog ..
Binlog Checksum enabled
Dumping binlog format description event, from position 0 to 154.. ok.
Dumping effective binlog data from /server/mysql_data/mysql-bin.000010 position 306 to tail(329).. ok.
Binlog Checksum enabled
Concat succeeded.
Wed Jan 16 16:41:36 2019 - [info] scp from [email protected]:/server/mha_tmp/saved_master_binlog_from_10.238.162.31_3306_201
90116164133.binlog to local:/server/mha/app1/saved_master_binlog_from_10.238.162.31_3306_20190116164133.binlog succeeded.
Wed Jan 16 16:41:36 2019 - [info] HealthCheck: SSH to 10.238.162.33 is reachable.
Wed Jan 16 16:41:37 2019 - [info] HealthCheck: SSH to 10.238.162.34 is reachable.
Wed Jan 16 16:41:37 2019 - [info]
Wed Jan 16 16:41:37 2019 - [info] * Phase 3.3: Determining New Master Phase..
Wed Jan 16 16:41:37 2019 - [info]
Wed Jan 16 16:41:37 2019 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Wed Jan 16 16:41:37 2019 - [info] All slaves received relay logs to the same position. No need to resync each other.
Wed Jan 16 16:41:37 2019 - [info] Searching new master from slaves..
Wed Jan 16 16:41:37 2019 - [info] Candidate masters from the configuration file:
Wed Jan 16 16:41:37 2019 - [info] 10.238.162.33(10.238.162.33:3306) Version=5.7.21-log (oldest major version between slave
s) log-bin:enabled
Wed Jan 16 16:41:37 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 16:41:37 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Wed Jan 16 16:41:37 2019 - [info] Non-candidate masters:
Wed Jan 16 16:41:37 2019 - [info] Searching from candidate_master slaves which have received the latest relay log events..
Wed Jan 16 16:41:37 2019 - [info] New master is 10.238.162.33(10.238.162.33:3306)
Wed Jan 16 16:41:37 2019 - [info] Starting master failover..
Wed Jan 16 16:41:37 2019 - [info]
From:
10.238.162.31(10.238.162.31:3306) (current master)
+--10.238.162.33(10.238.162.33:3306)
+--10.238.162.34(10.238.162.34:3306)

To:
10.238.162.33(10.238.162.33:3306) (new master)
+--10.238.162.34(10.238.162.34:3306)
Wed Jan 16 16:41:37 2019 - [info]
Wed Jan 16 16:41:37 2019 - [info] * Phase 3.4: New Master Diff Log Generation Phase..
Wed Jan 16 16:41:37 2019 - [info]
Wed Jan 16 16:41:37 2019 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Wed Jan 16 16:41:37 2019 - [info] Sending binlog..
Wed Jan 16 16:41:37 2019 - [info] scp from local:/server/mha/app1/saved_master_binlog_from_10.238.162.31_3306_20190116164133.
binlog to [email protected]:/server/mha_tmp/saved_master_binlog_from_10.238.162.31_3306_20190116164133.binlog succeeded.
Wed Jan 16 16:41:37 2019 - [info]
Wed Jan 16 16:41:37 2019 - [info] * Phase 3.5: Master Log Apply Phase..
Wed Jan 16 16:41:37 2019 - [info]
Wed Jan 16 16:41:37 2019 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Wed Jan 16 16:41:37 2019 - [info] Starting recovery on 10.238.162.33(10.238.162.33:3306)..
Wed Jan 16 16:41:37 2019 - [info] Generating diffs succeeded.
Wed Jan 16 16:41:37 2019 - [info] Waiting until all relay logs are applied.
Wed Jan 16 16:41:37 2019 - [info] done.
Wed Jan 16 16:41:37 2019 - [info] Getting slave status..
Wed Jan 16 16:41:37 2019 - [info] This slave(10.238.162.33)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000
010:306). No need to recover from Exec_Master_Log_Pos.
Wed Jan 16 16:41:37 2019 - [info] Connecting to the target slave host 10.238.162.33, running recover script..
Wed Jan 16 16:41:37 2019 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='mha' --slave_host=10
.238.162.33 --slave_ip=10.238.162.33 --slave_port=3306 --apply_files=/server/mha_tmp/saved_master_binlog_from_10.238.162.31_
3306_20190116164133.binlog --workdir=/server/mha_tmp --target_version=5.7.21-log --timestamp=20190116164133 --handle_raw_binl
og=1 --disable_log_bin=0 --manager_version=0.58 --slave_pass=xxx
Wed Jan 16 16:41:38 2019 - [info]
MySQL client version is 5.7.21. Using --binary-mode.
Applying differential binary/relay log files /server/mha_tmp/saved_master_binlog_from_10.238.162.31_3306_20190116164133.binlo
g on 10.238.162.33:3306. This may take long time...
Applying log files succeeded.
Wed Jan 16 16:41:38 2019 - [info] All relay logs were successfully applied.
Wed Jan 16 16:41:38 2019 - [info] Getting new master's binlog name and position..
Wed Jan 16 16:41:38 2019 - [info] Slave-mysql-bin.000004:458
Wed Jan 16 16:41:38 2019 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO
MASTER_HOST='10.238.162.33', MASTER_PORT=3306, MASTER_LOG_FILE='Slave-mysql-bin.000004', MASTER_LOG_POS=458, MASTER_USER='re
pl', MASTER_PASSWORD='xxx';
Wed Jan 16 16:41:38 2019 - [info] Executing master IP activate script:
Wed Jan 16 16:41:38 2019 - [info] /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=10.2
38.162.31 --orig_master_ip=10.238.162.31 --orig_master_port=3306 --new_master_host=10.238.162.33 --new_master_ip=10.238.162.3
3 --new_master_port=3306 --new_master_user='mha' --new_master_password=xxx
Unknown option: new_master_user
Unknown option: new_master_password

IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===

Enabling the VIP - 10.238.162.100 on the new master - 10.238.162.33
Wed Jan 16 16:41:38 2019 - [info] OK.
Wed Jan 16 16:41:38 2019 - [info] Setting read_only=0 on 10.238.162.33(10.238.162.33:3306)..
Wed Jan 16 16:41:38 2019 - [info] ok.
Wed Jan 16 16:41:38 2019 - [info] ** Finished master recovery successfully.
Wed Jan 16 16:41:38 2019 - [info] * Phase 3: Master Recovery Phase completed.
Wed Jan 16 16:41:38 2019 - [info]
Wed Jan 16 16:41:38 2019 - [info] * Phase 4: Slaves Recovery Phase..
Wed Jan 16 16:41:38 2019 - [info]
Wed Jan 16 16:41:38 2019 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Wed Jan 16 16:41:38 2019 - [info]
Wed Jan 16 16:41:38 2019 - [info] -- Slave diff file generation on host 10.238.162.34(10.238.162.34:3306) started, pid: 28388
. Check tmp log /server/mha/app1/10.238.162.34_3306_20190116164133.log if it takes time..
Wed Jan 16 16:41:39 2019 - [info]
Wed Jan 16 16:41:39 2019 - [info] Log messages from 10.238.162.34 ...
Wed Jan 16 16:41:39 2019 - [info]
Wed Jan 16 16:41:38 2019 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Wed Jan 16 16:41:39 2019 - [info] End of log messages from 10.238.162.34.
Wed Jan 16 16:41:39 2019 - [info] -- 10.238.162.34(10.238.162.34:3306) has the latest relay log events.
Wed Jan 16 16:41:39 2019 - [info] Generating relay diff files from the latest slave succeeded.
Wed Jan 16 16:41:39 2019 - [info]
Wed Jan 16 16:41:39 2019 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Wed Jan 16 16:41:39 2019 - [info]
Wed Jan 16 16:41:39 2019 - [info] -- Slave recovery on host 10.238.162.34(10.238.162.34:3306) started, pid: 28390. Check tmp
log /server/mha/app1/10.238.162.34_3306_20190116164133.log if it takes time..
Wed Jan 16 16:41:40 2019 - [info]
Wed Jan 16 16:41:40 2019 - [info] Log messages from 10.238.162.34 ...
Wed Jan 16 16:41:40 2019 - [info]
Wed Jan 16 16:41:39 2019 - [info] Sending binlog..
Wed Jan 16 16:41:39 2019 - [info] scp from local:/server/mha/app1/saved_master_binlog_from_10.238.162.31_3306_20190116164133.
binlog to [email protected]:/server/mha_tmp/saved_master_binlog_from_10.238.162.31_3306_20190116164133.binlog succeeded.
Wed Jan 16 16:41:39 2019 - [info] Starting recovery on 10.238.162.34(10.238.162.34:3306)..
Wed Jan 16 16:41:39 2019 - [info] Generating diffs succeeded.
Wed Jan 16 16:41:39 2019 - [info] Waiting until all relay logs are applied.
Wed Jan 16 16:41:39 2019 - [info] done.
Wed Jan 16 16:41:39 2019 - [info] Getting slave status..
Wed Jan 16 16:41:39 2019 - [info] This slave(10.238.162.34)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000
010:306). No need to recover from Exec_Master_Log_Pos.
Wed Jan 16 16:41:39 2019 - [info] Connecting to the target slave host 10.238.162.34, running recover script..
Wed Jan 16 16:41:39 2019 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='mha' --slave_host=10
.238.162.34 --slave_ip=10.238.162.34 --slave_port=3306 --apply_files=/server/mha_tmp/saved_master_binlog_from_10.238.162.31_
3306_20190116164133.binlog --workdir=/server/mha_tmp --target_version=5.7.21-log --timestamp=20190116164133 --handle_raw_binl
og=1 --disable_log_bin=0 --manager_version=0.58 --slave_pass=xxx
Wed Jan 16 16:41:40 2019 - [info]
MySQL client version is 5.7.21. Using --binary-mode.
Applying differential binary/relay log files /server/mha_tmp/saved_master_binlog_from_10.238.162.31_3306_20190116164133.binlo
g on 10.238.162.34:3306. This may take long time...
Applying log files succeeded.
Wed Jan 16 16:41:40 2019 - [info] All relay logs were successfully applied.
Wed Jan 16 16:41:40 2019 - [info] Resetting slave 10.238.162.34(10.238.162.34:3306) and starting replication from the new ma
ster 10.238.162.33(10.238.162.33:3306)..
Wed Jan 16 16:41:40 2019 - [info] Executed CHANGE MASTER.
Wed Jan 16 16:41:40 2019 - [info] Slave started.
Wed Jan 16 16:41:40 2019 - [info] End of log messages from 10.238.162.34.
Wed Jan 16 16:41:40 2019 - [info] -- Slave recovery on host 10.238.162.34(10.238.162.34:3306) succeeded.
Wed Jan 16 16:41:40 2019 - [info] All new slave servers recovered successfully.
Wed Jan 16 16:41:40 2019 - [info]
Wed Jan 16 16:41:40 2019 - [info] * Phase 5: New master cleanup phase..
Wed Jan 16 16:41:40 2019 - [info]
Wed Jan 16 16:41:40 2019 - [info] Resetting slave info on the new master..
Wed Jan 16 16:41:40 2019 - [info] 10.238.162.33: Resetting slave info succeeded.
Wed Jan 16 16:41:40 2019 - [info] Master failover to 10.238.162.33(10.238.162.33:3306) completed successfully.
Wed Jan 16 16:41:40 2019 - [info] Deleted server1 entry from /etc/mha/app1.cnf .
Wed Jan 16 16:41:40 2019 - [info]

----- Failover Report -----

app1: MySQL Master failover 10.238.162.31(10.238.162.31:3306) to 10.238.162.33(10.238.162.33:3306) succeeded

Master 10.238.162.31(10.238.162.31:3306) is down!

Check MHA Manager logs at Master:/server/mha/app1/manager.log for details.

Started automated(non-interactive) failover.
Invalidated master IP address on 10.238.162.31(10.238.162.31:3306)
The latest slave 10.238.162.33(10.238.162.33:3306) has all relay logs for recovery.
Selected 10.238.162.33(10.238.162.33:3306) as a new master.
10.238.162.33(10.238.162.33:3306): OK: Applying all logs succeeded.
10.238.162.33(10.238.162.33:3306): OK: Activated master IP address.
10.238.162.34(10.238.162.34:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
10.238.162.34(10.238.162.34:3306): OK: Applying all logs succeeded. Slave started, replicating from 10.238.162.33(10.238.162.
33:3306)
10.238.162.33(10.238.162.33:3306): Resetting slave info succeeded.
Master failover to 10.238.162.33(10.238.162.33:3306) completed successfully.
Wed Jan 16 16:41:40 2019 - [info] Sending mail..
sh: /usr/local/bin/send_report: 权限不够
Wed Jan 16 16:41:40 2019 - [error][/usr/local/share/perl5/MHA/MasterFailover.pm, ln2089] Failed to send mail with return code
126:0

从上面的输出可以看出整个MHA的切换过程，共包括以下的步骤：
1.配置文件检查阶段，这个阶段会检查整个集群配置文件配置
2.宕机的master处理，这个阶段包括虚拟ip摘除操作，主机关机操作（这个我这里还没有实现，需要研究）
3.复制dead maste和最新slave相差的relay log，并保存到MHA Manger具体的目录下
4.识别含有最新更新的slave
5.应用从master保存的二进制日志事件（binlog events）
6.提升一个slave为新的master进行复制
7.使其他的slave连接新的master进行复制

查看VIP漂移状况

31服务器 VIP已经不在

33服务器 VIP已经漂移到33服务器

查看MySQL复制状况在34服务上查看，33已经成为34的master。

查看MHA 配置文件，发现注释和[server1]部分已经没有了。

完成切换后MHA manager进程会自动退出。查看官档找到原因

2 手动failover

MHA Manager必须没有运行
手动failover，这种场景意味着在业务上没有启用MHA自动切换功能，当主服务器故障时，人工手动调用MHA来进行故障切换操作，具体命令如下：
注意：如果，MHA manager检测到没有dead的server，将报错，并结束failover：

31服务器停掉MySQL模拟宕机或者服务异常停止。 /etc/init.d/mysqld stop

手动切换命令如下：

masterha_master_switch --master_state=dead --conf=/etc/mha/app1.cnf --dead_master_host=10.238.162.31 --dead_master_port=3306 --new_master_host=10.238.162.33 --new_master_port=3306 --ignore_last_failover

输出信息是交互式，会询问你是否进行切换：建议阅读输出以理解切换手动切换过程。

# masterha_master_switch --master_state=dead --conf=/etc/mha/app1.cnf --dead_master_host=10.238.162.31 --dead_master_port=3306 --new_master_host=10.238.162.33 --new_master_port=3306 --ignore_last_failover
--dead_master_ip= is not set. Using 10.238.162.31.
Wed Jan 16 20:01:16 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed Jan 16 20:01:16 2019 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Wed Jan 16 20:01:16 2019 - [info] Reading server configuration from /etc/mha/app1.cnf..
Wed Jan 16 20:01:16 2019 - [info] MHA::MasterFailover version 0.58.
Wed Jan 16 20:01:16 2019 - [info] Starting master failover.
Wed Jan 16 20:01:16 2019 - [info]
Wed Jan 16 20:01:16 2019 - [info] * Phase 1: Configuration Check Phase..
Wed Jan 16 20:01:16 2019 - [info]
Wed Jan 16 20:01:17 2019 - [info] GTID failover mode = 0
Wed Jan 16 20:01:17 2019 - [info] Dead Servers:
Wed Jan 16 20:01:17 2019 - [info] 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 20:01:17 2019 - [info] Checking master reachability via MySQL(double check)...
Wed Jan 16 20:01:17 2019 - [info] ok.
Wed Jan 16 20:01:17 2019 - [info] Alive Servers:
Wed Jan 16 20:01:17 2019 - [info] 10.238.162.33(10.238.162.33:3306)
Wed Jan 16 20:01:17 2019 - [info] 10.238.162.34(10.238.162.34:3306)
Wed Jan 16 20:01:17 2019 - [info] Alive Slaves:
Wed Jan 16 20:01:17 2019 - [info] 10.238.162.33(10.238.162.33:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
Wed Jan 16 20:01:17 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 20:01:17 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Wed Jan 16 20:01:17 2019 - [info] 10.238.162.34(10.238.162.34:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
Wed Jan 16 20:01:17 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Master 10.238.162.31(10.238.162.31:3306) is dead. Proceed? (yes/NO): yes
Wed Jan 16 20:04:10 2019 - [info] Starting Non-GTID based failover.
Wed Jan 16 20:04:10 2019 - [info]
Wed Jan 16 20:04:10 2019 - [info] ** Phase 1: Configuration Check Phase completed.
Wed Jan 16 20:04:10 2019 - [info]
Wed Jan 16 20:04:10 2019 - [info] * Phase 2: Dead Master Shutdown Phase..
Wed Jan 16 20:04:10 2019 - [info]
Wed Jan 16 20:04:10 2019 - [info] HealthCheck: SSH to 10.238.162.31 is reachable.
Wed Jan 16 20:04:11 2019 - [info] Forcing shutdown so that applications never connect to the current master..
Wed Jan 16 20:04:11 2019 - [info] Executing master IP deactivation script:
Wed Jan 16 20:04:11 2019 - [info] /usr/local/bin/master_ip_failover --orig_master_host=10.238.162.31 --orig_master_ip=10.238.162.31 --orig_master_port=3306 --command=stopssh --ssh_user=root

IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===

Disabling the VIP on old master: 10.238.162.31
Wed Jan 16 20:04:11 2019 - [info] done.
Wed Jan 16 20:04:11 2019 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Wed Jan 16 20:04:11 2019 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Wed Jan 16 20:04:11 2019 - [info]
Wed Jan 16 20:04:11 2019 - [info] * Phase 3: Master Recovery Phase..
Wed Jan 16 20:04:11 2019 - [info]
Wed Jan 16 20:04:11 2019 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Wed Jan 16 20:04:11 2019 - [info]
Wed Jan 16 20:04:11 2019 - [info] The latest binary log file/position on all slaves is mysql-bin.000012:154
Wed Jan 16 20:04:11 2019 - [info] Latest slaves (Slaves that received relay log files to the latest):
Wed Jan 16 20:04:11 2019 - [info] 10.238.162.33(10.238.162.33:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
Wed Jan 16 20:04:11 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 20:04:11 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Wed Jan 16 20:04:11 2019 - [info] 10.238.162.34(10.238.162.34:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
Wed Jan 16 20:04:11 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 20:04:11 2019 - [info] The oldest binary log file/position on all slaves is mysql-bin.000012:154
Wed Jan 16 20:04:11 2019 - [info] Oldest slaves:
Wed Jan 16 20:04:11 2019 - [info] 10.238.162.33(10.238.162.33:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
Wed Jan 16 20:04:11 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 20:04:11 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Wed Jan 16 20:04:11 2019 - [info] 10.238.162.34(10.238.162.34:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
Wed Jan 16 20:04:11 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 20:04:11 2019 - [info]
Wed Jan 16 20:04:11 2019 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Wed Jan 16 20:04:11 2019 - [info]
Wed Jan 16 20:04:11 2019 - [info] Fetching dead master's binary logs..
Wed Jan 16 20:04:11 2019 - [info] Executing command on the dead master 10.238.162.31(10.238.162.31:3306): save_binary_logs --command=save --start_file=mysql-bin.000012 --start_pos=154 --binlog_dir=/server/mysql_data --output_file=/server/mha_tmp/saved_master_binlog_from_10.238.162.31_3306_20190116200116.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.58
Creating /server/mha_tmp if not exists.. ok.
Concat binary/relay logs from mysql-bin.000012 pos 154 to mysql-bin.000012 EOF into /server/mha_tmp/saved_master_binlog_from_10.238.162.31_3306_20190116200116.binlog ..
Binlog Checksum enabled
Dumping binlog format description event, from position 0 to 154.. ok.
Dumping effective binlog data from /server/mysql_data/mysql-bin.000012 position 154 to tail(177).. ok.
Binlog Checksum enabled
Concat succeeded.
saved_master_binlog_from_10.238.162.31_3306_20190116200116.binlog 100% 177 123.2KB/s 00:00
Wed Jan 16 20:04:12 2019 - [info] scp from [email protected]:/server/mha_tmp/saved_master_binlog_from_10.238.162.31_3306_20190116200116.binlog to local:/server/mha/app1/saved_master_binlog_from_10.238.162.31_3306_20190116200116.binlog succeeded.
Wed Jan 16 20:04:12 2019 - [info] HealthCheck: SSH to 10.238.162.33 is reachable.
Wed Jan 16 20:04:13 2019 - [info] HealthCheck: SSH to 10.238.162.34 is reachable.
Wed Jan 16 20:04:13 2019 - [info]
Wed Jan 16 20:04:13 2019 - [info] * Phase 3.3: Determining New Master Phase..
Wed Jan 16 20:04:13 2019 - [info]
Wed Jan 16 20:04:13 2019 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Wed Jan 16 20:04:13 2019 - [info] All slaves received relay logs to the same position. No need to resync each other.
Wed Jan 16 20:04:13 2019 - [info] 10.238.162.33 can be new master.
Wed Jan 16 20:04:13 2019 - [info] New master is 10.238.162.33(10.238.162.33:3306)
Wed Jan 16 20:04:13 2019 - [info] Starting master failover..
Wed Jan 16 20:04:13 2019 - [info]
From:
10.238.162.31(10.238.162.31:3306) (current master)
+--10.238.162.33(10.238.162.33:3306)
+--10.238.162.34(10.238.162.34:3306)

To:
10.238.162.33(10.238.162.33:3306) (new master)
+--10.238.162.34(10.238.162.34:3306)

Starting master switch from 10.238.162.31(10.238.162.31:3306) to 10.238.162.33(10.238.162.33:3306)? (yes/NO): yes
Wed Jan 16 20:06:21 2019 - [info] New master decided manually is 10.238.162.33(10.238.162.33:3306)
Wed Jan 16 20:06:21 2019 - [info]
Wed Jan 16 20:06:21 2019 - [info] * Phase 3.4: New Master Diff Log Generation Phase..
Wed Jan 16 20:06:21 2019 - [info]
Wed Jan 16 20:06:21 2019 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Wed Jan 16 20:06:21 2019 - [info] Sending binlog..
saved_master_binlog_from_10.238.162.31_3306_20190116200116.binlog 100% 177 110.1KB/s 00:00
Wed Jan 16 20:06:21 2019 - [info] scp from local:/server/mha/app1/saved_master_binlog_from_10.238.162.31_3306_20190116200116.binlog to [email protected]:/server/mha_tmp/saved_master_binlog_from_10.238.162.31_3306_20190116200116.binlog succeeded.
Wed Jan 16 20:06:21 2019 - [info]
Wed Jan 16 20:06:21 2019 - [info] * Phase 3.5: Master Log Apply Phase..
Wed Jan 16 20:06:21 2019 - [info]
Wed Jan 16 20:06:21 2019 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Wed Jan 16 20:06:21 2019 - [info] Starting recovery on 10.238.162.33(10.238.162.33:3306)..
Wed Jan 16 20:06:21 2019 - [info] Generating diffs succeeded.
Wed Jan 16 20:06:21 2019 - [info] Waiting until all relay logs are applied.
Wed Jan 16 20:06:21 2019 - [info] done.
Wed Jan 16 20:06:21 2019 - [info] Getting slave status..
Wed Jan 16 20:06:21 2019 - [info] This slave(10.238.162.33)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000012:154). No need to recover from Exec_Master_Log_Pos.
Wed Jan 16 20:06:21 2019 - [info] Connecting to the target slave host 10.238.162.33, running recover script..
Wed Jan 16 20:06:21 2019 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='mha' --slave_host=10.238.162.33 --slave_ip=10.238.162.33 --slave_port=3306 --apply_files=/server/mha_tmp/saved_master_binlog_from_10.238.162.31_3306_20190116200116.binlog --workdir=/server/mha_tmp --target_version=5.7.21-log --timestamp=20190116200116 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.58 --slave_pass=xxx
Wed Jan 16 20:06:22 2019 - [info]
MySQL client version is 5.7.21. Using --binary-mode.
Applying differential binary/relay log files /server/mha_tmp/saved_master_binlog_from_10.238.162.31_3306_20190116200116.binlog on 10.238.162.33:3306. This may take long time...
Applying log files succeeded.
Wed Jan 16 20:06:22 2019 - [info] All relay logs were successfully applied.
Wed Jan 16 20:06:22 2019 - [info] Getting new master's binlog name and position..
Wed Jan 16 20:06:22 2019 - [info] Slave-mysql-bin.000004:458
Wed Jan 16 20:06:22 2019 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.238.162.33', MASTER_PORT=3306, MASTER_LOG_FILE='Slave-mysql-bin.000004', MASTER_LOG_POS=458, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Wed Jan 16 20:06:22 2019 - [info] Executing master IP activate script:
Wed Jan 16 20:06:22 2019 - [info] /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=10.238.162.31 --orig_master_ip=10.238.162.31 --orig_master_port=3306 --new_master_host=10.238.162.33 --new_master_ip=10.238.162.33 --new_master_port=3306 --new_master_user='mha' --new_master_password=xxx
Unknown option: new_master_user
Unknown option: new_master_password

IN SCRIPT TEST====/etc/init.d/keepalived stop==/etc/init.d/keepalived start===

Enabling the VIP - 10.238.162.100 on the new master - 10.238.162.33
Wed Jan 16 20:06:22 2019 - [info] OK.
Wed Jan 16 20:06:22 2019 - [info] ** Finished master recovery successfully.
Wed Jan 16 20:06:22 2019 - [info] * Phase 3: Master Recovery Phase completed.
Wed Jan 16 20:06:22 2019 - [info]
Wed Jan 16 20:06:22 2019 - [info] * Phase 4: Slaves Recovery Phase..
Wed Jan 16 20:06:22 2019 - [info]
Wed Jan 16 20:06:22 2019 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Wed Jan 16 20:06:22 2019 - [info]
Wed Jan 16 20:06:22 2019 - [info] -- Slave diff file generation on host 10.238.162.34(10.238.162.34:3306) started, pid: 30439. Check tmp log /server/mha/app1/10.238.162.34_3306_20190116200116.log if it takes time..
Wed Jan 16 20:06:23 2019 - [info]
Wed Jan 16 20:06:23 2019 - [info] Log messages from 10.238.162.34 ...
Wed Jan 16 20:06:23 2019 - [info]
Wed Jan 16 20:06:22 2019 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Wed Jan 16 20:06:23 2019 - [info] End of log messages from 10.238.162.34.
Wed Jan 16 20:06:23 2019 - [info] -- 10.238.162.34(10.238.162.34:3306) has the latest relay log events.
Wed Jan 16 20:06:23 2019 - [info] Generating relay diff files from the latest slave succeeded.
Wed Jan 16 20:06:23 2019 - [info]
Wed Jan 16 20:06:23 2019 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Wed Jan 16 20:06:23 2019 - [info]
Wed Jan 16 20:06:23 2019 - [info] -- Slave recovery on host 10.238.162.34(10.238.162.34:3306) started, pid: 30441. Check tmp log /server/mha/app1/10.238.162.34_3306_20190116200116.log if it takes time..
saved_master_binlog_from_10.238.162.31_3306_20190116200116.binlog 100% 177 179.1KB/s 00:00
Wed Jan 16 20:06:24 2019 - [info]
Wed Jan 16 20:06:24 2019 - [info] Log messages from 10.238.162.34 ...
Wed Jan 16 20:06:24 2019 - [info]
Wed Jan 16 20:06:23 2019 - [info] Sending binlog..
Wed Jan 16 20:06:23 2019 - [info] scp from local:/server/mha/app1/saved_master_binlog_from_10.238.162.31_3306_20190116200116.binlog to [email protected]:/server/mha_tmp/saved_master_binlog_from_10.238.162.31_3306_20190116200116.binlog succeeded.
Wed Jan 16 20:06:23 2019 - [info] Starting recovery on 10.238.162.34(10.238.162.34:3306)..
Wed Jan 16 20:06:23 2019 - [info] Generating diffs succeeded.
Wed Jan 16 20:06:23 2019 - [info] Waiting until all relay logs are applied.
Wed Jan 16 20:06:23 2019 - [info] done.
Wed Jan 16 20:06:23 2019 - [info] Getting slave status..
Wed Jan 16 20:06:23 2019 - [info] This slave(10.238.162.34)'s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(mysql-bin.000012:154). No need to recover from Exec_Master_Log_Pos.
Wed Jan 16 20:06:23 2019 - [info] Connecting to the target slave host 10.238.162.34, running recover script..
Wed Jan 16 20:06:23 2019 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user='mha' --slave_host=10.238.162.34 --slave_ip=10.238.162.34 --slave_port=3306 --apply_files=/server/mha_tmp/saved_master_binlog_from_10.238.162.31_3306_20190116200116.binlog --workdir=/server/mha_tmp --target_version=5.7.21-log --timestamp=20190116200116 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.58 --slave_pass=xxx
Wed Jan 16 20:06:24 2019 - [info]
MySQL client version is 5.7.21. Using --binary-mode.
Applying differential binary/relay log files /server/mha_tmp/saved_master_binlog_from_10.238.162.31_3306_20190116200116.binlog on 10.238.162.34:3306. This may take long time...
Applying log files succeeded.
Wed Jan 16 20:06:24 2019 - [info] All relay logs were successfully applied.
Wed Jan 16 20:06:24 2019 - [info] Resetting slave 10.238.162.34(10.238.162.34:3306) and starting replication from the new master 10.238.162.33(10.238.162.33:3306)..
Wed Jan 16 20:06:24 2019 - [info] Executed CHANGE MASTER.
Wed Jan 16 20:06:24 2019 - [info] Slave started.
Wed Jan 16 20:06:24 2019 - [info] End of log messages from 10.238.162.34.
Wed Jan 16 20:06:24 2019 - [info] -- Slave recovery on host 10.238.162.34(10.238.162.34:3306) succeeded.
Wed Jan 16 20:06:24 2019 - [info] All new slave servers recovered successfully.
Wed Jan 16 20:06:24 2019 - [info]
Wed Jan 16 20:06:24 2019 - [info] * Phase 5: New master cleanup phase..
Wed Jan 16 20:06:24 2019 - [info]
Wed Jan 16 20:06:24 2019 - [info] Resetting slave info on the new master..
Wed Jan 16 20:06:24 2019 - [info] 10.238.162.33: Resetting slave info succeeded.
Wed Jan 16 20:06:24 2019 - [info] Master failover to 10.238.162.33(10.238.162.33:3306) completed successfully.
Wed Jan 16 20:06:24 2019 - [info]

----- Failover Report -----

app1: MySQL Master failover 10.238.162.31(10.238.162.31:3306) to 10.238.162.33(10.238.162.33:3306) succeeded

Master 10.238.162.31(10.238.162.31:3306) is down!

Check MHA Manager logs at Master for details.

Started manual(interactive) failover.
Invalidated master IP address on 10.238.162.31(10.238.162.31:3306)
The latest slave 10.238.162.33(10.238.162.33:3306) has all relay logs for recovery.
Selected 10.238.162.33(10.238.162.33:3306) as a new master.
10.238.162.33(10.238.162.33:3306): OK: Applying all logs succeeded.
10.238.162.33(10.238.162.33:3306): OK: Activated master IP address.
10.238.162.34(10.238.162.34:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
10.238.162.34(10.238.162.34:3306): OK: Applying all logs succeeded. Slave started, replicating from 10.238.162.33(10.238.162.33:3306)
10.238.162.33(10.238.162.33:3306): Resetting slave info succeeded.
Master failover to 10.238.162.33(10.238.162.33:3306) completed successfully.

3 在线切换

在许多情况下，需要将现有的主服务器迁移到另外一台服务器上。比如主服务器硬件故障，RAID 控制卡需要重建，将主服务器移到性能更好的服务器上等等。维护主服务器引起性能下降，导致停机时间至少无法写入数据。另外，阻塞或杀掉当前运行的会话会导致主主之间数据不一致的问题发生。 MHA 提供快速切换和优雅的阻塞写入，这个切换过程只需要 0.5-2s 的时间，这段时间内数据是无法写入的。在很多情况下，0.5-2s 的阻塞写入是可以接受的。因此切换主服务器不需要计划分配维护时间窗口。

MHA在线切换的大概过程：
1.检测复制设置和确定当前主服务器
2.确定新的主服务器
3.阻塞写入到当前主服务器
4.等待所有从服务器赶上复制
5.授予写入到新的主服务器
6.重新设置从服务器

注意，在线切换的时候应用架构需要考虑以下两个问题：
1.自动识别master和slave的问题（master的机器可能会切换），如果采用了vip的方式，基本可以解决这个问题。
2.负载均衡的问题（可以定义大概的读写比例，每台机器可承担的负载比例，当有机器离开集群时，需要考虑这个问题）

为了保证数据完全一致性，在最快的时间内完成切换，MHA的在线切换必须满足以下条件才会切换成功，否则会切换失败。

1.所有slave的IO线程都在运行
2.所有slave的SQL线程都在运行
3.所有的show slave status的输出中Seconds_Behind_Master参数小于或者等于running_updates_limit秒，如果在切换过程中不指定running_updates_limit,那么默认情况下running_updates_limit为1秒。
4.在master端，通过show processlist输出，没有一个更新花费的时间大于running_updates_limit秒。

在线切换步骤如下：
首先，停掉MHA监控：

注意：由于在线进行切换需要调用到master_ip_online_change这个脚本，但是由于该脚本不完整，需要自己进行相应的修改，我google到后发现还是有问题，脚本中new_master_password这个变量获取不到，导致在线切换失败，所以进行了相关的硬编码，直接把mysql的root用户密码赋值给变量new_master_password，如果有哪位大牛知道原因，请指点指点。这个脚本还可以管理vip。下面贴出脚本：

# cat master_ip_online_change
#!/usr/bin/env perl

# Copyright (C) 2011 DeNA Co.,Ltd.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA

## Note: This is a sample script and is not complete. Modify the script based on your environment.

use strict;
use warnings FATAL => 'all';

use Getopt::Long;
use MHA::DBHelper;
use MHA::NodeUtil;
use Time::HiRes qw( sleep gettimeofday tv_interval );
use Data::Dumper;

my $_tstart;
my $_running_interval = 0.1;
my (
$command, $orig_master_host, $orig_master_ip,
$orig_master_port, $orig_master_user,
$new_master_host, $new_master_ip, $new_master_port,
$new_master_user,
);

my $vip = '10.238.162.100/32'; # Virtual IP
my $key = "1";
my $ssh_start_vip = "/sbin/ifconfig ens160:$key $vip";
my $ssh_stop_vip = "/sbin/ifconfig ens160:$key down";
my $ssh_user = "root";
my $new_master_password='123456';
my $orig_master_password='123456';
GetOptions(
'command=s' => \$command,
#'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'orig_master_user=s' => \$orig_master_user,
#'orig_master_password=s' => \$orig_master_password,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
'new_master_user=s' => \$new_master_user,
#'new_master_password=s' => \$new_master_password,
);

exit &main();

sub current_time_us {
my ( $sec, $microsec ) = gettimeofday();
my $curdate = localtime($sec);
return $curdate . " " . sprintf( "%06d", $microsec );
}

sub sleep_until {
my $elapsed = tv_interval($_tstart);
if ( $_running_interval > $elapsed ) {
sleep( $_running_interval - $elapsed );
}
}

sub get_threads_util {
my $dbh = shift;
my $my_connection_id = shift;
my $running_time_threshold = shift;
my $type = shift;
$running_time_threshold = 0 unless ($running_time_threshold);
$type = 0 unless ($type);
my @threads;

my $sth = $dbh->prepare("SHOW PROCESSLIST");
$sth->execute();

while ( my $ref = $sth->fetchrow_hashref() ) {
my $id = $ref->{Id};
my $user = $ref->{User};
my $host = $ref->{Host};
my $command = $ref->{Command};
my $state = $ref->{State};
my $query_time = $ref->{Time};
my $info = $ref->{Info};
$info =~ s/^\s*(.*?)\s*$/$1/ if defined($info);
next if ( $my_connection_id == $id );
next if ( defined($query_time) && $query_time < $running_time_threshold );
next if ( defined($command) && $command eq "Binlog Dump" );
next if ( defined($user) && $user eq "system user" );
next
if ( defined($command)
&& $command eq "Sleep"
&& defined($query_time)
&& $query_time >= 1 );

if ( $type >= 1 ) {
next if ( defined($command) && $command eq "Sleep" );
next if ( defined($command) && $command eq "Connect" );
}

if ( $type >= 2 ) {
next if ( defined($info) && $info =~ m/^select/i );
next if ( defined($info) && $info =~ m/^show/i );
}

push @threads, $ref;
}
return @threads;
}

sub main {
if ( $command eq "stop" ) {
## Gracefully killing connections on the current master
# 1. Set read_only= 1 on the new master
# 2. DROP USER so that no app user can establish new connections
# 3. Set read_only= 1 on the current master
# 4. Kill current queries
# * Any database access failure will result in script die.
my $exit_code = 1;
eval {
## Setting read_only=1 on the new master (to avoid accident)
my $new_master_handler = new MHA::DBHelper();

# args: hostname, port, user, password, raise_error(die_on_error)_or_not
$new_master_handler->connect( $new_master_ip, $new_master_port,
$new_master_user, $new_master_password, 1 );
print current_time_us() . " Set read_only on the new master.. ";
$new_master_handler->enable_read_only();
if ( $new_master_handler->is_read_only() ) {
print "ok.\n";
}
else {
die "Failed!\n";
}
$new_master_handler->disconnect();

# Connecting to the orig master, die if any database error happens
my $orig_master_handler = new MHA::DBHelper();
$orig_master_handler->connect( $orig_master_ip, $orig_master_port,
$orig_master_user, $orig_master_password, 1 );

## Drop application user so that nobody can connect. Disabling per-session binlog beforehand
#$orig_master_handler->disable_log_bin_local();
#print current_time_us() . " Drpping app user on the orig master..\n";
#FIXME_xxx_drop_app_user($orig_master_handler);

## Waiting for N * 100 milliseconds so that current connections can exit
my $time_until_read_only = 15;
$_tstart = [gettimeofday];
my @threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
while ( $time_until_read_only > 0 && $#threads >= 0 ) {
if ( $time_until_read_only % 5 == 0 ) {
printf
"%s Waiting all running %d threads are disconnected.. (max %d milliseconds)\n",
current_time_us(), $#threads + 1, $time_until_read_only * 100;
if ( $#threads < 5 ) {
print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"
foreach (@threads);
}
}
sleep_until();
$_tstart = [gettimeofday];
$time_until_read_only--;
@threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
}

## Setting read_only=1 on the current master so that nobody(except SUPER) can write
print current_time_us() . " Set read_only=1 on the orig master.. ";
$orig_master_handler->enable_read_only();
if ( $orig_master_handler->is_read_only() ) {
print "ok.\n";
}
else {
die "Failed!\n";
}

## Waiting for M * 100 milliseconds so that current update queries can complete
my $time_until_kill_threads = 5;
@threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
while ( $time_until_kill_threads > 0 && $#threads >= 0 ) {
if ( $time_until_kill_threads % 5 == 0 ) {
printf
"%s Waiting all running %d queries are disconnected.. (max %d milliseconds)\n",
current_time_us(), $#threads + 1, $time_until_kill_threads * 100;
if ( $#threads < 5 ) {
print Data::Dumper->new( [$_] )->Indent(0)->Terse(1)->Dump . "\n"
foreach (@threads);
}
}
sleep_until();
$_tstart = [gettimeofday];
$time_until_kill_threads--;
@threads = get_threads_util( $orig_master_handler->{dbh},
$orig_master_handler->{connection_id} );
}

print "Disabling the VIP on old master: $orig_master_host \n";
&stop_vip();

## Terminating all threads
print current_time_us() . " Killing all application threads..\n";
$orig_master_handler->kill_threads(@threads) if ( $#threads >= 0 );
print current_time_us() . " done.\n";
#$orig_master_handler->enable_log_bin_local();
$orig_master_handler->disconnect();

## After finishing the script, MHA executes FLUSH TABLES WITH READ LOCK
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {
## Activating master ip on the new master
# 1. Create app user with write privileges
# 2. Moving backup script if needed
# 3. Register new master's ip to the catalog database

# We don't return error even though activating updatable accounts/ip failed so that we don't interrupt slaves' recovery.
# If exit code is 0 or 10, MHA does not abort
my $exit_code = 10;
eval {
my $new_master_handler = new MHA::DBHelper();

# args: hostname, port, user, password, raise_error_or_not
$new_master_handler->connect( $new_master_ip, $new_master_port,
$new_master_user, $new_master_password, 1 );

## Set read_only=0 on the new master
#$new_master_handler->disable_log_bin_local();
print current_time_us() . " Set read_only=0 on the new master.\n";
$new_master_handler->disable_read_only();

## Creating an app user on the new master
#print current_time_us() . " Creating app user on the new master..\n";
#FIXME_xxx_create_app_user($new_master_handler);
#$new_master_handler->enable_log_bin_local();
$new_master_handler->disconnect();

## Update master ip on the catalog database, etc
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {

# do nothing
exit 0;
}
else {
&usage();
exit 1;
}
}

# A simple system call that enable the VIP on the new master
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}

sub usage {
print
"Usage: master_ip_online_change --command=start|stop|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
die;
}

masterha_stop --conf=/etc/mha/app1.cnf

其中参数的意思：

--orig_master_is_new_slave 切换时加上此参数是将原 master 变为 slave 节点，如果不加此参数，原来的 master 将不启动

--running_updates_limit=10000,故障切换时,候选master 如果有延迟的话， mha 切换不能成功，加上此参数表示延迟在此时间范围内都可切换（单位为s），但是切换的时间长短是由recover 时relay 日志的大小决定

# masterha_master_switch --conf=/etc/mha/app1.cnf --master_state=alive --new_master_host=10.238.162.33 --new_master_port=3306 --orig_master_is_new_slave --running_updates_limit=10000
Wed Jan 16 22:16:17 2019 - [info] MHA::MasterRotate version 0.58.
Wed Jan 16 22:16:17 2019 - [info] Starting online master switch..
Wed Jan 16 22:16:17 2019 - [info]
Wed Jan 16 22:16:17 2019 - [info] * Phase 1: Configuration Check Phase..
Wed Jan 16 22:16:17 2019 - [info]
Wed Jan 16 22:16:17 2019 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Wed Jan 16 22:16:17 2019 - [info] Reading application default configuration from /etc/mha/app1.cnf..
Wed Jan 16 22:16:17 2019 - [info] Reading server configuration from /etc/mha/app1.cnf..
Wed Jan 16 22:16:18 2019 - [info] GTID failover mode = 0
Wed Jan 16 22:16:18 2019 - [info] Current Alive Master: 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 22:16:18 2019 - [info] Alive Slaves:
Wed Jan 16 22:16:18 2019 - [info] 10.238.162.33(10.238.162.33:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
Wed Jan 16 22:16:18 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)
Wed Jan 16 22:16:18 2019 - [info] Primary candidate for the new Master (candidate_master is set)
Wed Jan 16 22:16:18 2019 - [info] 10.238.162.34(10.238.162.34:3306) Version=5.7.21-log (oldest major version between slaves) log-bin:enabled
Wed Jan 16 22:16:18 2019 - [info] Replicating from 10.238.162.31(10.238.162.31:3306)

It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 10.238.162.31(10.238.162.31:3306)? (YES/no): yes
Wed Jan 16 22:16:21 2019 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Wed Jan 16 22:16:21 2019 - [info] ok.
Wed Jan 16 22:16:21 2019 - [info] Checking MHA is not monitoring or doing failover..
Wed Jan 16 22:16:21 2019 - [info] Checking replication health on 10.238.162.33..
Wed Jan 16 22:16:21 2019 - [info] ok.
Wed Jan 16 22:16:21 2019 - [info] Checking replication health on 10.238.162.34..
Wed Jan 16 22:16:21 2019 - [info] ok.
Wed Jan 16 22:16:21 2019 - [info] 10.238.162.33 can be new master.
Wed Jan 16 22:16:21 2019 - [info]
From:
10.238.162.31(10.238.162.31:3306) (current master)
+--10.238.162.33(10.238.162.33:3306)
+--10.238.162.34(10.238.162.34:3306)

To:
10.238.162.33(10.238.162.33:3306) (new master)
+--10.238.162.34(10.238.162.34:3306)
+--10.238.162.31(10.238.162.31:3306)

Starting master switch from 10.238.162.31(10.238.162.31:3306) to 10.238.162.33(10.238.162.33:3306)? (yes/NO): yes
Wed Jan 16 22:16:23 2019 - [info] Checking whether 10.238.162.33(10.238.162.33:3306) is ok for the new master..
Wed Jan 16 22:16:23 2019 - [info] ok.
Wed Jan 16 22:16:23 2019 - [info] 10.238.162.31(10.238.162.31:3306): SHOW SLAVE STATUS returned empty result. To check replication filtering rules, temporarily executing CHANGE MASTER to a dummy host.
Wed Jan 16 22:16:23 2019 - [info] 10.238.162.31(10.238.162.31:3306): Resetting slave pointing to the dummy host.
Wed Jan 16 22:16:23 2019 - [info] ** Phase 1: Configuration Check Phase completed.
Wed Jan 16 22:16:23 2019 - [info]
Wed Jan 16 22:16:23 2019 - [info] * Phase 2: Rejecting updates Phase..
Wed Jan 16 22:16:23 2019 - [info]
Wed Jan 16 22:16:23 2019 - [info] Executing master ip online change script to disable write on the current master:
Wed Jan 16 22:16:23 2019 - [info] /usr/local/bin/master_ip_online_change --command=stop --orig_master_host=10.238.162.31 --orig_master_ip=10.238.162.31 --orig_master_port=3306 --orig_master_user='mha' --new_master_host=10.238.162.33 --new_master_ip=10.238.162.33 --new_master_port=3306 --new_master_user='mha' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave --orig_master_password=xxx --new_master_password=xxx
Unknown option: orig_master_ssh_user
Unknown option: new_master_ssh_user
Unknown option: orig_master_is_new_slave
Unknown option: orig_master_password
Unknown option: new_master_password
Wed Jan 16 22:16:23 2019 254185 Set read_only on the new master.. ok.
Wed Jan 16 22:16:23 2019 257941 Set read_only=1 on the orig master.. ok.
Disabling the VIP on old master: 10.238.162.31
SIOCSIFFLAGS: 无法指定被请求的地址
Wed Jan 16 22:16:23 2019 477838 Killing all application threads..
Wed Jan 16 22:16:23 2019 477953 done.
Wed Jan 16 22:16:23 2019 - [info] ok.
Wed Jan 16 22:16:23 2019 - [info] Locking all tables on the orig master to reject updates from everybody (including root):
Wed Jan 16 22:16:23 2019 - [info] Executing FLUSH TABLES WITH READ LOCK..
Wed Jan 16 22:16:23 2019 - [info] ok.
Wed Jan 16 22:16:23 2019 - [info] Orig master binlog:pos is mysql-bin.000013:154.
Wed Jan 16 22:16:23 2019 - [info] Waiting to execute all relay logs on 10.238.162.33(10.238.162.33:3306)..
Wed Jan 16 22:16:23 2019 - [info] master_pos_wait(mysql-bin.000013:154) completed on 10.238.162.33(10.238.162.33:3306). Executed 0 events.
Wed Jan 16 22:16:23 2019 - [info] done.
Wed Jan 16 22:16:23 2019 - [info] Getting new master's binlog name and position..
Wed Jan 16 22:16:23 2019 - [info] Slave-mysql-bin.000004:458
Wed Jan 16 22:16:23 2019 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.238.162.33', MASTER_PORT=3306, MASTER_LOG_FILE='Slave-mysql-bin.000004', MASTER_LOG_POS=458, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Wed Jan 16 22:16:23 2019 - [info] Executing master ip online change script to allow write on the new master:
Wed Jan 16 22:16:23 2019 - [info] /usr/local/bin/master_ip_online_change --command=start --orig_master_host=10.238.162.31 --orig_master_ip=10.238.162.31 --orig_master_port=3306 --orig_master_user='mha' --new_master_host=10.238.162.33 --new_master_ip=10.238.162.33 --new_master_port=3306 --new_master_user='mha' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave --orig_master_password=xxx --new_master_password=xxx
Unknown option: orig_master_ssh_user
Unknown option: new_master_ssh_user
Unknown option: orig_master_is_new_slave
Unknown option: orig_master_password
Unknown option: new_master_password
Wed Jan 16 22:16:23 2019 713106 Set read_only=0 on the new master.
Enabling the VIP - 10.238.162.100/32 on the new master - 10.238.162.33
Wed Jan 16 22:16:23 2019 - [info] ok.
Wed Jan 16 22:16:23 2019 - [info]
Wed Jan 16 22:16:23 2019 - [info] * Switching slaves in parallel..
Wed Jan 16 22:16:23 2019 - [info]
Wed Jan 16 22:16:23 2019 - [info] -- Slave switch on host 10.238.162.34(10.238.162.34:3306) started, pid: 31905
Wed Jan 16 22:16:23 2019 - [info]
Wed Jan 16 22:16:24 2019 - [info] Log messages from 10.238.162.34 ...
Wed Jan 16 22:16:24 2019 - [info]
Wed Jan 16 22:16:23 2019 - [info] Waiting to execute all relay logs on 10.238.162.34(10.238.162.34:3306)..
Wed Jan 16 22:16:23 2019 - [info] master_pos_wait(mysql-bin.000013:154) completed on 10.238.162.34(10.238.162.34:3306). Executed 0 events.
Wed Jan 16 22:16:23 2019 - [info] done.
Wed Jan 16 22:16:23 2019 - [info] Resetting slave 10.238.162.34(10.238.162.34:3306) and starting replication from the new master 10.238.162.33(10.238.162.33:3306)..
Wed Jan 16 22:16:23 2019 - [info] Executed CHANGE MASTER.
Wed Jan 16 22:16:24 2019 - [info] Slave started.
Wed Jan 16 22:16:24 2019 - [info] End of log messages from 10.238.162.34 ...
Wed Jan 16 22:16:24 2019 - [info]
Wed Jan 16 22:16:24 2019 - [info] -- Slave switch on host 10.238.162.34(10.238.162.34:3306) succeeded.
Wed Jan 16 22:16:24 2019 - [info] Unlocking all tables on the orig master:
Wed Jan 16 22:16:24 2019 - [info] Executing UNLOCK TABLES..
Wed Jan 16 22:16:24 2019 - [info] ok.
Wed Jan 16 22:16:24 2019 - [info] Starting orig master as a new slave..
Wed Jan 16 22:16:24 2019 - [info] Resetting slave 10.238.162.31(10.238.162.31:3306) and starting replication from the new master 10.238.162.33(10.238.162.33:3306)..
Wed Jan 16 22:16:24 2019 - [info] Executed CHANGE MASTER.
Wed Jan 16 22:16:25 2019 - [info] Slave started.
Wed Jan 16 22:16:25 2019 - [info] All new slave servers switched successfully.
Wed Jan 16 22:16:25 2019 - [info]
Wed Jan 16 22:16:25 2019 - [info] * Phase 5: New master cleanup phase..
Wed Jan 16 22:16:25 2019 - [info]
Wed Jan 16 22:16:25 2019 - [info] 10.238.162.33: Resetting slave info succeeded.
Wed Jan 16 22:16:25 2019 - [info] Switching master to 10.238.162.33(10.238.162.33:3306) completed successfully.

4 修复宕机master

通常情况下自动切换以后，原master可能已经废弃掉，待原master主机修复后，如果数据完整的情况下，可能想把原来master重新作为新主库的slave，这时我们可以借助当时自动切换时刻的MHA日志来完成对原master的修复。下面是提取相关日志的命令：

[root@Master app1]#  grep -i "All other slaves should start" manager.log 
Wed Jan 16 17:24:47 2019 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.238.162.33', MASTER_PORT=3306, MASTER_LOG_FILE='Slave-mysql-bin.000004', MASTER_LOG_POS=458, MASTER_USER='repl', MASTER_PASSWORD='xxx';

参考&感谢

主要参考了这位大神博客：站着巨人肩上可以看得更远 https://www.cnblogs.com/gomysql/p/3675429.html

Git官方文档： https://github.com/yoshinorim/mha4mysql-manager/wiki

你可能感兴趣的:(MySQL)

mysql禁用远程登录 igotyback mysql
去mysql库中的user表里，将host都改成localhost之后刷新权限FLUSHPRIVILEGES;
MYSQL面试系列-04 king01299 面试 mysql 面试
MYSQL面试系列-0417.关于redolog和binlog的刷盘机制、redolog、undolog作用、GTID是做什么的？innodb_flush_log_at_trx_commit及sync_binlog参数意义双117.1innodb_flush_log_at_trx_commit该变量定义了InnoDB在每次事务提交时，如何处理未刷入（flush）的重做日志信息（redolog）。它
关于Mysql 中 Row size too large (＞ 8126) 错误的解决和理解秋刀prince mysql mysql 数据库
提示：啰嗦一嘴，数据库的任何操作和验证前，一定要记得先备份！！！不会有错；文章目录问题发现一、问题导致的可能原因1、页大小2、行格式2.1compact格式2.2Redundant格式2.3Dynamic格式2.4Compressed格式3、BLOB和TEXT列二、解决办法1、修改页大小（不推荐）2、修改行格式3、修改数据类型为BLOB和TEXT列4、其他优化方式（可以参考使用）4.1合理设置数据
MongoDB知识概括 GeorgeLin98 持久层 mongodb
MongoDB知识概括MongoDB相关概念单机部署基本常用命令索引-IndexSpirngDataMongoDB集成副本集分片集群安全认证MongoDB相关概念业务应用场景：传统的关系型数据库（如MySQL），在数据操作的“三高”需求以及应对Web2.0的网站需求面前，显得力不从心。解释：“三高”需求：①Highperformance-对数据库高并发读写的需求。②HugeStorage-对海量数
入门MySQL——查询语法练习 K_un
前言：前面几篇文章为大家介绍了DML以及DDL语句的使用方法，本篇文章将主要讲述常用的查询语法。其实MySQL官网给出了多个示例数据库供大家实用查询，下面我们以最常用的员工示例数据库为准，详细介绍各自常用的查询语法。1.员工示例数据库导入官方文档员工示例数据库介绍及下载链接：https://dev.mysql.com/doc/employee/en/employees-installation.h
博客网站制作教程 2401_85194651 java maven
首先就是技术框架：后端：Java+SpringBoot数据库：MySQL前端：Vue.js数据库连接：JPA(JavaPersistenceAPI)1.项目结构blog-app/├──backend/│├──src/main/java/com/example/blogapp/││├──BlogApplication.java││├──config/│││└──DatabaseConfig.java
ubuntu安装wordpress lissettecarlr
1安装nginx网上安装方式很多，这就就直接用apt-get了apt-getinstallnginx不用启动啥，然后直接在浏览器里面输入IP:80就能看到nginx的主页了。如果修改了一些配置可以使用下列命令重启一下systemctlrestartnginx.service2安装mysql输入安装前也可以更新一下软件源，在安装过程中将会让你输入数据库的密码。sudoapt-getinstallmy
计算机毕业设计PHP仓储综合管理系统（源码+程序+VUE+lw+部署） java毕设程序源码王哥 php 课程设计 vue.js
该项目含有源码、文档、程序、数据库、配套开发软件、软件安装教程。欢迎交流项目运行环境配置：phpStudy+Vscode+Mysql5.7+HBuilderX+Navicat11+Vue+Express。项目技术：原生PHP++Vue等等组成，B/S模式+Vscode管理+前后端分离等等。环境需要1.运行环境：最好是小皮phpstudy最新版，我们在这个版本上开发的。其他版本理论上也可以。2.开发
You have an error in your SQL syntax； check the manual that corresponds to your MySQL server version 努力的菜鸟~ sql 数据库
YouhaveanerrorinyourSQLsyntax;checkthemanualthatcorrespondstoyourMySQLserverversionfortherightsyntaxtousenear‘IDENTIFIEDBY‘123456’WITHGRANTOPTION’atline1在mysql5.7之前GRANTALLPRIVILEGESON*.*TO'root'@'%'I
mysql学习教程，从入门到精通，TOP 和MySQL LIMIT 子句（15）知识分享小能手大数据数据库 MySQL mysql 学习 oracle 数据库开发语言 adb 大数据
1、TOP和MySQLLIMIT子句内容在SQL中，不同的数据库系统对于限制查询结果的数量有不同的实现方式。TOP关键字主要用于SQLServer和Access数据库中，而LIMIT子句则主要用于MySQL、PostgreSQL（通过LIMIT/OFFSET语法）、SQLite等数据库中。下面将分别详细介绍这两个功能的语法、语句以及案例。1.1、TOP子句（SQLServer和Access）1.1
鲲鹏 ARM 架构麒麟 Lylin v10 安装 Nginx (离线) 焚木灵 arm开发架构 nginx 服务器
最近做一个银行的项目，银行的服务器是鲲鹏ARM架构的服务器，并且是麒麟v10的系统，这里记录一下在无法访问外网安装Nginx的方法。其他文章：鲲鹏ARM架构麒麟Lylinv10安装Mysql8.3(离线)-CSDN博客鲲鹏ARM架构麒麟Lylinv10安装Node和NVM(离线)-CSDN博客鲲鹏ARM架构麒麟Lylinv10安装Pm2(离线)-CSDN博客鲲鹏ARM架构麒麟Lylinv10安装P
Kubernetes部署MySQL数据持久化沫殇-MS Kubernetes MySQL数据库 kubernetes mysql 容器
一、安装配置NFS服务端1、安装nfs-kernel-server：sudoapt-yinstallnfs-kernel-server2、服务端创建共享目录#列出所有可用块设备的信息lsblk#格式化磁盘sudomkfs-text4/dev/sdb#创建一个目录：sudomkdir-p/data/nfs/mysql#更改目录权限：sudochown-Rnobody:nogroup/data/nfs
MySQL事务隔离级别和MVCC 简书徐小耳
MySQL事务隔离级别和MVCC参考：https://mp.weixin.qq.com/s/Jeg8656gGtkPteYWrG5_Nw1.MVCC只对读已提交和可重复的读有效果，而未提交读和串行则无意义。2.每条记录都会有trx_id(事务修改记录的id）和roll_pointer是一个指针指向旧版本的undo日志链表（row_id不是必必要的，如果有主键存在就不需要了）3.版本链的头结点就是记
CentOS7 安装MySQL5.7.44 不要Null了 java centos mysql
1.下载mysql安装包，我放在百度网盘里(下方链接)链接：https://pan.baidu.com/s/1_Mn1XW_1mWdTV4mhnLG66A提取码：s31n2.首先看看以前是否安装过mysqlrpm-qa|grep-imysql如果已经安装过mysql会提示卸载mysqlrpm-emysql-…3.使用FinallShell或者Xftp进行上传放到/usr/local/mysql，没
非关系型数据库天秤-white nosql
一、为什么要用Nosql1.单机MySQL的时代。一个基本的网站访问量一般不会太大，单个数据库完全足够。那时候更多使用的静态网页html，服务器根本没有太大压力。这时候网站的瓶颈是什么？-数据量如果太大，一个机器放不下。-数据量太大需要建立数据的索引（B+Tree），一个服务器内存放不下。-访问量读写混合，一个服务器承受不了。2.memcached缓存+MySQL+垂直拆分（读写分离）。网站80%
六、全局锁和表锁：给表加个字段怎么有这么多阻碍 nieniemin
数据库锁设计的初衷是处理并发问题。作为多用户共享的资源，当出现并发访问的时候，数据库需要合理地控制资源的访问规则。而锁就是用来实现这些访问规则的重要数据结构。根据加锁的范围，MySQL里面的锁大致可以分成全局锁、表级锁和行锁三类。6.1全局锁全局锁就是对整个数据库实例加锁。MySQL提供了一个加全局读锁的方法，命令是Flushtableswithreadlock(FTWRL)。当你需要让整个库处于
Kubernetes 自定义控制器开发 IT回忆录 Kubenetes kubernetes
目录前言一、CRD二、创建数据库表（Mysql）二、控制器开发1.使用kubernetes的examplecontroller模板2.在controller.go中新增数据表监听方法3.修改tools工具生成资源对象结构体定义这里记录开发k8s控制器的一般方式，controller开发主要使用k8s提供的client-go库进行。前言Controller监听集群内部资源对象的变化，编辑资源对象(增
【K8s】专题十一：Kubernetes 集群证书过期处理方法行者Sun1989 Kubernetes kubernetes 云原生容器
本文内容均来自个人笔记并重新梳理，如有错误欢迎指正！如果对您有帮助，烦请点赞、关注、转发、订阅专栏！专栏订阅入口Linux专栏|Docker专栏|Kubernetes专栏往期精彩文章【Docker】（全网首发）KylinV10下MySQL容器内存占用异常的解决方法【Docker】（全网首发）KylinV10下MySQL容器内存占用异常的解决方法（续）【Docker】MySQL源码构建Docker镜
MySQl篇（SQL - 基本介绍）（持续更新迭代） wclass-zhengge mysql sql 数据库
目录一、简介二、SQL方言（分页查询为例）1.简介2.SQL方言大比拼2.1.Oracle2.1.1.使用ROWNUM实现分页查询2.1.2.使用ROW_NUMBER()实现分页查询2.2.MySQL2.3.PostgreSQL三、语法规范四、注释五、MySQL脚本中的标点符号一、简介1、SQL是结构化查询语言（StructureQueryLanguage），专门用来操作/访问关系型数据库的通用语
跟着黑马学mysql（5）小杜不吃糖 mysql 数据库
17.DQL-聚合函数DQL-聚合函数介绍将一列数据作为一个整体，进行纵向计算。常见聚合函数函数功能count统计数量max最大值min最小值avg平均值sum求和语法SELECT聚合函数(字段列表)FROM表名;注意：所有的null值不参与聚合函数的运算18.DQL-分组查询语法SELECT字段列表FROM表名[WHERE条件]GROUPBY分组字段名[HAVING分组后的过滤条件];where
梧桐数据库（WuTongDB）：数据库技术中都有哪些常见的优化器鲁鲁517 梧桐数据库梧桐数据库
以下是一些常见的数据库优化器：1.CBO（Cost-BasedOptimizer）应用场景：广泛应用于关系型数据库中，如Oracle、PostgreSQL、MySQL等。工作原理：通过计算不同执行计划的代价（如CPU、I/O等资源消耗），选择最低代价的执行计划。代表数据库：Oracle、PostgreSQL、MySQL。特点：CBO使用统计信息（如表大小、索引分布）来评估查询的代价。2.RBO（R
MySQL日志沉着冷静2024 MySQL mysql 数据库
MySQL日志文章目录MySQL日志MySQL三大日志binlog的三种格式redolog和binlog的区别和应用场景为什么崩溃恢复不用binlog而用redolog？redolog如何实现持久化redolog还能做什么？redolog的三种刷盘策略两阶段提交什么是？为什么？两阶段提交过程MySQL三大日志1.undologundolog是InnoDB存储引擎层的日志，实现了事务的原子性，主要用
MySQL锁沉着冷静2024 MySQL mysql 数据库
MySQL锁文章目录MySQL锁MySQL中锁的分类创建索引时会锁表吗线上修改表结构会加什么锁Innodb存储引擎的行级锁有哪些Update语句中，不带where条件，加什么锁？MySQL实现乐观锁MySQL死锁MySQL死锁是怎么发生的？检查死锁如何避免死锁MySQL中锁的分类全局锁：主要用于全库逻辑备份表级锁：表锁、元数据锁、意向锁表锁：通过locktables语句对表进行加锁，它不仅限制其他
Python实现mysql命令行 xu-jssy python mysql adb
一、源码importosimportpymysqldefsql_shell():password=input("EnterPassword:")#访问密码ifpassword.strip()!="yyds":print("Bye")return#清空控制台输出os.system("cls"ifos.name=="nt"else"clear")try:#连接到MySQL数据库conn=pymysql
mysql case 干货 lack倪酱
这是我舍友@某人提供的是为了做酒店时判断每个楼栋有几间空房子的。房间表与数据楼栋表与数据查询楼栋房间总数查询空房间数当时就在想怎么把后一个单列的数据合并到前面那三列的数据里呢？又是百度，又是谷歌，都是什么“unitall”。。表示我要的不是这个。有点难为我小脑瓜，实在是对数据库不了解，只会日常操作。最后是我舍友想出了解决办法直接贴图好了，如果想要了解case是什么，你可以自行查资料。学习查东西也是
计算机毕设Node.js+Vue校园易购二手交易平台（程序+LW+部署） Node程序源码强子 vue.js 课程设计 node.js
项目运行环境配置：Node.js最新版+Vscode+Mysql5.7+HBuilderX+Navicat11+Vue。项目技术：Express框架+Node.js+Vue等等组成，B/S模式+Vscode管理+前后端分离等等。环境需要1.运行环境：最好是Nodejs最新版，我们在这个版本上开发的。其他版本理论上也可以。2.开发
MySQL 源码｜55 - 语法解析(V2)：基础查询语句（query_specification）｜V20240915 长行 MySQL源码 mysql 源码语法解析查询
目录文档：MySQL源码｜源码剖析文档目录源码位置（版本=MySQL8.0.37）：sql/sql_yacc.yy前置文档：MySQL源码｜68-语法解析(V2)：LOCKING子句（锁定读取）｜V20240909MySQL源码｜39-语法解析(V2)：ORDERBY子句｜V20240814｜V20240912（第2版）MySQL源码｜40-语法解析(V2)：GROUPBY子句｜V20240814
老生常谈：MySQL高可用架构我有一头小花驴 mysql 架构数据库
引言“高可用”是互联网一个永恒的话题，先避开MySQL不谈，为了保证各种服务的高可用有几种常用的解决方案。服务冗余：把服务部署多份，当某个节点不可用时，切换到其他节点。服务冗余对于无状态的服务是相对容易的。服务备份：有些服务是无法同时存在多个运行时的，比如说：Nginx的反向代理，一些集群的leader节点。这时可以存在一个备份服务，处于随时待命状态。自动切换：服务冗余之后，当某个节点不可用时，要
mysql 隐秘后门_【技术分享】CVE-2016-5483：利用mysqldump备份可生成后门 Toby Dai mysql 隐秘后门
预估稿费：100RMB投稿方式：发送邮件至linwei#360.cn，或登陆网页版在线投稿前言mysqldump是用来创建MySQL数据库逻辑备份的一个常用工具。它在默认配置下可以生成一个.sql文件，其中包含创建/删除表和插入数据等。在导入转储文件的时候，攻击者可以通过制造恶意表名来实现任意SQL语句查询和shell命令执行的目的。另一个与之相关的漏洞利用场景可以参考。攻击场景攻击者已经能够访问
处理绿盟科技安全评估的系统漏洞自我修炼的小石头数据库运维开发工具
如下图一次扫描,中度风险39个,这个是必须要处理的.其中mysql占了36个,一看这个感觉整个人都不好了.但是最后解决的办法也很简单,就是升级.系统版本Redhat7.31.telnet因为要升级openssh,openssl,为了避免意外,首先要开启telnet服务,并把所有相关的包上传到服务器.1.1确认是否安装了telnet和xinetd（默认是没有安装）rpm-qatelnet*rpm-q
java类加载顺序 3213213333332132 java
package com.demo; /** * @Description 类加载顺序 * @author FuJianyong * 2015-2-6上午11:21:37 */ public class ClassLoaderSequence { String s1 = "成员属性"; static String s2 = "
Hibernate与mybitas的比较 BlueSkator sql Hibernate 框架 ibatis orm
第一章 Hibernate与MyBatis Hibernate 是当前最流行的O/R mapping框架，它出身于sf.net，现在已经成为Jboss的一部分。 Mybatis 是另外一种优秀的O/R mapping框架。目前属于apache的一个子项目。 MyBatis 参考资料官网：http:
php多维数组排序以及实际工作中的应用 dcj3sjt126com PHP usort uasort
自定义排序函数返回false或负数意味着第一个参数应该排在第二个参数的前面, 正数或true反之, 0相等usort不保存键名uasort 键名会保存下来uksort 排序是对键名进行的 <!doctype html> <html lang="en"> <head> <meta charset="utf-8&q
DOM改变字体大小周华华前端
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml&q
c3p0的配置 g21121 c3p0
c3p0是一个开源的JDBC连接池，它实现了数据源和JNDI绑定，支持JDBC3规范和JDBC2的标准扩展。c3p0的下载地址是：http://sourceforge.net/projects/c3p0/这里可以下载到c3p0最新版本。以在spring中配置dataSource为例：  <bean name="prope
Java获取工程路径的几种方法 510888780 java
第一种： File f = new File(this.getClass().getResource("/").getPath()); System.out.println(f); 结果: C:\Documents%20and%20Settings\Administrator\workspace\projectName\bin 获取当前类的所在工程路径; 如果不加“
在类Unix系统下实现SSH免密码登录服务器 Harry642 免密 ssh
1.客户机 (1)执行ssh-keygen -t rsa -C "[email protected]"生成公钥，xxx为自定义大email地址 (2)执行scp ~/.ssh/id_rsa.pub root@xxxxxxxxx:/tmp将公钥拷贝到服务器上，xxx为服务器地址 (3)执行cat
Java新手入门的30个基本概念一 aijuans java java 入门新手
在我们学习Java的过程中,掌握其中的基本概念对我们的学习无论是J2SE,J2EE,J2ME都是很重要的,J2SE是Java的基础,所以有必要对其中的基本概念做以归纳,以便大家在以后的学习过程中更好的理解java的精髓,在此我总结了30条基本的概念。　　Java概述:　　目前Java主要应用于中间件的开发(middleware)---处理客户机于服务器之间的通信技术,早期的实践证明,Java不适合
Memcached for windows 简单介绍 antlove java Web windows cache memcached
1. 安装memcached server a. 下载memcached-1.2.6-win32-bin.zip b. 解压缩，dos 窗口切换到 memcached.exe所在目录，运行memcached.exe -d install c.启动memcached Server,直接在dos窗口键入 net start "memcached Server&quo
数据库对象的视图和索引百合不是茶索引 oeacle数据库视图
视图视图是从一个表或视图导出的表，也可以是从多个表或视图导出的表。视图是一个虚表，数据库不对视图所对应的数据进行实际存储，只存储视图的定义，对视图的数据进行操作时,只能将字段定义为视图,不能将具体的数据定义为视图为什么oracle需要视图; &
Mockito(一) --入门篇 bijian1013 持续集成 mockito 单元测试
Mockito是一个针对Java的mocking框架，它与EasyMock和jMock很相似，但是通过在执行后校验什么已经被调用，它消除了对期望行为（expectations）的需要。其它的mocking库需要你在执行前记录期望行为（expectations），而这导致了丑陋的初始化代码。 &nb
精通Oracle10编程SQL(5)SQL函数 bijian1013 oracle 数据库 plsql
/* * SQL函数 */ --数字函数 --ABS(n):返回数字n的绝对值 declare v_abs number(6,2); begin v_abs:=abs(&no); dbms_output.put_line('绝对值：'||v_abs); end; --ACOS(n):返回数字n的反余弦值，输入值的范围是-1~1，输出值的单位为弧度
【Log4j一】Log4j总体介绍 bit1129 log4j
Log4j组件：Logger、Appender、Layout Log4j核心包含三个组件：logger、appender和layout。这三个组件协作提供日志功能：日志的输出目标日志的输出格式日志的输出级别(是否抑制日志的输出) logger继承特性 A logger is said to be an ancestor of anothe
Java IO笔记白糖_ java
public static void main(String[] args) throws IOException { //输入流 InputStream in = Test.class.getResourceAsStream("/test"); InputStreamReader isr = new InputStreamReader(in); Bu
Docker 监控 ronin47 docker监控
目前项目内部署了docker，于是涉及到关于监控的事情，参考一些经典实例以及一些自己的想法，总结一下思路。 1、关于监控的内容监控宿主机本身监控宿主机本身还是比较简单的，同其他服务器监控类似，对cpu、network、io、disk等做通用的检查，这里不再细说。额外的，因为是docker的
java-顺时针打印图形 bylijinnan java
一个画图程序要求打印出： 1.int i=5; 2.1 2 3 4 5 3.16 17 18 19 6 4.15 24 25 20 7 5.14 23 22 21 8 6.13 12 11 10 9 7. 8.int i=6 9.1 2 3 4 5 6 10.20 21 22 23 24 7 11.19
关于iReport汉化版强制使用英文的配置方法 Kai_Ge iReport汉化英文版
对于那些具有强迫症的工程师来说，软件汉化固然好用，但是汉化不完整却极为头疼，本方法针对iReport汉化不完整的情况，强制使用英文版，方法如下：在 iReport 安装路径下的 etc/ireport.conf 里增加红色部分启动参数，即可变为英文版。 # ${HOME} will be replaced by user home directory accordin
[并行计算]论宇宙的可计算性 comsci 并行计算
现在我们知道,一个涡旋系统具有并行计算能力.按照自然运动理论,这个系统也同时具有存储能力,同时具备计算和存储能力的系统,在某种条件下一般都会产生意识...... 那么,这种概念让我们推论出一个结论 &nb
用OpenGL实现无限循环的coverflow dai_lm android coverflow
网上找了很久，都是用Gallery实现的，效果不是很满意，结果发现这个用OpenGL实现的，稍微修改了一下源码，实现了无限循环功能源码地址： https://github.com/jackfengji/glcoverflow public class CoverFlowOpenGL extends GLSurfaceView implements GLSurfaceV
JAVA数据计算的几个解决方案1 datamachine java Hibernate 计算
老大丢过来的软件跑了10天，摸到点门道，正好跟以前攒的私房有关联，整理存档。 -----------------------------华丽的分割线------------------------------------- 数据计算层是指介于数据存储和应用程序之间，负责计算数据存储层的数据，并将计算结果返回应用程序的层次。J &nbs
简单的用户授权系统,利用给user表添加一个字段标识管理员的方式 dcj3sjt126com yii
怎么创建一个简单的(非 RBAC)用户授权系统通过查看论坛，我发现这是一个常见的问题，所以我决定写这篇文章。本文只包括授权系统.假设你已经知道怎么创建身份验证系统(登录)。数据库首先在 user 表创建一个新的字段(integer 类型),字段名 'accessLevel',它定义了用户的访问权限扩展 CWebUser 类在配置文件(一般为 protecte
未选之路 dcj3sjt126com 诗
作者:罗伯特*费罗斯特黄色的树林里分出两条路, 可惜我不能同时去涉足, 我在那路口久久伫立, 我向着一条路极目望去, 直到它消失在丛林深处. 但我却选了另外一条路, 它荒草萋萋,十分幽寂; 显得更诱人,更美丽, 虽然在这两条小路上, 都很少留下旅人的足迹. 那天清晨落叶满地, 两条路都未见脚印痕迹. 呵,留下一条路等改日再
Java处理15位身份证变18位蕃薯耀 18位身份证变15位 15位身份证变18位身份证转换
15位身份证变18位，18位身份证变15位 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 蕃薯耀 201
SpringMVC4零配置--应用上下文配置【AppConfig】 hanqunfeng springmvc4
从spring3.0开始，Spring将JavaConfig整合到核心模块，普通的POJO只需要标注@Configuration注解，就可以成为spring配置类，并通过在方法上标注@Bean注解的方式注入bean。 Xml配置和Java类配置对比如下： applicationContext-AppConfig.xml <!-- 激活自动代理功能参看：
Android中webview跟JAVASCRIPT中的交互 jackyrong JavaScript html android 脚本
在android的应用程序中,可以直接调用webview中的javascript代码,而webview中的javascript代码,也可以去调用ANDROID应用程序(也就是JAVA部分的代码).下面举例说明之: 1 JAVASCRIPT脚本调用android程序要在webview中,调用addJavascriptInterface(OBJ,int
8个最佳Web开发资源推荐 lampcy 编程 Web 程序员
Web开发对程序员来说是一项较为复杂的工作，程序员需要快速地满足用户需求。如今很多的在线资源可以给程序员提供帮助，比如指导手册、在线课程和一些参考资料，而且这些资源基本都是免费和适合初学者的。无论你是需要选择一门新的编程语言，或是了解最新的标准，还是需要从其他地方找到一些灵感，我们这里为你整理了一些很好的Web开发资源，帮助你更成功地进行Web开发。这里列出10个最佳Web开发资源，它们都是受
架构师之面试------jdk的hashMap实现 nannan408 HashMap
1.前言。如题。 2.详述。 (1)hashMap算法就是数组链表。数组存放的元素是键值对。jdk通过移位算法（其实也就是简单的加乘算法），如下代码来生成数组下标(生成后indexFor一下就成下标了）。 static int hash(int h) { h ^= (h >>> 20) ^ (h >>>
html禁止清除input文本输入缓存 Rainbow702 html 缓存 input 输入框 change
多数浏览器默认会缓存input的值，只有使用ctl+F5强制刷新的才可以清除缓存记录。如果不想让浏览器缓存input的值，有2种方法：方法一：在不想使用缓存的input中添加 autocomplete="off"; <input type="text" autocomplete="off" n
POJO和JavaBean的区别和联系 tjmljw POJO java beans
POJO 和JavaBean是我们常见的两个关键字，一般容易混淆，POJO全称是Plain Ordinary Java Object / Pure Old Java Object，中文可以翻译成：普通Java类，具有一部分getter/setter方法的那种类就可以称作POJO，但是JavaBean则比 POJO复杂很多， Java Bean 是可复用的组件，对 Java Bean 并没有严格的规
java中单例的五种写法 liuxiaoling java 单例
/** * 单例模式的五种写法： * 1、懒汉 * 2、恶汉 * 3、静态内部类 * 4、枚举 * 5、双重校验锁 */ /** * 五、双重校验锁，在当前的内存模型中无效 */ class LockSingleton { private volatile static LockSingleton singleton; pri