基于DOCKER的MySQL Group Replication + ProxySQL + keepalived 高可用集群搭建

基于DOCKER的MySQL Group Replication + ProxySQL + keepalived 高可用环境配置

整体规划

host	ip	port	remark
服务器域名	服务器外网ip	10088	服务器访问端口
	192.168.100.30		keepalived虚拟ip地址
mysql-proxy	192.168.100.21	6601-admin管理端口,6603-客户端连接端口	ProxySQL代理
mysql-proxy3	192.168.100.23	6601-admin管理端口,6603-客户端连接端口	ProxySQL代理
mysql-master	192.168.100.11	3306-mysql端口，33060-mgr端口	mysql初始主节点
mysql-slave1	192.168.100.12	3306-mysql端口，33060-mgr端口	mysql初始从节点
mysql-slave2	192.168.100.13	3306-mysql端口，33060-mgr端口	mysql初始从节点

基本流程

配置MGR

Step1 给3个mysql容器编写配置文件

master的my.cnf

# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html
[client]    
default-character-set=utf8  
  
[mysql]    
default-character-set=utf8

[mysqld]
skip-name-resolve
character-set-server=utf8
collation-server=utf8_general_ci

innodb_buffer_pool_size=2048M
innodb_log_file_size=128M
query_cache_size=64M
max_connections=128
max_allowed_packet = 50M
log_timestamps=SYSTEM
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

#服务器id
server-id=1
pid-file=/var/run/mysqld/mysqld.pid

gtid_mode=on
enforce_gtid_consistency=on
master_info_repository=table
relay_log_info_repository=table
binlog_checksum=none
log_slave_updates=on
log_bin=binlog
binlog_format=row
relay-log=bogn-relay-bin
#唯一确定事务影响行的主键，必须开启
transaction_write_set_extraction=XXHASH64
#唯一标识一个组
loose-group_replication_group_name="aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
loose-group_replication_allow_local_disjoint_gtids_join=on
#重启自动自动组复制，第一次初始化的时候可以off，不然进去初始化之前需要先关闭stop group_replication
loose-group_replication_start_on_boot=on
#unreachable的超时时间设置
loose-group_replication_unreachable_majority_timeout=10
#用于组间通信的地址
loose-group_replication_local_address="192.168.100.11:33060"
#donor地址
loose-group_replication_group_seeds="192.168.100.11:33060,192.168.100.12:33060,192.168.100.13:33060"
#引导节点设置
loose-group_replication_bootstrap_group=off

[mysqld_safe]
default-character-set=utf8

slave1的my.cnf，只有下面的与master不同

server-id=2
loose-group_replication_local_address= "192.168.100.12:33060"

slave2的my.cnf，同样也只有这两部分需要改动

server-id=3
loose-group_replication_local_address= "192.168.100.13:33060"

Step2 创建3个mysql容器

在docker创建容器之前，先创建一个网络

docker network create --subnet=192.168.100.0/24 mysqlnet

再用docker创建3个mysql容器

/home/mt/mysql/10001/conf/my.cnf是配置文件的存放位置

/etc/mysql/conf.d/my.cnf 是容器内部配置文件的位置

docker run -p 10001:3306 -p 10011:33060 --name master --hostname=mysql-master --net=mysqlnet --ip=192.168.100.11 --add-host mysql-master:192.168.100.11 --add-host mysql-slave1:192.168.100.12 --add-host mysql-slave2:192.168.100.13 -v /home/mt/mysql/10001/conf/my.cnf:/etc/mysql/conf.d/my.cnf -v /home/mt/mysql/10001/logs:/logs -v /home/mt/mysql/10001/data:/mysql_data -e MYSQL_ROOT_PASSWORD=123456 -d mysql:5.7.23

docker run -p 10002:3306 -p 10012:33060 --name slave1 --hostname=mysql-slave1 --net=mysqlnet --ip=192.168.100.12 --add-host mysql-master:192.168.100.11 --add-host mysql-slave1:192.168.100.12 --add-host mysql-slave2:192.168.100.13 -v /home/mt/mysql/10002/conf/my.cnf:/etc/mysql/conf.d/my.cnf -v /home/mt/mysql/10002/logs:/logs -v /home/mt/mysql/10002/data:/mysql_data -e MYSQL_ROOT_PASSWORD=123456 -d mysql:5.7.23

docker run -p 10003:3306 -p 10013:33060 --name slave2 --hostname=mysql-slave2 --net=mysqlnet --ip=192.168.100.13 --add-host mysql-master:192.168.100.11 --add-host mysql-slave1:192.168.100.12 --add-host mysql-slave2:192.168.100.13 -v /home/mt/mysql/10003/conf/my.cnf:/etc/mysql/conf.d/my.cnf -v /home/mt/mysql/10003/logs:/logs -v /home/mt/mysql/10003/data:/mysql_data -e MYSQL_ROOT_PASSWORD=123456 -d mysql:5.7.23

Step3 配置master节点

第一步：创建用于复制的用户
set sql_log_bin=0;
create user mgruser@’%’ identified by ‘123456’;
grant replication slave,replication client on . to mgruser@’%’;
create user mgruser@‘127.0.0.1’ identified by ‘123456’;
grant replication slave,replication client on . to mgruser@‘127.0.0.1’;
create user mgruser@‘localhost’ identified by ‘123456’;
grant replication slave,replication client on . to mgruser@‘localhost’;
set sql_log_bin=1;
flush privileges;

第二步：配置复制所使用的用户
change master to master_user=‘mgruser’,master_password=‘123456’ for channel ‘group_replication_recovery’;
flush privileges;
第三步：安装mysql group replication 这个插件

install plugin group_replication soname ‘group_replication.so’;
　
第四步：建个群(官方点的说法就是初始化一个复制组)

set global group_replication_bootstrap_group=on;
start group_replication;
set global group_replication_bootstrap_group=off;

最后，查看一下当前的组员

SELECT * FROM performance_schema.replication_group_members;

Step4 配置slave节点

步骤的前三步同上面master一样，最后一步只需要start group_replication;

成功之后再查看一下当前的组员

SELECT * FROM performance_schema.replication_group_members;

这里之前按照网上的配置使用的时候有个问题

2018-10-09T02:19:06.813142-00:00 0 [ERROR] Plugin group_replication reported: 'This member has more executed transactions than those present in the group. Local transactions: e1f0e952-cb68-11e8-acef-0242c0a8640c:1-6 > Group transactions: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-2,

解决方法就是在前面的配置文件里面的

loose-group_replication_allow_local_disjoint_gtids_join=on

如果已经进入了发生了这个问题，也可以通过设置

mysql> show variables like ‘group_replication_allow_local_disjoint_gtids_join’;
±--------------------------------------------------±------+
| Variable_name | Value |
±--------------------------------------------------±------+
| group_replication_allow_local_disjoint_gtids_join | OFF |
±--------------------------------------------------±------+
1 row in set (0.00 sec)

mysql> set global group_replication_allow_local_disjoint_gtids_join=1;
Query OK, 0 rows affected (0.00 sec)

mysql> START GROUP_REPLICATION;
Query OK, 0 rows affected (2.12 sec)

至此，GMR的配置就配置完成了，可以关掉master之后看到写节点就变成了slave1

配置ProxySQL 集群

Step5 安装ProxySQL

这里没有找到ProxySQL的官方镜像，于是使用了一个centos的镜像，在centos的镜像里面再安装ProxySQL

第一步，安装centos的镜像

docker run --privileged --name proxysql --hostname=mysql-proxy --net=mysqlnet --ip=192.168.100.21 -p 10021:6601 -p 10022:6602 -p 10023:6603 -v /home/mt/centos:/home -d -i -t centos:7.2.1511 /usr/sbin/init

第二步，安装ProxySQL

根据https://github.com/sysown/proxysql/wiki的官方文档进行安装

Adding repository:

cat <

Installing:

yum install proxysql OR yum install proxysql-version

安装好之后使用systemctl start proxysql即可

Step6 配置ProxySQL

第一步，在mgr的写节点导入视图，这个具体的内容可以在这里下https://github.com/lefred/mysql_gr_routing_check/blob/master/addition_to_sys.sql

mysql -p < addition_to_sys.sql

第二步，proxysql增加帐号

create user ‘monitor’@’%’ identified by ‘monitor’;
grant all privileges on . to ‘monitor’@’%’ with grant option;
create user ‘proxysql’@’%’ identified by ‘proxysql’;
grant all privileges on . to ‘proxysql’@’%’ with grant option;
flush privileges;

第三步，登陆proxysql，配置mysql节点信息

mysql -uadmin -padmin -h127.0.0.1 -P6601

注意admin只能在本地登陆

添加节点

insert into mysql_servers (hostgroup_id, hostname, port) values(1,‘192.168.100.11’, 3306);
insert into mysql_servers (hostgroup_id, hostname, port) values(3,‘192.168.100.12’, 3306);
insert into mysql_servers (hostgroup_id, hostname, port) values(3,‘192.168.100.13’, 3306);

配置用户（主要是添加程序端的这个用户，也就是run，将其设置到写组1里面）
insert into mysql_users(username,password,default_hostgroup) values(‘proxysql’,‘proxysql’,1);

添加mysql group replication 信息
insert into mysql_group_replication_hostgroups (writer_hostgroup,backup_writer_hostgroup,reader_hostgroup, offline_hostgroup,active,max_writers,writer_is_also_reader,max_transactions_behind) values (1,2,3,4,1,1,0,100);

proxysql 读写分离
insert into mysql_query_rules(rule_id,active,match_digest,destination_hostgroup,apply)values(1,1,’^SELECT.*FOR UPDATE$’,1,1);

insert into mysql_query_rules(rule_id,active,match_digest,destination_hostgroup,apply)values(2,1,’^SELECT’,3,1);

最后加载新的配置

load mysql servers to runtime;
load mysql users to runtime;
load mysql variables to runtime;
load mysql query to runtime;
load mysql rules to runtime;

保存新的配置

save mysql servers to disk;
save mysql users to disk;
save mysql variables to disk;
save mysql query to disk;
save mysql rules to disk;

这样整个环境搭建就完成了，登陆验证一下

mysql -uproxysql -pproxysql -h192.168.100.21 -P6603

可以正常增删查改

挂掉master节点，也可以正常增删查改

Step7 配置ProxySQL集群

update global_variables set variable_value='admin:admin;cluster:cluster' where variable_name='admin-admin_credentials';
 update global_variables set variable_value='cluster' where variable_name='admin-cluster_username';
 update global_variables set variable_value='cluster' where variable_name='admin-cluster_password';
 insert into proxysql_servers(hostname,port,weight,comment) values('192.168.100.21',6601,1,'primary'),('192.168.100.23',6601,1,'secondary');
 load admin variables to runtime;
 load proxysql servers to runtime;
save admin variables to disk;
save proxysql servers to disk;

Step8 配置第二台ProxySQL

第二台ProxySQL采用配置文件来配置

贴几个主要的配置部分

admin_variables=
{
	admin_credentials="admin:admin;cluster:cluster"
#	mysql_ifaces="127.0.0.1:6032;/tmp/proxysql_admin.sock"
	mysql_ifaces="0.0.0.0:6601"
#	refresh_interval=2000
#	debug=true
	cluster_username="cluster"
	cluster_password="cluster"
	cluster_check_interval_ms=200                       
	cluster_check_status_frequency=100
	cluster_mysql_query_rules_save_to_disk=true
	cluster_mysql_servers_save_to_disk=true
	cluster_mysql_users_save_to_disk=true
	cluster_proxysql_servers_save_to_disk=true
	cluster_mysql_query_rules_diffs_before_sync=3
	cluster_mysql_servers_diffs_before_sync=3
	cluster_mysql_users_diffs_before_sync=3
	cluster_proxysql_servers_diffs_before_sync=3
}
mysql_variables=
{
	threads=4
	max_connections=2048
	default_query_delay=0
	default_query_timeout=36000000
	have_compress=true
	poll_timeout=2000
#	interfaces="0.0.0.0:6033;/tmp/proxysql.sock"
	interfaces="0.0.0.0:6602"
	default_schema="information_schema"
	stacksize=1048576
	server_version="5.5.30"
	connect_timeout_server=3000
# make sure to configure monitor username and password
# https://github.com/sysown/proxysql/wiki/Global-variables#mysql-monitor_username-mysql-monitor_password
	monitor_username="monitor"
	monitor_password="monitor"
	monitor_history=600000
	monitor_connect_interval=60000
	monitor_ping_interval=10000
	monitor_read_only_interval=1500
	monitor_read_only_timeout=500
	ping_interval_server_msec=120000
	ping_timeout_server=500
	commands_stats=true
	sessions_sort=true
	connect_retries_on_failure=10
}
proxysql_servers =                                        
(
        {
                hostname="192.168.100.21"
                port=6601
                comment="primary"                        
        },
        {
                hostname="192.168.100.23"
                port=6601
                comment="secondary"
        }
)

配置keepalived

Step9 配置100.21上的keepalived

keepalived.conf

! Configuration File for keepalived

global_defs {
   router_id LVS_DEVEL_1
}

vrrp_script chk_proxysql {
    script "/home/check_proxysql.sh"
    interval 1
}

vrrp_instance proxysql {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 90
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass xxxx
    }

    track_script {
        chk_proxysql
    }

    virtual_ipaddress {
        192.168.100.30/24
    }
}

check_proxysql.sh

#!/bin/sh
echo "aaaaa"
peer_ip='192.168.100.23'
peer_port=22
proxysql='proxysql-1'
vip='192.168.100.30'
log=/home/keepalived.log
alias date='date +"%y-%m-%d_%H:%M:%S"'
echo "`date`  enter script." >> $log
#check if this keepalived is MASTER
ip a|grep $vip
if [ $? -ne 0 ];then
    exit 0
fi
echo "`date`  after check keepalived master script." >> $log
#check if data port(6033) is alive
data_port_stats=$(timeout 2  bash -c 'cat < /dev/null > /dev/tcp/0.0.0.0/6603' &> /dev/null;echo $?)
if [ $data_port_stats -eq 0 ];then
    exit 0
else
    #check if the other keepalived is running
    peer_keepalived=$(ssh -p$peer_port $peer_ip 'systemctl is-active keepalived.service')
    if [ "$peer_keepalived" != "active" ];then
        echo "`date`  data port of $proxysql is not available, but the BACKUP keepalived is not running, so can't do the failover" >> $log
    else
        echo "`date`  data port of proxysql is not available, now SHUTDOWN keepalived." >> $log
        systemctl stop keepalived.service
    fi
fi

Step10 配置100.23上的keepalived

keepalived.conf

! Configuration File for keepalived

global_defs {
   router_id LVS_DEVEL_3
}

vrrp_script chk_proxysql {
    script "/home/check_proxysql.sh"
    interval 1
}

vrrp_instance proxysql {
    state BACKUP
    interface eth0
    virtual_router_id 53
    priority 80
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass xxxx
    }

    track_script {
        chk_proxysql
    }

    virtual_ipaddress {
        192.168.100.30/24
    }
}

check_proxysql.sh

#!/bin/sh
echo "aaaaa"
peer_ip='192.168.100.21'
peer_port=22
proxysql='proxysql-2'
vip='192.168.100.30'
log=/home/keepalived.log
alias date='date +"%y-%m-%d_%H:%M:%S"'
echo "`date`  enter script." >> $log
#check if this keepalived is MASTER
ip a|grep $vip
if [ $? -ne 0 ];then
    exit 0
fi
echo "`date`  after check keepalived master script." >> $log
#check if data port(6033) is alive
data_port_stats=$(timeout 2  bash -c 'cat < /dev/null > /dev/tcp/0.0.0.0/6603' &> /dev/null;echo $?)
if [ $data_port_stats -eq 0 ];then
    exit 0
else
    #check if the other keepalived is running
    peer_keepalived=$(ssh -p$peer_port $peer_ip 'systemctl is-active keepalived.service')
    if [ "$peer_keepalived" != "active" ];then
        echo "`date`  data port of $proxysql is not available, but the BACKUP keepalived is not running, so can't do the failover" >> $log
    else
        echo "`date`  data port of proxysql is not available, now SHUTDOWN keepalived." >> $log
        systemctl stop keepalived.service
    fi
fi

配置宿主机端口转发

Step11 在主机配置端口转发

firewall-cmd --permanent --add-forward-port=port=10088:proto=tcp:toaddr=192.168.100.30:toport=6603
firewall-cmd --reload

最后在本地电脑远程通过ip:10088连接到了192.168.100.30的vip代理下的ProxySQL

参考

[1] http://www.enmotech.com/web/detail/1/171/1.html

[2] https://blog.csdn.net/jolly10/article/details/80418758

[3] https://blog.csdn.net/kai404/article/details/77524733

欢迎关注微信交流