ProxySQL读写分离和主节点故障无感知切换（接上一篇多写模式）
ProxySQL读写分离和主节点故障无感知切换
1) 安装mysql客户端，用于在本机连接到ProxySQL的管理接口
[root@proxysql-node ~]# vim /etc/yum.repos.d/mariadb.repo
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/5.5/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1
[root@proxysql-node yum.repos.d]# yum clean all
Loaded plugins: fastestmirror
Cleaning repos: base extras mariadb updates
Cleaning up list of fastest mirrors
[root@proxysql-node yum.repos.d]# yum makecache
[root@proxysql-node ~]# yum install -y MariaDB-client
proxysql的rpm包下载地址: # https://pan.baidu.com/s/1_XQt0kK7TRui5YV6kfp9wA
提取密码:# kb6q
登录数据库
[root@proxysql-node ~]# yum install -y perl-DBI perl-DBD-MySQL
[root@proxysql-node ~]# rpm -ivh proxysql-1.4.8-1-centos7.x86_64.rpm --force
启动proxysql
[root@proxysql-node ~]# /etc/init.d/proxysql start
Starting ProxySQL: DONE!
[root@proxysql-node ~]# ss -lntup|grep proxy
tcp LISTEN 0 128 *:6080 *:* users:(("proxysql",pid=29931,fd=11))
tcp LISTEN 0 128 *:6032 *:* users:(("proxysql",pid=29931,fd=28))
tcp LISTEN 0 128 *:6033 *:* users:(("proxysql",pid=29931,fd=27))
tcp LISTEN 0 128 *:6033 *:* users:(("proxysql",pid=29931,fd=26))
tcp LISTEN 0 128 *:6033 *:* users:(("proxysql",pid=29931,fd=25))
tcp LISTEN 0 128 *:6033 *:* users:(("proxysql",pid=29931,fd=24))
[root@proxysql-node ~]# mysql -uadmin -padmin -h127.0.0.1 -P6032
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.5.30 (ProxySQL Admin Module)
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> show databases;
+-----+---------------+-------------------------------------+
| seq | name | file |
+-----+---------------+-------------------------------------+
| 0 | main | |
| 2 | disk | /var/lib/proxysql/proxysql.db |
| 3 | stats | |
| 4 | monitor | |
| 5 | stats_history | /var/lib/proxysql/proxysql_stats.db |
+-----+---------------+-------------------------------------+
接着初始化Proxysql，将之前的proxysql数据都删除
MySQL [(none)]> delete from scheduler ;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> delete from mysql_servers ;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> delete from mysql_users ;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> delete from mysql_query_rules;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> delete from mysql_group_replication_hostgroups;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> LOAD MYSQL VARIABLES TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> SAVE MYSQL VARIABLES TO DISK;
Query OK, 94 rows affected (0.01 sec)
MySQL [(none)]> LOAD MYSQL SERVERS TO RUNTIME;
Query OK, 0 rows affected (0.01 sec)
MySQL [(none)]> SAVE MYSQL SERVERS TO DISK;
Query OK, 0 rows affected (0.01 sec)
MySQL [(none)]> LOAD MYSQL USERS TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> SAVE MYSQL USERS TO DISK;
Query OK, 0 rows affected (0.01 sec)
MySQL [(none)]> LOAD SCHEDULER TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> SAVE SCHEDULER TO DISK;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> LOAD MYSQL QUERY RULES TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> SAVE MYSQL QUERY RULES TO DISK;
Query OK, 0 rows affected (0.01 sec)
在数据库端建立proxysql登入需要的帐号 （在三个MGR任意一个节点上操作，会自动同步到其他节点）
mysql> set global validate_password_policy=0;
Query OK, 0 rows affected (0.00 sec)
mysql> set global validate_password_length=1;
Query OK, 0 rows affected (0.00 sec) #临时设置密码策略
mysql> CREATE USER 'proxysql'@'%' IDENTIFIED BY '123.com';
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL ON * . * TO 'proxysql'@'%';
Query OK, 0 rows affected (0.00 sec)
mysql> create user 'sbuser'@'%' IDENTIFIED BY '123.com';
Query OK, 0 rows affected (0.00 sec)
mysql> GRANT ALL ON * . * TO 'sbuser'@'%';
Query OK, 0 rows affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
创建检查MGR节点状态的函数和视图 （在三个MGR任意一个节点上操作，会自动同步到其他节点）
在MGR-node1节点上，创建系统视图sys.gr_member_routing_candidate_status，该视图将为ProxySQL提供组复制相关的监控状态指标。
下载addition_to_sys.sql脚本，在MGR-node1节点执行如下语句导入MySQL即可 (在mgr-node1节点的mysql执行后，会同步到其他两个节点上)。
链接#： https://pan.baidu.com/s/1WfQEH4m0VfDhZiEZmo4W-g
提取码#：3vsumysql> START GROUP_REPLICATION;
mysql> START GROUP_REPLICATION;
ERROR 3092 (HY000): The server is not configured properly to be an active member of the group. Please see more details on error log.
#分析查看日志信息，节点开始组复制后，会连接其他节点，获取信息，但是无法连接，因为此时集群中其他节点尚未开始组复制，集群中也没有主节点。在一节点上执行如下命令，让该节点成为主节点，其他节点复制该节点上的数据
mysql> set global group_replication_bootstrap_group = ON;
Query OK, 0 rows affected (0.00 sec)
mysql> START GROUP_REPLICATION;
Query OK, 0 rows affected (2.01 sec)
mysql> START GROUP_REPLICATION;
ERROR 3092 (HY000): The server is not configured properly to be an active member of the group. Please see more details on error log.
mysql> set global group_replication_bootstrap_group = ON;
Query OK, 0 rows affected (0.00 sec)
mysql> START GROUP_REPLICATION;
Query OK, 0 rows affected (2.01 sec)
mysql> select * from sys.gr_member_routing_candidate_status;
+------------------+-----------+---------------------+----------------------+
| viable_candidate | read_only | transactions_behind | transactions_to_cert |
+------------------+-----------+---------------------+----------------------+
| YES | NO | 0 | 0 |
+------------------+-----------+---------------------+----------------------+
1 row in set (0.01 sec)
在proxysql中增加帐号
MySQL [(none)]> INSERT INTO MySQL_users(username,password,default_hostgroup) VALUES ('proxysql','123.com',1);
Query OK, 1 row affected (0.00 sec)
MySQL [(none)]> UPDATE global_variables SET variable_value='proxysql' where variable_name='mysql-monitor_username';
Query OK, 1 row affected (0.00 sec)
MySQL [(none)]> UPDATE global_variables SET variable_value='123.com' where variable_name='mysql-monitor_password';
Query OK, 1 row affected (0.00 sec)
MySQL [(none)]> LOAD MYSQL SERVERS TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> SAVE MYSQL SERVERS TO DISK;
Query OK, 0 rows affected (0.01 sec)
测试一下能否正常登入数据库
如果下面测试登录时报错:
[root@proxysql-node ~]# mysql -uproxysql -p123.com -h 127.0.0.1 -P6033 -e"select @@hostname"
ERROR 1045 (28000): ProxySQL Error: Access denied for user 'proxysql'@'127.0.0.1' (using password: YES)
但是检查发现，明明用户名和密码已经修改成proxysql:proxysql了
MySQL [(none)]> select * from global_variables;
..........
| mysql-interfaces | 0.0.0.0:6033 |
| mysql-default_schema | information_schema |
| mysql-stacksize | 1048576 |
| mysql-server_version | 5.5.30 |
| mysql-connect_timeout_server | 3000 |
| mysql-monitor_username | proxysql |
| mysql-monitor_password | proxysql |
#解决办法: 依次执行下面的命令
MySQL [(none)]> delete from mysql_servers;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> insert into mysql_servers (hostgroup_id, hostname, port) values(1,'192.168.145.134',3306);
Query OK, 1 row affected (0.00 sec)
MySQL [(none)]> insert into mysql_servers (hostgroup_id, hostname, port) values(1,'192.168.145.135',3306);
Query OK, 1 row affected (0.00 sec)
MySQL [(none)]> insert into mysql_servers (hostgroup_id, hostname, port) values(1,'192.168.145.136',3306);
Query OK, 1 row affected (0.00 sec)
MySQL [(none)]> insert into mysql_servers (hostgroup_id, hostname, port) values(2,'192.168.145.134',3306);
Query OK, 1 row affected (0.00 sec)
MySQL [(none)]> insert into mysql_servers (hostgroup_id, hostname, port) values(2,'192.168.145.135',3306);
Query OK, 1 row affected (0.00 sec)
MySQL [(none)]> insert into mysql_servers (hostgroup_id, hostname, port) values(2,'192.168.145.136',3306);
Query OK, 1 row affected (0.00 sec)
MySQL [(none)]> select * from mysql_servers ;
+--------------+-----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | 192.168.145.134 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.145.135 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.145.136 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.134 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.135 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.136 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+-----------------+------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
6 rows in set (0.00 sec)
#hostgroup_id = 1代表write group，针对我们提出的限制，这个地方只配置了一个节点；
#hostgroup_id = 2代表read group，包含了MGR的所有节点,目前只是Onlinle的，等配置过scheduler后，status就会有变化 。
#对于上面的hostgroup配置，默认所有的写操作会发送到hostgroup_id为1的online节点，也就是发送到写节点上。所有的读操作，会发送为hostgroup_id为2的online节点。
#需要确认一下没有使用proxysql的读写分离规则（因为之前测试中配置了这个地方，所以需要删除，以免影响后面的测试）。
MySQL [(none)]> delete from mysql_query_rules;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> commit;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> LOAD MYSQL VARIABLES TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> SAVE MYSQL VARIABLES TO DISK;
Query OK, 94 rows affected (0.01 sec)
MySQL [(none)]> LOAD MYSQL SERVERS TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> SAVE MYSQL SERVERS TO DISK;
Query OK, 0 rows affected (0.01 sec)
MySQL [(none)]> LOAD MYSQL USERS TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> SAVE MYSQL USERS TO DISK;
Query OK, 0 rows affected (0.00 sec)
#再次验证proxysql登录
[root@proxysql-node ~]# mysql -uproxysql -p123.com -h 127.0.0.1 -P6033 -e"select @@hostname"
+------------+
| @@hostname |
+------------+
| MGR-node3 |
+------------+
配置scheduler
脚本proxysql_groupreplication_checker.sh：用于multi-primary模式，可以实现读写分离，以及故障切换，同一时间点多个节点可以多写；
#!/bin/bash
## inspired by proxysql_galera_checker.sh
# Author: Frédéric -lefred- Descamps
# version: 0.1
# 2016-08-25
# CHANGE THOSE
PROXYSQL_USERNAME="admin"
PROXYSQL_PASSWORD="admin"
PROXYSQL_HOSTNAME="127.0.0.1"
PROXYSQL_PORT="6032"
#
function usage()
{
echo "Usage: $0 [hostgroup_id read] [number writers] [writers are readers 0|1} [log_file]"
exit 0
}
if [ "$1" = '-h' -o "$1" = '--help' -o -z "$1" ]
then
usage
fi
if [ $# -lt 1 ]
then
echo "Invalid number of arguments"
usage
fi
HOSTGROUP_WRITER_ID="${1}"
HOSTGROUP_READER_ID="${2:--1}"
NUMBER_WRITERS="${3:-0}"
WRITER_IS_READER="${4:-1}"
ERR_FILE="${5:-/dev/null}"
#echo "Hostgroup writers $HOSTGROUP_WRITER_ID"
#echo "Hostgroup readers $HOSTGROUP_READER_ID"
#echo "Number of writers $NUMBER_WRITERS"
#echo "Writers are readers $WRITER_IS_READER"
#echo "log file $ERR_FILE"
#Timeout exists for instances where mysqld may be hung
TIMEOUT=10
PROXYSQL_CMDLINE="mysql -u$PROXYSQL_USERNAME -p$PROXYSQL_PASSWORD -h $PROXYSQL_HOSTNAME -P $PROXYSQL_PORT --protocol=tcp -Nse"
MYSQL_CREDENTIALS=$($PROXYSQL_CMDLINE "SELECT variable_value FROM global_variables WHERE variable_name IN ('mysql-monitor_username','mysql-monitor_password') ORDER BY variable_name DESC")
MYSQL_USERNAME=$(echo $MYSQL_CREDENTIALS | awk '{print $1}')
MYSQL_PASSWORD=$(echo $MYSQL_CREDENTIALS | awk '{print $2}')
#echo $MYSQL_CREDENTIALS
#echo $MYSQL_USERNAME $MYSQL_PASSWORD
MYSQL_CMDLINE="timeout $TIMEOUT mysql -u$MYSQL_USERNAME -p$MYSQL_PASSWORD "
$PROXYSQL_CMDLINE "SELECT hostgroup_id, hostname, port, status, max_replication_lag FROM mysql_servers WHERE hostgroup_id IN ($HOSTGROUP_WRITER_ID, $HOSTGROUP_READER_ID) AND status <> 'OFFLINE_HARD'" | while read hostgroup server port stat max_replication_lag
do
read GR_STATUS READONLY TRX_BEHIND <<<$($MYSQL_CMDLINE -h $server -P $port -Nse "SELECT viable_candidate, read_only, transactions_behind FROM sys.gr_member_routing_candidate_status" 2>>${ERR_FILE} | tail -1 2>>${ERR_FILE})
echo "`date` Check server $hostgroup:$server:$port , status $stat , GR_STATUS $GR_STATUS, READONLY $READONLY, TRX_BEHIND $TRX_BEHIND" >> ${ERR_FILE}
UPDATE_STATS_ONLINE_CMD="UPDATE mysql_servers SET status='ONLINE' WHERE hostgroup_id=$hostgroup AND hostname='$server' AND port='$port'; LOAD MYSQL SERVERS TO RUNTIME;"
UPDATE_STATS_OFFLINE_SOFT_CMD="UPDATE mysql_servers SET status='OFFLINE_SOFT' WHERE hostgroup_id=$hostgroup AND hostname='$server' AND port='$port'; LOAD MYSQL SERVERS TO RUNTIME;"
if [ "${GR_STATUS}" == "" -a "${READONLY}" == "" -a "${TRX_BEHIND}" == "" ] ; then
# case : mysql server can not reach
echo "`date` Changing server $hostgroup:$server:$port to status OFFLINE_SOFT" >> ${ERR_FILE}
$PROXYSQL_CMDLINE "${UPDATE_STATS_OFFLINE_SOFT_CMD}" 2>> ${ERR_FILE}
elif [ "${GR_STATUS}" == "YES" -a "${READONLY}" == "NO" -a "$stat" != "ONLINE" -a ${TRX_BEHIND} -le $max_replication_lag ] ; then
echo "`date` Changing server $hostgroup:$server:$port to status ONLINE" >> ${ERR_FILE}
$PROXYSQL_CMDLINE "${UPDATE_STATS_ONLINE_CMD}" 2>> ${ERR_FILE}
elif [ "${GR_STATUS}" == "YES" -a "${READONLY}" == "YES" -a "$stat" != "ONLINE" -a "$hostgroup" == "$HOSTGROUP_READER_ID" -a ${TRX_BEHIND} -le $max_replication_lag ] ; then
echo "`date` Changing server $hostgroup:$server:$port to status ONLINE" >> ${ERR_FILE}
$PROXYSQL_CMDLINE "${UPDATE_STATS_ONLINE_CMD}" 2>> ${ERR_FILE}
elif [ "${GR_STATUS}" == "NO" -o "${READONLY}" == "YES" -a "$stat" = "ONLINE" -a "$hostgroup" == "$HOSTGROUP_WRITER_ID" ] ; then
echo "`date` Changing server $hostgroup:$server:$port to status OFFLINE_SOFT" >> ${ERR_FILE}
$PROXYSQL_CMDLINE "${UPDATE_STATS_OFFLINE_SOFT_CMD}" 2>> ${ERR_FILE}
elif [ "${GR_STATUS}" == "YES" -a "${READONLY}" == "NO" -a "$stat" = "ONLINE" -a ${TRX_BEHIND} -gt $max_replication_lag ] ; then
echo "`date` Changing server $hostgroup:$server:$port to status OFFLINE_SOFT" >> ${ERR_FILE}
$PROXYSQL_CMDLINE "${UPDATE_STATS_OFFLINE_SOFT_CMD}" 2>> ${ERR_FILE}
fi
done
if [ $NUMBER_WRITERS -gt 0 ]
then
CONT=0
# Only check online servers
$PROXYSQL_CMDLINE "SELECT hostname, port FROM mysql_servers WHERE hostgroup_id = $HOSTGROUP_WRITER_ID AND status = 'ONLINE' order by hostname, port" | while read server port
do
if [ $CONT -ge $NUMBER_WRITERS ]
then
# Number of writers reached, disabling extra servers
echo "`date` Number of writers reached, disabling extra write server $HOSTGROUP_WRITER_ID:$server:$port to status OFFLINE_SOFT" >> ${ERR_FILE}
$PROXYSQL_CMDLINE "UPDATE mysql_servers set status = 'OFFLINE_SOFT' WHERE hostgroup_id = $HOSTGROUP_WRITER_ID AND hostname = '$server' AND port = $port;" 2>> ${ERR_FILE}
fi
CONT=$(( $CONT + 1 ))
done
fi
if [ $WRITER_IS_READER -eq 0 ]
then
# Writer is not a read node, but only if we have another read node online
READER_NON_WRITER=$($PROXYSQL_CMDLINE "SELECT count(*) FROM mysql_servers ms1 LEFT JOIN mysql_servers ms2 ON ms1.hostname = ms2.hostname AND ms1.port = ms2.port AND ms1.hostgroup_id <> ms2.hostgroup_id WHERE ms1.hostgroup_id = $HOSTGROUP_READER_ID AND ms1.status = 'ONLINE' AND (ms2.hostgroup_id = $HOSTGROUP_WRITER_ID OR ms2.hostgroup_id IS NULL) AND (ms2.status = 'OFFLINE_SOFT' OR ms2.hostgroup_id IS NULL);" 2>>${ERR_FILE})
if [ $READER_NON_WRITER -gt 0 ]
then
$PROXYSQL_CMDLINE "SELECT hostname, port FROM mysql_servers WHERE hostgroup_id = $HOSTGROUP_WRITER_ID AND status = 'ONLINE' order by hostname, port" | while read server port
do
echo "`date` Disabling read for write server $HOSTGROUP_READER_ID:$server:$port to status OFFLINE_SOFT" >> ${ERR_FILE}
$PROXYSQL_CMDLINE "UPDATE mysql_servers set status = 'OFFLINE_SOFT' WHERE hostgroup_id = $HOSTGROUP_READER_ID AND hostname = '$server' AND port = $port;" 2>> ${ERR_FILE}
done
else
echo "`date` Not enough read servers, we won't disable read in write servers" >> ${ERR_FILE}
fi
fi
echo "`date` Enabling config" >> ${ERR_FILE}
$PROXYSQL_CMDLINE "LOAD MYSQL SERVERS TO RUNTIME;" 2>> ${ERR_FILE}
exit 0
将脚本proxysql_groupreplication_checker.sh放到目录/var/lib/proxysql/下，并增加可以执行的权限:
[root@proxysql-node ~]# chmod a+x /var/lib/proxysql/proxysql_groupreplication_checker.sh
[root@proxysql-node ~]# ll /var/lib/proxysql/proxysql_groupreplication_checker.sh
-rwxr-xr-x 1 root root 6081 Jul 21 23:21 /var/lib/proxysql/proxysql_groupreplication_checker.sh
最后，在proxysql的scheduler表里面加载如下记录，然后加载到RUNTIME使其生效，同时还可以持久化到磁盘:
执行语句"
MySQL [(none)]> INSERT INTO scheduler(id,interval_ms,filename,arg1,arg2,arg3,arg4, arg5) VALUES (1,'10000','/var/lib/proxysql/proxysql_groupreplication_checker.sh','1','2','1','0','/var/lib/proxysql/proxysql_groupreplication_checker.log');
Query OK, 1 row affected (0.00 sec)
MySQL [(none)]> select * from scheduler
-> ;
+----+--------+-------------+--------------------------------------------------------+------+------+------+------+---------------------------------------------------------+---------+
| id | active | interval_ms | filename | arg1 | arg2 | arg3 | arg4 | arg5 | comment |
+----+--------+-------------+--------------------------------------------------------+------+------+------+------+---------------------------------------------------------+---------+
| 1 | 1 | 10000 | /var/lib/proxysql/proxysql_groupreplication_checker.sh | 1 | 2 | 1 | 0 | /var/lib/proxysql/proxysql_groupreplication_checker.log | |
+----+--------+-------------+--------------------------------------------------------+------+------+------+------+---------------------------------------------------------+---------+
1 row in set (0.00 sec)
MySQL [(none)]> LOAD SCHEDULER TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)
scheduler各column的说明：
active : 1: enable scheduler to schedule the script we provide
interval_ms : invoke one by one in cycle (eg: 5000(ms) = 5s represent every 5s invoke the script)
filename: represent the script file path
arg1~arg5: represent the input parameters the script received
脚本proxysql_groupreplication_checker.sh对应的参数说明如下:
arg1 is the hostgroup_id for write
arg2 is the hostgroup_id for read
arg3 is the number of writers we want active at the same time
arg4 represents if we want that the member acting for writes is also candidate for reads
arg5 is the log file
schedule信息加载后，就会分析当前的环境，mysql_servers中显示出当前只有192.168.145.134是可以写的，
192.168.145.135以及192.168.145.136是用来读的。
MySQL [(none)]> select * from mysql_servers ;
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | 192.168.145.134 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.145.135 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.145.136 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.134 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.135 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.136 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
6 rows in set (0.01 sec)
因为schedule的arg4，我这里设为了0，就表示可写的节点不能用于读。那我将arg4设置为1试一下：
MySQL [(none)]> update scheduler set arg4=1;
Query OK, 1 row affected (0.000 sec)
MySQL [(none)]> select * from scheduler;
+----+--------+-------------+--------------------------------------------------------+------+------+------+------+---------------------------------------------------------+---------+
| id | active | interval_ms | filename | arg1 | arg2 | arg3 | arg4 | arg5 | comment |
+----+--------+-------------+--------------------------------------------------------+------+------+------+------+---------------------------------------------------------+---------+
| 1 | 1 | 10000 | /var/lib/proxysql/proxysql_groupreplication_checker.sh | 1 | 2 | 1 | 1 | /var/lib/proxysql/proxysql_groupreplication_checker.log | |
+----+--------+-------------+--------------------------------------------------------+------+------+------+------+---------------------------------------------------------+---------+
1 row in set (0.000 sec)
MySQL [(none)]> SAVE SCHEDULER TO DISK;
Query OK, 0 rows affected (0.286 sec)
MySQL [(none)]> LOAD SCHEDULER TO RUNTIME;
Query OK, 0 rows affected (0.000 sec)
MySQL [(none)]> select * from mysql_servers;
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | 192.168.145.134 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.145.135 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.145.136 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.134 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.135 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.136 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
6 rows in set (0.00 sec)
arg4设置为1之后192.168.145.134节点用来写的同时，也可以被用来读。
便于下面的测试还是将arg4设为0：
MySQL [(none)]> update scheduler set arg4=0;
Query OK, 1 row affected (0.000 sec)
MySQL [(none)]> SAVE SCHEDULER TO DISK;
Query OK, 0 rows affected (0.197 sec)
MySQL [(none)]> LOAD SCHEDULER TO RUNTIME;
Query OK, 0 rows affected (0.000 sec)
MySQL [(none)]> select * from mysql_servers;
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | 192.168.145.134 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.145.135 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.145.136 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.134 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.135 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.136 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
6 rows in set (0.00 sec)
各个节点的gr_member_routing_candidate_status视图也显示了当前节点是否是正常状态的，
proxysql就是读取的这个视图的信息来决定此节点是否可用。
mysql> select * from sys.gr_member_routing_candidate_status\G;
*************************** 1. row ***************************
viable_candidate: YES
read_only: NO
transactions_behind: 0
transactions_to_cert: 0
1 row in set (0.00 sec)
ERROR:
No query specified
设置读写分离
MySQL [(none)]> insert into mysql_query_rules (active, match_pattern, destination_hostgroup, apply) values (1,"^SELECT",2,1);
Query OK, 1 row affected (0.00 sec)
MySQL [(none)]> LOAD MYSQL QUERY RULES TO RUNTIME;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> SAVE MYSQL QUERY RULES TO DISK;
Query OK, 0 rows affected (0.01 sec)
MySQL [(none)]> insert into mysql_query_rules(active,match_pattern,destination_hostgroup,apply) values(1,'^SELECT.*FOR UPDATE$',1,1);
Query OK, 1 row affected (0.00 sec)
[root@MGR-node3 ~]# mysql -uproxysql -p123.com -h192.168.145.137 -P6033 -e "select @@hostname"
mysql: [Warning] Using a password on the command line interface can be insecure.
+------------+
| @@hostname |
+------------+
| MGR-node3 |
+------------+
[root@MGR-node3 ~]# mysql -uproxysql -p123.com -h192.168.145.137 -P6033 -e "select @@hostname"
mysql: [Warning] Using a password on the command line interface can be insecure.
+------------+
| @@hostname |
+------------+
| MGR-node3 |
+------------+
[root@MGR-node3 ~]# mysql -uproxysql -p123.com -h192.168.145.137 -P6033 -e "select @@hostname"
mysql: [Warning] Using a password on the command line interface can be insecure.
+------------+
| @@hostname |
+------------+
| MGR-node2 |
+------------+
验证数据的读写分离效果
[root@proxysql-node ~]# mysql -uproxysql -p123.com -h192.168.145.137 -P6033 -e "select @@hostname"
+------------+
| @@hostname |
+------------+
| MGR-node3 |
+------------+
[root@proxysql-node ~]# mysql -uproxysql -p123.com -h192.168.145.137 -P6033 -e "select * from test.haha"
+----+----------+
| id | name |
+----+----------+
| 1 | wangwu |
| 2 | maliu |
| 11 | beijing |
| 12 | shanghai |
| 13 | anhui |
+----+----------+
[root@proxysql-node ~]# mysql -uproxysql -p123.com -h192.168.145.137 -P6033 -e "delete from test.haha where id=1;"
[root@proxysql-node ~]# mysql -uproxysql -p123.com -h192.168.145.137 -P6033 -e "delete from test.haha where id=2;"
[root@proxysql-node ~]# mysql -uproxysql -p123.com -h192.168.145.137 -P6033 -e "select * from test.haha"
+----+----------+
| id | name |
+----+----------+
| 11 | beijing |
| 12 | shanghai |
| 13 | anhui |
+----+----------+
[root@proxysql-node ~]# mysql -uproxysql -p123.com -h192.168.145.137 -P6033 -e "select * from test.haha"
+----+----------+
| id | name |
+----+----------+
| 11 | beijing |
| 12 | shanghai |
| 13 | anhui |
+----+----------+
[root@proxysql-node ~]# mysql -uproxysql -p123.com -h192.168.145.137 -P6033 -e 'insert into test.haha values(21,"test"),(22,"hello"),(23,"zhangsan");'
[root@proxysql-node ~]# mysql -uproxysql -p123.com -h192.168.145.137 -P6033 -e "select * from test.haha"
+----+----------+
| id | name |
+----+----------+
| 11 | beijing |
| 12 | shanghai |
| 13 | anhui |
| 21 | test |
| 22 | hello |
| 23 | zhangsan |
+----+----------+
最后在proxysql管理端查看读写分离情况
[root@proxysql-node ~]# mysql -uadmin -padmin -h 127.0.0.1 -P6032
MySQL [(none)]> select hostgroup,username,digest_text,count_star from stats_mysql_query_digest;
+-----------+----------+-----------------------------------------------+------------+
| hostgroup | username | digest_text | count_star |
+-----------+----------+-----------------------------------------------+------------+
| 1 | proxysql | insert into test.haha values(?,?),(?,?),(?,?) | 1 |
| 1 | proxysql | delete from test.haha where id=? | 2 |
| 2 | proxysql | select * from test.haha | 3 |
| 2 | proxysql | select @@hostname | 17 |
| 1 | proxysql | select @@version_comment limit ? | 23 |
+-----------+----------+-----------------------------------------------+------------+
5 rows in set (0.00 sec)
MySQL [(none)]> select * from mysql_servers;
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | 192.168.145.134 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.145.135 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.145.136 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.134 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.135 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.136 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
6 rows in set (0.00 sec)
通过上面可以看到：
写操作都分配到了group1组内，即写操作分配到192.168.145.134节点上。
读操作都分配到了group2组内，即读操作分配到192.168.145.135、192.168.145.136节点上。
设置故障应用无感应
在上面的读写分离规则中，我设置了192.168.145.134为可写节点，192.168.145.135、192.168.145.136为只读节点
现手动将192.168.145.134变成只读模式：
[root@MGR-node1 ~]# mysql -p123.com
mysql> set global read_only=1;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> select * from mysql_servers;
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | 192.168.145.134 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.145.135 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.145.136 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.134 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.135 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.136 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
6 rows in set (0.00 sec)
[root@MGR-node3 ~]# mysql -uproxysql -p123.com -h192.168.145.137 -P6033 -e "select @@hostname"
mysql: [Warning] Using a password on the command line interface can be insecure.
+------------+
| @@hostname |
+------------+
| MGR-node3 |
+------------+
[root@MGR-node3 ~]# mysql -uproxysql -p123.com -h192.168.145.137 -P6033 -e "select @@hostname"
mysql: [Warning] Using a password on the command line interface can be insecure.
+------------+
| @@hostname |
+------------+
| MGR-node3 |
+------------+
[root@MGR-node3 ~]# mysql -uproxysql -p123.com -h192.168.145.137 -P6033 -e "select @@hostname"
mysql: [Warning] Using a password on the command line interface can be insecure.
+------------+
| @@hostname |
+------------+
| MGR-node1 |
+------------+
将192.168.145.134的可读模式改回来
mysql> set global read_only=0;
Query OK, 0 rows affected (0.00 sec)
MySQL [(none)]> select * from mysql_servers;
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 1 | 192.168.145.134 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.145.135 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 1 | 192.168.145.136 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.134 | 3306 | OFFLINE_SOFT | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.135 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
| 2 | 192.168.145.136 | 3306 | ONLINE | 1 | 0 | 1000 | 0 | 0 | 0 | |
+--------------+-----------------+------+--------------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
6 rows in set (0.00 sec)
到此，ProxySQL就简单实现了MGR的读写分离和主节点故障无感知环境