MGR node1:192.168.222.76:3307
MGR node2:192.168.222.77:3307
MGR node3:192.168.222.78:3307
ProxySQL:192.168.222.75
分别在3台数据节点(192.168.222.76、77、78)上执行:
(1)设置hostname:
hostnamectl set-hostname mgr_node1 (node2/node3)
(2)编辑 /etc/hosts,新增host信息
192.168.222.76 mgr_node1
192.168.222.77 mgr_node2
192.168.222.78 mgr_node3
(3)关闭SELinux:setenforce 0
(4)修改ulimit值,vim /etc/security/limits.conf
* soft nofile 1024000
* hard nofile 1024000
rpm -e mariadb-devel
yum install mysql-community-common
yum install mysql-community-libs
yum install mysql-community-devel
yum install mysql-community-libs-compat
yum install libaio
groupadd mysql
useradd -r -g mysql -s /bin/false mysql
xz -d mysql-8.0.20-linux-glibc2.12-x86_64.tar.xz
tar xvf mysql-8.0.20-linux-glibc2.12-x86_64.tar -C /usr/local/
ln -s mysql-8.0.20-linux-glibc2.12-x86_64 mysql3307
cd mysql3307
mkdir mysql-files
chown mysql:mysql mysql-files
chmod 750 mysql-files
以第一节点为例,MGR关键配置如下
server_id=2220763307
report_host='mgr_node1'
gtid_mode=ON
enforce_gtid_consistency=ON
binlog_format=row
binlog_checksum=NONE
disabled_storage_engines="MyISAM,BLACKHOLE,FEDERATED,ARCHIVE,MEMORY"
#兼容老版本客户端
default_authentication_plugin=mysql_native_password
# 默认算法已经是XXHASH64,可以不配置
transaction_write_set_extraction=XXHASH64
loose-group_replication_group_name="aaaaaaaa-aaaa-aaaa-aaaaaaaaaaaaaaaa"
# 单主
loose-group_replication_single_primary_mode=ON
# 单主模式下,关闭该配置
loose-group_replication_enforce_update_everywhere_checks=OFF
# 不自动启动group replication
loose-group_replication_start_on_boot=OFF
loose-group_replication_local_address= "192.168.222.76:33071"
loose-group_replication_group_seeds= "192.168.222.76:33071,192.168.222.77:33071,192.168.222.78.33071"
# 设立当前节点为集群引导节点时短暂开启,默认OFF,可以不配置
loose-group_replication_bootstrap_group=OFF
运行uuidgen,生成的uuid填入:group_replication_group_name
./bin/mysqld --defaults-file=/usr/local/mysql3307/my.cnf --initialize-insecure --user=mysql
./bin/mysqld --defaults-file=/usr/local/mysql3307/my.cnf &
CREATE USER 'mgr_repl'@'%' IDENTIFIED WITH mysql_native_password BY '123456';
# 集群间同步数据用
GRANT REPLICATION SLAVE ON *.* TO 'mgr_repl'@'%';
INSTALL PLUGIN group_replication SONAME 'group_replication.so';
CHANGE MASTER TO MASTER_USER='mgr_repl', MASTER_PASSWORD='123456' FOR CHANNEL 'group_replication_recovery';
FLUSH PRIVILEGES;
set global group_replication_bootstrap_group=ON;
start group_replication;
set global group_replication_bootstrap_group=OFF;
确认集群状态:
root@testnode1 [14:38] > SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 3d35218d-94c5-11ea-b213-005056b8a4a4 | mgr_node1 | 3307 | ONLINE | PRIMARY | 8.0.20 |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
1 row in set (0.01 sec)
新节点正常初始化后启动,执行:
INSTALL PLUGIN group_replication SONAME 'group_replication.so';
(非必要,建议执行)INSTALL PLUGIN clone SONAME 'mysql_clone.so';
CHANGE MASTER TO MASTER_USER='mgr_repl', MASTER_PASSWORD='123456' FOR CHANNEL 'group_replication_recovery';
start group_replication;
查看集群中节点:
root@testnode2 [18:04] > SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 3d35218d-94c5-11ea-b213-005056b8a4a4 | mgr_node1 | 3307 | ONLINE | PRIMARY | 8.0.20 |
| group_replication_applier | 5da974a5-94f3-11ea-a178-005056b8c7db | mgr_node2 | 3307 | ONLINE | SECONDARY | 8.0.20 |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
2 rows in set (0.01 sec)
名词解释:Donor节点(集群中已有节点) = > Recipient节点(欲加入集群的节点)
首先,在donor节点检查是否已安装mysql_clone插件:
show plugins;
INSTALL PLUGIN clone SONAME 'mysql_clone.so';
然后,新节点正常初始化后启动,安装 group_replication 和 mysql_clone 插件:
INSTALL PLUGIN group_replication SONAME 'group_replication.so';
INSTALL PLUGIN clone SONAME 'mysql_clone.so';
recipient节点的group replication插件需要在克隆前就确保已安装,否则启动克隆时报错如下:
ERROR 3870 (HY000): Clone Donor plugin group_replication is not active in Recipient.
远程备份账号权限:BACKUP_ADMIN
CREATE USER 'clone_user'@'%' IDENTIFIED BY '123456';
GRANT BACKUP_ADMIN on *.* to 'clone_user'@'%';
flush privileges;
来到donor节点
# cd /data/
# mkdir clone_dir
# chown -R mysql:mysql clone_dir
mysql > clone local data directory = '/data/clone_dir/20200513';
备份进度可以通过如下sql查看:
root@testnode1 [21:59] > SELECT STAGE, STATE, END_TIME FROM performance_schema.clone_progress;
+-----------+-------------+----------------------------+
| STAGE | STATE | END_TIME |
+-----------+-------------+----------------------------+
| DROP DATA | Completed | 2020-05-13 21:48:34.584910 |
| FILE COPY | Completed | 2020-05-13 21:48:40.907383 |
| PAGE COPY | Completed | 2020-05-13 21:48:41.113312 |
| REDO COPY | Completed | 2020-05-13 21:48:41.315660 |
| FILE SYNC | Completed | 2020-05-13 21:48:51.580895 |
| RESTART | Not Started | NULL |
| RECOVERY | Not Started | NULL |
+-----------+-------------+----------------------------+
7 rows in set (0.00 sec)
Tips:备份的文件配合一份新的配置文件,就可以在本地快速引导一套新的mysql实例
注意修改以下配置项,避免冲突:
server_id=2220783317
report_host='mgr_node4'
port = 3317
mysqlx-port=33170
datadir = /tmp/clone_dir/20200514
loose-group_replication_local_address= "192.168.222.78:33171"
loose-group_replication_group_seeds= "192.168.222.76:33071,192.168.222.77:33071,192.168.222.78.33071,192.168.222.78:33171"
root@testnode3 [15:17] > SELECT * FROM performance_schema.replication_group_members;
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| CHANNEL_NAME | MEMBER_ID | MEMBER_HOST | MEMBER_PORT | MEMBER_STATE | MEMBER_ROLE | MEMBER_VERSION |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
| group_replication_applier | 3d35218d-94c5-11ea-b213-005056b8a4a4 | mgr_node1 | 3307 | ONLINE | PRIMARY | 8.0.20 |
| group_replication_applier | 5da974a5-94f3-11ea-a178-005056b8c7db | mgr_node2 | 3307 | ONLINE | SECONDARY | 8.0.20 |
| group_replication_applier | a553c26f-95b2-11ea-b31e-005056b854d7 | mgr_node4 | 3317 | ONLINE | SECONDARY | 8.0.20 |
| group_replication_applier | f1776459-958f-11ea-8dec-005056b854d7 | mgr_node3 | 3307 | ONLINE | SECONDARY | 8.0.20 |
+---------------------------+--------------------------------------+-------------+-------------+--------------+-------------+----------------+
4 rows in set (0.00 sec)
step1、来到recipient节点,使用至少权限为CLONE_ADMIN的账号登录(注:CLONE_ADMIN包含了BACKUP_ADMIN、SHUTDOWN权限)
step2、指向Donor节点
SET GLOBAL clone_valid_donor_list = '192.168.222.76:3307';
step3、填入Donor节点上的备份账号
CLONE INSTANCE FROM 'clone_user'@'192.168.222.76':3307 IDENTIFIED BY '123456';
step4、克隆结束后,根据原先recipient节点的启动方式执行关闭或自动重启新节点。
Tips: 若以mysqld方式启动,克隆完成后会有如下报错,属于正常现象(目前仅mysqld_safe方式支持自动重启):
ERROR 3707 (HY000): Restart server failed (mysqld is not managed by supervisor process)
cd /usr/local/mysql3307
./bin/mysqld --defaults-file=/usr/local/mysql3307/my.cnf &
CHANGE MASTER TO MASTER_USER='mgr_repl', MASTER_PASSWORD='123456' FOR CHANNEL 'group_replication_recovery';
start group_replication;
由于8.0.20的redo log格式有调整。xtrabackup-8.0.11基于8.0.18开发,故暂无法使用。官方回复如下:
更新:Percona XtraBackup 8.0.12版本已支持MySQL 8.0.20
如下地址下载rpm包,安装
https://github.com/sysown/proxysql/releases
systemctl start proxysql
默认用户名、密码:admin:admin
mysql -u admin -padmin -h 127.0.0.1 -P6032 --prompt='Admin> '
Admin> use main
Admin> update global_variables set variable_value = 'admin:adminpasswd' where variable_name = 'admin-admin_credentials';
Admin> LOAD ADMIN VARIABLES TO RUNTIME;
Admin> SAVE ADMIN VARIABLES TO DISK;
set mysql-default_charset='utf8mb4';
set mysql-default_collation_connection='utf8mb4_general_ci';
set mysql-max_allowed_packet=67108864; #对应后端数据库max_allowed_packet
#set mysql-ping_interval_server_msec=60000; #sleep会话保持时长120s
set mysql-max_connections=10000; #ProxySQL能处理前端请求的最大连接数
set mysql-server_version='8.0.20'; #proxysql响应给客户端的MySQL版本号
set mysql-set_query_lock_on_hostgroup=0; #set语法兼容
Tips:避免如下报错:
9006 - ProxySQL Error: connection is locked to hostgroup 10 but trying to reach hostgroup 30
https://github.com/sysown/proxysql/issues/2202
LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;
Admin> insert into mysql_group_replication_hostgroups(writer_hostgroup,backup_writer_hostgroup,reader_hostgroup, offline_hostgroup,active,max_writers,writer_is_also_reader)
values (10,20,30,40,1,1,1);
Tips:
Another similar option is currently being implemented in ProxySQL 2.0.9 for the transactions_behind check. See below:
mysql-monitor_groupreplication_max_transactions_behind
Currently, if group replication max_transactions_behind exceeds the threshold once, the node is evicted from the cluster.
The upcoming 2.0.9 release features another additional variable which will define a count for such checks so that max_transactions_behind would have to fail more than once (x number of times) before eviction.
mysql-monitor_groupreplication_max_transactions_behind_count默认3
将配置生效并存储:
Admin> load mysql servers to runtime;
Admin> save mysql servers to disk;
确认已生效:
Admin> select * from runtime_mysql_group_replication_hostgroups;
+------------------+-------------------------+------------------+-------------------+--------+-------------+-----------------------+-------------------------+---------+
| writer_hostgroup | backup_writer_hostgroup | reader_hostgroup | offline_hostgroup | active | max_writers | writer_is_also_reader | max_transactions_behind | comment |
+------------------+-------------------------+------------------+-------------------+--------+-------------+-----------------------+-------------------------+---------+
| 10 | 20 | 30 | 40 | 1 | 1 | 0 | 100 | NULL |
+------------------+-------------------------+------------------+-------------------+--------+-------------+-----------------------+-------------------------+---------+
1 row in set (0.00 sec)
Admin> insert into mysql_servers(hostgroup_id,hostname,port,max_connections) values(10,"192.168.222.76",3307,5000);
Admin> insert into mysql_servers(hostgroup_id,hostname,port,max_connections) values(30,"192.168.222.77",3307,5000);
Admin> insert into mysql_servers(hostgroup_id,hostname,port,max_connections) values(30,"192.168.222.78",3307,5000);
将配置生效并存储:
Admin> load mysql servers to runtime;
Admin> save mysql servers to disk;
确认已生效:
select * from runtime_mysql_servers;
+--------------+----------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| hostgroup_id | hostname | port | gtid_port | status | weight | compression | max_connections | max_replication_lag | use_ssl | max_latency_ms | comment |
+--------------+----------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
| 10 | 192.168.222.76 | 3307 | 0 | ONLINE | 1 | 0 | 5000 | 0 | 0 | 0 | |
| 30 | 192.168.222.78 | 3307 | 0 | ONLINE | 1 | 0 | 5000 | 0 | 0 | 0 | |
| 30 | 192.168.222.77 | 3307 | 0 | ONLINE | 1 | 0 | 5000 | 0 | 0 | 0 | |
+--------------+----------------+------+-----------+--------+--------+-------------+-----------------+---------------------+---------+----------------+---------+
3 rows in set (0.00 sec)
在后端DB执行
需要在后端DB添加如下监控视图:(来源https://github.com/lefred/mysql_gr_routing_check/blob/master/addition_to_sys.sql)
USE sys;
DELIMITER $$
CREATE FUNCTION my_id() RETURNS TEXT(36) DETERMINISTIC NO SQL RETURN (SELECT @@global.server_uuid as my_id);$$
CREATE FUNCTION gr_member_in_primary_partition()
RETURNS VARCHAR(3)
DETERMINISTIC
BEGIN
RETURN (SELECT IF( MEMBER_STATE='ONLINE' AND ((SELECT COUNT(*) FROM
performance_schema.replication_group_members WHERE MEMBER_STATE NOT IN ('ONLINE', 'RECOVERING')) >=
((SELECT COUNT(*) FROM performance_schema.replication_group_members)/2) = 0),
'YES', 'NO' ) FROM performance_schema.replication_group_members JOIN
performance_schema.replication_group_member_stats USING(member_id) where member_id=my_id());
END$$
CREATE VIEW gr_member_routing_candidate_status AS SELECT
sys.gr_member_in_primary_partition() as viable_candidate,
IF( (SELECT (SELECT GROUP_CONCAT(variable_value) FROM
performance_schema.global_variables WHERE variable_name IN ('read_only',
'super_read_only')) != 'OFF,OFF'), 'YES', 'NO') as read_only,
Count_Transactions_Remote_In_Applier_Queue as transactions_behind, Count_Transactions_in_queue as 'transactions_to_cert'
from performance_schema.replication_group_member_stats where member_id=my_id();$$
DELIMITER ;
create user 'proxysql_monitor'@'%' identified by '123456';
grant select on sys.gr_member_routing_candidate_status to 'proxysql_monitor'@'%';
flush privileges;
在proxysql中录入监控账号信息:
Admin> UPDATE global_variables SET variable_value='proxysql_monitor' WHERE variable_name='mysql-monitor_username';
Admin> UPDATE global_variables SET variable_value='123456' WHERE variable_name='mysql-monitor_password';
缩短监控巡检周期到2秒(##需与官方文档核对##)
(默认60000、10000、1500):
UPDATE global_variables SET variable_value='2000'
WHERE variable_name IN ('mysql-monitor_connect_interval','mysql-monitor_ping_interval','mysql-monitor_read_only_interval');
遇到: MySQL_Session.cpp:1454:handler_again___status_PINGING_SERVER(): [ERROR] Detected a broken connection during ping on (10,10.10.7.53,3306) , FD (Conn:2339 , MyDS:2339) : 2013, Lost connection to MySQL server during query
可适当调大mysql-monitor_ping_interval:2000=>5000
运行如下命令进行校对:
SELECT * FROM global_variables WHERE variable_name LIKE 'mysql-monitor_%';
无误后,将配置生效并存储
Admin> LOAD MYSQL VARIABLES TO RUNTIME;
Admin> SAVE MYSQL VARIABLES TO DISK;
确认监控是否可用(未配置前,是access denied状态)
Admin> SELECT * FROM monitor.mysql_server_connect_log ORDER BY time_start_us DESC LIMIT 10;
+----------------+------+------------------+-------------------------+-------------------------------------------------------------------------+
| hostname | port | time_start_us | connect_success_time_us | connect_error |
+----------------+------+------------------+-------------------------+-------------------------------------------------------------------------+
| 192.168.222.78 | 3307 | 1589546175511459 | 1723 | NULL |
| 192.168.222.77 | 3307 | 1589546175088734 | 1492 | NULL |
| 192.168.222.76 | 3307 | 1589546174666010 | 842 | NULL |
| 192.168.222.76 | 3307 | 1589546115653773 | 1281 | NULL |
| 192.168.222.77 | 3307 | 1589546115160901 | 1767 | NULL |
| 192.168.222.78 | 3307 | 1589546114667025 | 1431 | NULL |
| 192.168.222.78 | 3307 | 1589546090462462 | 0 | Access denied for user 'monitor'@'192.168.222.76' (using password: YES) |
| 192.168.222.76 | 3307 | 1589546089762699 | 0 | Access denied for user 'monitor'@'192.168.222.76' (using password: YES) |
| 192.168.222.77 | 3307 | 1589546089063100 | 0 | Access denied for user 'monitor'@'192.168.222.76' (using password: YES) |
| 192.168.222.77 | 3307 | 1589546030519571 | 0 | Access denied for user 'monitor'@'192.168.222.76' (using password: YES) |
+----------------+------+------------------+-------------------------+-------------------------------------------------------------------------+
10 rows in set (0.00 sec)
Admin> SELECT * FROM monitor.mysql_server_ping_log ORDER BY time_start_us DESC LIMIT 10;
insert into mysql_query_rules(rule_id,active,match_digest,destination_hostgroup,apply) VALUES (101,1,"^SELECT .* FOR UPDATE$",10,1),(102,1,"^SELECT",30,1);
将配置生效并存储:
load mysql query rules to runtime;
save mysql query rules to disk;
ProxySQL中配置:
insert into mysql_users(username,password,default_hostgroup) values('proxydb','proxydb',10);
LOAD MYSQL USERS TO RUNTIME;
load to runtime(memory=>runtime)时,会将runtime层的密码自动加密(admin-hash_passwords默认ON):
复测:SELECT username,password FROM runtime_mysql_users;
引用runtime层的加密密码到memory层:
SAVE MYSQL USERS FROM RUNTIME;
复测:SELECT username,password FROM mysql_users;
引用memory层的加密密码到disk层:
SAVE MYSQL USERS TO DISK;
后端DB配置:
create user 'proxydb'@'%' identified by 'proxydb';
grant XXX on *.* to 'proxydb'@'%';
通过ProxySQL的IP和P6033端口就能访问了,自动读写分离
注:用begin显示开始的事务,默认直接走写节点
附1:cnf文件备份方法,用于传递到备用proxysql上
how to copy runtime configuration to another ProxySQL server?
SELECT CONFIG INTO OUTFILE /tmp/f1;
alias:SAVE CONFIG TO FILE /tmp/f2;
备用proxysql配置生效:
法一、
先删除/var/lib/proxysql/proxysql.db,
再覆盖备份文件到/etc/proxysql.cnf,修改权限为root:proxysql
重启备用proxysql
附2:
sysbench --report-interval=5 --num-threads=4 --num-requests=0 --max-time=20 --test=tests/db/oltp.lua --mysql-user='msandbox' --mysql-password='msandbox' --oltp-table-size=10000 --mysql-host=127.0.0.1 --mysql-port=6033 run
sysbench --num-threads=4 --max-requests=0 --max-time=20 --test=tests/db/oltp.lua --mysql-user='msandbox' --mysql-password='msandbox' --oltp-table-size=10000 --mysql-host=127.0.0.1 --mysql-port=6033 --db-ps-mode=disable run
排版待编辑
已有节点完成配置后
ProxySQL Cluster信息配置
insert into proxysql_servers(hostname,port) values('10.10.7.51',6032),('10.10.7.52',6032);
load proxysql servers to runtime;
save proxysql servers to disk;
配置用于实例间通讯的账号(;前为本地登录账号,;后为实例间同步端口)
Admin> update global_variables set variable_value = 'admin:13&3biYvWHQ7;proxysql_cluster:17yFhXm62$zH' where variable_name = 'admin-admin_credentials';
Admin> update global_variables set variable_value = 'proxysql_cluster' where variable_name = 'admin-cluster_username';
Admin> update global_variables set variable_value = '17yFhXm62$zH' where variable_name = 'admin-cluster_password';
SELECT * FROM global_variables WHERE variable_name LIKE 'admin-cluster_%';
确认无误后:
Admin> LOAD ADMIN VARIABLES TO RUNTIME;
Admin> SAVE ADMIN VARIABLES TO DISK;
安装新节点并启动:
sudo systemctl start proxysql
# 登录新ProxySQL节点,录入以下cluster信息(sudo tail -f /var/lib/proxysql/proxysql.log观察老节点输出)
Admin> update global_variables set variable_value = 'admin:13&3biYvWHQ7;proxysql_cluster:17yFhXm62$zH' where variable_name = 'admin-admin_credentials';
Admin> update global_variables set variable_value = 'proxysql_cluster' where variable_name = 'admin-cluster_username';
Admin> update global_variables set variable_value = '17yFhXm62$zH' where variable_name = 'admin-cluster_password';
Admin> insert into proxysql_servers(hostname,port) values('10.10.7.51',6032),('10.10.7.52',6032);
Admin> load admin variables to runtime;
Admin> save admin variables to disk;
Admin> load proxysql servers to runtime;
Admin> save proxysql servers to disk;
目前,已确认以下值不会同步,需手动设置:
set mysql-default_charset='utf8mb4';
set mysql-default_collation_connection='utf8mb4_general_ci';
set mysql-max_connections=10000; #ProxySQL能处理前端请求的最大连接数
set mysql-server_version='8.0.20'; #proxysql响应给客户端的MySQL版本号
set mysql-set_query_lock_on_hostgroup=0; #set语法兼容
LOAD MYSQL VARIABLES TO RUNTIME;
SAVE MYSQL VARIABLES TO DISK;
参考文档:
MySQL 8.0 Reference Manual: 18.2.1 Deploying Group Replication in Single-Primary Mode
MySQL8.0新特性之clone
https://proxysql.com/documentation/ProxySQL-Configuration/
https://proxysql.com/documentation/proxysql-read-write-split-howto/
https://www.colabug.com/2019/1224/6764189/