基于MaxScale中间件的MySQL读写分离

基于MaxScale中间件的MySQL读写分离

概述

maxscale 基于keepalived的高可用，通过VIP提供服务

maxscale官网：https://mariadb.com/downloads/mariadb-tx/maxscale

maxscale文档（比官网查看方便）https://github.com/mariadb-corporation/MaxScale/tree/2.2/Documentation

mysql基于GTID模式的主从复制，可以在主库故障后快速修复复制状态。

故障切换示意图

 

环境介绍

NAME	VERSION	IP	PORT	COMMENT
maxscale	2.2.6	172.16.10.114	4306,6603	4306为读写分离端口，6603为管理端口
master	5.6.39	172.16.10.114	3308	GTID复制
slave	5.6.39	172.16.10.114	3309	GTID复制

 

maxscale安装配置

mysql GTID配置

开启GTID需要在配置文件中加入以下参数：

gtid_mode = ON

enforce_gtid_consistency = ON

log_slave_updates = ON

 

主从配置命令：

CHANGE MASTER TO MASTER_HOST=host, MASTER_PORT=port, MASTER_USER=user, MASTER_PASSWORD=password, MASTER_AUTO_POSITION=1;

注意：

1. 开启GTID的主从复制，在备份时，会自动加上GTID的相关信息，通过备份进行恢复时，会报错，其原因是备份中的GTID与实例执行的GTID冲突，同时也防止事件重复执行，解决些报错有2种办法：

法一，在备份时候添加参数--set-gtid-purged=OFF，不备份GTID相关信息。

法二，在进行还原时，先进入实例执行reset master，再进行数据还原

1. 开启GTID以后，无法使用sql_slave_skip_counter跳过事务，跳过错误的方法为：

set grid_next='xxxxxxxx'; #待跳过的GTID

begin;

commit; #通过产生一个空事务来占据此GTID

change master to master_auto_position=1;

MaxScale安装配置

wget -c https://downloads.mariadb.com/MaxScale/2.2.6/rhel/6/x86_64/maxscale-2.2.6-1.rhel.6.x86_64.rpm

yum install maxscale-2.2.6-1.rhel.6.x86_64.rpm

 

cat /etc/maxscale.cnf

[maxscale]

threads=8 #线程配置，默认为1

auth_connect_timeout=3600

auth_read_timeout=3600

auth_write_timeout=3600

 

[server1] #配置后端服务器

type=server

address=172.16.10.114

port=3308

protocol=MySQLBackend

server_weight=1

 

[server2]

type=server

address=172.16.10.114

port=3309

protocol=MySQLBackend

server_weight=1

 

#[server3]

#type=server

#address=172.16.10.114

#port=3310

#protocol=MySQLBackend

#server_weight=1

 

[readwritesplit] #读写分离配置

type=service

router=readwritesplit

servers=server1,server2

user=connect

passwd=connect

weightby=server_weight

max_slave_replication_lag=10 #允许最大主从延迟，当主从延迟超过该值时，不再向从库分发读请求

 

[Read Service] #配置读服务，虽然字面意思为读服务，也可以执行DML,DDL等操作，取决于对用户的授权，因此可以理解为连接服务

type=service

router=readconnroute

router_options=master

servers=server1,server2

user=connect

passwd=connect

weightby=server_weight

 

[MySQL Monitor] #监控配置

type=monitor

module=mariadbmon

servers=server1,server2

user=monitor

passwd=monitor

auto_failover=true #是否故障自动切换

auto_rejoin=true #故障实例恢复后自动加入集群

detect_standalone_master=true #探测独立的master,是否允许集群中最后一个实例成为主库

allow_cluster_recovery=false #是否允许集群自动恢复

#failcount=3 #在集群中最后一个实例成为主库前检查其它从库是否存活的次数,默认为5

#monitor_interval=10000 #探测间隔，单位毫秒，默认2000

detect_stale_master=true #当集群中只剩下主或主从复制全出错时，是否允许主提供服务

#detect_stale_slave=false

script=/tmp/reset_slave.sh #在下面的events发生时，执行的脚本

events=master_down #配置在发生什么事件时，执行上面的脚本

 

[Splitter-Service] #配置读写分离监听端口

type=listener

service=readwritesplit

protocol=MySQLClient

port=4306

 

[Read Listener] #配置读服务监听端口

type=listener

service=Read Service

protocol=MySQLClient

port=4307

 

[MaxAdmin Service] #配置管理服务

type=service

router=cli

 

[MaxAdmin Listener] #配置管理服务端口

type=listener

service=MaxAdmin Service

protocol=maxscaled

port=6603

 

上面的配置中涉及2个用户，一个是连接数据库的用户，一个是监控用户，其授权分别如下；

connect

CREATE USER 'connect'@'172.16.10.114' IDENTIFIED BY 'connect';

GRANT SELECT ON mysql.user TO 'connect'@'172.16.10.114';

GRANT SELECT ON mysql.db TO 'connect'@'172.16.10.114';

GRANT SELECT ON mysql.tables_priv TO 'connect'@'172.16.10.114';

GRANT SHOW DATABASES ON *.* TO 'connect'@'172.16.10.114';

 

monitor

CREATE USER 'monitor'@'172.16.10.114' IDENTIFIED BY 'monitor';

GRANT RELOAD, SUPER, REPLICATION CLIENT ON *.* to 'monitor'@'172.16.10.114';

 

注意：

1. 如果配置maxscale的高可用，还需要配置用户能通过另外一台机器到mysql库的相关权限。
2. 应用对数据库的连接权限，因中间多了一层maxscale,需要同一用户，同时允许应用IP,maxscale IP都可以连接到数据库，并且密码相同。举例如下：

maxscale IP为172.16.10.114,应用IP 172.16.10.238,访问数据库test,其授权为：

grant select,update,delete,insert on test.* to test_rw@'172.16.10.114' identified by 'test_rw';

grant select,update,delete,insert on test.* to test_rw@'172.16.10.238' identified by 'test_rw';

grant select on test.* to test_r@'172.16.10.114' identified by 'test_r';

grant select on test.* to test_r@'172.16.10.238' identified by 'test_r';

1. reset_salve.sh脚本内容，其目的主要是在master宕掉后，执行的脚本，清除从库的复制信息，使从库可以提升为主库，如果从库有复制的相关信息，其不能提升为主库提供写服务。

此脚本在主从切换后，需要修改连接IP为新从库的IP

cat /tmp/reset_slave.sh

mysql -h127.0.0.1 -P3309 -uthunder -pthunder -Nse 'stop slave;reset slave all;'

启动maxscale

/etc/init.d/maxscale start

管理maxscale

通过maxadmin命令，默认用户名和密码为admin/mariadb

maxadmin -h127.0.0.1 -P6603 -uadmin -p

也可能通过maxctrl命令通过API来管理，大致命令相同，maxadmin属于交互式，maxctrl属于非交互

MaxScale> help

Available commands:

add:

    add user - Add an administrative account for using maxadmin over the network

    add readonly-user - Add a read-only account for using maxadmin over the network

    add server - Add a new server to a service

 

remove:

    remove user - Remove account for using maxadmin over the network

    remove server - Remove a server from a service or a monitor

 

create:

    create server - Create a new server

    create listener - Create a new listener for a service

    create monitor - Create a new monitor

 

destroy:

    destroy server - Destroy a server

    destroy listener - Destroy a listener

    destroy monitor - Destroy a monitor

 

alter:

    alter server - Alter server parameters

    alter monitor - Alter monitor parameters

    alter service - Alter service parameters

    alter maxscale - Alter maxscale parameters

 

set:

    set server - Set the status of a server

    set pollsleep - Set poll sleep period

    set nbpolls - Set non-blocking polls

    set log_throttling - Set the log throttling configuration

 

clear:

    clear server - Clear server status

 

disable:

    disable log-priority - Disable a logging priority

    disable sessionlog-priority - [Deprecated] Disable a logging priority for a particular session

    disable root - Disable root access

    disable syslog - Disable syslog logging

    disable maxlog - Disable MaxScale logging

    disable account - Disable Linux user

 

enable:

    enable log-priority - Enable a logging priority

    enable sessionlog-priority - [Deprecated] Enable a logging priority for a session

    enable root - Enable root user access to a service

    enable syslog - Enable syslog logging

    enable maxlog - Enable MaxScale logging

    enable account - Activate a Linux user account for administrative MaxAdmin use

    enable readonly-account - Activate a Linux user account for read-only MaxAdmin use

 

flush:

    flush log - Flush the content of a log file and reopen it

    flush logs - Flush the content of a log file and reopen it

 

list:

    list clients - List all the client connections to MaxScale

    list dcbs - List all active connections within MaxScale

    list filters - List all filters

    list listeners - List all listeners

    list modules - List all currently loaded modules

    list monitors - List all monitors

    list services - List all services

    list servers - List all servers

    list sessions - List all the active sessions within MaxScale

    list threads - List the status of the polling threads in MaxScale

    list commands - List registered commands

 

reload:

    reload config - [Deprecated] Reload the configuration

    reload dbusers - Reload the database users for a service

 

restart:

    restart monitor - Restart a monitor

    restart service - Restart a service

    restart listener - Restart a listener

 

shutdown:

    shutdown maxscale - Initiate a controlled shutdown of MaxScale

    shutdown monitor - Stop a monitor

    shutdown service - Stop a service

    shutdown listener - Stop a listener

 

show:

    show dcbs - Show all DCBs

    show dbusers - [deprecated] Show user statistics

    show authenticators - Show authenticator diagnostics for a service

    show epoll - Show the polling system statistics

    show eventstats - Show event queue statistics

    show filter - Show filter details

    show filters - Show all filters

    show log_throttling - Show the current log throttling setting (count, window (ms), suppression (ms))

    show modules - Show all currently loaded modules

    show monitor - Show monitor details

    show monitors - Show all monitors

    show persistent - Show the persistent connection pool of a server

    show server - Show server details

    show servers - Show all servers

    show serversjson - Show all servers in JSON

    show services - Show all configured services in MaxScale

    show service - Show a single service in MaxScale

    show session - Show session details

    show sessions - Show all active sessions in MaxScale

    show tasks - Show all active housekeeper tasks in MaxScale

    show threads - Show the status of the worker threads in MaxScale

    show users - Show enabled Linux accounts

    show version - Show the MaxScale version number

 

sync:

    sync logs - Flush log files to disk

 

call:

    call command - Call module command

 

ping:

    ping workers - Ping Workers

 

 

Type `help COMMAND` to see details of each command.

Where commands require names as arguments and these names contain

whitespace either the \ character may be used to escape the whitespace

or the name may be enclosed in double quotes ".

查看当前mysql实例状态

MaxScale> list servers

Servers.

-------------------+-----------------+-------+-------------+--------------------

Server             | Address         | Port  | Connections | Status             

-------------------+-----------------+-------+-------------+--------------------

server1            | 172.16.10.114   |  3308 |           0 | Master, Running

server2            | 172.16.10.114   |  3309 |           0 | Slave, Running

-------------------+-----------------+-------+-------------+--------------------

[root@Hexindai-BJ-DFGC-114 ~]# maxctrl list servers

┌─────────┬───────────────┬──────┬─────────────┬────

│ Server  │ Address       │ Port │ Connections │ State             │ GTID │

├─────────┼───────────────┼──────┼─────────────┼────

│ server1 │ 172.16.10.114 │ 3308 │ 0           │ Master, Running   │      │

├─────────┼───────────────┼──────┼─────────────┼────

│ server2 │ 172.16.10.114 │ 3309 │ 0           │ Slave, Running   │      │

└─────────┴───────────────┴──────┴─────────────┴────

 

复制故障后状态：

MaxScale> list servers

Servers.

-------------------+-----------------+-------+-------------+--------------------

Server             | Address         | Port  | Connections | Status             

-------------------+-----------------+-------+-------------+--------------------

server1            | 172.16.10.114   |  3308 |           0 | Master, Running

server2            | 172.16.10.114   |  3309 |           0 | Running

-------------------+-----------------+-------+-------------+--------------------

解决主从复制问题

从库宕机恢复步骤：

MaxScale> list servers

Servers.

-------------------+-----------------+-------+-------------+--------------------

Server             | Address         | Port  | Connections | Status             

-------------------+-----------------+-------+-------------+--------------------

server1            | 172.16.10.114   |  3308 |           0 | Master, Running

server2            | 172.16.10.114   |  3309 |           0 | Maintenance, Down

-------------------+-----------------+-------+-------------+--------------------

将从库重新加到读写分离集群中

启动从库

启动复制start slave;

等主从追赶上后，在maxscale里面执行

MaxScale> clear server server2 maintenance

MaxScale> list servers

Servers.

-------------------+-----------------+-------+-------------+--------------------

Server             | Address         | Port  | Connections | Status             

-------------------+-----------------+-------+-------------+--------------------

server1            | 172.16.10.114   |  3308 |           0 | Master, Running

server2            | 172.16.10.114   |  3309 |           0 | Slave, Running

-------------------+-----------------+-------+-------------+--------------------

主库宕机后恢复步骤：

MaxScale> list servers

Servers.

-------------------+-----------------+-------+-------------+--------------------

Server             | Address         | Port  | Connections | Status             

-------------------+-----------------+-------+-------------+--------------------

server1            | 172.16.10.114   |  3308 |           0 | Down

server2            | 172.16.10.114   |  3309 |           0 | Slave, Running

-------------------+-----------------+-------+-------------+--------------------

MaxScale> list servers

Servers.

-------------------+-----------------+-------+-------------+--------------------

Server             | Address         | Port  | Connections | Status             

-------------------+-----------------+-------+-------------+--------------------

server1            | 172.16.10.114   |  3308 |           0 | Maintenance, Down

server2            | 172.16.10.114   |  3309 |           0 | Master, Running

-------------------+-----------------+-------+-------------+--------------------

切换时间约为10s, 为以下2个参数的乘积，最长时间不超过monitor_interval*(failcount+1)

#failcount=3 #在集群中最后一个实例成为主库前检查其它从库是否存活的次数,默认为5

#monitor_interval=10000 #探测间隔，单位毫秒，默认2000

此时从库接替原主库接受读写请求，恢复主从架构，原主库将成为新的从库，启动实例，做主从复制：

change master to  master_host='172.16.10.114', master_port=3309, master_user='repl', master_password='repl4slave', master_auto_position=1;

检查复制状态

show slave status;

将新从库加入到读写分离集群中：

MaxScale> list servers

Servers.

-------------------+-----------------+-------+-------------+--------------------

Server             | Address         | Port  | Connections | Status             

-------------------+-----------------+-------+-------------+--------------------

server1            | 172.16.10.114   |  3308 |           0 | Maintenance, Down

server2            | 172.16.10.114   |  3309 |           0 | Master, Running

-------------------+-----------------+-------+-------------+--------------------

MaxScale> list servers

Servers.

-------------------+-----------------+-------+-------------+--------------------

Server             | Address         | Port  | Connections | Status             

-------------------+-----------------+-------+-------------+--------------------

server1            | 172.16.10.114   |  3308 |           0 | Maintenance, Down

server2            | 172.16.10.114   |  3309 |           0 | Master, Running

-------------------+-----------------+-------+-------------+--------------------

MaxScale> clear server server1 maintenance

MaxScale> list servers

Servers.

-------------------+-----------------+-------+-------------+--------------------

Server             | Address         | Port  | Connections | Status             

-------------------+-----------------+-------+-------------+--------------------

server1            | 172.16.10.114   |  3308 |           0 | Slave, Running

server2            | 172.16.10.114   |  3309 |           0 | Master, Running

-------------------+-----------------+-------+-------------+--------------------

keepalived安装与配置

如不考虑maxscale的高可用，可忽略以下内容

INSTALL

#yum install keepalived

CONFIG ON MASTER

#cat /etc/keepalived/keepalived.conf

vrrp_script chk_myscript {

  script "/opt/soft/is_maxscale_running.sh"

  interval 2 # check every 2 seconds

  fall 2 # require 2 failures for KO

  rise 2 # require 2 successes for OK

}

 

vrrp_instance VI_1 {

  state MASTER

  interface em1

  virtual_router_id 51

  priority 150

  advert_int 1

  authentication {

    auth_type PASS

    auth_pass mypass

  }

  virtual_ipaddress {

    192.168.1.13 #VIP

  }

  track_script {

    chk_myscript

  }

  notify "/opt/soft/notify_script.sh"

}

 

CONFIG ON STANDBY

#cat /etc/keepalived/keepalived.conf

vrrp_script chk_myscript {

  script "/opt/soft/is_maxscale_running.sh"

  interval 2 # check every 2 seconds

  fall 2 # require 2 failures for KO

  rise 2 # require 2 successes for OK

}

 

vrrp_instance VI_1 {

  state MASTER

  interface em1

  virtual_router_id 51

  priority 100

  advert_int 1

  authentication {

    auth_type PASS

    auth_pass mypass

  }

  virtual_ipaddress {

    192.168.1.13

  }

  track_script {

    chk_myscript

  }

  notify "/opt/soft/notify_script.sh"

}

 

#cat is_maxscale_running.sh

#!/bin/bash

fileName="maxadmin_output.txt"

rm $fileName

timeout 2s maxadmin -h127.0.0.1 -uadmin -pmariadb list servers > $fileName

to_result=$?

if [ $to_result -ge 1 ]

then

  echo Timed out or error, timeout returned $to_result

  exit 3

else

  echo MaxAdmin success, rval is $to_result

  echo Checking maxadmin output sanity

  grep1=$(grep server1 $fileName)

  grep2=$(grep server2 $fileName)

 

  if [ "$grep1" ] && [ "$grep2" ]

  then

    echo All is fine

    exit 0

  else

    echo Something is wrong

    exit 3

  fi

fi

 

#cat notify_script.sh

#!/bin/bash

 

TYPE=$1

NAME=$2

STATE=$3

 

OUTFILE=./state.txt

touch $OUTFILE

case $STATE in

  "MASTER") echo "Setting this MaxScale node to active mode" > $OUTFILE

                  maxctrl alter maxscale passive false

                  exit 0

                  ;;

  "BACKUP") echo "Setting this MaxScale node to passive mode" > $OUTFILE

                  maxctrl alter maxscale passive true

                  exit 0

                  ;;

  "FAULT")  echo "MaxScale failed the status check." > $OUTFILE

                  maxctrl alter maxscale passive true

                  exit 0

                  ;;

        *)        echo "Unknown state" > $OUTFILE

                  exit 1

                  ;;

esac

启动keepalived

/etc/init.d/keepalived start

分别停掉keepalived与maxscale，观察VIP漂移