Mysql-proxy
实现对于前端请求的读写分离,利用LVS
实现对于后端读请求的负载均衡,后端的读写服务器之间实现主从复制功能;规划
实验环境:rhel6.5,内核版本:2.6.32-431.el6.x86_64;
读写分离主机:172.25.23.82 主机名:mysql-proxy
写服务器+主从架构的主服务器:172.25.23.78/24 主机名:mysql-master
LVS所在主机:172.25.23.79/24 主机名:lvs-server
LVS后端主机1: sql-slave-reallvs 主机名:sql-slave-reallvs1
LVS后端主机2:172.25.23.81 主机名:sql-slave2-realvs2
读写分离采用的软件为mysql-proxy;
步骤简介
mysql-master
配置磁盘阵列,这里采用软件raid 5
并且提供一块冗余磁盘;mysql-5.7.11
并且采用Innodb
存储引擎,启用每表一个表空间文件;mysql-master
的配置文件,配置master
端的复制用户;sql-slave-realvs1
安装软件,并且将数据文件单独保存在一块磁盘上面;sql-slave-realvs2
安装软件,并且将数据文件单独保存在一块磁盘上面;slave
在实验环境下,不再使用raid
阵列,仅仅使用单独的分区;slave
向master
同步数据;sql-slave-realvs
的负载均衡;LVS-DR
模型配置sql-slave-realvs
LVS
后端服务检查的脚本来实现对于Mysql
后端服务健康状况的检查,并且能够实现自动移除故障节点,并且在服务正常后,自动的添加正常节点;接下来配置前端的mysql-proxy
,采用rpm
安装软件,通过配置实现读写分离,将写请求调度到Mysql-master
,将读请求调度到lvs-server
,并且最终实现调度到sql-slave-realvs
上面;
正式的配置过程开始
raid 5
阵列mysql-master
添加一块40G
的硬盘 [root@mysql-master ~]# fdisk /dev/vdb
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-83220, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-83220, default 83220): +10G
Command (m for help): p
Disk /dev/vdb: 42.9 GB, 42949672960 bytes
16 heads, 63 sectors/track, 83220 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xad29dba2
Device Boot Start End Blocks Id System
/dev/vdb1 1 20806 10486192+ fd Linux raid autodetect
/dev/vdb2 20807 41612 10486224 fd Linux raid autodetect
/dev/vdb3 41613 62418 10486224 fd Linux raid autodetect
/dev/vdb4 62419 83220 10484208 fd Linux raid autodetect
[root@mysql-master ~]# partprobe /dev/vdb
fd
[root@mysql-master ~]# mdadm -C /dev/md5 -l5 -n 3 -x 1 /dev/vdb1 /dev/vdb2 /dev/vdb3 /dev/vdb4
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md5 started.
raid
阵列的状态 Number Major Minor RaidDevice State
0 252 17 0 active sync /dev/vdb1
1 252 18 1 active sync /dev/vdb2
4 252 19 2 spare rebuilding /dev/vdb3
3 252 20 - spare /dev/vdb4
[root@mysql-master ~]# mke2fs -j /dev/md5
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=128 blocks, Stripe width=256 blocks
1310720 inodes, 5237760 blocks
261888 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
160 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000
Writing inode tables: done
Creating journal (32768 blocks): done
[root@mysql-master ~]# mkdir /mysql/data -pv
mkdir: created directory `/mysql'
mkdir: created directory `/mysql/data'
[root@mysql-master ~]# mount /dev/md5 /mysql/data/
[root@mysql-master ~]# vim /etc/fstab
/dev/md5 /mydata/data ext3 defaults 0 0
mysql-5.7-11
的Yum
源,所以使用软件包可以直接安装;mysql-community-client.x86_64 : MySQL database client applications and tools
mysql-community-common.x86_64 : MySQL database common files for server and
: client libs
mysql-community-devel.x86_64 : Development header files and libraries for MySQL
: database client applications
mysql-community-embedded.x86_64 : MySQL embedded library
mysql-community-embedded-devel.x86_64 : Development header files and libraries
: for MySQL as an embeddable library
mysql-community-libs.x86_64 : Shared libraries for MySQL database client
: applications
mysql-community-libs-compat.x86_64 : Shared compat libraries for MySQL 5.1.72
: database client applications
mysql-community-server.x86_64 : A very fast and reliable SQL database server
mysql-community-test.x86_64 : Test suite for the MySQL database server
Mysql
的初始化[root@mysql-master ~]# mysql_install_db --datadir=/mysql/data
2018-08-10 07:42:20 [WARNING] mysql_install_db is deprecated. Please consider switching to mysqld --initialize
2018-08-10 07:42:50 [WARNING] select() timed out.
Mysql
datadir=/mysql/data
socket=/mysql/data/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
log-error=/mysql/data/mysqld.log
pid-file=/mysql/data/mysqld.pid
[root@mysql-master mysql]# chown mysql.mysql ./data/ -R
Mysql
[root@mysql-master mysql]# /etc/init.d/mysqld start
Starting mysqld: [ OK ]
3306
端口已经启动tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1095/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1462/master
tcp 0 0 172.25.23.78:22 172.25.23.250:45360 ESTABLISHED 1593/sshd
tcp 0 0 :::22 :::* LISTEN 1095/sshd
tcp 0 0 ::1:25 :::* LISTEN 1462/master
tcp 0 0 :::3306 :::* LISTEN 5202/mysqld
Mysql
安全安装脚本[root@mysql-master ~]# mysql_secure_installation
Securing the MySQL server deployment.
Connecting to MySQL server using password in '/root/.mysql_secret'
VALIDATE PASSWORD PLUGIN can be used to test passwords
and improve security. It checks the strength of password
and allows the users to set only those passwords which are
secure enough. Would you like to setup VALIDATE PASSWORD plugin?
Press y|Y for Yes, any other key for No: y
There are three levels of password validation policy:
LOW Length >= 8
MEDIUM Length >= 8, numeric, mixed case, and special characters
STRONG Length >= 8, numeric, mixed case, special characters and dictionary file
Please enter 0 = LOW, 1 = MEDIUM and 2 = STRONG: 0
Using existing password for root.
Estimated strength of the password: 50
Change the password for root ? ((Press y|Y for Yes, any other key for No) : y
New password:
Re-enter new password:
Estimated strength of the password: 50
Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No) : y
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.
Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
Success.
Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.
Disallow root login remotely? (Press y|Y for Yes, any other key for No) : y
Success.
By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.
Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
- Dropping test database...
Success.
- Removing privileges on test database...
Success.
Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.
Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y
Success.
All done!
[root@mysql-master ~]# mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 5.7.11
Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
4 rows in set (0.00 sec)
innodb_file_per_table = 1 //必须的参数
server-id = 1 //必须的参数
log-bin=master-bin //master必须的参数
binlog-format=ROW //二进制日志的记录格式
log-slave-updates=true //建议的参数
gtid-mode=ON //非常建议的参数
enforce-gtid-consistency=true
master-info-repository=TABLE
relay-log-info-repository=TABLE
sync-master-info=1
slave-parallel-workers=2
binlog-checksum=CRC32
master-verify-checksum=1
slave-sql-verify-checksum=1
binlog-rows-query-log_events=1
report-port=3306 //作用暂时未知
report-host=172.25.23.78
Mysql
服务器,并且创建允许有复制权限的用户[root@mysql-master data]# /etc/init.d/mysqld restart
Stopping mysqld: [ OK ]
Starting mysqld: [ OK ]
mysql5.7
默认密码策略比较严格,也是为了安全; grant replication slave on *.* to 'repluser'@'172.25.23.%' identified by 'Replpass121314..';
Query OK, 0 rows affected, 1 warning (0.11 sec)
[root@mysql-master data]# mysql -h172.25.23.78 -urepluser -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
//如果这个过程比较长,建议打开选项skip_name_resolve
master-salve
的master
端就已经配置完成了;slave
节点sql-slave-realvs1
首先安装软件包[root@sql-slave-reallvs1 ~]# yum install mysql-community-client.x86_64 mysql-community-common.x86_64 mysql-community-devel.x86_64 mysql-community-embedded.x86_64 mysql-community-libs.x86_64 mysql-community-server.x86_64 -y
[root@sql-slave-reallvs1 ~]# mkdir /mysql/data -pv
mkdir: created directory `/mysql'
mkdir: created directory `/mysql/data'
vim /etc/my.cnf
datadir=/mysql/mydata
[root@sql-slave-reallvs1 ~]# /etc/init.d/mysqld start
Initializing MySQL database: [ OK ]
Installing validate password plugin: [ OK ]
Starting mysqld: [ OK ]
[root@sql-slave-reallvs1 ~]# mysql_secure_installation
Securing the MySQL server deployment.
Enter password for user root:
The existing password for the user account root has expired. Please set a new password.
New password:
Re-enter new password:
The 'validate_password' plugin is installed on the server.
The subsequent steps will run with the existing configuration
of the plugin.
Using existing password for root.
Estimated strength of the password: 100
Change the password for root ? ((Press y|Y for Yes, any other key for No) : y
New password:
Re-enter new password:
Estimated strength of the password: 100
Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No) : y
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.
Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
Success.
Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.
Disallow root login remotely? (Press y|Y for Yes, any other key for No) : y
Success.
By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.
Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
- Dropping test database...
Success.
- Removing privileges on test database...
Success.
Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.
Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y
Success.
All done!
尝试使用新密码登陆成功;
在sql-slave-realvs2
上面进行同样的操作
[root@sql-slave2-realvs2 ~]# yum install mysql-community-client.x86_64 mysql-community-common.x86_64 mysql-community-devel.x86_64 mysql-community-libs.x86_64 mysql-community-server.x86_64 -y
[root@sql-slave2-realvs2 ~]# mkdir /mysql/data -pv
mkdir: created directory `/mysql'
mkdir: created directory `/mysql/data'
datadir=/mysql/data
Initializing MySQL database: [ OK ]
Installing validate password plugin: [ OK ]
Starting mysqld: [ OK ]
[root@sql-slave2-realvs2 ~]# mysql_secure_installation
Securing the MySQL server deployment.
Enter password for user root:
The existing password for the user account root has expired. Please set a new password.
New password:
Re-enter new password:
... Failed! Error: Your password does not satisfy the current policy requirements
New password:
Re-enter new password:
The 'validate_password' plugin is installed on the server.
The subsequent steps will run with the existing configuration
of the plugin.
Using existing password for root.
Estimated strength of the password: 100
Change the password for root ? ((Press y|Y for Yes, any other key for No) : y
New password:
Re-enter new password:
Estimated strength of the password: 100
Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No) : y
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.
Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
Success.
Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.
Disallow root login remotely? (Press y|Y for Yes, any other key for No) : y
Success.
By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.
Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
- Dropping test database...
Success.
- Removing privileges on test database...
Success.
Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.
Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y
Success.
All done!
尝试使用新密码登陆成功;
这里的主从服务架构,都是一主多从,并且是两个从向主节点进行数据复制
sql-salve-realvs1
配置信息如下innodb_file_per_table = 1
server-id = 2
log-bin=slave-bin
binlog-format=ROW
log-slave-updates=true
gtid-mode=on
enforce-gtid-consistency=true
master-info-repository=TABLE
sync-master-info=1
slave-parallel-workers=2
binlog-checksum=CRC32
master-verify-checksum=1
slave-sql-verify-checksum=1
binlog-rows-query-log_events=1
report-port=3306
port=3306
report-host=172.25.23.80
sql-slave-realvs2
的配置信息如下innodb_file_per_table = 1
server-id = 3
log-bin=slave-bin
binlog-format=ROW
log-slave-updates=true
gtid-mode=on
enforce-gtid-consistency=true
master-info-repository=TABLE
sync-master-info=1
slave-parallel-workers=2
binlog-checksum=CRC32
master-verify-checksum=1
slave-sql-verify-checksum=1
binlog-rows-query-log_events=1
report-port=3306
port=3306
report-host=172.25.23.81
[root@sql-slave-reallvs1 ~]# /etc/init.d/mysqld restart
Stopping mysqld: [ OK ]
Starting mysqld: [ OK ]
sql-slave-realvs2
节点上面的服务[root@sql-slave2-realvs2 ~]# /etc/init.d/mysqld restart
Stopping mysqld: [ OK ]
Starting mysqld: [ OK ]
sql-salve-realvs1
上面验证复制用户能够登陆
* 连接sql-slave-realvs1
上面,配置并且启动从节点
* 查看从节点状态
* 然后启动从节点,再次查看状态
* 配置sql-slave-realvs2
节点
* 尝试在主节点上面创建数据库
* sql-slave-reallvs1
上面已经创建,并且事务号码已经更改
* sql-slave-reallvs2
上面也已经创建,并且事务执行号码也已经更改
[root@mysql-lvs ~]# ip addr add 172.25.23.100/24 dev eth0:0
RIP
地址,RIP
有三块网卡,其中一块作为主从复制,所以不进行修改,这里只修改lo
以及eth1
网卡的配置信息sql-slave-reallvs1
的配合i过程:eth1
配置一个地址 [root@sql-slave-reallvs1 ~]# sysctl -w net.ipv4.conf.eth1.arp_announce=2
net.ipv4.conf.eth1.arp_announce = 2
[root@sql-slave-reallvs1 ~]# sysctl -w net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.lo.arp_announce = 2
[root@sql-slave-reallvs1 ~]# echo 1 > /proc/sys/net/ipv4/conf/eth1/arp_ignore
[root@sql-slave-reallvs1 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
[root@sql-slave-reallvs1 ~]# ifconfig lo:0 172.25.23.100 broadcast 172.25.23.100 netmask 255.255.255.255
* 接下来配置sql-slave-reallvs2
上面的网卡信息
* 网卡配置成功之后的信息
* 配置arp
响应的相关信息
[root@sql-slave2-realvs2 ~]# sysctl -w net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.lo.arp_announce = 2
[root@sql-slave2-realvs2 ~]# sysctl -w net.ipv4.conf.eth1.arp_announce=2
net.ipv4.conf.eth1.arp_announce = 2
[root@sql-slave2-realvs2 ~]# echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
[root@sql-slave2-realvs2 ~]# echo 1 > /proc/sys/net/ipv4/conf/eth1/arp_ignore
[root@sql-slave2-realvs2 ~]# ifconfig lo:0 172.25.23.100 broadcast 172.25.23.100 netmask 255.255.255.255
[root@sql-slave-reallvs1 ~]# route add -host 172.25.23.100 dev lo:0
[root@sql-slave2-realvs2 ~]# route add -host 172.25.23.100 dev lo:0
sql-slave-reallvs
上面的Mysql
服务已经配置,所以这里来指定调度规则;[root@mysql-lvs ~]# ipvsadm -A -t 172.25.23.100:3306 -s wlc
[root@mysql-lvs ~]# ipvsadm -a -t 172.25.23.100:3306 -r 172.25.23.1 -g -w 2
[root@mysql-lvs ~]# ipvsadm -a -t 172.25.23.100:3306 -r 172.25.23.2 -g -w 1
mysql-master
上面创建一个可以用于远程登陆连接的用户,这里使用这个用户来再次验证lvs
调度显示的信息 LVS
已经可以正常工作了,由于Mysql
对于后端的服务没有进行后端服务的检查,所以这里来提供一个脚本,希望能够检测Mysql
后台服务器的状况,并且根据状况作出相应的策略;[root@mysql-lvs /]# cat heath_check1.sh
#!/bin/bash
VIP=172.25.23.100
CPORT=3306
RIP=("172.25.23.1" "172.25.23.2")
RW=("1" "2")
RSTATUS=("1" "1")
FALL_BACK=127.0.0.1
RPORT=3306
TYPE=wlc
add() {
ipvsadm -a -t $VIP:$CPORT -r $1:$RPORT -g -w $2
echo "add $I successful"
[ $? -eq 0 ] && return 0 || return 1
}
del() {
ipvsadm -d -t $VIP:$CPORT -r $1:$RPORT
echo "del $I successful"
[ $? -eq 0 ] && return 0 || return 1
}
while :; do
let COUNT=0
for I in ${RIP[*]}; do
result=`mysqladmin ping -h$I -utestuser -pLixun121314.. 2> /dev/null | awk '{print $3}'`
if [ $result == "alive" ]; then
if [ ${RSTATUS[$COUNT]} -eq 0 ]; then
echo "add start"
add $I ${RW[$COUNT]}
[ $? -eq 0 ] && RSTATUS[$COUNT]=1
fi
else
if [ ${RSTATUS[$COUNT]} -eq 1 ]; then
echo "del start"
del $I
[ $? -eq 0 ] && RSTATUS[$COUNT]=0
fi
let COUNT++
fi
done
sleep 5
done
172.25.23.1
以及172.25.23.2
的服务的健康状况,并且能够动态的添加,删除后台的服务;LVS
负载两台sql-slave-reallvs
的任务就已经完成,接下来完成读写分离;mysql-proxy-0.8.3
,并且使用的是编译汉的二进制源码软件包[root@localhost mnt]# tar -xf mysql-proxy-0.8.3-linux-glibc2.3-x86-64bit.tar.gz -C /usr/local/mysql-proxy/
Mysql-proxy
./mysql-proxy --daemon --log-level=debug --plugins="proxy" --log-file=/var/log/mysql-proxy.log --proxy-backend-addresses="172.25.23.78:3306" --proxy-read-only-backend-addresses="172.25.23.100:3306" --proxy-lua-script=../mysql-proxy-0.8.3-linux-glibc2.3-x86-64bit/share/doc/mysql-proxy/rw-splitting.lua //这个脚本一定需要进行指定,否则无法完成读写分离功能;
[root@lvs-server Desktop]# mysql -h172.25.23.82 -utestuser -p --port=4040
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 28
Server version: 5.7.11-log MySQL Community Server (GPL)
Copyright (c) 2000, 2015, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [hello]> create table teacher(id int, name char);
Query OK, 0 rows affected (1.66 sec)
* 可以验证的是写操作在mysql-master
上面执行,查看从服务器是否同步到数据
* 测试读写分离,因为只有mysql-master
可以进行写入操作,并且sql-slave-reallvs
只能够向mysql-master
进行数据同步,所以只需要检查两个sql-slave-reallvs
上面是否存在数据更新,就可以,上面已经进行了这些操作;
* 接下来提供验证lvs
调度的查询操作的方法,通过观察
* 当进行查询操作时,这些节点的数据是会存在变化的
* 数据发生了改变
* 根据实验的规划,功能都已经完成了;