mysql主从复制，半同步，并行复制

1、mysql主从复制

复制原理：
Mysql 中有一种日志叫做 bin 日志（二进制日志）。这个日志会记录下所有修改了数据库的SQL 语句（insert,update,delete,create/alter/drop table, grant 等等）。
主从复制的原理其实就是把主服务器上的 bin 日志复制到从服务器上执行一遍，这样从服务器上的数据就和主服务器上的数据相同了。

[root@server1 conf]# vim /etc/my.cnf   

[root@server1 conf]# /etc/init.d/mysqld start   启动数据库
Starting MySQL SUCCESS! 
[root@server1 conf]# mysql -p   登陆mysql
Enter password:
mysql> show databases;  查看库名
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.00 sec)
mysql> show master status;  查看master状态

[root@server1 ~]# cd /usr/local/mysql/data/   mysql数据存放位置
[root@server1 data]# ls
auto.cnf         ib_buffer_pool  mysql             performance_schema  server-cert.pem
ca-key.pem       ibdata1         mysql-bin.000001  private_key.pem     server-key.pem
ca.pem           ib_logfile0     mysql-bin.index   public_key.pem      sys
client-cert.pem  ib_logfile1     mysql.sock        server1.err
client-key.pem   ibtmp1          mysql.sock.lock   server1.pid
[root@server1 data]# mysqlbinlog mysql-bin.000001  mysql日志有专门的查看方式用mysqlbinlog查看

[root@server1 data]# mysql -p  登陆mysql
Enter password:   输入密码
mysql> grant REPLICATION SLAVE ON *.* TO 'repl'@'%' IDENTIFIED BY 'westos';
Query OK, 0 rows affected, 1 warning (0.05 sec)
 REPLICATION SLAVE  表示复制权限
*.*  表示所有库.所有表
to 表示给谁
repl 表示用户
@ 分隔符
% 表示所有主机，除了localhost
IDENTIFIED BY 'westos' 表示给用户一个密码为westos
注意在8.0版本需要两条命令，先创建用户，再授权

mysql> show master status;  查看master状态

在server2上安装mysql：

[root@server1 local]# scp -r mysql/ server2:/usr/local/    将mysql文件拷贝到server2的/usr/local/ 里
[root@server1 local]# scp /etc/my.cnf server2:/etc/  将文件拷贝到server2 的/etc里
[root@server1 local]# cd /etc/init.d/   
[root@server1 init.d]# scp mysqld server2:/etc/init.d/  将启动脚本拷贝到server2的/etc/init.d/里
[root@server2 etc]# vim /etc/my.cnf   编辑my.cnf 文件
[mysqld]
datadir=/usr/local/mysql/data
socket=/usr/local/mysql/data/mysql.sock
symbolic-links=0

server-id=2     整数倍增长即可，不需要指定二进制名称并激活
[root@server2 tomcat]# cd /usr/local/mysql/data/   进入数据目录
[root@server2 data]# rm -fr *    删除
[root@server2 ~]# vim .bash_profile  编辑文件，添加环境变量

[root@server2 ~]# source .bash_profile   生效
[root@server2 ~]# useradd -u 1001 -g mysql -M -d /usr/local/nysql/ -s /sbin/nologin mysql  建立的用户uid gid 和server1保持一致
[root@server2 ~]# mysqld --initialize --user=mysql  初始化数据库

[root@server2 ~]# /etc/init.d/mysqld start   启动数据库
Starting MySQL.Logging to '/usr/local/mysql/data/server2.err'.
 SUCCESS! 
 [root@server2 ~]# mysql_secure_installation  数据库设置
 Securing the MySQL server deployment.

Enter password for user root:     生成的密码输入

The existing password for the user account root has expired. Please set a new password.

New password: 

Re-enter new password: 

VALIDATE PASSWORD PLUGIN can be used to test passwords
and improve security. It checks the strength of password
and allows the users to set only those passwords which are
secure enough. Would you like to setup VALIDATE PASSWORD plugin?

Press y|Y for Yes, any other key for No: 
Using existing password for root.
Change the password for root ? ((Press y|Y for Yes, any other key for No) : 

 ... skipping.
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.

Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
Success.


Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.

Disallow root login remotely? (Press y|Y for Yes, any other key for No) : y
Success.

By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.


Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
 - Dropping test database...
Success.

 - Removing privileges on test database...
Success.

Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.

Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y
Success.

All done! 
[root@server2 ~]# mysql -pwestos   登陆
mysql> show databases;    可以查看库
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.00 sec)

目前主从数据库上没有数据都是一致的，在生产环境中需要锁表备份

mysql> FLUSH TABLES WITH READ LOCK;  锁表
 mysqldump --all-databases --master-data > dbdump.db  备份数局库所有的数据到dbdump.db
 将dbdump.db拷贝到从机上
 mysql> UNLOCK TABLES;  解表

配置从库

[root@server2 ~]# mysql -pwestos  登陆数据库
mysql> CHANGE MASTER TO MASTER_HOST='172.25.50.1', MASTER_USER='repl', MASTER_PASSWORD='westos', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=437;
Query OK, 0 rows affected, 2 warnings (0.43 sec)
MASTER_HOST  表示主机的ip或者域名
MASTER_LOG_FILE 表示二进制日志
MASTER_LOG_POS 表示从那块开始执行
mysql> start slave;   开启slave
Query OK, 0 rows affected (0.05 sec)

mysql> show slave status\G;  查看slave状态

IO_Running 负责复制二进制日志
有两个yes表示成功，如果IO不对可能是 用户和网络（火墙）问题，如果sql不对可能是两个主机上的sql不一致

测试：

mysql> create database westos;   在主库上建立数据库
Query OK, 1 row affected (0.05 sec)
mysql> show databases;   查看从库数据库

2、多节点线性复制server1 ->server2->server3

在server3上装mysql

[root@server1 local]# /etc/init.d/mysqld stop  停掉mysql
[root@server1 local]# scp -r mysql/ server3:/usr/local/   将server1上的mysql目录拷贝到server3 的/usr/local/ 
[root@server1 etc]# scp my.cnf server3:/etc/    从server1拷贝my.cnf 配置文件
[root@server1 etc]# scp /etc/init.d/mysqld  server3:/etc/init.d/  从server1拷贝启动脚本
[root@server1 etc]# /etc/init.d/mysqld start  启动server1的mysql
[root@server3 local]# vim /etc/my.cnf   编辑配置文件
[mysqld]
datadir=/usr/local/mysql/data
socket=/usr/local/mysql/data/mysql.sock
symbolic-links=0

server-id=3
[root@server3 local]# cd /usr/local/mysql/data/  进入mysql数据目录
[root@server3 data]# rm -fr *  删除
[root@server3 ~]# vim .bash_profile   添加环境变量

[root@server3 ~]# source .bash_profile  使环境变量生效
[root@server3 ~]# groupadd -g 1001 mysql 创建mysql组
[root@server3 ~]# useradd -u 1001 -g mysql -M -d /usr/local/nysql/ -s /sbin/nologin mysql   创建mysql用户
[root@server3 ~]# mysqld --initialize --user=mysql  初始化
[root@server3 ~]# /etc/init.d/mysqld start  启动mysql
[root@server3 ~]# mysql_secure_installation  配置mysql

Securing the MySQL server deployment.

Enter password for user root: 

The existing password for the user account root has expired. Please set a new password.

New password: 

Re-enter new password: 

VALIDATE PASSWORD PLUGIN can be used to test passwords
and improve security. It checks the strength of password
and allows the users to set only those passwords which are
secure enough. Would you like to setup VALIDATE PASSWORD plugin?

Press y|Y for Yes, any other key for No: 
Using existing password for root.
Change the password for root ? ((Press y|Y for Yes, any other key for No) : 

 ... skipping.
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.

Remove anonymous users? (Press y|Y for Yes, any other key for No) : y
Success.


Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.

Disallow root login remotely? (Press y|Y for Yes, any other key for No) : y
Success.

By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.


Remove test database and access to it? (Press y|Y for Yes, any other key for No) : y
 - Dropping test database...
Success.

 - Removing privileges on test database...
Success.

Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.

Reload privilege tables now? (Press y|Y for Yes, any other key for No) : y
Success.

All done! 

[root@server1 etc]# mysql -pwestos  登陆server1的mysql
mysql> use westos;  进入westos库
mysql> create table user_tb (           创建表
    -> username  varchar(50) not null,
    -> password  varchar(50) not null);
Query OK, 0 rows affected (0.45 sec)
mysql> desc user_tb;    查看表
+----------+-------------+------+-----+---------+-------+
| Field    | Type        | Null | Key | Default | Extra |
+----------+-------------+------+-----+---------+-------+
| username | varchar(50) | NO   |     | NULL    |       |
| password | varchar(50) | NO   |     | NULL    |       |
+----------+-------------+------+-----+---------+-------+
2 rows in set (0.03 sec)

mysql> insert into user_tb values ('user1','123');   插入数据
Query OK, 1 row affected (0.11 sec)

mysql> insert into user_tb values ('user2','456');    插入数据
Query OK, 1 row affected (0.09 sec)
mysql> select * from user_tb;   查询表所有字段
+----------+----------+
| username | password |
+----------+----------+
| user1    | 123      |
| user2    | 456      |
+----------+----------+
2 rows in set (0.00 sec)

server1上mysql建立的数据，server2上也已经同步了
[root@server2 ~]# mysql -pwestos  登陆server2数据库
mysql> use westos;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select * from user_tb;    sever1上建立的表，server2上也有了
+----------+----------+
| username | password |
+----------+----------+
| user1    | 123      |
| user2    | 456      |
+----------+----------+
2 rows in set (0.00 sec)

让server3也与server1、server2保持数据同步

[root@server1 ~]# mysqldump -pwestos westos > dump.db   在server1上将westos库导出
mysqldump: [Warning] Using a password on the command line interface can be insecure.
[root@server1 ~]# scp dump.db server3:   导出的数据拷贝到server3上
注意：
[root@server1 ~]# cat dump.db   查看导出的数据文件，发现表会先删除后建立

[root@server3 ~]# mysqladmin -pwestos create  westos  在server3上创建westos库
[root@server3 ~]# mysql -pwestos westos < dump.db   将dump.db导入到westos库
[root@server3 ~]# mysql -pwestos  登陆数据库
mysql> select * from westos.user_tb;  发现数据已经同步了
+----------+----------+
| username | password |
+----------+----------+
| user1    | 123      |
| user2    | 456      |
+----------+----------+
2 rows in set (0.00 sec)

[root@server2 ~]# vim /etc/my.cnf 编辑配置文件
从库做为其他从库的主库时 log-slave-updates参数是必须要添加的

[root@server2 data]# mysql -pwestos  登陆数据库
mysql> grant REPLICATION SLAVE ON *.* TO 'repl'@'%' IDENTIFIED BY 'westos'      创建用户并授权复制权限
[root@server3 ~]# mysql -h 172.25.50.2 -u repl -p  从server3上用repl用户登陆mysql
mysql>    发现可以登陆 ，说明用户创建成成功
[root@server3 ~]# mysql -pwestos  登陆数据库
mysql> change master to master_host='172.25.50.2' , master_user='repl', master_password='westos', master_log_file='mysql-bin.000001', master_log_pos=437;
查看master_log_file 日志名，master_log_pos日志号，在server2上查看
mysql> show master status;   
+------------------+----------+--------------+------------------+-------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+-------------------+
| mysql-bin.000001 |      437 |              |                  |                   |
+------------------+----------+--------------+------------------+-------------------+
mysql> start slave;  开启slave
Query OK, 0 rows affected (0.07 sec)
mysql> show slave status\G;

测试：

[root@server1 ~]# msyql -pwestos  登陆
 mysql> use westos;  进入westos数据库
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> insert into user_tb values ('user3','789');  插入数据
Query OK, 1 row affected (0.11 sec)

sever1上插入的数据，看看server3上有没有，如果有就复制成功了
在server3上查看user_tb表
mysql> select * from westos.user_tb;
+----------+----------+
| username | password |
+----------+----------+
| user1    | 123      |
| user2    | 456      |
| user3    | 789      |        server1上插入的数据，在server3上有，成功
+----------+----------+
3 rows in set (0.00 sec)

线性复制有缺点：当sever1master挂掉了，其中slave延迟最低的一个会接管，其他slave奔着这个接管的slave去，但是在设置change master 日志文件，和日志号时， 不知道是多少，还需要自己去推断增加了运维维护的难度
面对上述问题，引入gtid，他时全局id，所有id都一样，而且还是单调递增的

3、gtid主从复制

server1配置：
[root@server1 ~]# vim /etc/my.cnf 修改配置文件

[root@server1 ~]# /etc/init.d/mysqld restart    重启服务
Shutting down MySQL............ SUCCESS! 
Starting MySQL.. SUCCESS! 

server2配置：
[root@server2 ~]# vim /etc/my.cnf   编辑配置文件

[root@server2 ~]# /etc/init.d/mysqld restart   重启数据库
 Shutting down MySQL............ SUCCESS! 
Starting MySQL. SUCCESS!  
[root@server2 data]# mysql -p  登陆数据库
mysql> stop slave  停止slave
mysql> change master to master_host='172.25.50.1' , master_user='repl', master_password='westos', MASTER_AUTO_POSITION = 1;
Query OK, 0 rows affected, 2 warnings (0.22 sec)
MASTER_AUTO_POSITION = 1 表示自动
mysql> start slave;    启动slave
Query OK, 0 rows affected (0.02 sec)
mysql> show slave status\G;

server3配置：

[root@server3 ~]# vim /etc/my.cnf

[root@server3 ~]# /etc/init.d/mysqld restart   重启数据库
Shutting down MySQL.. SUCCESS! 
Starting MySQL. SUCCESS! 
[root@server3 ~]# mysql -pwestos  登陆数据库
mysql> stop slave;  停止slave
mysql> change master to master_host='172.25.50.1' , master_user='repl', master_password='westos', MASTER_AUTO_POSITION = 1 （一主两从模式）
mysql> start slave;  启动slave
Query OK, 0 rows affected (0.02 sec)
mysql> show slave status\G;  查看slave状态

测试：

[root@server1 ~]# mysql -pwestos  登陆数据库
mysql> use westos;  进入westos数据库
mysql> insert into user_tb values ('user4','666');  插入数据
Query OK, 1 row affected (0.10 sec)
[root@server3 ~]# mysql -pwestos 登陆
mysql> show slave status\G;  查看slave状态

mysql> select * from westos.user_tb;   查询表内容

如果当servert1坏掉，server2接管，只需改server3上change master to master_host=‘172.25.50.2’ ；如果server3接管，只需改
server2上change master to master_host='172.25.50.3 ‘

4、查看主库、从库运行线程

[root@server1 ~]# mysql -pwestos  登陆主库
mysql> show processlist ;     查看server1 mysql主库上运行的线程

[root@server2 ~]# mysql -pwestos  登陆从库
mysql> show processlist;   查看从库运行线程

5、mysql半同步流程：

master主库上负责读写操作。当binlog dump 接收到主库的变化时，会给slave发送二进制日志，io接收到二进制日志后他会把日志存到本地的relaylog，当io线程已经把日志存到relaylog并保存到disk磁盘，io线程会发送一个确认ack，告诉master数据已经存下来了，此过程叫半同步，slave里的sql线程读去relaylog，master半同步的时间是10s，如果在10s内slave没有正常给master发送ack确认，master就会切成异步模式，就会导致主从数据不一致；

如果master有多个用户并行写入，salve端io是多线程的读取的数度快，sql是单线程的回放的满，导致io存的越来越多，sql读取的慢，就会造成延迟

mysql> show slave status\G;

6、AFTER_COMMITM模式 mysql半同步流程图

此版本适用于5.6版本

用户做一个引擎提交首先写入binlog日志当binglog dump线程发现binglog变更了会做一个push动作，将二进制日志发送给islave端
io线程，提交如果是异步模式，直接就进行sync（同步），进行刷磁盘动作，刷完以后进行引擎提交 ，最后silave端io发出一个ack确认当master收到确认之后，才给客户端返回ok；
缺点：master刷完磁盘后就直接进行引擎提交，不等slave端io线程发ack确认就直接提交，如果这时候slave端还没发确认，没存入数据，master就挂掉了，主备切换会被slave接管成为新的master，此时新的master没有之前master写入的数据，数据就丢失了，如下图：

7、AFTER_SYNC模式mysql半同步流程图

AFTER_SYNC版本的半同步模式和AFTER_COMMITM版本的半同步模式流程大致相同，唯一区别是引擎提交是在slave端io线程发送ack确认，master确认之后才会引擎提交，保证slave没有接收到数据，master也看不到，此方式称为无损模式

8、mysql 5、7半同步配置

主服务器master配置

安装插件：
[root@server1 ~]# msyql -pwestos  登陆数据库
mysql> INSTALL PLUGIN rpl_semi_sync_master SONAME 'semisync_master.so';  msater端安装插件
Query OK, 0 rows affected (0.11 sec)        
mysql> show plugins;  查看服务器有关插件信息

设置主服务器master的半同步复制全局变量：
[root@server1 ~]# msyql -pwestos  登陆数据库
mysql> SET GLOBAL rpl_semi_sync_master_enabled =1; 设置master端全局变量，开启半同步复制
mysql> show variables like 'rpl%'  查看服务器复制配置信息，

mysql> set global  rpl_semi_sync_master_wait_for_slave_count =2;   表示master要等待多少个slave，目前我们是一主两从，所以设置为2;   表示master要等待多少个slave，目前我们是一主两从，所以设置为2

从服务器slave配置

安装插件：
[root@server2 ~]# mysql -pwestos  server2上登陆数据库
mysql> INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';   slave端安装插件
Query OK, 0 rows affected (0.13 sec)
[root@server3 ~]# mysql -pwestos    server3上登陆数据库
mysql> INSTALL PLUGIN rpl_semi_sync_slave SONAME 'semisync_slave.so';   slave端安装插件

注：SET GLOBAL rpl_semi_sync_master_timeout = N; 生产环境中半同步时间一般设置为无穷大

设置从服务器slave的半同步复制全局变量：
[root@server2~]# mysql -pwestos  server2上登陆数据库
mysql> SET GLOBAL rpl_semi_sync_slave_enabled =1;  设置slave端全局变量， 开启半同步复制
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like 'rpl%'  查看服务器复制配置信息

[root@server3 ~]# mysql -pwestos  server3上登陆数据库
mysql> SET GLOBAL rpl_semi_sync_slave_enabled =1;  设置slave端全局变量， 开启半同步复制
Query OK, 0 rows affected (0.00 sec)
mysql> show variables like 'rpl%';  

slave端重启IO线程

server2、server3：
mysql> STOP SLAVE IO_THREAD;   
Query OK, 0 rows affected (0.02 sec)

mysql> START SLAVE IO_THREAD;
Query OK, 0 rows affected (0.00 sec)
mysql> show status like 'rpl%';

半同步配置完成在server1上查看配置是否成功
server1:
mysql> show status like ‘rpl%’ ; 查看

测试：

[root@server1 ~]# mysql -pwestos
mysql> use westos;
mysql> insert into user_tb values ('user6','999');   主库上插入数据
[root@server2 ~]# mysql -pwestos登陆从机数据库
mysql> select * from westos.user_tb; 查询，从库上已经有server1插入的数据

如果server2 slave端停掉，会怎样？

mysql> STOP SLAVE IO_THREAD;   停掉slave
Query OK, 0 rows affected (0.03 sec)

在server1上插入数据：
mysql> insert into user_tb values ('user7','5555');  发现插入数度变慢了
mysql>  show status like 'rpl%' ;  查看master状态发现

9、 sql线程优化-mysq并行复制（解决延迟问题）

master有大量的并行写入，sql单个线程回放。是不够用的，该如何解决？？
需要用到mysql并行复制

[root@server2 ~]# mysql -pwestos
mysql> STOP SLAVE SQL_THREAD;   停掉sql线程
Query OK, 0 rows affected (0.05 sec)
mysql> set global slave_parallel_type=LOGICAL_CLOCK;  开启并行复制
Query OK, 0 rows affected (0.01 sec)
mysql> set global slave_parallel_workers=16;   开启16个workers
Query OK, 0 rows affected (0.00 sec)
[root@server2 ~]# cd /usr/local/mysql/data/  进入slave端数据库目录

master.info 和 relay-log.info 文件记录了master和slave之间的信息，当开启并行复制，文件变更的速度会比原来快的多，当频繁的刷新文件，一种是以文件形式保存，性能比较低，一种以数据库表形式保存，这种性能高。所以将maste.info和relay-log.info以数据库表的形式储存：

server2:
mysql> stop slave ;   关闭slave
mysql> set global master_info_repository='TABLE';   设置为以表的形式保存
Query OK, 0 rows affected (0.05 sec)
mysql> set global relay_log_info_repository='TABLE';   设置为以表的形式保存
Query OK, 0 rows affected (0.04 sec)
mysql> start slave;  开启slave
Query OK, 0 rows affected (0.74 sec)
mysql> use mysql; 进入mysql数据库
mysql> show  tables;  显示数据库中所有表

以上设置是为热备模式，也可以将参数写在主配置文件里

[root@server3 data]# vim /etc/my.cnf

[root@server3 data]# /etc/init.d/mysqld restart   重启数据库
Shutting down MySQL.. SUCCESS! 
Starting MySQL. SUCCESS! 

10、设置mysql主从复制延迟

作用：1、假如主库上误操作，如果有延迟，可以及时数据回滚
2、主从复制必然会造成延迟，只是可以优化减少延迟，我想知道延迟多长时间对对我们的业务造成影响。可以提前测试预估
在server3上设置延迟：

[root@server3 data]# mysql -pwestos 
mysql> stop slave;
Query OK, 0 rows affected (0.13 sec)
 mysql> CHANGE MASTER TO MASTER_DELAY = 30;  设置30s延迟
Query OK, 0 rows affected (0.38 sec)
 mysql> start slave;   开启slave
Query OK, 0 rows affected (0.76 sec)
mysql> show slave status\G;      查看slave状态

11、设置慢查询

[root@server1 ~]# mysql -pwestos   server1上登陆数据库
mysql> set global slow_query_log=ON;  打开慢查询
Query OK, 0 rows affected (0.88 sec)
root@server1 ~]# cd /usr/local/mysql/data/  进入mysql数据目录

mysql> show variables like 'long%';   查询时间

mysql> select sleep(10);  执行一个10s的语句
[root@server1 data]# cat server1-slow.log    查询日志

将运行慢的查询语句都过滤出来，分析为啥慢，然后优化