数据库11总结2

InnoDB 存储引擎核心特性

5.1 种类
InnoDB
MyISAM
CSV
Memory

tokudb
myrocks

influxdb--->时序类数据库,数据不能手工删除,只能设置自动清理策略.

5.2 InnoDB 核心特性
事务
行锁
热备
MVCC
一致性hash
insert buffer
ACSR
聚簇索引
主从复制(GTID)
外键

5.3 事务 ACID
A: 原子性.事务的最小工作单元,不可以被拆分.不能产生中间状态,要么全成功,要么全失败.
UNDO(回滚)+REDO(前滚)
C: 一致性. 在一个事务更新逻辑中,不管是处于事务更新前,中,结束.
本事务中操作的数据都终将保证一致状态,不会受到其他事务影响.
AID+锁+MVCC等机制都是对数据的最终一致起到关键作用
I: 隔离性 . 隔离级别+锁+MVCC
D: 持久性. 一旦被提交成功的数据,不管宕机与否,都不会导致事务的更新丢失.
Redo

5.4 Redo 和 UNDO

5.5 MVCC 多版本并发控制
利用的是UnDO的快照.InnoDB引擎中,维护着多个版本的UnDO快照.
多事务并发环境中,互相不影响.只要多事务之间更新的数据锁不互相冲突.事务是可以并发执行的.

特性:

在RR级别下的,一致性快照读是由MVCC保证的.在同一个事务生命周期内,获取的一定是同一个版本数据.
对于读操作,实现的是非锁定快照读. 读的操作不加锁.

5.6 隔离级别
RC : 事务并发读会更高一些. 防止脏读. 会出现不可重复读和幻读
RR : 可以防止脏读和不可重复读(一致性快照读.),配合GAP和Next-lock实现防止幻读

5.7 锁机制
RL
GL
NL
MDL ----> DDL DCL
TL ----> DDL DCL
FTWRL ---> 全局锁 ----> XBK备份非InnoDB

IS
S select xxx from xxx lock in shared mode;
IX
X select xxx from xxx for update;

5.8 碎片整理
alter table t1 engine=innodb
Pt-archiver
optimize table t1;

5.8 表空间迁移

create table t1
alter table t1 discard tablespace
alter table t1 import tablespace

5.9 自适应hash
加速索引读取性能, 索引的索引

5.10 insert buffer
为了大量insert业务场景设计.

日志管理

6.1 log_error
错误 ----> [ERROR]
启动
宕机
主从
锁
工作状态

6.2 binlog 二进制日志
如何截取二进制日志
show master status ; ----> 正在使用的二进制日志
show binlog events in 'xxxx' ----> 看事件
mysqlbinlog --start-position --stop-position xxxx.000009 > /backup/bin.sql
mysqlbinlog --include-gtids --exclude-gtids --skip-gtids > /backup/bin.sql

set sql_log_bin=0;
source
set sql_log_bin=1;

binlog_format :
SBR 语句模式
RBR 行模式
MBR 混合模式

滚动
flush logs
重启
MySQLadmin flush-logs
达到1G

备份时会应用
binlog_server: mysqlbinlog

清理
expire_logs_days=15

参数:
innodb_flush_log_at_trx_commit=1 ----> redo 刷写策略
innodb_flush_method=O_direct
sync_binlog=1 ----> binlog 刷写策略

6.3 slowlog
slow_query_log=1 # 慢日志开关
slow_query_log_file # 慢日志位置
long_query_time # 慢日志记录条件(时间)
log_queries_not_using_indexes # 慢日志记录条件(不走索引的)

mysqldumpslow
mysqlsla
pt-query-digest+Anemometer 图形化展示慢日志

备份恢复
7.1 mysqldump
-A -B --master-data=2 --single-transaction --triggers -R -E --max_allowed_packet=64M

7.2 xbk
备份过程:
(1) ckpt,记录LSN
(2) 备份innodb表
(3) 非InnoDB表,开启FTWRL(Flush Tbales with read lock)全局读锁,拷贝文件,自动解锁
(4) 并将备份过程中redo日志进行备份,记录结束LSN
(5) 增量都是基于上次last_lsn,检查所有变化的数据页进行备份
备份策略:
1. full+inc+binlog
2. full+binlog

恢复过程:
单独全备:
1. prepare 全备 : --apply-log
2. cp
全备+增量:
1. prepare 基础全备 : --apply-log --redo-only
2. 增量到基础全备,prepare增量: --apply-log --redo-only
3. 最后增量到基础全备,prepare增量: --apply-log
4. prepare 终极全备 : --apply-log
5. 恢复全备
This option should be used when preparing the base full
backup and when merging all incrementals except the last one.

主从复制
8.1 主从复制搭建
传统
gtid
8.2 主从复制原理

8.3 主从监控
show slave status\G

8.4 主从故障分析

8.5 主从延时

8.6 演变
延时从:
SQL线程延时一段时间,回放日志.
怎么用? 解决逻辑损坏.

半同步概念 :

过滤复制配置:

MHA高可用
9.1 搭建
9.2 原理
9.3 故障处理
高可用配合读写分离
分布式架构-MyCat
优化
NoSQL

==========================================
MHA 故障环境修复:

修复主从
51/53节点:
mysql -S /tmp/mysql.sock -e "show slave status \G"|grep Running:

2.主库修复vip
52节点:
[root@db02 ~]# ifconfig eth0:1 10.0.0.55/24

说明: 删除vip?
ifconfig eth0:1 down

修复binlog_server
53节点:
[root@db03 ~]# cd /data/mysql/binlog/
[root@db03 /data/mysql/binlog]# rm -rf *
[root@db03 /data/mysql/binlog]# mysqlbinlog -R --host=10.0.0.52 --user=mha --password=mha --raw --stop-never mysql-bin.000001 &
检查配置文件
53节点:
[root@db03 /data/mysql/binlog]# vim /etc/mha/app1.cnf
检查SSH和主从
53节点:
masterha_check_ssh --conf=/etc/mha/app1.cnf
masterha_check_repl --conf=/etc/mha/app1.cnf
启动MHA
53节点:
nohup masterha_manager --conf=/etc/mha/app1.cnf --remove_dead_master_conf --ignore_last_failover < /dev/null > /var/log/mha/app1/manager.log 2>&1 &

================================
读写分离

安装
略.
2.配置
2.1 软件路径
[root@db03 ~]# cd /usr/local/mysql-proxy/
[root@db03 /usr/local/mysql-proxy]# ll
总用量 0
drwxr-xr-x 2 root root 75 11月 26 16:54 bin
drwxr-xr-x 2 root root 22 11月 26 16:54 conf
drwxr-xr-x 3 root root 331 11月 26 16:54 lib
drwxr-xr-x 2 root root 6 12月 17 2014 log
[root@db03 /usr/local/mysql-proxy]#
2.2 环境变量设置
[root@db03 /usr/local/mysql-proxy]# vim /etc/profile
export PATH=/usr/local/mysql-proxy/bin:$PATH

source /etc/profile

2.3 配置文件
[root@db03 /usr/local/mysql-proxy/conf]# cd /usr/local/mysql-proxy/conf
[root@db03 /usr/local/mysql-proxy/conf]# vim oldguo.cnf
[mysql-proxy]
admin-username = user
admin-password = pwd
proxy-backend-addresses = 10.0.0.55:3306
proxy-read-only-backend-addresses = 10.0.0.51:3306,10.0.0.53:3306
pwds = repl:3yb5jEku5h4=,mha:O2jBXONX098=
daemon = true
keepalive = true
event-threads = 8
log-level = message
log-path = /usr/local/mysql-proxy/log
sql-log=ON
proxy-address = 0.0.0.0:33060
admin-address = 0.0.0.0:2345
charset=utf8

2.4 启动atlas
mysql-proxyd oldguo start

2.5 测试读写分离
读:
[root@db03 ~]# mysql -umha -pmha -h10.0.0.53 -P33060
db03 [(none)]>select @@server_id;

写:
db03 [(none)]>begin;
db03 [(none)]>select @@server_id;
db03 [(none)]>commit;

Atlas 的管理(在线)
db03 [(none)]>select * from help;
+----------------------------+---------------------------------------------------------+
| command | description |
+----------------------------+---------------------------------------------------------+
| SELECT * FROM help | shows this help |
| SELECT * FROM backends | lists the backends and their state |
| SET OFFLINE backend_id is backend_ndx's id |
| SET ONLINE backend | example: "add master 127.0.0.1:3306", ... |
| ADD SLAVE backend_id | example: "remove backend 1", ... |
| SELECT * FROM clients | lists the clients |
| ADD CLIENT client | example: "remove client 192.168.1.2", ... |
| SELECT * FROM pwds | lists the pwds |
| ADD PWD pwd | example: "add enpwd user:encrypted_password", ... |
| REMOVE PWD $pwd | example: "remove pwd user", ... |
| SAVE CONFIG | save the backends to config file |
| SELECT VERSION | display the version of Atlas |
+----------------------------+---------------------------------------------------------+
16 rows in set (0.00 sec)

3.1 帮助查看
SELECT * FROM help

3.2 查看后端节点
db03 [(none)]>SELECT * FROM backends ;
+-------------+----------------+-------+------+
| backend_ndx | address | state | type |
+-------------+----------------+-------+------+
| 1 | 10.0.0.55:3306 | up | rw |
| 2 | 10.0.0.51:3306 | up | ro |
| 3 | 10.0.0.53:3306 | up | ro |
+-------------+----------------+-------+------+
3 rows in set (0.00 sec)

db03 [(none)]>SELECT * FROM backends ;
+-------------+----------------+---------+------+
| backend_ndx | address | state | type |
+-------------+----------------+---------+------+
| 1 | 10.0.0.55:3306 | up | rw |
| 2 | 10.0.0.51:3306 | up | ro |
| 3 | 10.0.0.53:3306 | offline | ro |
+-------------+----------------+---------+------+
3 rows in set (0.00 sec)

3.4 添加/删除节点
REMOVE BACKEND 2;
REMOVE BACKEND 1;
ADD SLAVE 10.0.0.51:3306
ADD master 10.0.0.55:3306

3.5 开用户
3.5.1. MySQL主库建用户,授权
db02 [(none)]>grant all on . to root@'localhost' identified by '123';

3.5.2. Atlas 添加用户
ADD PWD root:123

3.6 永久保存配置到文件
db03 [(none)]>save config;

MyCAT基础架构准备
2.1 环境准备：
两台虚拟机 db01 db02
每台创建四个mysql实例：3307 3308 3309 3310
2.2 删除历史环境：
pkill mysqld
rm -rf /data/330*
mv /etc/my.cnf /etc/my.cnf.bak
2.3 创建相关目录初始化数据
mkdir /data/33{07..10}/data -p
mysqld --initialize-insecure --user=mysql --datadir=/data/3307/data --basedir=/usr/local/mysql
mysqld --initialize-insecure --user=mysql --datadir=/data/3308/data --basedir=/usr/local/mysql
mysqld --initialize-insecure --user=mysql --datadir=/data/3309/data --basedir=/usr/local/mysql
mysqld --initialize-insecure --user=mysql --datadir=/data/3310/data --basedir=/usr/local/mysql
2.4 准备配置文件和启动脚本
========db01==============
cat >/data/3307/my.cnf< [mysqld]
basedir=/usr/local/mysql
datadir=/data/3307/data
socket=/data/3307/mysql.sock
port=3307
log-error=/data/3307/mysql.log
log_bin=/data/3307/mysql-bin
binlog_format=row
skip-name-resolve
server-id=7
gtid-mode=on
enforce-gtid-consistency=true
log-slave-updates=1
EOF

cat >/data/3308/my.cnf< [mysqld]
basedir=/usr/local/mysql
datadir=/data/3308/data
port=3308
socket=/data/3308/mysql.sock
log-error=/data/3308/mysql.log
log_bin=/data/3308/mysql-bin
binlog_format=row
skip-name-resolve
server-id=8
gtid-mode=on
enforce-gtid-consistency=true
log-slave-updates=1
EOF

cat >/data/3309/my.cnf< [mysqld]
basedir=/usr/local/mysql
datadir=/data/3309/data
socket=/data/3309/mysql.sock
port=3309
log-error=/data/3309/mysql.log
log_bin=/data/3309/mysql-bin
binlog_format=row
skip-name-resolve
server-id=9
gtid-mode=on
enforce-gtid-consistency=true
log-slave-updates=1
EOF
cat >/data/3310/my.cnf< [mysqld]
basedir=/usr/local/mysql
datadir=/data/3310/data
socket=/data/3310/mysql.sock
port=3310
log-error=/data/3310/mysql.log
log_bin=/data/3310/mysql-bin
binlog_format=row
skip-name-resolve
server-id=10
gtid-mode=on
enforce-gtid-consistency=true
log-slave-updates=1
EOF

cat >/etc/systemd/system/mysqld3307.service< [Unit]
Description=MySQL Server
Documentation=man:mysqld(8)
Documentation=http://dev.mysql.com/doc/refman/en/using-systemd.html
After=network.target
After=syslog.target
[Install]
WantedBy=multi-user.target
[Service]
User=mysql
Group=mysql
ExecStart=/usr/local/mysql/bin/mysqld --defaults-file=/data/3307/my.cnf
LimitNOFILE = 5000
EOF

cat >/etc/systemd/system/mysqld3308.service< [Unit]
Description=MySQL Server
Documentation=man:mysqld(8)
Documentation=http://dev.mysql.com/doc/refman/en/using-systemd.html
After=network.target
After=syslog.target
[Install]
WantedBy=multi-user.target
[Service]
User=mysql
Group=mysql
ExecStart=/usr/local/mysql/bin/mysqld --defaults-file=/data/3308/my.cnf
LimitNOFILE = 5000
EOF

[Install]
WantedBy=multi-user.target
[Service]
User=mysql
Group=mysql
ExecStart=/usr/local/mysql/bin/mysqld --defaults-file=/data/3310/my.cnf
LimitNOFILE = 5000
EOF
========db02===============
cat >/data/3307/my.cnf< [mysqld]
basedir=/usr/local/mysql
datadir=/data/3307/data
socket=/data/3307/mysql.sock
port=3307
log-error=/data/3307/mysql.log
log_bin=/data/3307/mysql-bin
binlog_format=row
skip-name-resolve
server-id=17
gtid-mode=on
enforce-gtid-consistency=true
log-slave-updates=1
EOF
cat >/data/3308/my.cnf< [mysqld]
basedir=/usr/local/mysql
datadir=/data/3308/data
port=3308
socket=/data/3308/mysql.sock
log-error=/data/3308/mysql.log
log_bin=/data/3308/mysql-bin
binlog_format=row
skip-name-resolve
server-id=18
gtid-mode=on
enforce-gtid-consistency=true
log-slave-updates=1
EOF
cat >/data/3309/my.cnf< [mysqld]
basedir=/usr/local/mysql
datadir=/data/3309/data
socket=/data/3309/mysql.sock
port=3309
log-error=/data/3309/mysql.log
log_bin=/data/3309/mysql-bin
binlog_format=row
skip-name-resolve
server-id=19
gtid-mode=on
enforce-gtid-consistency=true
log-slave-updates=1
EOF

cat >/data/3310/my.cnf< [mysqld]
basedir=/usr/local/mysql
datadir=/data/3310/data
socket=/data/3310/mysql.sock
port=3310
log-error=/data/3310/mysql.log
log_bin=/data/3310/mysql-bin
binlog_format=row
skip-name-resolve
server-id=20
gtid-mode=on
enforce-gtid-consistency=true
log-slave-updates=1
EOF

cat >/etc/systemd/system/mysqld3309.service< [Unit]
Description=MySQL Server
Documentation=man:mysqld(8)
Documentation=http://dev.mysql.com/doc/refman/en/using-systemd.html
After=network.target
After=syslog.target
[Install]
WantedBy=multi-user.target
[Service]
User=mysql
Group=mysql
ExecStart=/usr/local/mysql/bin/mysqld --defaults-file=/data/3309/my.cnf
LimitNOFILE = 5000
EOF
cat >/etc/systemd/system/mysqld3310.service< [Unit]
Description=MySQL Server
Documentation=man:mysqld(8)
Documentation=http://dev.mysql.com/doc/refman/en/using-systemd.html
After=network.target
After=syslog.target
[Install]
WantedBy=multi-user.target
[Service]
User=mysql
Group=mysql
ExecStart=/usr/local/mysql/bin/mysqld --defaults-file=/data/3310/my.cnf
LimitNOFILE = 5000
EOF
2.5 修改权限，启动多实例
chown -R mysql.mysql /data/*
systemctl start mysqld3307
systemctl start mysqld3308
systemctl start mysqld3309
systemctl start mysqld3310

mysql -S /data/3307/mysql.sock -e "show variables like 'server_id'"
mysql -S /data/3308/mysql.sock -e "show variables like 'server_id'"
mysql -S /data/3309/mysql.sock -e "show variables like 'server_id'"
mysql -S /data/3310/mysql.sock -e "show variables like 'server_id'"
2.6 节点主从规划
箭头指向谁是主库
10.0.0.51:3307 <-----> 10.0.0.52:3307
10.0.0.51:3309 ------> 10.0.0.51:3307
10.0.0.52:3309 ------> 10.0.0.52:3307

10.0.0.52:3308  <----->    10.0.0.51:3308
10.0.0.52:3310  ----->     10.0.0.52:3308
10.0.0.51:3310  ----->     10.0.0.51:3308

2.7 分片规划
shard1：
Master：10.0.0.51:3307
slave1：10.0.0.51:3309
Standby Master：10.0.0.52:3307
slave2：10.0.0.52:3309
shard2：
Master：10.0.0.52:3308
slave1：10.0.0.52:3310
Standby Master：10.0.0.51:3308
slave2：10.0.0.51:3310
2.8 开始配置
shard1
10.0.0.51:3307 <-----> 10.0.0.52:3307
db02
mysql -S /data/3307/mysql.sock -e "grant replication slave on . to repl@'10.0.0.%' identified by '123';"
mysql -S /data/3307/mysql.sock -e "grant all on . to root@'10.0.0.%' identified by '123' with grant option;"
db01
mysql -S /data/3307/mysql.sock -e "CHANGE MASTER TO MASTER_HOST='10.0.0.52', MASTER_PORT=3307, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='123';"
mysql -S /data/3307/mysql.sock -e "start slave;"
mysql -S /data/3307/mysql.sock -e "show slave status\G"
db02
mysql -S /data/3307/mysql.sock -e "CHANGE MASTER TO MASTER_HOST='10.0.0.51', MASTER_PORT=3307, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='123';"
mysql -S /data/3307/mysql.sock -e "start slave;"
mysql -S /data/3307/mysql.sock -e "show slave status\G"
10.0.0.51:3309 ------> 10.0.0.51:3307
db01
mysql -S /data/3309/mysql.sock -e "CHANGE MASTER TO MASTER_HOST='10.0.0.51', MASTER_PORT=3307, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='123';"
mysql -S /data/3309/mysql.sock -e "start slave;"
mysql -S /data/3309/mysql.sock -e "show slave status\G"
10.0.0.52:3309 ------> 10.0.0.52:3307
db02
mysql -S /data/3309/mysql.sock -e "CHANGE MASTER TO MASTER_HOST='10.0.0.52', MASTER_PORT=3307, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='123';"
mysql -S /data/3309/mysql.sock -e "start slave;"
mysql -S /data/3309/mysql.sock -e "show slave status\G"
shard2
10.0.0.52:3308 <-----> 10.0.0.51:3308
db01
mysql -S /data/3308/mysql.sock -e "grant replication slave on . to repl@'10.0.0.%' identified by '123';"
mysql -S /data/3308/mysql.sock -e "grant all on . to root@'10.0.0.%' identified by '123' with grant option;"
db02
mysql -S /data/3308/mysql.sock -e "CHANGE MASTER TO MASTER_HOST='10.0.0.51', MASTER_PORT=3308, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='123';"
mysql -S /data/3308/mysql.sock -e "start slave;"
mysql -S /data/3308/mysql.sock -e "show slave status\G"
db01
mysql -S /data/3308/mysql.sock -e "CHANGE MASTER TO MASTER_HOST='10.0.0.52', MASTER_PORT=3308, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='123';"
mysql -S /data/3308/mysql.sock -e "start slave;"
mysql -S /data/3308/mysql.sock -e "show slave status\G"
10.0.0.52:3310 -----> 10.0.0.52:3308
db02
mysql -S /data/3310/mysql.sock -e "CHANGE MASTER TO MASTER_HOST='10.0.0.52', MASTER_PORT=3308, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='123';"
mysql -S /data/3310/mysql.sock -e "start slave;"
mysql -S /data/3310/mysql.sock -e "show slave status\G"
10.0.0.51:3310 -----> 10.0.0.51:3308
db01
mysql -S /data/3310/mysql.sock -e "CHANGE MASTER TO MASTER_HOST='10.0.0.51', MASTER_PORT=3308, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='123';"
mysql -S /data/3310/mysql.sock -e "start slave;"
mysql -S /data/3310/mysql.sock -e "show slave status\G"
2.9 检测主从状态
mysql -S /data/3307/mysql.sock -e "show slave status\G"|grep Yes
mysql -S /data/3308/mysql.sock -e "show slave status\G"|grep Yes
mysql -S /data/3309/mysql.sock -e "show slave status\G"|grep Yes
mysql -S /data/3310/mysql.sock -e "show slave status\G"|grep Yes
注：如果中间出现错误，在每个节点进行执行以下命令
mysql -S /data/3307/mysql.sock -e "stop slave; reset slave all;"
mysql -S /data/3308/mysql.sock -e "stop slave; reset slave all;"
mysql -S /data/3309/mysql.sock -e "stop slave; reset slave all;"
mysql -S /data/3310/mysql.sock -e "stop slave; reset slave all;"

数据库11总结2

你可能感兴趣的:(数据库11总结2)