canal（mysql数据库 binlog的增量订阅，消费组件）
环境准备
1、jdk 环境变量配置
vim /etc/profile
export JAVA_HOME=/root/data/jdk/jdk
export PATH=$JAVA_HOME/bin:$PATH
source /etc/profile
2、jdk配置校验
root@never-PAF4:~# date
2018年 03月 11日 星期日 00:00:30 CST
root@never-PAF4:~# java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
root@never-PAF4:~#
3、canal
阿里巴巴mysql数据库 binlog的增量订阅，消费组件
- 基于日志增量订阅和消费支持的业务
- 数据库镜像
- 数据库实时备份
- 多级索引
- search build
- 业务cache刷新
- 价格变化等重要业务消息
- 定位：基于数据库增量日志解析，提供增量数据订阅 消费
工作原理
- canal 模拟mysql slave的交互协议，伪装自己为myql slave ，向mysql master 发送 dump协议
- mysql master收到 dump请求，开始推送binary log 给slave （canal）
- canal 解析 binary log 对象（原始为byte流）
下载
wget https://github.com/alibaba/canal/releases/download/canal-1.0.26-preview-1/canal.deployer-1.0.26-SNAPSHOT.tar.gz
解压：tar -xvf *.tar.gz
配置修改(使用默认配置需要对参数进行本地化，创建对应的测试库)
root@never-PAF4:~/data/canal/conf/example# pwd
/root/data/canal/conf/example
root@never-PAF4:~/data/canal/conf/example# vim instance.properties
root@never-PAF4:~/data/canal/conf/example# cat instance.properties
#################################################
## mysql serverId
canal.instance.mysql.slaveId=10
# position info
canal.instance.master.address=127.0.0.1:3302
canal.instance.master.journal.name=
canal.instance.master.position=
canal.instance.master.timestamp=
# table meta tsdb info
canal.instance.tsdb.enable=true
canal.instance.tsdb.dir=${canal.file.data.dir:../conf}/${canal.instance.destination:}
canal.instance.tsdb.url=jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3302/canal_tsdb
canal.instance.tsdb.dbUsername=root
canal.instance.tsdb.dbPassword=root
#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
# username/password
canal.instance.dbUsername=root
canal.instance.dbPassword=root
canal.instance.defaultDatabaseName=test
canal.instance.connectionCharset=UTF-8
# table regex
canal.instance.filter.regex=.*\\..*
# table black regex
canal.instance.filter.black.regex=
#################################################
root@never-PAF4:~/data/canal/conf/example#
docker 启动mysql（将本地的配置文件挂载到容器中..）
root@never-PAF4:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
mysql latest 5d4d51c57ea8 11 days ago 374MB
root@never-PAF4:~# docker run -p 3306:3306 --name slave -e MYSQL_ROOT_PASSWORD=canal -d mysql
47292b980e11e2e4e6bfdee7722474303adc8b6a236a427713d1088550682be8
root@never-PAF4:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
47292b980e11 mysql "docker-entrypoint.s…" 5 seconds ago Up 4 seconds 0.0.0.0:3306->3306/tcp slave进入镜像，修改配置文件，发现无法使用宿主机器的命令，选择构建新的镜像？还是将宿主机文件挂在容器中
root@never-PAF4:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
efb8f779ff41 mysql "docker-entrypoint.s…" 30 minutes ago Up 30 minutes 0.0.0.0:3305->3305/tcp, 3306/tcp master
47292b980e11 mysql "docker-entrypoint.s…" 31 minutes ago Up 31 minutes 0.0.0.0:3306->3306/tcp slave
root@never-PAF4:~# docker exec -it master /bin/bash
root@never-PAF4:~/data/mysql/master# docker run -p 3302:3306 --name ms4 -v ~/data/mysql/master/my.conf:/etc/mysql/my.cnf -e MYSQL_ROOT_PASSWORD=root -d mysql
my.cnf
root@never-PAF4:~/data/mysql/master# cat my.conf
[mysqld]
log-bin=mysql-bin
binlog-format=ROW
server_id=1
bind-address=0.0.0.0
root@never-PAF4:~/data/mysql/master#
监听的mysql服务器创建校验
root@never-PAF4:~/data/mysql/master# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a7252065ea95 mysql "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 0.0.0.0:3302->3306/tcp ms4
47292b980e11 mysql "docker-entrypoint.s…" About an hour ago Up About an hour 0.0.0.0:3306->3306/tcp slave
root@never-PAF4:~/data/mysql/master# docker exec -it ms4 mysql -uroot -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.21-log MySQL Community Server (GPL)
Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> create user canal identified by 'canal';
Query OK, 0 rows affected (0.06 sec)
mysql> grant select,replication slave,replication client on *.* to 'canal'@'%';
Query OK, 0 rows affected (0.05 sec)
mysql> flush privileges
->
-> ;
Query OK, 0 rows affected (0.05 sec)
mysql>
mysql>
结果，binlog 发生变化，也就是所监听的数据库发生变化 canal 客户端就会监听到
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=96m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: UseCMSCompactAtFullCollection is deprecated and will likely be removed in a future release.
2018-03-11 12:04:46.340 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2018-03-11 12:04:46.384 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
2018-03-11 12:04:46.385 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.
2018-03-11 12:04:46.423 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[172.17.0.1:11111]
2018-03-11 12:04:46.838 [main] WARN o.s.beans.GenericTypeAwarePropertyDescriptor - Invalid JavaBean property 'connectionCharset' being accessed! Ambiguous write methods found next to actually used [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.nio.charset.Charset)]: [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.lang.String)]
2018-03-11 12:04:47.040 [main] ERROR com.alibaba.druid.pool.DruidDataSource - testWhileIdle is true, validationQuery not set
2018-03-11 12:04:47.321 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......
2018-03-11 12:04:48.222 [destination = example , address = /127.0.0.1:3302 , EventParser] WARN c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - prepare to find start position just show master status
2018-03-11 12:53:51.976 [destination = example , address = /127.0.0.1:3302 , EventParser] WARN c.a.o.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta - dup apply for sql : CREATE TABLE `a` (
`a` varchar(255) NULL
)