canal（mysql数据库 binlog的增量订阅，消费组件）

环境准备

1、jdk 环境变量配置

vim /etc/profile


export JAVA_HOME=/root/data/jdk/jdk
export PATH=$JAVA_HOME/bin:$PATH

source /etc/profile

2、jdk配置校验

root@never-PAF4:~# date
2018年 03月 11日 星期日 00:00:30 CST
root@never-PAF4:~# java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
root@never-PAF4:~# 

3、canal

阿里巴巴mysql数据库 binlog的增量订阅，消费组件

1. 基于日志增量订阅和消费支持的业务
2. 数据库镜像
3. 数据库实时备份
4. 多级索引
5. search build
6. 业务cache刷新
7. 价格变化等重要业务消息
8. 定位：基于数据库增量日志解析，提供增量数据订阅 消费

工作原理

1. canal 模拟mysql slave的交互协议，伪装自己为myql slave ，向mysql master 发送 dump协议
2. mysql master收到 dump请求，开始推送binary log 给slave （canal）
3. canal 解析 binary log 对象（原始为byte流）

下载

wget https://github.com/alibaba/canal/releases/download/canal-1.0.26-preview-1/canal.deployer-1.0.26-SNAPSHOT.tar.gz

解压：tar -xvf *.tar.gz

配置修改(使用默认配置需要对参数进行本地化，创建对应的测试库)

root@never-PAF4:~/data/canal/conf/example# pwd
/root/data/canal/conf/example
root@never-PAF4:~/data/canal/conf/example# vim instance.properties 


root@never-PAF4:~/data/canal/conf/example# cat instance.properties 
#################################################
## mysql serverId
canal.instance.mysql.slaveId=10
# position info
canal.instance.master.address=127.0.0.1:3302
canal.instance.master.journal.name=
canal.instance.master.position=
canal.instance.master.timestamp=


# table meta tsdb info
canal.instance.tsdb.enable=true
canal.instance.tsdb.dir=${canal.file.data.dir:../conf}/${canal.instance.destination:}
canal.instance.tsdb.url=jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3302/canal_tsdb
canal.instance.tsdb.dbUsername=root
canal.instance.tsdb.dbPassword=root


#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position = 
#canal.instance.standby.timestamp = 
# username/password
canal.instance.dbUsername=root
canal.instance.dbPassword=root
canal.instance.defaultDatabaseName=test
canal.instance.connectionCharset=UTF-8
# table regex
canal.instance.filter.regex=.*\\..*
# table black regex
canal.instance.filter.black.regex=
#################################################
root@never-PAF4:~/data/canal/conf/example# 

docker 启动mysql（将本地的配置文件挂载到容器中..）

root@never-PAF4:~# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
mysql               latest              5d4d51c57ea8        11 days ago         374MB
root@never-PAF4:~# docker run -p 3306:3306 --name slave -e MYSQL_ROOT_PASSWORD=canal -d mysql
47292b980e11e2e4e6bfdee7722474303adc8b6a236a427713d1088550682be8
root@never-PAF4:~# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
47292b980e11        mysql               "docker-entrypoint.s…"   5 seconds ago       Up 4 seconds        0.0.0.0:3306->3306/tcp   slave

进入镜像，修改配置文件，发现无法使用宿主机器的命令，选择构建新的镜像？还是将宿主机文件挂在容器中

root@never-PAF4:~# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                              NAMES
efb8f779ff41        mysql               "docker-entrypoint.s…"   30 minutes ago      Up 30 minutes       0.0.0.0:3305->3305/tcp, 3306/tcp   master
47292b980e11        mysql               "docker-entrypoint.s…"   31 minutes ago      Up 31 minutes       0.0.0.0:3306->3306/tcp             slave
root@never-PAF4:~# docker exec -it master /bin/bash 

root@never-PAF4:~/data/mysql/master# docker run -p 3302:3306 --name ms4 -v ~/data/mysql/master/my.conf:/etc/mysql/my.cnf -e MYSQL_ROOT_PASSWORD=root -d mysql


my.cnf
root@never-PAF4:~/data/mysql/master# cat my.conf 
[mysqld]
log-bin=mysql-bin
binlog-format=ROW
server_id=1
bind-address=0.0.0.0
root@never-PAF4:~/data/mysql/master# 

监听的mysql服务器创建校验

root@never-PAF4:~/data/mysql/master# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
a7252065ea95        mysql               "docker-entrypoint.s…"   2 minutes ago       Up 2 minutes        0.0.0.0:3302->3306/tcp   ms4
47292b980e11        mysql               "docker-entrypoint.s…"   About an hour ago   Up About an hour    0.0.0.0:3306->3306/tcp   slave
root@never-PAF4:~/data/mysql/master# docker exec -it ms4 mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.21-log MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create user canal identified by 'canal';
Query OK, 0 rows affected (0.06 sec)

mysql> grant select,replication slave,replication client on *.* to 'canal'@'%';
Query OK, 0 rows affected (0.05 sec)

mysql> flush privileges
    -> 
    -> ;
Query OK, 0 rows affected (0.05 sec)

mysql> 

mysql> 

结果，binlog 发生变化，也就是所监听的数据库发生变化 canal 客户端就会监听到

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=96m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: UseCMSCompactAtFullCollection is deprecated and will likely be removed in a future release.
2018-03-11 12:04:46.340 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2018-03-11 12:04:46.384 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
2018-03-11 12:04:46.385 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.
2018-03-11 12:04:46.423 [main] INFO  com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[172.17.0.1:11111]
2018-03-11 12:04:46.838 [main] WARN  o.s.beans.GenericTypeAwarePropertyDescriptor - Invalid JavaBean property 'connectionCharset' being accessed! Ambiguous write methods found next to actually used [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.nio.charset.Charset)]: [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.lang.String)]
2018-03-11 12:04:47.040 [main] ERROR com.alibaba.druid.pool.DruidDataSource - testWhileIdle is true, validationQuery not set
2018-03-11 12:04:47.321 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......
2018-03-11 12:04:48.222 [destination = example , address = /127.0.0.1:3302 , EventParser] WARN  c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - prepare to find start position just show master status
2018-03-11 12:53:51.976 [destination = example , address = /127.0.0.1:3302 , EventParser] WARN  c.a.o.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta - dup apply for sql : CREATE TABLE `a` (
`a`  varchar(255) NULL 
)