开源数据库迁移工具canal
1. canal
1.1. 产品特性
canal是阿里巴巴旗下的一款开源项目，纯Java开发。基于数据库增量日志解析，提供增量数据订阅&消费，目前主要支持了MySQL（也支持mariaDB）
是一款基于Mysql或者Mariadb关系型数据库的全量/增量数据实时复制的工具，实现原理canal伪装自己为mysql的slave节点，监听mysql的binlog日志来获取数据，binlog设置为row模式以后，不仅能获取到执行的每一个增删改的脚本，同时还是获取到修改前和修改后的数据，基于这个特性，canal就能高性能的获取到mysql数据的变更
1.1.1. 知识科普
1.1.1.1. mysql Binlog介绍
- http://dev.mysql.com/doc/refman/5.5/en/binary-log.html
- http://www.taobaodba.com/html/474_mysqls-binary-log_details.html
1.1.1.2. mysql主备复制实现
从上层来看，复制分成三步：
（1）master将改变记录到二进制日志(binary log)中（这些记录叫做二进制日志事件，binary log events，可以通过show binlog events进行查看）；
（2）slave将master的binary log events拷贝到它的中继日志(relay log)；
（3）slave重做中继日志中的事件，将改变反映它自己的数据。
1.1.2. 工作原理
原理相对比较简单：
（1）canal模拟mysql slave的交互协议，伪装自己为mysql slave，向mysql master发送dump协议
（2）mysql master收到dump请求，开始推送binary log给slave(也就是canal)
（3）canal解析binary log对象(原始为byte流)
1.1.3. 部署使用
canal的部署主要分为server端和client端。
server端部署好以后，可以直接监听mysql binlog，因为server端是把自己模拟成了mysql slave，所以，只能接受数据，没有进行任何逻辑的处理，具体的逻辑处理，需要client端进行处理。
client端一般是需要大家进行简单的开发，下面有一个简单的示例，相对比较简单，可以试着去理解一下；
示例：https://github.com/alibaba/canal/wiki/ClientAPI
1.2. 支持场景
目前数据源主要支持Mysql（Mariadb），因为canal只作为服务端伪装为slave，接收并记录二进制binlog，其他数据转换可以依靠其他client工具完成，下面是canal官方编写的一个独立的组件Adapter，可以完成不同目标数据源的转换
1.2.1. 中间件数据源 - canal Adapter
为了便于大家的使用，官方做了一个独立的组件Adapter，Adapter是可以将canal server端获取的数据转换成几个常用的中间件数据源，现在支持kafka、rocketmq、hbase、elasticsearch，针对这几个中间件的支持，直接配置即可，无需开发。上文中，如果需要将mysql的数据同步到elasticsearch，直接运行 canal Adapter，修改相关的配置即可。
1.3. 迁移场景解决方案
1.3.1. 迁移测试场景
数据源与目标均为阿里云RDS数据库之间的数据迁移
1.3.1.1. 迁移程序安装环境
本次测试环境：CentOS7.5，2c4G，下载canal server端以及canal Adapter中间件数据源组件作为接收端，可以部署在同一台虚拟机
1.3.1.2. 待迁移源端数据库
- 数据库基础设施信息
| 数据库类型 | 数据源 | 数据库版本 | 资源大小 | 部署方式 |
|---|---|---|---|---|
| 关系型数据库 | 阿里云RDS | Mysql 5.6 | 2C-4G-50G | 单主库 |
- 数据库信息
| 数据库名 | 数据表名 | 表数据 | 备注 |
|---|---|---|---|
| coredb | seepcore_table | 220万 | 大小预估在2GB |
- 表结构信息
1.3.1.3. 待同步目标端数据库
- 数据库基础设施信息
| 数据库类型 | 数据源 | 数据库版本 | 资源大小 | 部署方式 |
|---|---|---|---|---|
| 关系型数据库 | 阿里云RDS | Mysql 5.6 | 2C-4G-50G | 单主库 |
- 数据库信息
| 数据库名 | 数据表名 | 表数据 | 备注 |
|---|---|---|---|
| coredb | seepcore_table | 0 | 新建立数据库，表为空，表结构与原表结构一致 |
- 表结构信息
1.3.1.4. 场景模拟
源端阿里云RDS数据库建立1个数据库coredb，下面产生1个表seepcore_table，表里面存放220万条数据
使用canal服务获取源数据库表信息，通过canal-Adapter组件来消费并全量增量数据到目标数据库（单表 – 单表）复制
| 场景 | 场景描述 |
|---|---|
| 场景一 | 启动canal服务，查看可以自动同步源端数据库表数据到目标数据库 |
| 场景二 | 在数据全量同步过程中，源端数据库seepcore_table表持续进行读、update、insert、delete操作，看是否可以完整进行数据同步并更新到目标数据库 |
| 场景三 | 在数据库增量同步过程中，源端数据库seepcore_table表持续进行读、update、insert、delete操作、看是否可以实进行数据同步并更新到目标数据库 |
1.3.2. 安装部署
1.3.2.1. 提前条件
对于自建 MySQL , 需要先开启 Binlog 写入功能，配置 binlog-format 为 ROW 模式，my.cnf 中配置如下
（自建数据库需开启，本次测试不涉及）
[mysqld]
log-bin=mysql-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定义，不要和 canal 的 slaveId 重复
- 授权 canal 链接 MySQL 账号具有作为 MySQL slave 的权限, 如果已有账户可直接 grant
CREATE USER canal IDENTIFIED BY 'canal';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
-- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;
FLUSH PRIVILEGES;
这个第一步还是蛮简单的，就是要自己搭建一个mysql，修改一下mysql的配置,这个配置一般是再/etc/my.cnf中
注意⚠️ ：如果数据源为阿里云RDS数据库实例，有关binlog的内容，需要注意以下几点，请查看官方链接：
https://github.com/alibaba/canal/wiki/aliyun-RDS-QuickStart
1.3.2.2. 安装JDK环境
canal deployer依赖Java >= 1.5，我们安装java 1.8.0版本，直接使用CentOS7.5默认yum源安装即可
yum install -y java-1.8.0
1.3.2.3. 下载canal deployer安装包
目前canal官方release最新版本为canal 1.1.4版本，我们就使用最新release版本进行测试
- canal deployer包下载 - canal 服务端
https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.deployer-1.1.4.tar.gz
- 解压缩canal deployer - canal 服务端
mkdir canal-deployer
tar xf canal.deployer-1.1.4.tar.gz -C canal-deployer/
1.3.2.4. 下载canal adapter安装包
- canal adapter包下载 - canal 客户端
https://github.com/alibaba/canal/releases/download/canal-1.1.4/canal.adapter-1.1.4.tar.gz
- 解压缩canal adapter - canal 客户端
mkdir canal-adapter
tar xf canal.adapter-1.1.4.tar.gz -C canal-adapter/
1.3.3. 迁移使用
1.3.3.1. 目录结构
.
|-- bin # canal程序起停服务脚本目录
| |-- restart.sh
| |-- startup.bat
| |-- startup.sh
| `-- stop.sh
|-- conf # canal配置文件存放目录，我们主要修改此目录下的配置文件
| |-- canal_local.properties
| |-- canal.properties
| |-- example
| | `-- instance.properties
| |-- logback.xml
| |-- metrics
| | `-- Canal_instances_tmpl.json
| `-- spring
| |-- base-instance.xml
| |-- default-instance.xml
| |-- file-instance.xml
| |-- group-instance.xml
| |-- memory-instance.xml
| `-- tsdb
| |-- h2-tsdb.xml
| |-- mysql-tsdb.xml
| |-- sql
| | `-- create_table.sql
| `-- sql-map
| |-- sqlmap-config.xml
| |-- sqlmap_history.xml
| `-- sqlmap_snapshot.xml
|-- lib # canal使用的java库文件存放目录
| |-- aopalliance-1.0.jar
| |-- aviator-2.2.1.jar
| |-- canal.common-1.1.4.jar
| |-- canal.deployer-1.1.4.jar
| |-- canal.filter-1.1.4.jar
| |-- canal.instance.core-1.1.4.jar
| |-- canal.instance.manager-1.1.4.jar
| |-- canal.instance.spring-1.1.4.jar
| |-- canal.meta-1.1.4.jar
| |-- canal.parse-1.1.4.jar
| |-- canal.parse.dbsync-1.1.4.jar
| |-- canal.parse.driver-1.1.4.jar
| |-- canal.prometheus-1.1.4.jar
| |-- canal.protocol-1.1.4.jar
| |-- canal.server-1.1.4.jar
| |-- canal.sink-1.1.4.jar
| |-- canal.store-1.1.4.jar
| |-- commons-beanutils-1.8.2.jar
| |-- commons-cli-1.2.jar
| |-- commons-codec-1.9.jar
| |-- commons-compress-1.9.jar
| |-- commons-io-2.4.jar
| |-- commons-lang-2.6.jar
| |-- commons-lang3-3.4.jar
| |-- commons-logging-1.1.3.jar
| |-- disruptor-3.4.2.jar
| |-- druid-1.1.9.jar
| |-- fastjson-1.2.58.jar
| |-- fastsql-2.0.0_preview_973.jar
| |-- guava-18.0.jar
| |-- h2-1.4.196.jar
| |-- httpclient-4.5.1.jar
| |-- httpcore-4.4.3.jar
| |-- ibatis-sqlmap-2.3.4.726.jar
| |-- jackson-annotations-2.9.0.jar
| |-- jackson-core-2.9.6.jar
| |-- jackson-databind-2.9.6.jar
| |-- javax.annotation-api-1.3.2.jar
| |-- jcl-over-slf4j-1.7.12.jar
| |-- jctools-core-2.1.2.jar
| |-- jopt-simple-5.0.4.jar
| |-- jsr305-3.0.2.jar
| |-- kafka_2.11-1.1.1.jar
| |-- kafka-clients-1.1.1.jar
| |-- logback-classic-1.1.3.jar
| |-- logback-core-1.1.3.jar
| |-- lz4-java-1.4.1.jar
| |-- metrics-core-2.2.0.jar
| |-- mysql-connector-java-5.1.47.jar
| |-- netty-3.2.2.Final.jar
| |-- netty-all-4.1.6.Final.jar
| |-- netty-tcnative-boringssl-static-1.1.33.Fork26.jar
| |-- oro-2.0.8.jar
| |-- protobuf-java-3.6.1.jar
| |-- rocketmq-acl-4.5.2.jar
| |-- rocketmq-client-4.5.2.jar
| |-- rocketmq-common-4.5.2.jar
| |-- rocketmq-logging-4.5.2.jar
| |-- rocketmq-remoting-4.5.2.jar
| |-- rocketmq-srvutil-4.5.2.jar
| |-- scala-library-2.11.12.jar
| |-- scala-logging_2.11-3.8.0.jar
| |-- scala-reflect-2.11.12.jar
| |-- simpleclient-0.4.0.jar
| |-- simpleclient_common-0.4.0.jar
| |-- simpleclient_hotspot-0.4.0.jar
| |-- simpleclient_httpserver-0.4.0.jar
| |-- simpleclient_pushgateway-0.4.0.jar
| |-- slf4j-api-1.7.12.jar
| |-- snakeyaml-1.19.jar
| |-- snappy-java-1.1.7.1.jar
| |-- spring-aop-3.2.18.RELEASE.jar
| |-- spring-beans-3.2.18.RELEASE.jar
| |-- spring-context-3.2.18.RELEASE.jar
| |-- spring-core-3.2.18.RELEASE.jar
| |-- spring-expression-3.2.18.RELEASE.jar
| |-- spring-jdbc-3.2.18.RELEASE.jar
| |-- spring-orm-3.2.18.RELEASE.jar
| |-- spring-tx-3.2.18.RELEASE.jar
| |-- zkclient-0.10.jar
| `-- zookeeper-3.4.5.jar
`-- logs # 程序启动日志输出都存在这个目录
1.3.3.2. 配置canal deployer服务端
修改canal deployer服务的配置文件，添加源端数据库连接信息 - 源端数据库
vim /root/canal-deployer/config/example/instance.properties
#################################################
## mysql serverId , v1.0.26+ will autoGen
# canal.instance.mysql.slaveId=0
# enable gtid use true/false
canal.instance.gtidon=false
# position info
canal.instance.master.address=rm-uf6xxpgxxhaxxo.mysql.rds.aliyuncs.com:3306 #填写源端数据库连接地址以及端口号，默认3306
canal.instance.master.journal.name=
canal.instance.master.position=
canal.instance.master.timestamp=
canal.instance.master.gtid=
# rds oss binlog
canal.instance.rds.accesskey=
canal.instance.rds.secretkey=
canal.instance.rds.instanceId=
# table meta tsdb info
canal.instance.tsdb.enable=true
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3306/canal_tsdb
#canal.instance.tsdb.dbUsername=canal
#canal.instance.tsdb.dbPassword=canal
#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#canal.instance.standby.gtid=
# username/password
canal.instance.dbUsername=xuxingzhuang # 填写源端数据库连接用户名
canal.instance.dbPassword=******** # 填写源端数据库连接用户密码
canal.instance.connectionCharset = UTF-8
# enable druid Decrypt database password
canal.instance.enableDruid=false
#canal.instance.pwdPublicKey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBALK4BUxdDltRRE5/zXpVEVPUgunvscYFtEip3pmLlhrWpacX7y7GCMo2/JM6LeHmiiNdH1FWgGCpUfircSwlWKUCAwEAAQ==
# table regex
canal.instance.filter.regex=.*\\..*
# table black regex
canal.instance.filter.black.regex=mysql..* #RDS Mysql限制用户访问mysql.slave_master_info表的权限，需要加入到黑名单才可以，其他自建数据库可以不配置
# table field filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.field=test1.t_product:id/subject/keywords,test2.t_company:id/name/contact/ch
# table field black filter(format: schema1.tableName1:field1/field2,schema2.tableName2:field1/field2)
#canal.instance.filter.black.field=test1.t_product:subject/product_image,test2.t_company:id/name/contact/ch
# mq config
canal.mq.topic=example
# dynamic topic route by schema or table regex
#canal.mq.dynamicTopic=mytest1.user,mytest2\\..*,.*\\..*
canal.mq.partition=0
# hash partition config
#canal.mq.partitionsNum=3
#canal.mq.partitionHash=test.table:id^name,.*\\..*
#################################################
1.3.3.3. 启动canal deployer服务
/root/canal-deployer/bin/startup.sh
- 查看启动日志，数据库连接是否正常
tail -f /root/canal-deployer/logs/example/example.log
- 日志输出一下内容则表示canal服务端已经连接源数据正常，等待binlog dump指令
2020-07-20 09:36:30.381 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2020-07-20 09:36:30.386 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
2020-07-20 09:36:30.557 [main] WARN o.s.beans.GenericTypeAwarePropertyDescriptor - Invalid JavaBean property 'connectionCharset' being accessed! Ambiguous write methods found next to actually used [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.nio.charset.Charset)]: [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.lang.String)]
2020-07-20 09:36:30.597 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2020-07-20 09:36:30.598 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
2020-07-20 09:36:30.955 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example
2020-07-20 09:36:30.962 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2020-07-20 09:36:30.962 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter : ^mysql..*$
2020-07-20 09:36:30.969 [main] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - start successful....
2020-07-20 09:36:34.307 [destination = example , address = rm-uf6b0pg70y7h9lha9qo.mysql.rds.aliyuncs.com/8.133.15.188:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2020-07-20 09:36:34.307 [destination = example , address = rm-uf6b0pg70y7h9lha9qo.mysql.rds.aliyuncs.com/8.133.15.188:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just show master status
2020-07-20 09:36:54.083 [destination = example , address = rm-uf6b0pg70y7h9lha9qo.mysql.rds.aliyuncs.com/8.133.15.188:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.000006,position=83537126,serverId=2614176453,gtid=,timestamp=1595237786000] cost : 19764ms , the next step is binlog dump
如出现权限问题请查看：RDS Mysql限制用户访问mysql.slave_master_info表权限
1.3.3.4. 配置canal Adapter 客户端服务 - 启动器
修改canal adapter客户端中间件启动器服务配置文件，添加源端以及目标端数据库连接信息
- application.yaml
vim /root/canal-adapter/conf/application.yml
server:
port: 8081
logging: # 添加服务启动级别为debug模式
level:
com.alibaba.otter.canal.client.adapter.hbase: DEBUG
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
default-property-inclusion: non_null
canal.conf:
mode: tcp # kafka rocketMQ
canalServerHost: 127.0.0.1:11111 #本地canal deployer服务端启动地址以及端口号，默认不需要修改
# zookeeperHosts: slave1:2181
# mqServers: 127.0.0.1:9092 #or rocketmq
# flatMessage: true
batchSize: 500
syncBatchSize: 1000
retries: 0
timeout:
accessKey:
secretKey:
srcDataSources:
defaultDS:
url: jdbc:mysql://8.133.15.188:3306/coredb?useUnicode=true # 源数据库连接地址以及端口号，地址建议使用IP，不要使用域名
username: xuxingzhuang # 源数据库连接用户
password: Abc999@1 # 源数据库连接密码
canalAdapters:
- instance: example # canal instance Name or mq topic name # 对应canal destination
groups: # 对应适配器分组, 分组间的适配器并行运行
- groupId: g1
outerAdapters: # 适配器列表, 分组内的适配串行运行
- name: logger # 适配器SPI名
- name: rdb # 指定为rdb类型同步
key: mysql1 # 指定adapter的唯一key, 与表映射配置中outerAdapterKey对应
properties: # HBase相关连接参数
jdbc.driverClassName: com.mysql.jdbc.Driver # 使用的jdbc库名称
jdbc.url: jdbc:mysql://172.19.134.199:3306/coredb?useUnicode=true # 目标数据库连接地址以及端口号，地址建议使用IP，不要使用域名
jdbc.username: xuxingzhuang # 目标数据库连接用户
jdbc.password: Abc999@1 # 目标数据库连接密码
threads: 5 # 并行执行的线程数, 默认为1
commitSize: 3000 # 批次提交的最大行数
# - name: rdb
# key: oracle1
# properties:
# jdbc.driverClassName: oracle.jdbc.OracleDriver
# jdbc.url: jdbc:oracle:thin:@localhost:49161:XE
# jdbc.username: mytest
# jdbc.password: m121212
# - name: rdb
# key: postgres1
# properties:
# jdbc.driverClassName: org.postgresql.Driver
# jdbc.url: jdbc:postgresql://localhost:5432/postgres
# jdbc.username: postgres
# jdbc.password: 121212
# threads: 2
# commitSize: 3000
# - name: hbase
# properties:
# hbase.zookeeper.quorum: 127.0.0.1
# hbase.zookeeper.property.clientPort: 2181
# zookeeper.znode.parent: /hbase
# - name: es
# hosts: 127.0.0.1:9300 # 127.0.0.1:9200 for rest mode
# properties:
# mode: transport # or rest
# # security.auth: test:123456 # only used for rest mode
# cluster.name: elasticsearch
注意⚠️ ：其中 outAdapter 的配置: name统一为rdb, key为对应的数据源的唯一标识需和下面的表映射文件中的outerAdapterKey对应, properties为目标库jdb的相关参数 adapter将会自动加载 conf/rdb 下的所有.yml结尾的表映射配置文件
canal adapter 中间件相关配置参数请参考：https://github.com/alibaba/canal/tree/master/client-adapter
1.3.3.5. 配置canal Adapter 客户端服务 - 适配器
修改canal adapter客户端中间件适配器服务配置文件，添加目标端数据库信息与2.3.3.4启动器配置相对应
- mytest_user.yml # 可以自定义配置文件，adapter会自动加载conf/rdb目录下的所有.yml结尾的表映射配置文件
vim /root/canal-adapter/conf/rdb/mytest_user.yml
dataSourceKey: defaultDS # 源数据源的key, 对应上面配置的srcDataSources中的值
destination: example # cannal的instance或者MQ的topic
groupId: g1
outerAdapterKey: mysql1 # adapter key, 对应上面配置outAdapters中的key
concurrent: true # 是否按主键hase并行同步, 并行同步的表必须保证主键不会更改及主键不能为其他同步表的外键!!
dbMapping:
database: coredb # 源数据源的database/shcema
table: seepcore_table # 源数据源表名
targetTable: coredb.seepcore_table # 目标数据源的库名.表名
targetPk: # 主键映射
id: id # 如果是复合主键可以换行映射多个
# mapAll: true # 是否整表映射, 要求源表和目标表字段名一模一样 (如果targetColumns也配置了映射,则以targetColumns配置为准)
# targetColumns: # 字段映射, 格式: 目标表字段: 源表字段, 如果字段名一样源表字段名可不填
# id:
# name:
# role_id:
# c_time:
# test1:
etlCondition: "where c_time>={}"
commitBatch: 3000 # 批量提交的大小
## Mirror schema synchronize config
#dataSourceKey: defaultDS
#destination: example
#groupId: g1
#outerAdapterKey: mysql1
#concurrent: true
#dbMapping:
# mirrorDb: true
# database: mytest
1.3.3.6. 启动RDB数据同步
启动canal-adapter启动器
/root/canal-adapter/bin/startup.sh
服务启动日志 - 启动成功
2020-07-20 10:27:13.078 [main] INFO o.s.c.annotation.AnnotationConfigApplicationContext - Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@3c3d9b6b: startup date [Mon Jul 20 10:27:13 UTC 2020]; root of context hierarchy
2020-07-20 10:27:13.391 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'configurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$fbd7a6d0] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-07-20 10:27:13.717 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - No active profile set, falling back to default profiles: default
2020-07-20 10:27:13.740 [main] INFO o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext - Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@6eb2384f: startup date [Mon Jul 20 10:27:13 UTC 2020]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@3c3d9b6b
2020-07-20 10:27:14.311 [main] INFO org.springframework.cloud.context.scope.GenericScope - BeanFactory id=ba9c0aec-0105-3f1f-b89e-e85c68567039
2020-07-20 10:27:14.373 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$fbd7a6d0] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2020-07-20 10:27:14.612 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8081 (http)
2020-07-20 10:27:14.627 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8081"]
2020-07-20 10:27:14.640 [main] INFO org.apache.catalina.core.StandardService - Starting service [Tomcat]
2020-07-20 10:27:14.641 [main] INFO org.apache.catalina.core.StandardEngine - Starting Servlet Engine: Apache Tomcat/8.5.29
2020-07-20 10:27:14.662 [localhost-startStop-1] INFO org.apache.catalina.core.AprLifecycleListener - The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
2020-07-20 10:27:14.783 [localhost-startStop-1] INFO o.a.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext
2020-07-20 10:27:14.784 [localhost-startStop-1] INFO org.springframework.web.context.ContextLoader - Root WebApplicationContext: initialization completed in 1044 ms
2020-07-20 10:27:14.913 [localhost-startStop-1] INFO o.s.boot.web.servlet.ServletRegistrationBean - Servlet dispatcherServlet mapped to [/]
2020-07-20 10:27:14.917 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'characterEncodingFilter' to: [/*]
2020-07-20 10:27:14.918 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'hiddenHttpMethodFilter' to: [/*]
2020-07-20 10:27:14.918 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'httpPutFormContentFilter' to: [/*]
2020-07-20 10:27:14.919 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'requestContextFilter' to: [/*]
2020-07-20 10:27:15.645 [main] INFO com.alibaba.druid.pool.DruidDataSource - {dataSource-1} inited
2020-07-20 10:27:15.820 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2020-07-20 10:27:16.182 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerAdapter - Looking for @ControllerAdvice: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@6eb2384f: startup date [Mon Jul 20 10:27:13 GMT+00:00 2020]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@3c3d9b6b
2020-07-20 10:27:16.276 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/count/{type}/{key}/{task}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String,java.lang.String)
2020-07-20 10:27:16.278 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/count/{type}/{task}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String)
2020-07-20 10:27:16.278 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/syncSwitch/{destination}/{status}],methods=[PUT]}" onto public com.alibaba.otter.canal.client.adapter.support.Result com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String)
2020-07-20 10:27:16.279 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/syncSwitch/{destination}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.String> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String)
2020-07-20 10:27:16.280 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/etl/{type}/{key}/{task}],methods=[POST]}" onto public com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.String,java.lang.String)
2020-07-20 10:27:16.280 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/etl/{type}/{task}],methods=[POST]}" onto public com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.String)
2020-07-20 10:27:16.281 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/destinations],methods=[GET]}" onto public java.util.List<java.util.Map<java.lang.String, java.lang.String>> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.destinations()
2020-07-20 10:27:16.285 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.http.HttpServletRequest)
2020-07-20 10:27:16.286 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse)
2020-07-20 10:27:16.330 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2020-07-20 10:27:16.330 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]
2020-07-20 10:27:16.575 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Registering beans for JMX exposure on startup
2020-07-20 10:27:16.584 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'refreshScope' has been autodetected for JMX exposure
2020-07-20 10:27:16.585 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'configurationPropertiesRebinder' has been autodetected for JMX exposure
2020-07-20 10:27:16.586 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'environmentManager' has been autodetected for JMX exposure
2020-07-20 10:27:16.589 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'environmentManager': registering with JMX server as MBean [org.springframework.cloud.context.environment:name=environmentManager,type=EnvironmentManager]
2020-07-20 10:27:16.608 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'refreshScope': registering with JMX server as MBean [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope]
2020-07-20 10:27:16.620 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=6eb2384f,type=ConfigurationPropertiesRebinder]
2020-07-20 10:27:16.634 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters.
2020-07-20 10:27:16.636 [main] INFO c.a.otter.canal.client.adapter.support.ExtensionLoader - extension classpath dir: /root/canal-adapter/plugin
2020-07-20 10:27:16.668 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: logger succeed
2020-07-20 10:27:16.671 [main] INFO c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Start loading rdb mapping config ...
2020-07-20 10:27:16.729 [main] INFO c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Rdb mapping config loaded
1.3.3.7. 发起全量数据同步（手动发起）
通过调用canal adapter的REST API接口进行全量数据同步
curl http://127.0.0.1:8081/etl/rdb/mysql1/mytest_user.yml -X POST # etl/rdb//.yml
2020-07-20 11:14:45.287 [http-nio-8081-exec-1] INFO c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - <E6><95><B0><E6><8D><AE><E5><85><A8><E9><87><8F><E5><AF><BC><E5><85><A5><E5><AE><8C><E6><88><90>, <E4><B8><80><E5><85><B1><E5><AF><BC><E5><85>
<A5> 0 <E6><9D><A1><E6><95><B0><E6><8D><AE>, <E8><80><97><E6><97><B6>: 312166
2020-07-20 11:14:45.329 [Thread-2] INFO org.springframework.boot.SpringApplication - No active profile set, falling back to default profiles: default
2020-07-20 11:14:45.331 [Thread-2] INFO o.s.c.annotation.AnnotationConfigApplicationContext - Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@7c463fa3: startup date [Mon Jul 20 11:14:45 GMT+00:00 2020]; parent: org.sp
ringframework.context.annotation.AnnotationConfigApplicationContext@131a9b8b
2020-07-20 11:14:45.354 [Thread-2] INFO org.springframework.boot.SpringApplication - Started application in 0.156 seconds (JVM running for 2856.752)
2020-07-20 11:14:45.355 [Thread-2] INFO o.s.c.annotation.AnnotationConfigApplicationContext - Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@7c463fa3: startup date [Mon Jul 20 11:14:45 GMT+00:00 2020]; parent: org.sprin
gframework.context.annotation.AnnotationConfigApplicationContext@131a9b8b
2020-07-20 11:14:45.355 [Thread-2] INFO o.s.c.annotation.AnnotationConfigApplicationContext - Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@131a9b8b: startup date [Mon Jul 20 11:14:45 GMT+00:00 2020]; root of context h
ierarchy
2020-07-20 11:14:45.597 [Thread-2] INFO com.alibaba.druid.pool.DruidDataSource - {dataSource-11} inited
2020-07-20 11:14:47.605 [Thread-2] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters.
2020-07-20 11:14:47.606 [Thread-2] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: logger succeed
2020-07-20 11:14:47.606 [Thread-2] INFO c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Start loading rdb mapping config ...
2020-07-20 11:14:47.620 [Thread-2] INFO c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Rdb mapping config loaded
2020-07-20 11:14:47.660 [Thread-2] INFO com.alibaba.druid.pool.DruidDataSource - {dataSource-12} inited
2020-07-20 11:14:47.661 [Thread-2] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: rdb succeed
2020-07-20 11:14:47.662 [Thread-2] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal instance: example succeed
2020-07-20 11:14:47.662 [Thread-2] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ......
2020-07-20 11:14:47.662 [Thread-21] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to connect destination: example <=============
2020-07-20 11:14:47.662 [Thread-2] INFO c.a.o.c.a.launcher.monitor.ApplicationConfigMonitor - ## adapter application config reloaded.
2020-07-20 11:14:47.665 [Thread-21] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to subscribe destination: example <=============
2020-07-20 11:14:47.705 [Thread-21] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Subscribe destination: example succeed <=============
2020-07-20 11:15:31.521 [http-nio-8081-exec-5] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - etl sql : SELECT * FROM coredb.seepcore_table
2020-07-20 11:15:37.095 [http-nio-8081-exec-5] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - workerCnt 212 for cnt 2120000 threadCount 4
2020-07-20 11:15:37.182 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:1
2020-07-20 11:15:37.198 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:2
2020-07-20 11:15:37.216 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:3
2020-07-20 11:15:37.232 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:4
2020-07-20 11:15:37.247 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:5
2020-07-20 11:15:37.262 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:6
2020-07-20 11:15:37.277 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:7
2020-07-20 11:15:37.291 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:8
2020-07-20 11:15:37.306 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:9
2020-07-20 11:15:37.321 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:10
2020-07-20 11:15:37.336 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:11
2020-07-20 11:15:37.352 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:12
2020-07-20 11:15:37.367 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:13
2020-07-20 11:15:37.382 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:14
2020-07-20 11:15:37.398 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:15
2020-07-20 11:15:37.413 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:16
2020-07-20 11:15:37.415 [pool-64-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:17
2020-07-20 11:15:37.428 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:18
2020-07-20 11:15:37.429 [pool-64-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:19
2020-07-20 11:15:37.443 [pool-64-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:20
2020-07-20 11:15:37.443 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:21
2020-07-20 11:15:37.457 [pool-64-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:22
2020-07-20 11:15:37.458 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:23
2020-07-20 11:15:37.471 [pool-64-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:24
2020-07-20 11:15:37.473 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:25
2020-07-20 11:15:37.484 [pool-64-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:26
2020-07-20 11:15:37.488 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:27
2020-07-20 11:15:37.498 [pool-64-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:28
2020-07-20 11:15:37.503 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:29
2020-07-20 11:15:37.512 [pool-64-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:30
2020-07-20 11:15:37.517 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:31
2020-07-20 11:15:37.525 [pool-64-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:32
2020-07-20 11:15:37.531 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:33
2020-07-20 11:15:37.539 [pool-64-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:34
2020-07-20 11:15:37.545 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:35
2020-07-20 11:15:37.553 [pool-64-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:36
2020-07-20 11:15:37.560 [pool-64-thread-1] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:37
2020-07-20 11:15:37.568 [pool-64-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:38
通过以上log输入可以看到已经在全量数据同步了
1.3.4. 场景模拟测试
1.3.4.1 场景一 —> 自动同步数据源到目标数据库
1）启动canal服务，查看可以自动同步源端数据库表数据到目标数据库
/root/canal-adapter/bin/startup.sh
- 模拟测试结果
全量数据获取到了2120044，但是全量数据import只插入了1760000条数据，具体原因还需要调查；
2020-07-20 18:03:16.143 [pool-7-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:1759995
2020-07-20 18:03:16.156 [pool-7-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:1759996
2020-07-20 18:03:16.169 [pool-7-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:1759997
2020-07-20 18:03:16.182 [pool-7-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:1759998
2020-07-20 18:03:16.194 [pool-7-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:1759999
2020-07-20 18:03:16.207 [pool-7-thread-2] DEBUG c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - successful import count:1760000
2020-07-20 18:03:16.241 [http-nio-8081-exec-2] INFO c.a.otter.canal.client.adapter.rdb.service.RdbEtlService - 数据全量导入完成, 一共导入 1760000 条数据, 耗时: 6665526
2020-07-20 18:03:16.246 [Thread-4] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to connect destination: example <=============
2020-07-20 18:03:16.263 [Thread-4] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to subscribe destination: example <=============
2020-07-20 18:03:16.304 [Thread-4] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Subscribe destination: example succeed <=============
1.3.4.2. 场景二 —> 全量同步过程中可以持续同步DML操作产生增量数据
在数据全量同步过程中，源端数据库seepcore_table表持续进行读、update、insert、delete操作，看是否可以完整进行数据同步并更新到目标数据库
1）源端数据库继续插入数据 -（源数据2120000）
通过批量数据插入脚本插入50条数据（当前数据：2120050）
python test-database.py -i rm-uf6b0pg70y7h9lha9qo.mysql.rds.aliyuncs.com -u xuxingzhuang -p Abc999@1 -t mysql -d coredb -c 50
A total of 2120050 data in Streetest_table table
Timer: 1.39316892624
- 模拟测试结果：
后台日志发现全量同步期间产生的所有插入新数据均已同步到目标数据库（需要等第一次全量同步任务结束后，才会将全量期间产生的操作全部更新）
2020-07-20 13:10:13.069 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120001,"date_time":"2020-07-20 11:38:32.843714","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245112000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612935,"type":"INSERT"}
2020-07-20 13:10:13.070 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120002,"date_time":"2020-07-20 11:38:32.857229","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245112000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612938,"type":"INSERT"}
2020-07-20 13:10:13.070 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120003,"date_time":"2020-07-20 11:38:32.852962","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245112000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612938,"type":"INSERT"}
2020-07-20 13:10:13.070 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120004,"date_time":"2020-07-20 11:38:32.864206","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245112000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612938,"type":"INSERT"}
2020-07-20 13:10:13.070 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120008,"date_time":"2020-07-20 11:38:32.869364","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245112000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612938,"type":"INSERT"}
2020-07-20 13:10:13.070 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120005,"date_time":"2020-07-20 11:38:32.854505","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245112000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612939,"type":"INSERT"}
2020-07-20 13:10:13.071 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120007,"date_time":"2020-07-20 11:38:32.863452","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245112000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612939,"type":"INSERT"}
2020-07-20 13:10:13.071 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120006,"date_time":"2020-07-20 11:38:32.862800","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245112000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612939,"type":"INSERT"}
2020-07-20 13:10:13.071 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120010,"date_time":"2020-07-20 11:38:32.854709","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245112000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612940,"type":"INSERT"}
2020-07-20 13:10:13.071 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120009,"date_time":"2020-07-20 11:38:32.865199","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245112000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612940,"type":"INSERT"}
2020-07-20 13:10:13.071 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120011,"date_time":"2020-07-20 11:38:32.862053","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245112000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612940,"type":"INSERT"}
2020-07-20 13:10:13.071 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120013,"date_time":"2020-07-20 11:38:32.870220","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245112000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612941,"type":"INSERT"}
2020-07-20 13:10:13.072 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120012,"date_time":"2020-07-20 11:38:32.861193","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245112000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612941,"type":"INSERT"}
2020-07-20 13:10:13.072 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120015,"date_time":"2020-07-20 11:38:32.856384","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245113000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612941,"type":"INSERT"}
2020-07-20 13:10:13.072 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120016,"date_time":"2020-07-20 11:38:32.871605","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245113000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612941,"type":"INSERT"}
2020-07-20 13:10:13.072 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120018,"date_time":"2020-07-20 11:38:32.880963","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245113000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612942,"type":"INSERT"}
2020-07-20 13:10:13.072 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120017,"date_time":"2020-07-20 11:38:32.877266","line_3":"0.857429472221","line_4":"0.190442092634","line_10":"0.391573329888","line_11":"0.0523556316908","line_12":"0.397537615338","line_13":"0.871978472905","line_14":"0.849247058151","line_15":"0.98314988525","line_16":"0.792365307466","line_17":"0.145223249989","line_18":"0.884171002156","line_19":"0.152193908175","line_20":"0.187707927696"}],"database":"coredb","destination":"example","es":1595245113000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612942,"type":"INSERT"}
2）源端数据库更新数据
通过DMS控制台，将源数据库表前10条数据line_11字段的值更新成666，然后提交
- 模拟测试结果
后台日志查看发现全量同步期间产生的所有更新数据均已经同步到目标数据库（需要等第一次全量同步任务结束后，才会将全量期间产生的操作全部更新**）**
2020-07-20 13:10:13.078 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":1,"date_time":"2020-07-20 05:29:51.491012","line_3":"0.105045469676","line_4":"0.196150758915","line_10":"0.166098156586","line_11":"666","line_12":"0.216416529335","line_13":"0.700279620682","line_14":"0.371665511121","line_15":"0.556910860997","line_16":"0.631409869478","line_17":"0.592778046917","line_18":"0.410662814747","line_19":"0.543671281044","line_20":"0.2446974235"}],"database":"coredb","destination":"example","es":1595245695000,"groupId":null,"isDdl":false,"old":[{"line_11":"0.773814645105"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612949,"type":"UPDATE"}
2020-07-20 13:10:13.078 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2,"date_time":"2020-07-20 05:29:51.571999","line_3":"0.105045469676","line_4":"0.196150758915","line_10":"0.166098156586","line_11":"666","line_12":"0.216416529335","line_13":"0.700279620682","line_14":"0.371665511121","line_15":"0.556910860997","line_16":"0.631409869478","line_17":"0.592778046917","line_18":"0.410662814747","line_19":"0.543671281044","line_20":"0.2446974235"}],"database":"coredb","destination":"example","es":1595245695000,"groupId":null,"isDdl":false,"old":[{"line_11":"0.773814645105"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612950,"type":"UPDATE"}
2020-07-20 13:10:13.078 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":3,"date_time":"2020-07-20 05:29:51.500105","line_3":"0.105045469676","line_4":"0.196150758915","line_10":"0.166098156586","line_11":"666","line_12":"0.216416529335","line_13":"0.700279620682","line_14":"0.371665511121","line_15":"0.556910860997","line_16":"0.631409869478","line_17":"0.592778046917","line_18":"0.410662814747","line_19":"0.543671281044","line_20":"0.2446974235"}],"database":"coredb","destination":"example","es":1595245696000,"groupId":null,"isDdl":false,"old":[{"line_11":"0.773814645105"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612950,"type":"UPDATE"}
2020-07-20 13:10:13.078 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":4,"date_time":"2020-07-20 05:29:51.501608","line_3":"0.105045469676","line_4":"0.196150758915","line_10":"0.166098156586","line_11":"666","line_12":"0.216416529335","line_13":"0.700279620682","line_14":"0.371665511121","line_15":"0.556910860997","line_16":"0.631409869478","line_17":"0.592778046917","line_18":"0.410662814747","line_19":"0.543671281044","line_20":"0.2446974235"}],"database":"coredb","destination":"example","es":1595245697000,"groupId":null,"isDdl":false,"old":[{"line_11":"0.773814645105"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612950,"type":"UPDATE"}
2020-07-20 13:10:13.079 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":5,"date_time":"2020-07-20 05:29:51.514419","line_3":"0.105045469676","line_4":"0.196150758915","line_10":"0.166098156586","line_11":"666","line_12":"0.216416529335","line_13":"0.700279620682","line_14":"0.371665511121","line_15":"0.556910860997","line_16":"0.631409869478","line_17":"0.592778046917","line_18":"0.410662814747","line_19":"0.543671281044","line_20":"0.2446974235"}],"database":"coredb","destination":"example","es":1595245697000,"groupId":null,"isDdl":false,"old":[{"line_11":"0.773814645105"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612951,"type":"UPDATE"}
2020-07-20 13:10:13.079 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":6,"date_time":"2020-07-20 05:29:51.496030","line_3":"0.105045469676","line_4":"0.196150758915","line_10":"0.166098156586","line_11":"666","line_12":"0.216416529335","line_13":"0.700279620682","line_14":"0.371665511121","line_15":"0.556910860997","line_16":"0.631409869478","line_17":"0.592778046917","line_18":"0.410662814747","line_19":"0.543671281044","line_20":"0.2446974235"}],"database":"coredb","destination":"example","es":1595245698000,"groupId":null,"isDdl":false,"old":[{"line_11":"0.773814645105"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612951,"type":"UPDATE"}
2020-07-20 13:10:13.079 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":7,"date_time":"2020-07-20 05:29:51.514790","line_3":"0.105045469676","line_4":"0.196150758915","line_10":"0.166098156586","line_11":"666","line_12":"0.216416529335","line_13":"0.700279620682","line_14":"0.371665511121","line_15":"0.556910860997","line_16":"0.631409869478","line_17":"0.592778046917","line_18":"0.410662814747","line_19":"0.543671281044","line_20":"0.2446974235"}],"database":"coredb","destination":"example","es":1595245699000,"groupId":null,"isDdl":false,"old":[{"line_11":"0.773814645105"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612951,"type":"UPDATE"}
2020-07-20 13:10:13.079 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":8,"date_time":"2020-07-20 05:29:51.513115","line_3":"0.105045469676","line_4":"0.196150758915","line_10":"0.166098156586","line_11":"666","line_12":"0.216416529335","line_13":"0.700279620682","line_14":"0.371665511121","line_15":"0.556910860997","line_16":"0.631409869478","line_17":"0.592778046917","line_18":"0.410662814747","line_19":"0.543671281044","line_20":"0.2446974235"}],"database":"coredb","destination":"example","es":1595245699000,"groupId":null,"isDdl":false,"old":[{"line_11":"0.773814645105"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612951,"type":"UPDATE"}
2020-07-20 13:10:13.079 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":9,"date_time":"2020-07-20 05:29:51.517106","line_3":"0.105045469676","line_4":"0.196150758915","line_10":"0.166098156586","line_11":"666","line_12":"0.216416529335","line_13":"0.700279620682","line_14":"0.371665511121","line_15":"0.556910860997","line_16":"0.631409869478","line_17":"0.592778046917","line_18":"0.410662814747","line_19":"0.543671281044","line_20":"0.2446974235"}],"database":"coredb","destination":"example","es":1595245700000,"groupId":null,"isDdl":false,"old":[{"line_11":"0.773814645105"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612952,"type":"UPDATE"}
2020-07-20 13:10:13.079 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":10,"date_time":"2020-07-20 05:29:51.503948","line_3":"0.105045469676","line_4":"0.196150758915","line_10":"0.166098156586","line_11":"666","line_12":"0.216416529335","line_13":"0.700279620682","line_14":"0.371665511121","line_15":"0.556910860997","line_16":"0.631409869478","line_17":"0.592778046917","line_18":"0.410662814747","line_19":"0.543671281044","line_20":"0.2446974235"}],"database":"coredb","destination":"example","es":1595245700000,"groupId":null,"isDdl":false,"old":[{"line_11":"0.773814645105"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612952,"type":"UPDATE"}
3）源端数据库删除数据
通过DMS控制台，将源数据库表id >= ‘10000’ and id <= ‘10005’ 6条数据进行删除（当前数据：2120044）
- 模拟测试结果
后台日志查看发现全量同步期间产生的所有删除数据均已经同步到目标数据库（需要等第一次全量同步任务结束后，才会将全量期间产生的操作全部更新**）**
2020-07-20 13:10:13.080 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":10000,"date_time":"2020-07-20 05:30:33.342526","line_3":"0.906797258488","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595246083000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612952,"type":"DELETE"}
2020-07-20 13:10:13.080 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":10001,"date_time":"2020-07-20 05:30:33.344901","line_3":"0.906797258488","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595246083000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612952,"type":"DELETE"}
2020-07-20 13:10:13.080 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":10002,"date_time":"2020-07-20 05:30:33.346536","line_3":"0.906797258488","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595246083000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612952,"type":"DELETE"}
2020-07-20 13:10:13.080 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":10003,"date_time":"2020-07-20 05:30:33.349987","line_3":"0.906797258488","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595246083000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612953,"type":"DELETE"}
2020-07-20 13:10:13.080 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":10004,"date_time":"2020-07-20 05:30:33.344492","line_3":"0.906797258488","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595246084000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612953,"type":"DELETE"}
2020-07-20 13:10:13.081 [pool-63-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":10005,"date_time":"2020-07-20 05:30:33.355474","line_3":"0.906797258488","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595246084000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595250612953,"type":"DELETE"}
1.3.4.3. 场景三 —> 实时同步DML操作增量数据
在数据库增量同步过程中，源端数据库seepcore_table表持续进行读、update、insert、delete操作、看是否可以实进行数据同步并更新到目标数据库
1）INSERT 插入表格10条
python test-database.py -i rm-uf6b0pg70y7h9lha9qo.mysql.rds.aliyuncs.com -u xuxingzhuang -p Abc999@1 -t mysql -d coredb -c 10
A total of 2120054 data in Streetest_table table
Timer: 1.32490301132
- 模拟测试结果
增量数据期间实时检测后台adapter服务，发现源端插入数据检测到之后并立刻更新到了目标数据库
tail -f /root/canal-adapter/logs/adapter/adapter.log
2020-07-21 02:00:13.457 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120051,"date_time":"2020-07-21 02:00:12.791486","line_3":"0.880327705382","line_4":"0.375869063853","line_10":"0.261958352865","line_11":"0.70745432179","line_12":"0.302732663155","line_13":"0.847648090347","line_14":"0.457910966175","line_15":"0.922193365002","line_16":"0.532943728403","line_17":"0.255739836367","line_18":"0.266689706937","line_19":"0.688414807242","line_20":"0.176631788009"}],"database":"coredb","destination":"example","es":1595296812000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595296813296,"type":"INSERT"}
2020-07-21 02:00:13.460 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120052,"date_time":"2020-07-21 02:00:12.805528","line_3":"0.880327705382","line_4":"0.375869063853","line_10":"0.261958352865","line_11":"0.70745432179","line_12":"0.302732663155","line_13":"0.847648090347","line_14":"0.457910966175","line_15":"0.922193365002","line_16":"0.532943728403","line_17":"0.255739836367","line_18":"0.266689706937","line_19":"0.688414807242","line_20":"0.176631788009"}],"database":"coredb","destination":"example","es":1595296812000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595296813302,"type":"INSERT"}
2020-07-21 02:00:13.460 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120053,"date_time":"2020-07-21 02:00:12.819156","line_3":"0.880327705382","line_4":"0.375869063853","line_10":"0.261958352865","line_11":"0.70745432179","line_12":"0.302732663155","line_13":"0.847648090347","line_14":"0.457910966175","line_15":"0.922193365002","line_16":"0.532943728403","line_17":"0.255739836367","line_18":"0.266689706937","line_19":"0.688414807242","line_20":"0.176631788009"}],"database":"coredb","destination":"example","es":1595296812000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595296813302,"type":"INSERT"}
2020-07-21 02:00:13.460 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120054,"date_time":"2020-07-21 02:00:12.816509","line_3":"0.880327705382","line_4":"0.375869063853","line_10":"0.261958352865","line_11":"0.70745432179","line_12":"0.302732663155","line_13":"0.847648090347","line_14":"0.457910966175","line_15":"0.922193365002","line_16":"0.532943728403","line_17":"0.255739836367","line_18":"0.266689706937","line_19":"0.688414807242","line_20":"0.176631788009"}],"database":"coredb","destination":"example","es":1595296812000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595296813303,"type":"INSERT"}
2020-07-21 02:00:13.460 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120055,"date_time":"2020-07-21 02:00:12.808008","line_3":"0.880327705382","line_4":"0.375869063853","line_10":"0.261958352865","line_11":"0.70745432179","line_12":"0.302732663155","line_13":"0.847648090347","line_14":"0.457910966175","line_15":"0.922193365002","line_16":"0.532943728403","line_17":"0.255739836367","line_18":"0.266689706937","line_19":"0.688414807242","line_20":"0.176631788009"}],"database":"coredb","destination":"example","es":1595296812000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595296813303,"type":"INSERT"}
2020-07-21 02:00:13.461 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120057,"date_time":"2020-07-21 02:00:12.789634","line_3":"0.880327705382","line_4":"0.375869063853","line_10":"0.261958352865","line_11":"0.70745432179","line_12":"0.302732663155","line_13":"0.847648090347","line_14":"0.457910966175","line_15":"0.922193365002","line_16":"0.532943728403","line_17":"0.255739836367","line_18":"0.266689706937","line_19":"0.688414807242","line_20":"0.176631788009"}],"database":"coredb","destination":"example","es":1595296812000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595296813303,"type":"INSERT"}
2020-07-21 02:00:13.461 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120056,"date_time":"2020-07-21 02:00:12.821414","line_3":"0.880327705382","line_4":"0.375869063853","line_10":"0.261958352865","line_11":"0.70745432179","line_12":"0.302732663155","line_13":"0.847648090347","line_14":"0.457910966175","line_15":"0.922193365002","line_16":"0.532943728403","line_17":"0.255739836367","line_18":"0.266689706937","line_19":"0.688414807242","line_20":"0.176631788009"}],"database":"coredb","destination":"example","es":1595296812000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595296813304,"type":"INSERT"}
2020-07-21 02:00:13.461 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120058,"date_time":"2020-07-21 02:00:12.820272","line_3":"0.880327705382","line_4":"0.375869063853","line_10":"0.261958352865","line_11":"0.70745432179","line_12":"0.302732663155","line_13":"0.847648090347","line_14":"0.457910966175","line_15":"0.922193365002","line_16":"0.532943728403","line_17":"0.255739836367","line_18":"0.266689706937","line_19":"0.688414807242","line_20":"0.176631788009"}],"database":"coredb","destination":"example","es":1595296812000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595296813304,"type":"INSERT"}
2020-07-21 02:00:13.461 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120059,"date_time":"2020-07-21 02:00:12.808557","line_3":"0.880327705382","line_4":"0.375869063853","line_10":"0.261958352865","line_11":"0.70745432179","line_12":"0.302732663155","line_13":"0.847648090347","line_14":"0.457910966175","line_15":"0.922193365002","line_16":"0.532943728403","line_17":"0.255739836367","line_18":"0.266689706937","line_19":"0.688414807242","line_20":"0.176631788009"}],"database":"coredb","destination":"example","es":1595296812000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595296813304,"type":"INSERT"}
2020-07-21 02:00:13.461 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":2120060,"date_time":"2020-07-21 02:00:12.806750","line_3":"0.880327705382","line_4":"0.375869063853","line_10":"0.261958352865","line_11":"0.70745432179","line_12":"0.302732663155","line_13":"0.847648090347","line_14":"0.457910966175","line_15":"0.922193365002","line_16":"0.532943728403","line_17":"0.255739836367","line_18":"0.266689706937","line_19":"0.688414807242","line_20":"0.176631788009"}],"database":"coredb","destination":"example","es":1595296812000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595296813304,"type":"INSERT"}
2）UPDATE 表格6条
在DMS控制台对源数据库coredb.seepcore_table更新ID>=‘5000’ and ID <= ‘50005’ 的5条数据，将line_3值更新成888
- 模拟测试结果
增量数据期间实时检测后台adapter服务，发现源端更新数据后，立刻接到消息并更新目标数据库
tail -f /root/canal-adapter/logs/adapter/adapter.log
2020-07-21 02:09:45.438 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":5000,"date_time":"2020-07-20 05:30:21.441670","line_3":"888","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595297385000,"groupId":null,"isDdl":false,"old":[{"line_3":"0.906797258488"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595297385435,"type":"UPDATE"}
2020-07-21 02:09:45.439 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":5001,"date_time":"2020-07-20 05:30:21.443971","line_3":"888","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595297385000,"groupId":null,"isDdl":false,"old":[{"line_3":"0.906797258488"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595297385437,"type":"UPDATE"}
2020-07-21 02:09:46.089 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":5002,"date_time":"2020-07-20 05:30:21.445624","line_3":"888","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595297385000,"groupId":null,"isDdl":false,"old":[{"line_3":"0.906797258488"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595297386088,"type":"UPDATE"}
2020-07-21 02:09:46.089 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":5003,"date_time":"2020-07-20 05:30:21.446878","line_3":"888","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595297385000,"groupId":null,"isDdl":false,"old":[{"line_3":"0.906797258488"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595297386088,"type":"UPDATE"}
2020-07-21 02:09:46.693 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":5004,"date_time":"2020-07-20 05:30:21.448033","line_3":"888","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595297386000,"groupId":null,"isDdl":false,"old":[{"line_3":"0.906797258488"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595297386692,"type":"UPDATE"}
2020-07-21 02:09:46.693 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":5005,"date_time":"2020-07-20 05:30:21.449714","line_3":"888","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595297386000,"groupId":null,"isDdl":false,"old":[{"line_3":"0.906797258488"}],"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595297386692,"type":"UPDATE"}
3）DELETE 操作6条
在DMS控制台对源数据库coredb.seepcore_table将ID>=‘100’ and ID <= ‘105’ 的6条数据进行删除操作
- 模拟测试结果
增量数据期间实时检测后台adapter服务，发现源端数据被删除后，立刻接到消息并更新目标数据库
2020-07-21 05:45:42.753 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":100,"date_time":"2020-07-20 05:29:51.514023","line_3":"0.105045469676","line_4":"0.196150758915","line_10":"0.166098156586","line_11":"0.773814645105","line_12":"0.216416529335","line_13":"0.700279620682","line_14":"0.371665511121","line_15":"0.556910860997","line_16":"0.631409869478","line_17":"0.592778046917","line_18":"0.410662814747","line_19":"0.543671281044","line_20":"0.2446974235"}],"database":"coredb","destination":"example","es":1595310342000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595310342750,"type":"DELETE"}
2020-07-21 05:45:42.756 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":101,"date_time":"2020-07-20 05:30:08.813811","line_3":"0.906797258488","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595310342000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595310342751,"type":"DELETE"}
2020-07-21 05:45:42.871 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":102,"date_time":"2020-07-20 05:30:08.896812","line_3":"0.906797258488","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595310342000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595310342871,"type":"DELETE"}
2020-07-21 05:45:43.471 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":103,"date_time":"2020-07-20 05:30:08.823369","line_3":"0.906797258488","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595310342000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595310343470,"type":"DELETE"}
2020-07-21 05:45:43.471 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":104,"date_time":"2020-07-20 05:30:08.829203","line_3":"0.906797258488","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595310343000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595310343470,"type":"DELETE"}
2020-07-21 05:45:43.471 [pool-6-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":105,"date_time":"2020-07-20 05:30:08.832929","line_3":"0.906797258488","line_4":"0.235910042687","line_10":"0.690050411974","line_11":"0.301257468709","line_12":"0.226111184348","line_13":"0.0238471144231","line_14":"0.171261640521","line_15":"0.288332965066","line_16":"0.771738776395","line_17":"0.660060472907","line_18":"0.908136300875","line_19":"0.0228070071957","line_20":"0.433860792991"}],"database":"coredb","destination":"example","es":1595310343000,"groupId":null,"isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"seepcore_table","ts":1595310343470,"type":"DELETE"}
1.3.4.4. 场景模拟测试结果
| 场景 | 场景描述 | 具体DML操作 | 测试结果 | 备注 |
|---|---|---|---|---|
| 场景一 | 启动canal服务，查看可以自动同步源端数据库表数据到目标数据库 | INSERT | 部分通过 | 需要手动通过REST API触发，具体为什么不能自动还需调研 |
| 场景二 | 在数据全量同步过程中，源端数据库seepcore_table表持续进行读、update、insert、delete操作，看是否可以完整进行数据同步并更新到目标数据库 | UPDATE | 通过 | 需要等第一次全量同步任务结束后，才会将全量期间产生的操作全部更新 |
| 场景三 | 在数据库增量同步过程中，源端数据库seepcore_table表持续进行读、update、insert、delete操作、看是否可以实进行数据同步并更新到目标数据库 | UPDATE | 通过 | 可以持续将增量数据实时同步到目标端 |
1.3.4.5. canal adapter REST API 汇总
- 通过etl进行全量、选择性以及策略性增量数据同步
curl http://127.0.0.1:8081/etl/rdb/mysql1/mytest_user.yml -X POST
# @param type 类型 hbase, es, rdb ---> rdb
# @param key adapter key # conf/rdb/mytest_user.yml对应的值 ---> mysql1
# @param task 任务名对应配置文件名 mytest_user.yml ---> mytest_user.yml
# @param params etl # where条件参数, 为空全部导入 ---> 空表示全量数据同步
-
canal条件同步(etl功能，手动触发)
如果想执行从某一个时刻的数据同步，在上面的测试URL后加上对应的参数就可以了，其参数由rbd的配置项决定的，也就是conf/rdb/mytest_user.yml配置文件中的etlCondition，比如我这里写的where id>={}，而{}之内的就是条件
dataSourceKey: defaultDS
destination: example
groupId: g1
outerAdapterKey: mysql1
concurrent: true
dbMapping:
database: coredb
table: seepcore_table
targetTable: coredb.seepcore_table
targetPk: # 主键映射 --> 在条件同步中，一定要定义，不然会失败
id: id # 如果是复合主键可以换行映射多个 --> 要开启主键
mapAll: true # 是否整表映射, 要求源表和目标表字段名一模一样 (如果targetColumns也配置了映射,则以targetColumns配置为准)，如果映射了那么表的所有字段都将会从源端复制到目标数据库
# targetColumns: # 如果开启了此选项，表示下面的字段写几个就会只同步几个字段的数据，其他多余的字段将会被重制为Null
# id: # 例如只开启了这一选项，那么只会同步源端id的值到目标数据库id字段中，其他字段均为Null
# date_time:
etlCondition: "where id>={}"
commitBatch: 3000 # 批量提交的大小
同步id>=300000的数据，可以执行下面的指令进行条件同步
curl http://127.0.0.1:8081/etl/rdb/mysql1/mytest_user.yml -X POST -d 'params=300000'
- 查询所有的实例
curl http://127.0.0.1:8081/destinations
- 实例同步开关
curl http://127.0.0.1:8081/syncSwitch/example/off -X PUT
*
* @param destination 实例名称 ---> example 可以从canal deployer配置conf/canal.properties中<canal.destinations>值
* @param status 开关状态: off on --> 全量或者增量同步时，此值自动重制为off，防止再有其他sync task进来，如果有请求会返回其他进程正在同步中
- **获取实例开关状态 **
curl http://127.0.0.1:8081/syncSwitch/example
*
* @param destination 实例名称 ---> example 可以从canal deployer配置conf/canal.properties中<canal.destinations>值
2.4. 迁移测试总结
2.4.1. canal优缺点
1）优点
- 可以支持全量以及实时增量数据同步
- 有良好的管理控制平台与监控系统（如果你已经有promethus监控，可以秒接canal监控以及开源的canal admin项目进行管理）
- 通过开源的canal adapter中间client可以将数据同步到多个数据源
2）缺点
- 单instance/订阅通道只支持订阅单个数据库，并只能支持单客户端消费。每当我们需要新增一个消费端->MySQL的订阅：对于Canal而言，就要给MySQL接一个“Slave”，可能会对主库有一定影响
- 消息的Schema很弱，所有消息的Schema均相同，客户端需要提前知道各个表消息的Schema与各字段的上下文才能正确消费
1.5. 测试问题处理
1.5.1. 启动canal deployer服务，连接数据库失败
- 问题分析
由于RDS MySQL限制了用户访问mysql.slave_master_info表的权限，所以有此报错
/root/canal-deployer/logs/example/example.log
###########################
2020-07-20 09:15:52.085 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2020-07-20 09:15:52.089 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
2020-07-20 09:15:52.269 [main] WARN o.s.beans.GenericTypeAwarePropertyDescriptor - Invalid JavaBean property 'connectionCharset' being accessed! Ambiguous write methods found next to actually used [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.nio.charset.Charset)]: [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.lang.String)]
2020-07-20 09:15:52.309 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [canal.properties]
2020-07-20 09:15:52.310 [main] INFO c.a.o.c.i.spring.support.PropertyPlaceholderConfigurer - Loading properties file from class path resource [example/instance.properties]
2020-07-20 09:15:52.854 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example
2020-07-20 09:15:52.863 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2020-07-20 09:15:52.863 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter :
2020-07-20 09:15:52.872 [main] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - start successful....
2020-07-20 09:15:53.272 [destination = example , address = rm-uf6b0pg70y7h9lha9qo.mysql.rds.aliyuncs.com/8.133.15.188:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2020-07-20 09:15:53.273 [destination = example , address = rm-uf6b0pg70y7h9lha9qo.mysql.rds.aliyuncs.com/8.133.15.188:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just show master status
2020-07-20 09:16:10.885 [destination = example , address = rm-uf6b0pg70y7h9lha9qo.mysql.rds.aliyuncs.com/8.133.15.188:3306 , EventParser] ERROR c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - dump address rm-uf6b0pg70y7h9lha9qo.mysql.rds.aliyuncs.com/8.133.15.188:3306 has an error, retrying. caused by
com.alibaba.otter.canal.parse.exception.CanalParseException: java.io.IOException: ErrorPacket [errorNumber=1142, fieldCount=-1, message=SHOW command denied to user 'xuxingzhuang'@'61.49.138.26' for table 'slave_master_info', sqlState=42000, sqlStateMarker=#]
with command: show create table `mysql`.`columns_priv`;show create table `mysql`.`db`;show create table `mysql`.`event`;show create table `mysql`.`failover_info`;show create table `mysql`.`func`;show create table `mysql`.`general_log`;show create table `mysql`.`ha_health_check`;show create table `mysql`.`help_category`;show create table `mysql`.`help_keyword`;show create table `mysql`.`help_relation`;show create table `mysql`.`help_topic`;show create table `mysql`.`innodb_index_stats`;show create table `mysql`.`innodb_table_stats`;show create table `mysql`.`ndb_binlog_index`;show create table `mysql`.`plugin`;show create table `mysql`.`proc`;show create table `mysql`.`procs_priv`;show create table `mysql`.`proxies_priv`;show create table `mysql`.`rds_extra`;show create table `mysql`.`servers`;show create table `mysql`.`slave_gtid_info`;show create table `mysql`.`slave_master_info`;show create table `mysql`.`slave_relay_log_info`;show create table `mysql`.`slave_worker_info`;show create table `mysql`.`slow_log`;show create table `mysql`.`tables_priv`;show create table `mysql`.`time_zone`;show create table `mysql`.`time_zone_leap_second`;show create table `mysql`.`time_zone_name`;show create table `mysql`.`time_zone_transition`;show create table `mysql`.`time_zone_transition_type`;show create table `mysql`.`user`;
Caused by: java.io.IOException: ErrorPacket [errorNumber=1142, fieldCount=-1, message=SHOW command denied to user 'xuxingzhuang'@'61.49.138.26' for table 'slave_master_info', sqlState=42000, sqlStateMarker=#]
with command: show create table `mysql`.`columns_priv`;show create table `mysql`.`db`;show create table `mysql`.`event`;show create table `mysql`.`failover_info`;show create table `mysql`.`func`;show create table `mysql`.`general_log`;show create table `mysql`.`ha_health_check`;show create table `mysql`.`help_category`;show create table `mysql`.`help_keyword`;show create table `mysql`.`help_relation`;show create table `mysql`.`help_topic`;show create table `mysql`.`innodb_index_stats`;show create table `mysql`.`innodb_table_stats`;show create table `mysql`.`ndb_binlog_index`;show create table `mysql`.`plugin`;show create table `mysql`.`proc`;show create table `mysql`.`procs_priv`;show create table `mysql`.`proxies_priv`;show create table `mysql`.`rds_extra`;show create table `mysql`.`servers`;show create table `mysql`.`slave_gtid_info`;show create table `mysql`.`slave_master_info`;show create table `mysql`.`slave_relay_log_info`;show create table `mysql`.`slave_worker_info`;show create table `mysql`.`slow_log`;show create table `mysql`.`tables_priv`;show create table `mysql`.`time_zone`;show create table `mysql`.`time_zone_leap_second`;show create table `mysql`.`time_zone_name`;show create table `mysql`.`time_zone_transition`;show create table `mysql`.`time_zone_transition_type`;show create table `mysql`.`user`;
at com.alibaba.otter.canal.parse.driver.mysql.MysqlQueryExecutor.queryMulti(MysqlQueryExecutor.java:109) ~[canal.parse.driver-1.1.4.jar:na]
at com.alibaba.otter.canal.parse.inbound.mysql.MysqlConnection.queryMulti(MysqlConnection.java:111) ~[canal.parse-1.1.4.jar:na]
at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta.dumpTableMeta(DatabaseTableMeta.java:232) ~[canal.parse-1.1.4.jar:na]
at com.alibaba.otter.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta.rollback(DatabaseTableMeta.java:182) ~[canal.parse-1.1.4.jar:na]
at com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.processTableMeta(AbstractMysqlEventParser.java:137) ~[canal.parse-1.1.4.jar:na]
at com.alibaba.otter.canal.parse.inbound.AbstractEventParser$3.run(AbstractEventParser.java:200) ~[canal.parse-1.1.4.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_252]
- 解决办法
将对应的库表添加进黑名单即可（instance.properties ）
文件位置：/root/canal-deployer/conf/example/instance.properties
canal.instance.filter.black.regex=mysql..*