canal搭建

下载地址：https://github.com/alibaba/canal/releases

搭建准备：

1.数据库权限账户申请 

 CREATE USER canal IDENTIFIED BY 'canal'; 

        GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';

        -- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ;

        FLUSH PRIVILEGES;

2.数据库配置修改

开启binlog并选择row模式（binlog以行的形式储存容易解析）

在my.cnf 加入如下：

         [mysqld]

         log-bin=mysql-bin #添加这一行就ok

         binlog-format=ROW #选择row模式

         server_id=1 #配置mysql replaction需要定义，不能和canal的slaveId重复

3.配置配置文件

3.1 canal.properties

#################################################
######### 		common argument		############# 
#################################################
canal.id= 这里配置id 不要与数据库的id冲突
canal.ip= 这里配置部署canal的ip
canal.port= 11111
canal.zkServers=
# flush data to zk
canal.zookeeper.flush.period = 1000
# flush meta cursor/parse position to file
canal.file.data.dir = ${canal.conf.dir}
canal.file.flush.period = 1000
## memory store RingBuffer size, should be Math.pow(2,n)
canal.instance.memory.buffer.size = 16384
## memory store RingBuffer used memory unit size , default 1kb
canal.instance.memory.buffer.memunit = 1024 
## meory store gets mode used MEMSIZE or ITEMSIZE
canal.instance.memory.batch.mode = MEMSIZE

## detecing config
canal.instance.detecting.enable = false
#canal.instance.detecting.sql = insert into retl.xdual values(1,now()) on duplicate key update x=now()
canal.instance.detecting.sql = select 1
canal.instance.detecting.interval.time = 3
canal.instance.detecting.retry.threshold = 3
canal.instance.detecting.heartbeatHaEnable = false

# support maximum transaction size, more than the size of the transaction will be cut into multiple transactions delivery
canal.instance.transaction.size =  1024
# mysql fallback connected to new master should fallback times
canal.instance.fallbackIntervalInSeconds = 60

# network config
canal.instance.network.receiveBufferSize = 16384
canal.instance.network.sendBufferSize = 16384
canal.instance.network.soTimeout = 30

# binlog filter config
canal.instance.filter.query.dcl = false
canal.instance.filter.query.dml = false
canal.instance.filter.query.ddl = false
canal.instance.filter.table.error = false
canal.instance.filter.rows = false

# binlog format/image check
canal.instance.binlog.format = ROW,STATEMENT,MIXED 
canal.instance.binlog.image = FULL,MINIMAL,NOBLOB

# binlog ddl isolation
canal.instance.get.ddl.isolation = false

#################################################
######### 		destinations		############# 
#################################################
canal.destinations=这里配置要开启的实例，多个以逗号隔开
# conf root dir
canal.conf.dir = ../conf
# auto scan instance dir add/remove and start/stop instance
canal.auto.scan = true
canal.auto.scan.interval = 5

canal.instance.global.mode = spring 
canal.instance.global.lazy = false
#canal.instance.global.manager.address = 127.0.0.1:1099
#canal.instance.global.spring.xml = classpath:spring/memory-instance.xml
canal.instance.global.spring.xml = classpath:spring/file-instance.xml
#canal.instance.global.spring.xml = classpath:spring/default-instance.xml

3.2 实例配置文件instance.properties

#################################################
## mysql serverId
canal.instance.mysql.slaveId = 多个实例与数据库id 都不要重复

# position info
canal.instance.master.address = 数据库地址：3306
canal.instance.master.journal.name = 
canal.instance.master.position = 
canal.instance.master.timestamp = 

#canal.instance.standby.address = 
#canal.instance.standby.journal.name =
#canal.instance.standby.position = 
#canal.instance.standby.timestamp = 

# username/password
canal.instance.dbUsername = 申请的数据库权限账户
canal.instance.dbPassword = 密码
canal.instance.defaultDatabaseName =要监听的数据库
canal.instance.connectionCharset = UTF-8

# table regex
canal.instance.filter.regex = .*\\..*  要监听的表 不用正则的话直接 库名.表名 多个之间用逗号隔开
# table black regex
canal.instance.filter.black.regex =  

#################################################

4.配置多个实例

直接在conf文件下 增加对应实例的文件夹与配置文件即可

5.启动与关闭

bin目录下的 startup.sh  与stop.sh

6.解析

编写canal客户端进行binlog解析，这一块不是自己写的不沾代码了 ；

参考：https://blog.csdn.net/kaition/article/details/78801513

7.踩过的坑

7.1数据库权限问题

    一定要是上边说的数据库权限账户，搭建过程中由于都是向同事申请数据库，出现过各种问题。

    检查数据库账户权限 show grants for username@hostname; 或者直接用show master status 查看是否有权限

7.2数据库配置问题

  检查数据库是否开启binlog 与binlog模式 show variables like '%binlog_format%';

  为数据库的binlog更改模式 set global binlog_format='ROW' （最好是改配置文件重启数据库）

  还报过一个 server_id was not set 的错误，运维给的模板建的数据库 没有设置server_id

  mysql动态设置server_id     ：   SET GLOBAL server_id=1；

7.3canal报错

 canal KILL DUMP 275 failure: 在某次重启了canal后报的错。原因是找不到binlog节点了 ，删除对应实例的conf下的dat文件 重新启动就可以了

7.4还有一些踩过的忘记的坑 想起来后续再补上