canal(mysql数据库 binlog的增量订阅,消费组件)

环境准备

1、jdk 环境变量配置

vim /etc/profile


export JAVA_HOME=/root/data/jdk/jdk
export PATH=$JAVA_HOME/bin:$PATH

source /etc/profile

2、jdk配置校验

root@never-PAF4:~# date
20180311日 星期日 00:00:30 CST
root@never-PAF4:~# java -version
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
root@never-PAF4:~# 

3、canal

阿里巴巴mysql数据库 binlog的增量订阅,消费组件

  1. 基于日志增量订阅和消费支持的业务
  2. 数据库镜像
  3. 数据库实时备份
  4. 多级索引
  5. search build
  6. 业务cache刷新
  7. 价格变化等重要业务消息
  8. 定位:基于数据库增量日志解析,提供增量数据订阅 消费

工作原理

  1. canal 模拟mysql slave的交互协议,伪装自己为myql slave ,向mysql master 发送 dump协议
  2. mysql master收到 dump请求,开始推送binary log 给slave (canal)
  3. canal 解析 binary log 对象(原始为byte流)

下载

wget https://github.com/alibaba/canal/releases/download/canal-1.0.26-preview-1/canal.deployer-1.0.26-SNAPSHOT.tar.gz

解压:tar -xvf *.tar.gz

配置修改(使用默认配置需要对参数进行本地化,创建对应的测试库)

root@never-PAF4:~/data/canal/conf/example# pwd
/root/data/canal/conf/example
root@never-PAF4:~/data/canal/conf/example# vim instance.properties 


root@never-PAF4:~/data/canal/conf/example# cat instance.properties 
#################################################
## mysql serverId
canal.instance.mysql.slaveId=10
# position info
canal.instance.master.address=127.0.0.1:3302
canal.instance.master.journal.name=
canal.instance.master.position=
canal.instance.master.timestamp=


# table meta tsdb info
canal.instance.tsdb.enable=true
canal.instance.tsdb.dir=${canal.file.data.dir:../conf}/${canal.instance.destination:}
canal.instance.tsdb.url=jdbc:h2:${canal.instance.tsdb.dir}/h2;CACHE_SIZE=1000;MODE=MYSQL;
#canal.instance.tsdb.url=jdbc:mysql://127.0.0.1:3302/canal_tsdb
canal.instance.tsdb.dbUsername=root
canal.instance.tsdb.dbPassword=root


#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position = 
#canal.instance.standby.timestamp = 
# username/password
canal.instance.dbUsername=root
canal.instance.dbPassword=root
canal.instance.defaultDatabaseName=test
canal.instance.connectionCharset=UTF-8
# table regex
canal.instance.filter.regex=.*\\..*
# table black regex
canal.instance.filter.black.regex=
#################################################
root@never-PAF4:~/data/canal/conf/example# 

docker 启动mysql(将本地的配置文件挂载到容器中..)

root@never-PAF4:~# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
mysql               latest              5d4d51c57ea8        11 days ago         374MB
root@never-PAF4:~# docker run -p 3306:3306 --name slave -e MYSQL_ROOT_PASSWORD=canal -d mysql
47292b980e11e2e4e6bfdee7722474303adc8b6a236a427713d1088550682be8
root@never-PAF4:~# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
47292b980e11        mysql               "docker-entrypoint.s…"   5 seconds ago       Up 4 seconds        0.0.0.0:3306->3306/tcp   slave

进入镜像,修改配置文件,发现无法使用宿主机器的命令,选择构建新的镜像?还是将宿主机文件挂在容器中

root@never-PAF4:~# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                              NAMES
efb8f779ff41        mysql               "docker-entrypoint.s…"   30 minutes ago      Up 30 minutes       0.0.0.0:3305->3305/tcp, 3306/tcp   master
47292b980e11        mysql               "docker-entrypoint.s…"   31 minutes ago      Up 31 minutes       0.0.0.0:3306->3306/tcp             slave
root@never-PAF4:~# docker exec -it master /bin/bash 

root@never-PAF4:~/data/mysql/master# docker run -p 3302:3306 --name ms4 -v ~/data/mysql/master/my.conf:/etc/mysql/my.cnf -e MYSQL_ROOT_PASSWORD=root -d mysql


my.cnf
root@never-PAF4:~/data/mysql/master# cat my.conf 
[mysqld]
log-bin=mysql-bin
binlog-format=ROW
server_id=1
bind-address=0.0.0.0
root@never-PAF4:~/data/mysql/master# 

监听的mysql服务器创建校验

root@never-PAF4:~/data/mysql/master# docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
a7252065ea95        mysql               "docker-entrypoint.s…"   2 minutes ago       Up 2 minutes        0.0.0.0:3302->3306/tcp   ms4
47292b980e11        mysql               "docker-entrypoint.s…"   About an hour ago   Up About an hour    0.0.0.0:3306->3306/tcp   slave
root@never-PAF4:~/data/mysql/master# docker exec -it ms4 mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.7.21-log MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> create user canal identified by 'canal';
Query OK, 0 rows affected (0.06 sec)

mysql> grant select,replication slave,replication client on *.* to 'canal'@'%';
Query OK, 0 rows affected (0.05 sec)

mysql> flush privileges
    -> 
    -> ;
Query OK, 0 rows affected (0.05 sec)

mysql> 

mysql> 

结果,binlog 发生变化,也就是所监听的数据库发生变化 canal 客户端就会监听到

Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=96m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: UseCMSCompactAtFullCollection is deprecated and will likely be removed in a future release.
2018-03-11 12:04:46.340 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler
2018-03-11 12:04:46.384 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations
2018-03-11 12:04:46.385 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## start the canal server.
2018-03-11 12:04:46.423 [main] INFO  com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[172.17.0.1:11111]
2018-03-11 12:04:46.838 [main] WARN  o.s.beans.GenericTypeAwarePropertyDescriptor - Invalid JavaBean property 'connectionCharset' being accessed! Ambiguous write methods found next to actually used [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.nio.charset.Charset)]: [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.lang.String)]
2018-03-11 12:04:47.040 [main] ERROR com.alibaba.druid.pool.DruidDataSource - testWhileIdle is true, validationQuery not set
2018-03-11 12:04:47.321 [main] INFO  com.alibaba.otter.canal.deployer.CanalLauncher - ## the canal server is running now ......
2018-03-11 12:04:48.222 [destination = example , address = /127.0.0.1:3302 , EventParser] WARN  c.a.otter.canal.parse.inbound.mysql.MysqlEventParser - prepare to find start position just show master status
2018-03-11 12:53:51.976 [destination = example , address = /127.0.0.1:3302 , EventParser] WARN  c.a.o.canal.parse.inbound.mysql.tsdb.DatabaseTableMeta - dup apply for sql : CREATE TABLE `a` (
`a`  varchar(255) NULL 
)

你可能感兴趣的:(mysql)