MySql + Canal实时同步（kafka）

官网文档：https://github.com/alibaba/canal/wiki/Canal-Kafka-RocketMQ-QuickStart

一、下载安装kafka

kafka下载地址：http://kafka.apache.org/downloads

image.png

kafka启动需要zookeeper，无需额外下载，下载包自带有

1.zookeeper配置与启动

vim config/zookeeper.properties

dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
# Disable the adminserver by default to avoid port conflicts.
# Set the port to something non-conflicting if choosing to enable this
admin.enableServer=false

启动命令：./bin/zookeeper-server-start.sh config/zookeeper.properties

2.kafka服务配置与启动

vim config/server.properties

#将Broker的Listener信息发布到Zookeeper中
#192.168.1.1 当前kafka的IP (用户发现kafka 服务)
advertised.listeners=PLAINTEXT://192.168.1.1:9092
zookeeper.connect=localhost:2181

启动命令：./bin/kafka-server-start.sh config/server.properties

二、MySql 配置与添加账号

1.添加 Canal 所需 MySQL 配置

vim /etc/my.cnf

[mysqld] #在此处下方新添配置
log-bin=mysql-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定义，不要和 canal 的 slaveId 重复

2.允许 root 远程连接并添加 Canal 所需的用户

use mysql;
UPDATE user SET Host = '%' WHERE user = 'root';
CREATE user canal IDENTIFIED BY 'Acanal123456!';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%';
FLUSH PRIVILEGES;
EXIT

三、下载安装Canal

下载地址：https://github.com/alibaba/canal/releases

image.png

修改配置1：/conf/canal.properties

canal.serverMode = kafka
# 这个对应的是 conf目录下的example
canal.destinations = example
#kafka地址
kafka.bootstrap.servers = 127.0.0.1:9092

修改配置2：/conf/example/instance.properties

#数据库地址
canal.instance.master.address=192.168.1.1:3306
#mysql账号密码
canal.instance.dbUsername=canal
canal.instance.dbPassword=Acanal123456!
#kafka的topic
canal.mq.topic=topic1

启动 sh bin/startup.sh
关闭 sh bin/stop.sh
日志：
vi logs/canal/canal.log
vi logs/example/example.log