目录
一、环境配置
二、myslq安装
三、ES安装
四、canal-admin安装
4.1 配置修改
4.2 启动canal-admin
五、canal-deployer安装
5.1配置canal-admin
5.2配置mysql的连接配置
5.3启动canal服务
六、配置canal-adapter
6.1配置application.yml
6.2适配器表映射文件
6.3启动adapter
6.4测试
Mysql和ElasticSearch实现数据同步的插件比较多,对比过来还是选择了阿里的canal,下面是几个同步插件的对比图。
目前canal的最新稳定版1.1.4不支持ES7.X,如果需要配合使用ES7.X,可以去官网下载最新的1.1.5的测试版本,也可以在这个地址下载:
canal-1.1.4:https://download.csdn.net/download/doupengzp/12672559
canal-1.1.5https://download.csdn.net/download/doupengzp/12695456
系统环境 | win10企业版 |
mysql版本 | 5.7 |
ES版本 | 6.8.11 |
canal版本 | 1.1.4 |
参考:https://blog.csdn.net/doupengzp/article/details/103801062
mysql安装完成,需要先开启 Binlog 写入功能,配置 binlog-format 为 ROW 模式,my.cnf 中配置如下
log-bin=mysql-bin #添加这一行就ok
binlog-format=ROW #选择row模式
server_id=1 #配置mysql replaction需要定义,不能和canal的slaveId重复
expire_logs_days=5 #日志过期时间为5天
创建测试表
CREATE TABLE `user` (
`id` int(10) NOT NULL,
`name` varchar(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL DEFAULT NULL,
`role_id` int(10) NOT NULL,
`c_time` timestamp(0) NULL DEFAULT CURRENT_TIMESTAMP(0),
`c_utime` timestamp(0) NULL DEFAULT CURRENT_TIMESTAMP(0) ON UPDATE CURRENT_TIMESTAMP(0),
PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = utf8mb4 COLLATE = utf8mb4_general_ci ROW_FORMAT = Dynamic;
参考:https://www.jianshu.com/p/8910c4869df5
创建索引并配置mapping,我是elasticsearch-head插件里面直接进行可视化创建的
{"properties":
{
"name":{"type":"text","fields":{"keyword":{"type":"keyword"}}},
"role_id":{"type":"long"},
"c_time":{"type":"date"}
}
}
修改 conf/application.yml,需要注意的是如果数据库使用的mysql8,需要下载mysql8的连接驱动jar包放在lib文件夹下,然后修改application.yml里面数据的连接驱动名称,数据库的连接地址url上需要指定时区(serverTimezone=GMT%2B8)
server:
port: 8089
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
spring.datasource:
address: 192.168.0.15:3306
database: canal_manager
username: root
password: 123456
#driver-class-name: com.mysql.jdbc.Driver
#mysql8数据库连接驱动
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://${spring.datasource.address}/${spring.datasource.database}?useUnicode=true&characterEncoding=UTF-8&serverTimezone = GMT&useSSL=false
hikari:
maximum-pool-size: 30
minimum-idle: 1
canal:
adminUser: admin
adminPasswd: admin
启动bin目录下的startup.bat
启动后控制台无报错,浏览器地址输入http://127.0.0.1:8089/ 访问,默认密码:admin/123456
如果安装了canal-admin,需要先修改conf/canal.properties的canal admin config配置如下
# canal admin config
canal.admin.manager = 127.0.0.1:8089
canal.admin.port = 11110
canal.admin.user = admin
canal.admin.passwd = 4ACFE3202A5FF5CF467898FC58AAB1D615029441
修改conf/example/instance.properties
## mysql serverId
canal.instance.mysql.slaveId = 1234
#position info,需要改成自己的数据库信息
canal.instance.master.address = 192.168.0.15:3306
canal.instance.master.journal.name =
canal.instance.master.position =
canal.instance.master.timestamp =
#canal.instance.standby.address =
#canal.instance.standby.journal.name =
#canal.instance.standby.position =
#canal.instance.standby.timestamp =
#username/password,需要改成自己的数据库信息
canal.instance.dbUsername = root
canal.instance.dbPassword = 123456
canal.instance.defaultDatabaseName =
canal.instance.connectionCharset = UTF-8
#table regex
canal.instance.filter.regex = .\*\\\\..\*
#可根据需要修改主题名称
canal.mq.topic=example
启动bin目录下的startup.bat,查看logs目录下的日志看是否有报错
这里面比较重要的配置就是elasticserach的连接端口需要配置成9300,9200 是 HTTP 协议的 RESTful 接口,9300 是 TCP 通讯端口,集群间和 TCPClient 都走的它。集群的命名cluster.name,如果ealsticsearch没有配置,不需要修改,默认集群名称为ealsticsearch。也可以参照canal的官网进行配置https://github.com/alibaba/canal/wiki/Sync-ES
server:
port: 8081
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
default-property-inclusion: non_null
canal.conf:
mode: tcp
canalServerHost: 192.168.0.15:11111
# zookeeperHosts: slave1:2181
# mqServers: 127.0.0.1:9092 #or rocketmq
# flatMessage: true
batchSize: 500
syncBatchSize: 1000
retries: 0
timeout:
accessKey:
secretKey:
srcDataSources:
defaultDS:
url: jdbc:mysql://192.168.0.5:3306/admin?useUnicode=true
username: root
password: 123456
canalAdapters:
- instance: example
groups:
- groupId: g1
outerAdapters:
- name: logger
# - name: rdb
# key: mysql1
# properties:
# jdbc.driverClassName: com.mysql.jdbc.Driver
# jdbc.url: jdbc:mysql://127.0.0.1:3306/mytest2?useUnicode=true
# jdbc.username: root
# jdbc.password: 121212
# - name: rdb
# key: oracle1
# properties:
# jdbc.driverClassName: oracle.jdbc.OracleDriver
# jdbc.url: jdbc:oracle:thin:@localhost:49161:XE
# jdbc.username: mytest
# jdbc.password: m121212
# - name: rdb
# key: postgres1
# properties:
# jdbc.driverClassName: org.postgresql.Driver
# jdbc.url: jdbc:postgresql://localhost:5432/postgres
# jdbc.username: postgres
# jdbc.password: 121212
# threads: 1
# commitSize: 3000
# - name: hbase
# properties:
# hbase.zookeeper.quorum: 127.0.0.1
# hbase.zookeeper.property.clientPort: 2181
# zookeeper.znode.parent: /hbase
- name: es
key: exampleKey
hosts: 192.168.0.15:9300
properties:
#mode: rest
# security.auth: test:123456 # only used for rest mode
#cluster.name: my-application
cluster.name: elasticsearch
修改 conf/es/mytest_user.yml文件:
dataSourceKey: defaultDS # 源数据源的key, 对应上面配置的srcDataSources中的值
destination: example # cannal的instance或者MQ的topic
groupId: g1
esMapping:
_index: mytest_user
_id: _id
# upsert: true
# pk: id
sql: "select a.id as _id, a.name from user a"
# objFields:
# _labels: array:;
etlCondition: "where a.c_time>={}"
commitBatch: 3000
启动bin目录下的startup.bat
在mysql的user表中添加数据