canal
工作原理
- canal 模拟 MySQL slave 的交互协议，伪装自己为 MySQL slave ，向 MySQL master 发送 dump 协议
- MySQL master 收到 dump 请求，开始推送 binary log 给 slave (即 canal )
- canal 解析 binary log 对象(原始为 byte 流)
优点： 可以完全和业务代码解耦，增量日志订阅。
缺点：实时性不高，订阅mysql日志，DB中数据事务成功后，开始同步至canal。
配置Mysql
set global validate_password_policy=0;
set global validate_password_length=1;
use mysql;
create user 'canal'@'%' identified by 'canal';
grant SELECT, REPLICATION SLAVE, REPLICATION CLIENT on *.* to 'canal'@'%' identified by 'canal';
flush privileges;
SHOW VARIABLES LIKE 'BINLOG_format';
SHOW VARIABLES LIKE 'log_bin';
show master status;
SHOW GRANTS FOR 'canal'@'192.168.199.105'
创建测试数据库es_test
创建表
CREATE TABLE `user` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`name` varchar(256) COLLATE utf8mb4_unicode_ci NOT NULL,
`detail` text COLLATE utf8mb4_unicode_ci NOT NULL,
`age` int(3) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;
下载Canal
canal下载地址：https://github.com/alibaba/canal/releases
下载最新的cana1.1.5，1.1.5 才支持es7
mkdir -p /data/canal
canal.deployer-1.1.5-SNAPSHOT.tar.gz
canal.adapter-1.1.5-SNAPSHOT.tar.gz
canal.admin-1.1.5-SNAPSHOT.tar.gz
解压
mkdir -p /data/canal/canal-adapter/ && tar -zxvf canal.adapter-1.1.5-SNAPSHOT.tar.gz -C ./canal-adapter/
mkdir -p /data/canal/canal-deployer/ && tar -zxvf canal.deployer-1.1.5-SNAPSHOT.tar.gz -C ./canal-deployer/
mkdir -p /data/canal/canal-admin/ && tar -zxvf canal.admin-1.1.5-SNAPSHOT.tar.gz -C ./canal-admin/
canal-admin
https://github.com/alibaba/canal/wiki/Canal-Admin-QuickStart
设置密码
select password('12345678')
84AAC12F54AB666ECFC2A83C676908C8BBC381B1
https://www.bookstack.cn/read/canal-v1.1.4/5fb1e3d9c31b6e21.md
启动
sh bin/startup.sh
canal-deployer
cd canal-deployer
vim conf/example/instance.properties
配置mysql的相关信息
canal.instance.master.address=192.168.199.101:3306
canal.instance.dbUsername=canal
canal.instance.dbPassword=canal
#table regex
canal.instance.filter.regex=es_test\\..*
启动Canal-server，并查看日志。
./bin/startup.sh
cat logs/canal/canal.log
2021-04-17 23:39:07.773 [main] INFO com.alibaba.otter.canal.deployer.CanalStarter - ## the canal server is running now ......
com.alibaba.otter.canal.common.CanalException: instance : example config is not found
sh bin/startup.sh local
2021-04-19 20:16:29.028 [New I/O server worker #1-1] ERROR c.a.o.c.common.zookeeper.running.ServerRunningMonitor - start failed
com.google.common.util.concurrent.UncheckedExecutionException: com.alibaba.otter.canal.server.exception.CanalServerException: can't find destination:example
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
删掉/data/canal/canal-deployer/conf/example/meta.dat
mv conf/example/meta.dat conf/example/meta.dat.bak
mv conf/example/h2.mv.db conf/example/h2.mv.db.bak
或者修改canal.instance.global.lazy = true
还是不行就得手工去canal-admin新建instance
查看日志
tail -100f logs/canal/canal.log
tail -100f logs/example/example.log
canal-adapter
cd canal-adapter
vim conf/application.yml
srcDataSources:
defaultDS:
url: jdbc:mysql://192.168.199.101:3306/es_test?useUnicode=true
username: root
password: 123456
canalAdapters:
- instance: example # canal instance Name or mq topic name
groups:
- groupId: g1
outerAdapters:
- name: logger
- name: es7
key: es71 #这个很重要，必须设置唯一，不然有坑
hosts: 127.0.0.1:9200 # 127.0.0.1:9200 for rest mode
properties:
mode: rest # or rest
# security.auth: test:123456 # only used for rest mode
cluster.name: es-cluster
在conf/es7/目录下新建user.yml,定义MySQL数据到ES数据的映射字段。
dataSourceKey: defaultDS
outerAdapterKey: es71
destination: example
groupId: g1
esMapping:
_index: user
_type: _doc
_id: _id
upsert: true
# pk: id
sql: "select id as _id,id,name,detail,age from user"
etlCondition: "where id<='{0}'" #etl的条件参数，可以将之前没能同步的数据同步，数据量大的话可以用logstash
commitBatch: 3000
chmod 777 user.yml
启动Canal-adapter服务，并查看日志。
./bin/startup.sh
tail -100f logs/adapter/adapter.log
测试验证
es安装中文分词
下载地址
https://github.com/medcl/elasticsearch-analysis-ik/releases
https://github.com/medcl/elasticsearch-analysis-pinyin/releases/download/v7.10.2/elasticsearch-analysis-pinyin-7.10.2.zip
root安装
./bin/elasticsearch-plugin install file:///data/efk/elasticsearch-analysis-ik-7.10.2.zip
./bin/elasticsearch-plugin install file:///data/efk/elasticsearch-analysis-pinyin-7.10.2.zip
[root@yfm05 elasticsearch-7.10.2]# ./bin/elasticsearch-plugin list
analysis-icu
analysis-ik
analysis-pinyin
重启es
Likely root cause: java.nio.file.AccessDeniedException: /data/efk/elasticsearch-7.10.2/config/analysis-ik
chown -R elastic:elastic /data/efk/elasticsearch-7.10.2
创建索引，索引一旦建立，主分片数量不可改变
创建mapping
PUT user/_mapping
{
"settings": {
"index": {
"number_of_shards": "2",
"number_of_replicas": "1"
}
},
"mappings": {
"properties": {
"age": {
"type": "integer"
},
"id": {
"type": "long"
},
"name": {
"type": "text",
"analyzer": "ik_smart"
},
"detail": {
"type": "text",
"analyzer": "ik_max_word"
}
}
}
}
[root@yfm05 canal-deployer]# curl http://192.168.199.105:8081/etl/es7/es71/user.yml -X POST
{"succeeded":true,"resultMessage":"导入ES 数据：1 条"}
canal-adapter自动监听日志
2021-04-19 21:53:08.506 [pool-2-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":5,"name":"5","detail":"5","age":5}],"database":"es_test","destination":"example","es":1618840388000,"groupId":"g1","isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"user","ts":1618840388457,"type":"INSERT"}
2021-04-19 21:53:08.640 [pool-2-thread-1] DEBUG c.a.o.canal.client.adapter.es.core.service.ESSyncService - DML: {"data":[{"id":5,"name":"5","detail":"5","age":5}],"database":"es_test","destination":"example","es":1618840388000,"groupId":"g1","isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"user","ts":1618840388457,"type":"INSERT"}
Affected indexes: user
2021-04-19 21:53:30.780 [pool-2-thread-1] INFO c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"id":5,"name":"5","detail":"5","age":5}],"database":"es_test","destination":"example","es":1618840410000,"groupId":"g1","isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"user","ts":1618840410780,"type":"DELETE"}
2021-04-19 21:53:30.781 [pool-2-thread-1] DEBUG c.a.o.canal.client.adapter.es.core.service.ESSyncService - DML: {"data":[{"id":5,"name":"5","detail":"5","age":5}],"database":"es_test","destination":"example","es":1618840410000,"groupId":"g1","isDdl":false,"old":null,"pkNames":["id"],"sql":"","table":"user","ts":1618840410780,"type":"DELETE"}
Affected indexes: user
