canal同步mysql数据到es

1、下载Canal相关包（注意：MYSQL必须开启binlog，sql：show variables like 'log_bin';）

cd /home/canal
mkdir adapter
mkdir deployer
tar -zxvf canal.adapter-1.1.5.tar.gz -C adapter/
tar -zxvf canal.deployer-1.1.5.tar.gz -C deployer/
2、修改deployer配置

vim /home/canal/deployer/conf/example/instance.properties
修改要同步的数据库地址以及用户名密码

# position info
canal.instance.master.address=122.3.23.1:3306
# username/password
canal.instance.dbUsername=root
canal.instance.dbPassword=123456

3、修改adapter配置

vim /home/canal/adapter/conf/application.yml

修改数据库的地址以及用户名密码，以及ES连接信息

  srcDataSources:
    defaultDS:
      url: jdbc:mysql://127.0.0.1:3306/test?useUnicode=true
      username: root
      password: 123456
  canalAdapters:
  - instance: example # canal instance Name or mq topic name
    groups:
    - groupId: g1
      outerAdapters:
      - name: logger # 输出到日志，可以注释掉
      - name: es7 # 输出到ES，如果用的是ES6.x版本就写es6
        hosts: http://127.0.0.1:9200 # 127.0.0.1:9300 for transport mode
        properties:
          mode: rest # transport or rest
          # security.auth: test:123456 #  only used for rest mode
          cluster.name: my-application # ES里配置的集群名
 

cd /home/canal/adapter/conf/es7 

vim customer.yml

dataSourceKey: defaultDS
destination: example
groupId: g1
esMapping:
  _index: pay_order
  _id: PayOrderId
  sql: "select t.PayOrderId,t.PayOrderId as PayOrder, t.MchId,t.AppId from t_pay_order t"
  commitBatch: 30

4、先启动deployer 

sh /home/canal/deployer/bin/startup.sh

sh /home/canal/adapter/bin/startup.sh
tail -f /home/canal/adapter/logs/adapter/adapter.log

5、同步数据库中所有数据

curl -X POST http://127.0.0.1:8081/etl/es7/sys_log.yml

打印出如下结果表示同步成功了

执行成功后请求的响应：
{"succeeded":true,"resultMessage":"导入ES 数据：687 条"}
adapter 日志中打印：
start etl to import data to index: sys_log
数据全量导入完成, 一共导入 687 条数据, 耗时: 1255