Canal解析binlog发送Kafka

官方链接

https://github.com/alibaba/canal/wiki/QuickStart

mysql的搭建：https://blog.csdn.net/NOAH_123456/article/details/84203781
kafka的搭建：https://blog.csdn.net/NOAH_123456/article/details/103260001

配置文件canal/conf/canal.properties 关键点

## 服务模式kafka，默认为tcp
canal.serverMode = kafka 
## kafka服务地址
canal.mq.servers = 127.0.0.1:9092 

配置文件canal/conf/example/instance.properties

## mysql数据库ip端口
canal.instance.master.address=134.175.145.11:3306
canal.instance.dbUsername=canal
canal.instance.dbPassword=canal
canal.instance.connectionCharset = UTF-8
canal.mq.topic=example

kafka消费方收到都数据：

代码地址：https://github.com/sijunx/mySpider/blob/feature_word_dic_20191001001/spider-base/spider-base-kafka/src/test/java/KafkaConsumerTest.java

{
“data”: [{
“id”: “25”,
“line”: “0”,
“channel_name”: “xxxxyyyy”,
“channel_type”: “2”,
“on_flag”: “0”,
“hide_flag”: “0”,
“created_at”: “2019-11-26 16:35:19”,
“updated_at”: “2019-11-26 16:35:24”,
“create_user_id”: “0”,
“create_user_name”: “”,
“create_user_type”: “”,
“update_user_id”: “0”,
“update_user_name”: “”,
“update_user_type”: “”,
“is_delete”: “0”,
“drc_check_time”: “2019-11-26 16:35:24.814”,
“ezone_shard_info”: null
}],
“database”: “spider_dev”,
“es”: 1574757324000,
“id”: 8,
“isDdl”: false,
“mysqlType”: {
“id”: “bigint(20)”,
“line”: “tinyint(4)”,
“channel_name”: “varchar(20)”,
“channel_type”: “tinyint(4)”,
“on_flag”: “tinyint(4)”,
“hide_flag”: “tinyint(4)”,
“created_at”: “timestamp”,
“updated_at”: “timestamp”,
“create_user_id”: “bigint(20)”,
“create_user_name”: “varchar(100)”,
“create_user_type”: “varchar(100)”,
“update_user_id”: “bigint(20)”,
“update_user_name”: “varchar(100)”,
“update_user_type”: “varchar(100)”,
“is_delete”: “tinyint(4)”,
“drc_check_time”: “timestamp(3)”,
“ezone_shard_info”: “bigint(20)”
},
“old”: [{
“channel_type”: “0”,
“updated_at”: “2019-11-26 16:35:19”,
“drc_check_time”: “2019-11-26 16:35:19.524”
}],
“pkNames”: [“id”],
“sql”: “”,
“sqlType”: {
“id”: -5,
“line”: -6,
“channel_name”: 12,
“channel_type”: -6,
“on_flag”: -6,
“hide_flag”: -6,
“created_at”: 93,
“updated_at”: 93,
“create_user_id”: -5,
“create_user_name”: 12,
“create_user_type”: 12,
“update_user_id”: -5,
“update_user_name”: 12,
“update_user_type”: 12,
“is_delete”: -6,
“drc_check_time”: 93,
“ezone_shard_info”: -5
},
“table”: “channel_info”,
“ts”: 1574757325237,
“type”: “UPDATE”
}

canal的日志

问题

com.alibaba.otter.canal.parse.exception.PositionNotFoundException: can’t find start position for example

参照：https://blog.csdn.net/my201110lc/article/details/77885720

解决方案：删除meta.dat删除，再重启canal，问题解决；

集群操作：进入canal对应的zookeeper集群下，删除节点/otter/canal/destinations/xxxxx/1001/cursor ；重启canal即可恢复；