用kafka接收mysql的binlog消息

1、更改mysql的参数，并重新启动mysql
更改my.cnf，主要更改如下的选项
binlog_format=row

2、下载maxwell
源代码:https://github.com/zendesk/maxwell
直接下载安装包:https://github.com/zendesk/maxwell/releases/download/v1.19.0/maxwell-1.19.0.tar.gz
解压

 

3、在kafka中创建一个topic
./bin/kafka-topics.sh --create --zookeeper localhost:2181 -replication-factor 3 --partitions 5 --topic mysql_binlog

4、在后台启动mysql和maxwell
bin/maxwell --user='root' --password='123456' --port=3306 --host='127.0.0.1' --producer=kafka \
--kafka.bootstrap.servers=localhost:9092 --kafka_topic=mysql_binlog &

5、监控kafka中的数据
./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mysql_binlog --from-beginning

6、做insert时候的数据

{"database":"localinfo","table":"fang_house_price","type":"insert","ts":1540887971,"xid":2574,"commit":true,"data":{"city_id":"6D3323DA5A4763EFC1C06EA2DE21DFCE","statistics_date":201810,"house_price":58575.0}}
{"database":"localinfo","table":"fang_house_price","type":"insert","ts":1540887971,"xid":2588,"commit":true,"data":{"city_id":"10D0597C95A533D4DD1FAF0CEE27700A","statistics_date":201808,"house_price":70638.0}}
{"database":"localinfo","table":"fang_house_price","type":"insert","ts":1540887971,"xid":2589,"commit":true,"data":{"city_id":"10D0597C95A533D4DD1FAF0CEE27700A","statistics_date":201809,"house_price":70238.0}}

做delete的时候：
{"database":"mtime_dw","table":"flow_app_day","type":"delete","ts":1540888635,"xoffset":369,"data":{"device_id":"c7d4d0a627e02e8LJ9SWSGUT8FUS4PJ","time":"20170709144959","ip":"223.104.108.158","log_type":"2","event_type":"0","type":"app","dt":20170709}}
{"database":"mtime_dw","table":"flow_app_day","type":"delete","ts":1540888635,"commit":true,"data":{"device_id":"c7d4d0a627e02e8LJ9SWSGUT8FUS4PJ","time":"20170709145001","ip":"223.104.108.158","log_type":"2","event_type":"0","type":"app","dt":20170709}}

做update操作的时候：
{"database":"mtime_dw","table":"flow_app_day","type":"update","ts":1540888780,"xoffset":94,"data":{"device_id":"c8e9c2d5f4a0726bAHK0217111005110","time":"20170709134113","ip":"10.101.16.203","log_type":"2","event_type":"0","type":"app","dt":20170709},"old":{"ip":"36.5.96.145"}}
{"database":"mtime_dw","table":"flow_app_day","type":"update","ts":1540888780,"xoffset":95,"data":{"device_id":"c8e9c2d5f4a0726bAHK0217111005110","time":"20170709134110","ip":"10.101.16.203","log_type":"2","event_type":"0","type":"app","dt":20170709},"old":{"ip":"36.5.96.145"}}
{"database":"mtime_dw","table":"flow_app_day","type":"update","ts":1540888780,"commit":true,"data":{"device_id":"c8e9c2d5f4a0726bAHK0217111005110","time":"20170709134106","ip":"10.101.16.203","log_type":"2","event_type":"1","type":"app","dt":20170709},"old":{"ip":"36.5.96.145"}}

做drop和truncate操作的时候：
kafka中未收到任何的消息记录。