目录
MySQL
-
Canal
Message Queue
Client APIs
Client Adapter
意义
MySQL
docker run --name mysql-canal -p 3306:3306 -e MYSQL_ROOT_PASSWORD=123456 -d mysql:5.7.17
docker cp mysql-canal:/etc/mysql/mysql.conf.d/mysqld.cnf ~/Downloads/mysqld.cnf
vim ~/Downloads/mysqld.cnf
[mysqld]
log-bin=mysql-bin #添加这一行就ok
binlog-format=ROW #选择row模式
server_id=1 #配置mysql replaction需要定义，不能和canal的slaveId重复
docker cp ~/Downloads/mysqld.cnf mysql-canal:/etc/mysql/mysql.conf.d/mysqld.cnf
docker restart mysql-canal
docker exec -i mysql-canal mysql -uroot -p123456 <<< "CREATE USER canal IDENTIFIED BY 'canal';"
docker exec -i mysql-canal mysql -uroot -p123456 <<< "GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%';"
docker exec -i mysql-canal mysql -uroot -p123456 <<< "FLUSH PRIVILEGES;"
Canal
wget https://github.com/alibaba/canal/releases/download/canal-1.1.0/canal.kafka-1.1.0.tar.gz
mkdir /opt/canal
tar zxvf canal.kafka-1.1.0.tar.gz -C /opt/canal
cd /opt/canal
vim conf/example/instance.properties
canal.instance.master.address=127.0.0.1:3306
canal.instance.dbUsername=canal
canal.instance.dbPassword=canal
sh bin/startup.sh
tail -f logs/canal/canal.log
2019-06-18 11:35:07.377 [main] INFO com.alibaba.otter.canal.deployer.CanalStater - ## start the canal server.
2019-06-18 11:35:07.405 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[192.168.1.80:11111]
2019-06-18 11:35:07.931 [main] WARN o.s.beans.GenericTypeAwarePropertyDescriptor - Invalid JavaBean property 'connectionCharset' being accessed! Ambiguous write methods found next to actually used [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.nio.charset.Charset)]: [public void com.alibaba.otter.canal.parse.inbound.mysql.AbstractMysqlEventParser.setConnectionCharset(java.lang.String)]
2019-06-18 11:35:08.204 [main] ERROR com.alibaba.druid.pool.DruidDataSource - testWhileIdle is true, validationQuery not set
2019-06-18 11:35:08.397 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^.*\..*$
2019-06-18 11:35:08.397 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter :
2019-06-18 11:35:08.439 [main] INFO com.alibaba.otter.canal.deployer.CanalStater - ## the canal server is running now ......
2019-06-18 11:35:08.469 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2019-06-18 11:35:08.470 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just show master status
2019-06-18 11:35:09.339 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=mysql-bin.000003,position=4,serverId=1,gtid=,timestamp=1560828705000] cost : 860ms , the next step is binlog dump
Message Queue
这里以Kafka为例 关于RocketMQ可以参考Canal Kafka RocketMQ QuickStart
brew install kafka
brew services start zookeeper
brew services start kafka
brew services list
/usr/local/bin/kafka-topics --create --zookeeper 127.0.0.1:2181 --replication-factor 1 --partitions 1 --topic canal
/usr/local/bin/kafka-topics --list --zookeeper 127.0.0.1:2181
/usr/local/bin/kafka-console-producer --broker-list 127.0.0.1:9092 --topic canal
/usr/local/bin/kafka-console-consumer --bootstrap-server 127.0.0.1:9092 --from-beginning --topic canal
更多关于Kafka 可以参考Kafka入门
vim conf/canal.properties
canal.zkServers=127.0.0.1:2181
canal.withoutNetty=true
canal.destinations=example
vim conf/kafka.yml
servers: 127.0.0.1:9092
topic: canal
sh bin/stop.sh
sh bin/startup.sh
CREATE DATABASE canal;
USE canal;
CREATE TABLE IF NOT EXISTS `test`(
`id` INT UNSIGNED AUTO_INCREMENT,
`title` VARCHAR(100) NOT NULL,
PRIMARY KEY ( `id` )
);
INSERT INTO test (title) VALUES ("title1");
Client APIs
mkdir canal-apis && cd canal-apis
gradle init --type java-application
gradle run
# Hello world.
vim settings.gradle
dependencies {
compile group: 'com.alibaba.otter', name: 'canal.client', version: '1.0.19'
}
vim src/main/java/App.java
import com.alibaba.otter.canal.client.CanalConnector;
import com.alibaba.otter.canal.client.CanalConnectors;
import com.alibaba.otter.canal.common.utils.AddressUtils;
import com.alibaba.otter.canal.protocol.CanalEntry.Column;
import com.alibaba.otter.canal.protocol.CanalEntry.Entry;
import com.alibaba.otter.canal.protocol.CanalEntry.EntryType;
import com.alibaba.otter.canal.protocol.CanalEntry.EventType;
import com.alibaba.otter.canal.protocol.CanalEntry.RowChange;
import com.alibaba.otter.canal.protocol.CanalEntry.RowData;
import com.alibaba.otter.canal.protocol.Message;
import java.net.InetSocketAddress;
import java.util.List;
public class App {
public static void main(String[] args) {
String ip = AddressUtils.getHostIp();
try {
CanalConnector connector = CanalConnectors.newSingleConnector(new InetSocketAddress(ip,
11111), "example", "", "");
int batchSize = 1000;
int emptyCount = 0;
try {
connector.connect();
connector.subscribe(".*\\..*");
connector.rollback();
int totalEmptyCount = 120;
while (emptyCount < totalEmptyCount) {
Message message = connector.getWithoutAck(batchSize);
long batchId = message.getId();
int size = message.getEntries().size();
if (batchId == -1 || size == 0) {
emptyCount++;
System.out.println("empty count : " + emptyCount);
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
}
} else {
emptyCount = 0;
printEntry(message.getEntries());
}
connector.ack(batchId);
}
System.out.println("empty too many times, exit");
} finally {
connector.disconnect();
}
} catch (Exception e) {
System.out.println("exception: " + e.toString());
}
}
private static void printEntry(List entrys) {
for (Entry entry : entrys) {
if (entry.getEntryType() == EntryType.TRANSACTIONBEGIN || entry.getEntryType() == EntryType.TRANSACTIONEND) {
continue;
}
RowChange rowChage = null;
try {
rowChage = RowChange.parseFrom(entry.getStoreValue());
} catch (Exception e) {
throw new RuntimeException("ERROR ## parser of eromanga-event has an error , data:" + entry.toString(),
e);
}
EventType eventType = rowChage.getEventType();
System.out.println(String.format("================> binlog[%s:%s] , name[%s,%s] , eventType : %s",
entry.getHeader().getLogfileName(), entry.getHeader().getLogfileOffset(),
entry.getHeader().getSchemaName(), entry.getHeader().getTableName(),
eventType));
for (RowData rowData : rowChage.getRowDatasList()) {
if (eventType == EventType.DELETE) {
printColumn(rowData.getBeforeColumnsList());
} else if (eventType == EventType.INSERT) {
printColumn(rowData.getAfterColumnsList());
} else {
System.out.println("-------> before");
printColumn(rowData.getBeforeColumnsList());
System.out.println("-------> after");
printColumn(rowData.getAfterColumnsList());
}
}
}
}
private static void printColumn(List columns) {
for (Column column : columns) {
System.out.println(column.getName() + " : " + column.getValue() + " update=" + column.getUpdated());
}
}
}
Client Adapter
canal 1.1.1版本之后 增加客户端数据落地的适配及启动功能 目前支持功能
客户端启动器
同步管理REST接口
日志适配器 作为DEMO
关系型数据库的数据同步(表对表同步) ETL功能
HBase的数据同步(表对表同步) ETL功能
(后续支持) ElasticSearch多表数据同步 ETL功能
更多关于Client Adapter 可以参考ClientAdapter
意义
开发 => 运维
压力 => 延时
事务
参考
Canal - QuickStart
MySQL binlog解析canal + kafka实践|ZooKeeper集群部署
基于docker的canal-server部署作业