Flink CDC 2.0 SQL初体验(MySql版)

1.MySql开启binlog:
修改my.cnf文件，增加如下内容：

server_id=1
log_bin=mysql-bin
binlog_format=ROW
expire_logs_days=5
binlog_do_db=flinktest1
binlog_do_db=flinktest2

• server_id：MySQL5.7及以上版本开启binlog必须要配置这个选项。对于MySQL集群，不同节点的server_id必须不同。对于单实例部署则没有要求。
• log_bin：指定binlog文件名和储存位置。如果不指定路径，默认位置为/var/lib/mysql/。
• binlog_format：binlog格式。有3个值可以选择：ROW：记录哪条数据被修改和修改之后的数据，会产生大量日志。STATEMENT：记录修改数据的SQL，日志量较小。MIXED：混合使用上述两个模式。CDC要求必须配置为ROW。
• expire_logs_days：bin_log过期时间，超过该时间的log会自动删除。
• binlog_do_db：binlog记录哪些数据库。如果需要配置多个库，如例子中配置多项。切勿使用逗号分隔。

查看binlog是否开启成功：

show variables like 'log_bin';

如上即为成功开启。

MySql创建表：

CREATE TABLE `student`  (
  `id` int(10) NOT NULL,
  `name` varchar(128) CHARACTER SET latin1 COLLATE latin1_swedish_ci DEFAULT NULL,
  `age` int(10) DEFAULT NULL,
  PRIMARY KEY (`id`) USING BTREE
) ENGINE = InnoDB CHARACTER SET = latin1;

2.maven依赖：

	<properties>
        <flink-version>1.13.0</flink-version>
        <maven.compiler.source>1.8</maven.compiler.source>
        <maven.compiler.target>1.8</maven.compiler.target>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-java</artifactId>
            <version>${flink-version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-streaming-java_2.12</artifactId>
            <version>${flink-version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-clients_2.12</artifactId>
            <version>${flink-version}</version>
        </dependency>

        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>3.1.3</version>
        </dependency>

        <dependency>
            <groupId>mysql</groupId>
            <artifactId>mysql-connector-java</artifactId>
            <version>5.1.49</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-table-planner-blink_2.12</artifactId>
            <version>${flink-version}</version>
        </dependency>

        <dependency>
            <groupId>com.ververica</groupId>
            <artifactId>flink-connector-mysql-cdc</artifactId>
            <version>2.0.2</version>
        </dependency>

        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.75</version>
        </dependency>

        <dependency>
            <groupId>org.apache.flink</groupId>
            <artifactId>flink-connector-jdbc_2.12</artifactId>
            <version>1.13.3</version>
        </dependency>
    </dependencies>

3.代码实现：

public class FlinkCDCToMySql {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);
        StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
        String sourceSql = "CREATE TABLE IF NOT EXISTS mySqlSource (" +
                "id Int primary key, " +
                "name String ," +
                "age Int" +
                ") with ( " +
                " 'connector' = 'mysql-cdc', " +
                " 'scan.startup.mode' = 'latest-offset', " +
                " 'hostname' = '192.168.0.101', " +
                " 'port' = '3306', " +
                " 'username' = 'root', " +
                " 'password' = 'szzzcheng', " +
                " 'database-name' = 'flinktest', " +
                " 'table-name' = 'student' " +
                ")";

        String sinkSql = " CREATE TABLE IF NOT EXISTS mySqlSink (" +
                "id Int primary key , " +
                "name String ," +
                "age Int" +
                ") with (" +
                " 'connector' = 'jdbc'," +
                " 'url' = 'jdbc:mysql://192.168.0.101:3306/flinktest'," +
                "'table-name' = 'flinktest_cdc'," +
                " 'username' = 'root'," +
                " 'password' = 'szzzcheng' " +
                " )";
        tableEnv.executeSql(sourceSql);
        tableEnv.executeSql(sinkSql);
        tableEnv.executeSql("insert  into  mySqlSink select * from mySqlSource ");
    }
}

之后在student表里增删改数据，flinktest_cdc表都会体现出来。
当然也可以改为打印在控制台的方式，只需要把sinkSql改为：

String sinkSql = " CREATE TABLE IF NOT EXISTS mySqlSink (" +
                "id Int primary key , " +
                "name String ," +
                "age Int" +
                ") with (" +
                " 'connector' = 'print'" +
                " )";

结果如下：

+I[3, tom, 12]
+I[6, jack, 16]
-U[6, jack, 16]
+U[6, jack2, 18]