flink数据sink入mysql（exactly-once）

前言：本文使用flink 1.15.0版本消费数据到mysql，从 1.13 开始，Flink JDBC sink 支持exactly-once 模式，需要数据库也支持 XA。所以本文介绍link + MySQL8+，使用exactly-once 模式flink事务写入mysql。

依赖：

        
            org.apache.flink
            flink-connector-jdbc
            1.15.0
        

话不多说，直接上完整代码：

public class JdbcSinkMysql {

    @Data
    static class Student {

        public Student(Integer id, String name, Boolean sex,Integer age, Integer score, Timestamp createTime, Timestamp updateTime) {
            this.id = id;
            this.name = name;
            this.sex = sex;
            this.age = age;
            this.score = score;
            this.createTime = createTime;
            this.updateTime = updateTime;
        }

        final Integer id;
        final String name;
        final Boolean sex;
        final Integer age;
        final Integer score;
        final Timestamp createTime;
        final Timestamp updateTime;
    }

    public static void main(String[] args) throws Exception {
        ArrayList<Student> students = new ArrayList<>();
        Random random = new Random();

        for (int i = 0; i <= 20; i++) {
            Boolean flag = true;
            if (i % 2 == 0) {
                flag = false;
            }
            Student student = new Student(i, "name_" + i, flag,random.nextInt(20) ,random.nextInt(100), new Timestamp(new Date().getTime()), new Timestamp(new Date().getTime()));
            students.add(student);
            System.out.println(student);
        }

        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        DataStreamSource<Student> dataStreamSource = env.fromCollection(students);
        env.enableCheckpointing(3000);
        env.setParallelism(2);

        dataStreamSource.addSink(JdbcSink.exactlyOnceSink(
                "insert into student (id,name,sex,age,score,create_time,update_time) values(?,?,?,?,?,?,?)",
                (ps, s) -> {
                    ps.setInt(1, s.id);
                    ps.setString(2, s.name);
                    ps.setBoolean(3, s.sex);
                    ps.setInt(4,s.age);
                    ps.setInt(5, s.score);
                    ps.setTimestamp(6, s.createTime);
                    ps.setTimestamp(7, s.updateTime);
                },
                JdbcExecutionOptions.builder()
                        .withMaxRetries(0)
                        .build(),
                JdbcExactlyOnceOptions.builder()
                        .withTransactionPerConnection(true)
                        .build(),
                () -> {
                    MysqlXADataSource xaDataSource = new MysqlXADataSource();
                    xaDataSource.setUrl("jdbc:mysql://localhost:3306/hc?characterEncoding=utf8&serverTimezone=UTC");
                    xaDataSource.setUser("flinkuser");
                    xaDataSource.setPassword("123456");
                    return xaDataSource;
                }
        )).name("sinkTomysql");
        env.execute("sink mysql");
    }
}

详细解析：

 // exactly-once 模式需要使用exactlyOnceSink(sql语句，sql语句注入值的映射关系，optional，exactlyOnceOptional，XA DataSource)方法创建一个接收器
JdbcSink.exactlyOnceSink(
                "insert into student (id,name,sex,age,score,create_time,update_time) values(?,?,?,?,?,?,?)",
                (ps, s) -> {
                    ps.setInt(1, s.id);
                    ps.setString(2, s.name);
                    ps.setBoolean(3, s.sex);
                    ps.setInt(4,s.age);
                    ps.setInt(5, s.score);
                    ps.setTimestamp(6, s.createTime);
                    ps.setTimestamp(7, s.updateTime);
                },
                JdbcExecutionOptions.builder()
                        .withMaxRetries(0)
                        .build(),
                JdbcExactlyOnceOptions.builder()
                        .withTransactionPerConnection(true)
                        .build(),
                () -> {
                    MysqlXADataSource xaDataSource = new MysqlXADataSource();
                    xaDataSource.setUrl("jdbc:mysql://localhost:3306/hc?characterEncoding=utf8&serverTimezone=UTC");
                    xaDataSource.setUser("flinkuser");
                    xaDataSource.setPassword("123456");
                    return xaDataSource;
                }
        )

**注意：**某些数据库只允许每个连接执行一个 XA 事务（例如 PostgreSQL、MySQL）。在这种情况下，请使用以下 API 来构造JdbcExactlyOnceOptions：

JdbcExactlyOnceOptions.builder()
.withTransactionPerConnection(true)
.build();

**注意：**目前，可以使用JdbcSink.exactlyOnceSink确保恰好一次语义。JdbcExecutionOptions.maxRetries == 0。否则，可能会产生重复的结果。

                JdbcExecutionOptions.builder()
                        .withMaxRetries(0)
                        .build(),

MySQLXADataSource示例：

MysqlXADataSource xaDataSource = new com.mysql.cj.jdbc.MysqlXADataSource();
xaDataSource.setUrl("jdbc:mysql://localhost:3306/");
xaDataSource.setUser(username);
xaDataSource.setPassword(password);

可能遇到问题：
1.时区问题：

serverTimezone=UTC

​ 2.XAER_RMERR 权限问题：mysql8以上需要设置

GRANT XA_RECOVER_ADMIN ON *.* TO 'flinkuser'@'localhost' 权限给flink用户;

​ 3.没有insert权限：这里缺什么权限就赋予什么权限

GRANT Insert ON hc.student TO 'flinkuser'@'localhost';
GRANT Update ON hc.student TO 'flinkuser'@'localhost';
刷新权限：
FLUSH PRIVILEGES;

​ 3.Public Key Retrieval is not allowed

allowPublicKeyRetrieval=true

​ 4.如遇其他问题，可访问flink官网查看flink-connector-jdbc：flink-connector-jdbc

欢迎分享讨论。