Flink Hudi 源码之HoodieTableSink

Flink源码分析系列文档目录

请点击：Flink 源码分析系列文档目录

源代码分支

release-0.9.0

Hudi 源代码GitHub地址：apache/hudi: Upserts, Deletes And Incremental Processing on Big Data. (github.com)

HoodieTableFactory

Flink通过SPI机制加载org.apache.flink.table.factories.Factory接口的实现类。Hudi的hudi-flink/src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory文件内容如下：

org.apache.hudi.table.HoodieTableFactory

这个类是Flink SQL创建Table Sink和Source的入口类。本篇我们从这个类开始，分析HoodieTableSink的创建过程。创建TableSink的入口方法逻辑如下：

@Override
public DynamicTableSink createDynamicTableSink(Context context) {
    // 获取create table是否with子句附带的参数
    Configuration conf = FlinkOptions.fromMap(context.getCatalogTable().getOptions());
    // 获取表的物理Schema，意思是不包含计算字段和元数据字段
    TableSchema schema = TableSchemaUtils.getPhysicalSchema(context.getCatalogTable().getSchema());
    // 检查参数合理性
    // 检查hoodie.datasource.write.recordkey.field和write.precombine.field配置项是否包含在表字段中，如果不包含则抛出异常
    sanityCheck(conf, schema);
    // 根据table定义和主键等配置，Hudi自动附加一些属性配置
    setupConfOptions(conf, context.getObjectIdentifier().getObjectName(), context.getCatalogTable(), schema);
    // 返回HoodieTableSink
    return new HoodieTableSink(conf, schema);
}

HoodieTableSink

Flink SQL在执行过程中最终被解析转换为Flink的TableSink或者TableSource。本篇我们关注数据写入Hudi的过程。HoodieTableSink写入数据的逻辑位于getSinkRuntimeProvider方法。它的内容和解析如下所示：

@Override
public SinkRuntimeProvider getSinkRuntimeProvider(Context context) {
    return (DataStreamSinkProvider) dataStream -> {

        // setup configuration
        // 获取checkpoint超时配置
        long ckpTimeout = dataStream.getExecutionEnvironment()
            .getCheckpointConfig().getCheckpointTimeout();
        // 设置Hudi的instant commit超时时间为Flink的checkpoint超时时间
        conf.setLong(FlinkOptions.WRITE_COMMIT_ACK_TIMEOUT, ckpTimeout);

        // 获取schema对应每列数据类型
        RowType rowType = (RowType) schema.toRowDataType().notNull().getLogicalType();

        // bulk_insert mode
        // 获取写入操作类型，默认是upsert
        final String writeOperation = this.conf.get(FlinkOptions.OPERATION);
        // 如果写入操作类型配置的为bulk_insert，进入这个if分支
        if (WriteOperationType.fromValue(writeOperation) == WriteOperationType.BULK_INSERT) {
            // 创建出批量插入operator工厂类
            BulkInsertWriteOperator.OperatorFactory operatorFactory = BulkInsertWriteOperator.getFactory(this.conf, rowType);
           // 获取分区字段
            final String[] partitionFields = FilePathUtils.extractPartitionKeys(this.conf);
            if (partitionFields.length > 0) {
                // 创建出key生成器，用于指定数据分组，keyBy算子使用
                RowDataKeyGen rowDataKeyGen = RowDataKeyGen.instance(conf, rowType);
                // 如果启用write.bulk_insert.shuffle_by_partition
                if (conf.getBoolean(FlinkOptions.WRITE_BULK_INSERT_SHUFFLE_BY_PARTITION)) {

                    // shuffle by partition keys
                    // 数据流按照分区字段值进行keyBy操作
                    dataStream = dataStream.keyBy(rowDataKeyGen::getPartitionPath);
                }
                // 如果需要按照分区排序
                if (conf.getBoolean(FlinkOptions.WRITE_BULK_INSERT_SORT_BY_PARTITION)) {
                    // 创建一个排序operator
                    SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, partitionFields);
                    // sort by partition keys
                    // 为datastream增加一个排序操作符
                    dataStream = dataStream
                        .transform("partition_key_sorter",
                                   TypeInformation.of(RowData.class),
                                   sortOperatorGen.createSortOperator())
                        .setParallelism(conf.getInteger(FlinkOptions.WRITE_TASKS));
                    ExecNode$.MODULE$.setManagedMemoryWeight(dataStream.getTransformation(),
                                                             conf.getInteger(FlinkOptions.WRITE_SORT_MEMORY) * 1024L * 1024L);
                }
            }
            // 为dataStream加入批量写入operator并返回
            return dataStream
                .transform("hoodie_bulk_insert_write",
                           TypeInformation.of(Object.class),
                           operatorFactory)
                // follow the parallelism of upstream operators to avoid shuffle
                .setParallelism(conf.getInteger(FlinkOptions.WRITE_TASKS))
                .addSink(new CleanFunction<>(conf))
                .setParallelism(1)
                .name("clean_commits");
        }
        // 对于非批量写入模式，采用流式写入
        // stream write
        int parallelism = dataStream.getExecutionConfig().getParallelism();
        // 创建流式写入operator
        StreamWriteOperatorFactory operatorFactory = new StreamWriteOperatorFactory<>(conf);

        // 将数据从RowData格式转换为HoodieRecord
        DataStream dataStream1 = dataStream
            .map(RowDataToHoodieFunctions.create(rowType, conf), TypeInformation.of(HoodieRecord.class));

        // bootstrap index
        // TODO: This is a very time-consuming operation, will optimization
        // 是否启动时加载索引
        if (conf.getBoolean(FlinkOptions.INDEX_BOOTSTRAP_ENABLED)) {
            // 如果启用，会在启动时自动加载索引，包装为IndexRecord发往下游
            dataStream1 = dataStream1.rebalance()
                .transform(
                "index_bootstrap",
                TypeInformation.of(HoodieRecord.class),
                new ProcessOperator<>(new BootstrapFunction<>(conf)))
                .setParallelism(conf.getOptional(FlinkOptions.INDEX_BOOTSTRAP_TASKS).orElse(parallelism))
                .uid("uid_index_bootstrap_" + conf.getString(FlinkOptions.TABLE_NAME));
        }

        // 按照record key分区，然后使用ucketAssignFunction分桶
        // 再按照分桶id分区，使用StreamWriteFunction流式写入
        DataStream