Flink StreamingFileSink源码解析

序

目前来说Flink流式写入文件系统，有两个方式BucketingSink以及StreamingFileSink。StreamingFileSink是在BucketingSink之后推出的。主要区别在StreamingFileSink可以用于故障恢复，保证exactly-once，但是要求hadoop版本必须在2.7以上，因为用到了hdfs的truncate方法。BucketingSink相对用法比较简单，并且没有版本要求，如果对于BucketingSink有兴趣，可以看笔者的另外一篇博客：https://blog.csdn.net/lvwenyuan_1/article/details/90608568

基本用法

        StreamingFileSink sink
                = StreamingFileSink.forRowFormat(new Path(path), new SimpleStringEncoder("UTF-8"))
                .withBucketAssigner(new DateTimeBucketAssigner<>("yyyy-MM-dd",ZoneId.of("Asia/Shanghai")))
                .build();

        stopData.addSink(sink);

StreamingFileSink分为forRowFormat和forBulkFormat。其中forRowFormat就是按行写入，可以用于写入字符串。forBulkFormat是按块写入，可以用于写入parquet这类的特殊文件。

源码解析

StreamingFileSink用于故障恢复，主要是依赖于checkpoint机制。所以StreamingFileSink实现了CheckpointedFunction, CheckpointListener这两个接口，所以如果是故障恢复，那么就是在initializeState方法里。

   @Override
    public void initializeState(FunctionInitializationContext context) throws Exception {
        //获取当前subtask index
        final int subtaskIndex = getRuntimeContext().getIndexOfThisSubtask();
        //初始化数据桶，并且创建hdfs的path路径，在创建的时候，会判断当前hadoop版本是否大于等于2.7,如果hadoop版本小于2.7就会报错
        this.buckets = bucketsBuilder.createBuckets(subtaskIndex);

        /**
         * 从状态中获取bucket列表，两个都是OperatreState的ListState
         * bucketStates是之前未被checkpoint记录为完成的文件
         * maxPartCountersState:一般文件格式是 part-subtaskIndex-counter-状态   这里取的就是counter，用getUnionListState取，是为了广播出去，到时候取最新处理counter
         */
        final OperatorStateStore stateStore = context.getOperatorStateStore();
        bucketStates = stateStore.getListState(BUCKET_STATE_DESC);
        maxPartCountersState = stateStore.getUnionListState(MAX_PART_COUNTER_STATE_DESC);

        //判断是否是恢复状态，就是在这里，进行的文件恢复
        if (context.isRestored()) {
            buckets.initializeState(bucketStates, maxPartCountersState);
        }
    }

继续追踪buckets.initializeState(bucketStates, maxPartCountersState);方法

    void initializeState(final ListState bucketStates, final ListState partCounterState) throws Exception {
        //获取最大的counter
        initializePartCounter(partCounterState);

        LOG.info("Subtask {} initializing its state (max part counter={}).", subtaskIndex, maxPartCounter);
        /**
         * 1.将所有原先是pending的文件，变为 finish文件
         * 2.重新写入之前的 in-progress文件
         * 3.如果接受到同一个bucket的多个状态，合并他们
         */
        initializeActiveBuckets(bucketStates);
    }

    private void initializeActiveBuckets(final ListState bucketStates) throws Exception {
        //遍历各个数据桶，反序列化数据，并出他们
        for (byte[] serializedRecoveredState : bucketStates.get()) {
            final BucketState recoveredState =
                    SimpleVersionedSerialization.readVersionAndDeSerialize(
                            bucketStateSerializer, serializedRecoveredState);
            handleRestoredBucketState(recoveredState);
        }
    }

追踪handleRestoredBucketState方法

    private void handleRestoredBucketState(final BucketState recoveredState) throws Exception {
        //获取对应数据桶的bucketId，默认情况下是(yy-MM-dd--HH) 这种形式，与传入的Assigner有关
        final BucketID bucketId = recoveredState.getBucketId();

        if (LOG.isDebugEnabled()) {
            LOG.debug("Subtask {} restoring: {}", subtaskIndex, recoveredState);
        }
        //用于恢复桶数据
        final Bucket restoredBucket = bucketFactory
                .restoreBucket(
                        fsWriter,
                        subtaskIndex,
                        maxPartCounter,
                        partFileWriterFactory,
                        rollingPolicy,
                        recoveredState
                );
        //更新活跃桶id，如果发现这bucketId，有重复，则合并两个文件
        updateActiveBucketId(bucketId, restoredBucket);
    }

继续追踪恢复数据的方法，可以看到最后到了一个Bucket类的一个私有构造器中

    /**
     * Constructor to restore a bucket from checkpointed state.
     * 从checkpoint中恢复的构造方法，是私有的
     */
    private Bucket(
            final RecoverableWriter fsWriter,
            final int subtaskIndex,
            final long initialPartCounter,
            final PartFileWriter.PartFileFactory partFileFactory,
            final RollingPolicy rollingPolicy,
            final BucketState bucketState) throws IOException {
        //重新创建一个数据桶对象
        this(
                fsWriter,
                subtaskIndex,
                bucketState.getBucketId(),
                bucketState.getBucketPath(),
                initialPartCounter,
                partFileFactory,
                rollingPolicy);

        //打开之前的in-progress文件
        restoreInProgressFile(bucketState);
        //从checkpoint中恢复
        commitRecoveredPendingFiles(bucketState);
    }

先看第一个方法

    private void restoreInProgressFile(final BucketState state) throws IOException {
        if (!state.hasInProgressResumableFile()) {
            return;
        }
        // we try to resume the previous in-progress file
        final ResumeRecoverable resumable = state.getInProgressResumableFile();
        //Flink 的 HadoopRecoverableWriter 支持中断后继续写，这里是true
        if (fsWriter.supportsResume()) {
            //重新打开之前的文件in-progress文件流，用于后续的更改和追加
            final RecoverableFsDataOutputStream stream = fsWriter.recover(resumable);
            inProgressPart = partFileFactory.resumeFrom(
                    bucketId, stream, resumable, state.getInProgressFileCreationTime());
        } else {
            // if the writer does not support resume, then we close the
            // in-progress part and commit it, as done in the case of pending files.
            fsWriter.recoverForCommit(resumable).commitAfterRecovery();
        }
        //是否需要清除或者覆盖，对于hadoop环境，这里是false
        if (fsWriter.requiresCleanupOfRecoverableState()) {
            fsWriter.cleanupRecoverableState(resumable);
        }
    }

再看第二个方法，从checkpoint中恢复的方法

    //将文件从何上个检查点恢复
    private void commitRecoveredPendingFiles(final BucketState state) throws IOException {

        // we commit pending files for checkpoints that precess the last successful one, from which we are recovering
        for (List committables: state.getCommittableFilesPerCheckpoint().values()) {
            //获取之前checkpoint记录成功写入的长度值
            for (CommitRecoverable committable: committables) {
                fsWriter.recoverForCommit(committable).commitAfterRecovery();
            }
        }
    }

其中recoverForCommit主要用于创建fscommit对象，用于commit写入hdfs的数据。commitAfterRecovery()就是最终的恢复方法。根据checkpoint中的记录的长度以及hdfs的truncate方法，去重新生成文件。

	private static boolean truncate(final FileSystem hadoopFs, final Path file, final long length) throws IOException {
		if (truncateHandle != null) {
			try {
				return (Boolean) truncateHandle.invoke(hadoopFs, file, length);
			}
			catch (InvocationTargetException e) {
				ExceptionUtils.rethrowIOException(e.getTargetException());
			}
			catch (Throwable t) {
				throw new IOException(
						"Truncation of file failed because of access/linking problems with Hadoop's truncate call. " +
								"This is most likely a dependency conflict or class loading problem.");
			}
		}
		else {
			throw new IllegalStateException("Truncation handle has not been initialized");
		}
		return false;
	}