flink分析使用之九通信框架
一、Flink的分布通信模型
flink做为一个分布式的应用，它基于的通信当然是分布式的通信框架。在Flink中，可以分为两种通信方式，一种是通过网络的传输通信，一种是基于本地的数据交换通信。网络通信主要是用来连接节点间的通信（包括客户端和服务端），本地主要是线程内交换一些数据。
本地主要是基于一系列的Oprator来实现的，这个在前面也已经看到过。这次主要分析这个。
二、线程间通信
在前面分析了任务和工作的启动和分发，知道数据流DataSet的整体流转的框架，现在分析一下本地的数据交换方法：
1、线程内
线程内使用Operator做为一种通信手段，它也可以用于线程间的通信。当然，在分布式中，跨线程的通信在上层是不区分的。在前面的学习过程中可以知道，Flink的数据模型其实是分为数据和操作两种类型的。Flink的各种工作和任务就围绕着数据和操作展开不同的工作形式。不过，同一线程内的Operator互相通信有一个优势，不需要进行序列化。直接可以进行数据的操作，它们通过共享的buffer来交换传递数据。比如KeyedStream.java中的函数sum:
public SingleOutputStreamOperator<T> sum(int positionToSum) {
return aggregate(new SumAggregator<>(positionToSum, getType(), getExecutionConfig()));
}
这个经常用于线程内的数据计算。
2、线程间
正如上面所讲，线程间有本地和异地线程间，但在上层其实是会被抽象开来的。但是在线程间的通信，就必须得进行序列化了。它们同样是通过buffer来传递数据，不过在缓冲区没有数据后，需要通过一些方式来得到数据才能进行操作，本地的线程间，在阻塞后，等待新数据的Flush到Buffer，而远程则需要发起RPC请求得到数据再返回给相关的Buffer。而运行不同的线程间的有：
public <R> SingleOutputStreamOperator<R> flatMap(FlatMapFunction<T, R> flatMapper) {
TypeInformation<R> outType = TypeExtractor.getFlatMapReturnTypes(clean(flatMapper),
getType(), Utils.getCallLocationName(), true);
return transform("Flat Map", outType, new StreamFlatMap<>(clean(flatMapper)));
}
当然，其它的如collection等也是经常用于线程间通信的。
三、内存的管理
Flink的优势就在于它的数据计算，数据计算中数据交换的内存管理是一个非常难解决的问题，在前面提到过，Flink没有直接使用JDK的内存管理，而是在其上又进行了一层的抽象，其中有两个重要的数据结构：
1、MemorySegment
@Internal
public abstract class MemorySegment {
/**
* The unsafe handle for transparent memory copied (heap / off-heap).
*/
@SuppressWarnings("restriction")
protected static final sun.misc.Unsafe UNSAFE = MemoryUtils.UNSAFE;
/**
* The beginning of the byte array contents, relative to the byte array object.
*/
@SuppressWarnings("restriction")
protected static final long BYTE_ARRAY_BASE_OFFSET = UNSAFE.arrayBaseOffset(byte[].class);
/**
* Constant that flags the byte order. Because this is a boolean constant, the JIT compiler can
* use this well to aggressively eliminate the non-applicable code paths.
*/
private static final boolean LITTLE_ENDIAN = (ByteOrder.nativeOrder() == ByteOrder.LITTLE_ENDIAN);
protected final byte[] heapMemory;
/**
* The address to the data, relative to the heap memory byte array. If the heap memory byte
* array is null, this becomes an absolute memory address outside the heap.
*/
protected long address;
/**
* The address one byte after the last addressable byte, i.e. address + size while the
* segment is not disposed.
*/
protected final long addressLimit;
/**
* The size in bytes of the memory segment.
*/
protected final int size;
/**
* Optional owner of the memory segment.
*/
private final Object owner;
MemorySegment(byte[] buffer, Object owner) {
if (buffer == null) {
throw new NullPointerException("buffer");
}
this.heapMemory = buffer;
this.address = BYTE_ARRAY_BASE_OFFSET;
this.size = buffer.length;
this.addressLimit = this.address + this.size;
this.owner = owner;
}
MemorySegment(long offHeapAddress, int size, Object owner) {
if (offHeapAddress <= 0) {
throw new IllegalArgumentException("negative pointer or size");
}
if (offHeapAddress >= Long.MAX_VALUE - Integer.MAX_VALUE) {
// this is necessary to make sure the collapsed checks are safe against numeric overflows
throw new IllegalArgumentException("Segment initialized with too large address: " + offHeapAddress
+ " ; Max allowed address is " + (Long.MAX_VALUE - Integer.MAX_VALUE - 1));
}
this.heapMemory = null;
this.address = offHeapAddress;
this.addressLimit = this.address + size;
this.size = size;
this.owner = owner;
}
public int size() {
return size;
}
public boolean isFreed() {
return address > addressLimit;
}
public void free() {
// this ensures we can place no more data and trigger
// the checks for the freed segment
address = addressLimit + 1;
}
public boolean isOffHeap() {
return heapMemory == null;
}
public byte[] getArray() {
if (heapMemory != null) {
return heapMemory;
} else {
throw new IllegalStateException("Memory segment does not represent heap memory");
}
}
public long getAddress() {
if (heapMemory == null) {
return address;
} else {
throw new IllegalStateException("Memory segment does not represent off heap memory");
}
}
public abstract ByteBuffer wrap(int offset, int length);
......
}
//子类之一：分配堆内存
@SuppressWarnings("unused")
@Internal
public final class HeapMemorySegment extends MemorySegment {
private byte[] memory;
HeapMemorySegment(byte[] memory) {
this(memory, null);
}
HeapMemorySegment(byte[] memory, Object owner) {
super(Objects.requireNonNull(memory), owner);
this.memory = memory;
}
@Override
public void free() {
super.free();
this.memory = null;
}
@Override
public ByteBuffer wrap(int offset, int length) {
try {
return ByteBuffer.wrap(this.memory, offset, length);
}
catch (NullPointerException e) {
throw new IllegalStateException("segment has been freed");
}
}
}
//子类之二：即可分配堆内存也可分配非堆内存
@Internal
public final class HybridMemorySegment extends MemorySegment {
private final ByteBuffer offHeapBuffer;
HybridMemorySegment(ByteBuffer buffer) {
this(buffer, null);
}
}
这正好是刚刚提到的Flink自己抽象的内存数据结构。它们一般都有一个wrap()的方法，将自己打包成相关的内存的Buffer。在Flink中重点是第二个，这样可以让JVM优化掉相关的虚表的查找操作。
2、Buffer
Buffer是个什么概念呢？在Flink中，数据交换是taskmanager管理task来进行的，为了提高网络的利用率，把Records的数据存储到Buffer中，是一种非常通用的方式。就和实际的网络通信一样，在各个环节上，都分布着一系列的缓冲区。要注意区分下面的两个BUFFER,一个是JAVA本身的抽象类，一个是FLINK的接口。
//jdk中的buffer
public abstract class Buffer {
static final int SPLITERATOR_CHARACTERISTICS =
Spliterator.SIZED | Spliterator.SUBSIZED | Spliterator.ORDERED;
// Invariants: mark <= position <= limit <= capacity
private int mark = -1;
private int position = 0;
private int limit;
private int capacity;
// Used only by direct buffers
// NOTE: hoisted here for speed in JNI GetDirectBufferAddress
long address;
// Creates a new buffer with the given mark, position, limit, and capacity,
// after checking invariants.
//
Buffer(int mark, int pos, int lim, int cap) { // package-private
if (cap < 0)
throw new IllegalArgumentException("Negative capacity: " + cap);
this.capacity = cap;
limit(lim);
position(pos);
if (mark >= 0) {
if (mark > pos)
throw new IllegalArgumentException("mark > position: ("
+ mark + " > " + pos + ")");
this.mark = mark;
}
}
......
}
//子类--JDK
public abstract class ByteBuffer
extends Buffer
implements Comparable<ByteBuffer>
{
final byte[] hb; // Non-null only for heap buffers
final int offset;
boolean isReadOnly; // Valid only for heap buffers
// Creates a new buffer with the given mark, position, limit, capacity,
// backing array, and array offset
//
ByteBuffer(int mark, int pos, int lim, int cap, // package-private
byte[] hb, int offset)
{
super(mark, pos, lim, cap);
this.hb = hb;
this.offset = offset;
}
// Creates a new buffer with the given mark, position, limit, and capacity
//
ByteBuffer(int mark, int pos, int lim, int cap) { // package-private
this(mark, pos, lim, cap, null, 0);
}
public static ByteBuffer allocateDirect(int capacity) {
return new DirectByteBuffer(capacity);
}
}
//=========================================================================================
//FLINK中的BUFFER
public interface Buffer {
boolean isBuffer();
void tagAsEvent();
@Deprecated
MemorySegment getMemorySegment();
@Deprecated
int getMemorySegmentOffset();
BufferRecycler getRecycler();
void recycleBuffer();
boolean isRecycled();
Buffer retainBuffer();
Buffer readOnlySlice();
Buffer readOnlySlice(int index, int length);
int getMaxCapacity();
int getReaderIndex();
void setReaderIndex(int readerIndex) throws IndexOutOfBoundsException;
int getSizeUnsafe();
int getSize();
void setSize(int writerIndex);
int readableBytes();
ByteBuffer getNioBufferReadable();
ByteBuffer getNioBuffer(int index, int length) throws IndexOutOfBoundsException;
void setAllocator(ByteBufAllocator allocator);
ByteBuf asByteBuf();
}
public interface BufferPoolFactory {
BufferPool createBufferPool(int numRequiredBuffers, int maxUsedBuffers) throws IOException;
BufferPool createBufferPool(int numRequiredBuffers, int maxUsedBuffers, Optional<BufferPoolOwner> owner) throws IOException;
/**
* Destroy callback for updating factory book keeping.
*/
void destroyBufferPool(BufferPool bufferPool) throws IOException;
}
public class NetworkBuffer extends AbstractReferenceCountedByteBuf implements Buffer {
/** The backing {@link MemorySegment} instance. */
private final MemorySegment memorySegment;
/** The recycler for the backing {@link MemorySegment}. */
private final BufferRecycler recycler;
/** Whether this buffer represents a buffer or an event. */
private boolean isBuffer;
/** Allocator for further byte buffers (needed by netty). */
private ByteBufAllocator allocator;
/**
* The current size of the buffer in the range from 0 (inclusive) to the
* size of the backing {@link MemorySegment} (inclusive).
*/
private int currentSize;
/**
* Creates a new buffer instance backed by the given memorySegment with 0 for
* the readerIndex and writerIndex.
*
* @param memorySegment
* backing memory segment (defines {@link #maxCapacity})
* @param recycler
* will be called to recycle this buffer once the reference count is 0
*/
public NetworkBuffer(MemorySegment memorySegment, BufferRecycler recycler) {
this(memorySegment, recycler, true);
}
......
}
public class NetworkBufferPool implements BufferPoolFactory {
private static final Logger LOG = LoggerFactory.getLogger(NetworkBufferPool.class);
private final int totalNumberOfMemorySegments;
private final int memorySegmentSize;
private final ArrayBlockingQueue<MemorySegment> availableMemorySegments;
private volatile boolean isDestroyed;
// ---- Managed buffer pools ----------------------------------------------
private final Object factoryLock = new Object();
private final Set<LocalBufferPool> allBufferPools = new HashSet<>();
private int numTotalRequiredBuffers;
......
}
看到AbstractReferenceCountedByteBuf没，这个里面有一个引用计数器，用来处理内存的GC的。NetworkBufferPool用来生产LocalBufferPool实现内存的池的管理 。
四、数据交换的方式
在Flink中，数据的交换方式主要还是以序列化的方式进行交换，这也是目前分布系统中主流的数据通信方式。在Flink中主要有以下几个类：
//记录序列化器
public interface RecordSerializer<T extends IOReadableWritable> {
/**
* Status of the serialization result.
*/
enum SerializationResult {
PARTIAL_RECORD_MEMORY_SEGMENT_FULL(false, true),
FULL_RECORD_MEMORY_SEGMENT_FULL(true, true),
FULL_RECORD(true, false);
private final boolean isFullRecord;
private final boolean isFullBuffer;
SerializationResult(boolean isFullRecord, boolean isFullBuffer) {
this.isFullRecord = isFullRecord;
this.isFullBuffer = isFullBuffer;
}
/**
* Whether the full record was serialized and completely written to
* a target buffer.
*
* @return true if the complete record was written
*/
public boolean isFullRecord() {
return this.isFullRecord;
}
/**
* Whether the target buffer is full after the serialization process.
*
* @return true if the target buffer is full
*/
public boolean isFullBuffer() {
return this.isFullBuffer;
}
}
/**
* Starts serializing the given record to an intermediate data buffer.
*
* @param record the record to serialize
*/
void serializeRecord(T record) throws IOException;
/**
* Copies the intermediate data serialization buffer to the given target buffer.
*
* @param bufferBuilder the new target buffer to use
* @return how much information was written to the target buffer and
* whether this buffer is full
*/
SerializationResult copyToBufferBuilder(BufferBuilder bufferBuilder);
/**
* Clears the buffer and checks to decrease the size of intermediate data serialization buffer
* after finishing the whole serialization process including
* {@link #serializeRecord(IOReadableWritable)} and {@link #copyToBufferBuilder(BufferBuilder)}.
*/
void prune();
/**
* Supports copying an intermediate data serialization buffer to multiple target buffers
* by resetting its initial position before each copying.
*/
void reset();
/**
* @return true if has some serialized data pending copying to the result {@link BufferBuilder}.
*/
boolean hasSerializedData();
}
//反序列化器
public interface RecordDeserializer<T extends IOReadableWritable> {
/**
* Status of the deserialization result.
*/
enum DeserializationResult {
PARTIAL_RECORD(false, true),
INTERMEDIATE_RECORD_FROM_BUFFER(true, false),
LAST_RECORD_FROM_BUFFER(true, true);
private final boolean isFullRecord;
private final boolean isBufferConsumed;
private DeserializationResult(boolean isFullRecord, boolean isBufferConsumed) {
this.isFullRecord = isFullRecord;
this.isBufferConsumed = isBufferConsumed;
}
public boolean isFullRecord () {
return this.isFullRecord;
}
public boolean isBufferConsumed() {
return this.isBufferConsumed;
}
}
DeserializationResult getNextRecord(T target) throws IOException;
void setNextBuffer(Buffer buffer) throws IOException;
Buffer getCurrentBuffer();
void clear();
boolean hasUnfinishedData();
}
//事件序列化器
public class EventSerializer {
// ------------------------------------------------------------------------
// Constants
// ------------------------------------------------------------------------
private static final int END_OF_PARTITION_EVENT = 0;
private static final int CHECKPOINT_BARRIER_EVENT = 1;
private static final int END_OF_SUPERSTEP_EVENT = 2;
private static final int OTHER_EVENT = 3;
private static final int CANCEL_CHECKPOINT_MARKER_EVENT = 4;
private static final int CHECKPOINT_TYPE_CHECKPOINT = 0;
private static final int CHECKPOINT_TYPE_SAVEPOINT = 1;
// ------------------------------------------------------------------------
// Serialization Logic
// ------------------------------------------------------------------------
public static ByteBuffer toSerializedEvent(AbstractEvent event) throws IOException {
final Class<?> eventClass = event.getClass();
if (eventClass == EndOfPartitionEvent.class) {
return ByteBuffer.wrap(new byte[] { 0, 0, 0, END_OF_PARTITION_EVENT });
}
else if (eventClass == CheckpointBarrier.class) {
return serializeCheckpointBarrier((CheckpointBarrier) event);
}
else if (eventClass == EndOfSuperstepEvent.class) {
return ByteBuffer.wrap(new byte[] { 0, 0, 0, END_OF_SUPERSTEP_EVENT });
}
else if (eventClass == CancelCheckpointMarker.class) {
CancelCheckpointMarker marker = (CancelCheckpointMarker) event;
ByteBuffer buf = ByteBuffer.allocate(12);
buf.putInt(0, CANCEL_CHECKPOINT_MARKER_EVENT);
buf.putLong(4, marker.getCheckpointId());
return buf;
}
else {
try {
final DataOutputSerializer serializer = new DataOutputSerializer(128);
serializer.writeInt(OTHER_EVENT);
serializer.writeUTF(event.getClass().getName());
event.write(serializer);
return serializer.wrapAsByteBuffer();
}
catch (IOException e) {
throw new IOException("Error while serializing event.", e);
}
}
}
......
}
通过这三个数据结构，来组织序列化的过程，在此过程中，需要写入和读取，主要有以下几个类：
//AbstractReader
public abstract class AbstractReader implements ReaderBase {
/** The input gate to read from. */
protected final InputGate inputGate;
/** The task event handler to manage task event subscriptions. */
private final TaskEventHandler taskEventHandler = new TaskEventHandler();
/** Flag indicating whether this reader allows iteration events. */
private boolean isIterative;
/**
* The current number of end of superstep events (reset for each superstep). A superstep is
* finished after an end of superstep event has been received for each input channel.
*/
private int currentNumberOfEndOfSuperstepEvents;
protected AbstractReader(InputGate inputGate) {
this.inputGate = inputGate;
}
@Override
public boolean isFinished() {
return inputGate.isFinished();
}
// ------------------------------------------------------------------------
// Events
// ------------------------------------------------------------------------
@Override
public void registerTaskEventListener(EventListener<TaskEvent> listener, Class<? extends TaskEvent> eventType) {
taskEventHandler.subscribe(listener, eventType);
}
@Override
public void sendTaskEvent(TaskEvent event) throws IOException {
inputGate.sendTaskEvent(event);
}
/**
* Handles the event and returns whether the reader reached an end-of-stream event (either the
* end of the whole stream or the end of an superstep).
*/
protected boolean handleEvent(AbstractEvent event) throws IOException {
final Class<?> eventType = event.getClass();
try {
// ------------------------------------------------------------
// Runtime events
// ------------------------------------------------------------
// This event is also checked at the (single) input gate to release the respective
// channel, at which it was received.
if (eventType == EndOfPartitionEvent.class) {
return true;
}
else if (eventType == EndOfSuperstepEvent.class) {
return incrementEndOfSuperstepEventAndCheck();
}
// ------------------------------------------------------------
// Task events (user)
// ------------------------------------------------------------
else if (event instanceof TaskEvent) {
taskEventHandler.publish((TaskEvent) event);
return false;
}
else {
throw new IllegalStateException("Received unexpected event of type " + eventType + " at reader.");
}
}
catch (Throwable t) {
throw new IOException("Error while handling event of type " + eventType + ": " + t.getMessage(), t);
}
}
public void publish(TaskEvent event){
taskEventHandler.publish(event);
}
// ------------------------------------------------------------------------
// Iterations
// ------------------------------------------------------------------------
@Override
public void setIterativeReader() {
isIterative = true;
}
@Override
public void startNextSuperstep() {
checkState(isIterative, "Tried to start next superstep in a non-iterative reader.");
checkState(currentNumberOfEndOfSuperstepEvents == inputGate.getNumberOfInputChannels(), "Tried to start next superstep before reaching end of previous superstep.");
currentNumberOfEndOfSuperstepEvents = 0;
}
@Override
public boolean hasReachedEndOfSuperstep() {
return isIterative && currentNumberOfEndOfSuperstepEvents == inputGate.getNumberOfInputChannels();
}
private boolean incrementEndOfSuperstepEventAndCheck() {
checkState(isIterative, "Tried to increment superstep count in a non-iterative reader.");
checkState(currentNumberOfEndOfSuperstepEvents + 1 <= inputGate.getNumberOfInputChannels(), "Received too many (" + currentNumberOfEndOfSuperstepEvents + ") end of superstep events.");
return ++currentNumberOfEndOfSuperstepEvents == inputGate.getNumberOfInputChannels();
}
}
//RecordReader：不可变读取记录器
public class RecordReader<T extends IOReadableWritable> extends AbstractRecordReader<T> implements Reader<T> {
private final Class<T> recordType;
private T currentRecord;
/**
* Creates a new RecordReader that de-serializes records from the given input gate and
* can spill partial records to disk, if they grow large.
*
* @param inputGate The input gate to read from.
* @param tmpDirectories The temp directories. USed for spilling if the reader concurrently
* reconstructs multiple large records.
*/
public RecordReader(InputGate inputGate, Class<T> recordType, String[] tmpDirectories) {
super(inputGate, tmpDirectories);
this.recordType = recordType;
}
@Override
public boolean hasNext() throws IOException, InterruptedException {
if (currentRecord != null) {
return true;
}
else {
T record = instantiateRecordType();
if (getNextRecord(record)) {
currentRecord = record;
return true;
}
else {
return false;
}
}
}
......
}
//MutableRecordReader：可变读取记录器
public class MutableRecordReader<T extends IOReadableWritable> extends AbstractRecordReader<T> implements MutableReader<T> {
/**
* Creates a new MutableRecordReader that de-serializes records from the given input gate and
* can spill partial records to disk, if they grow large.
*
* @param inputGate The input gate to read from.
* @param tmpDirectories The temp directories. USed for spilling if the reader concurrently
* reconstructs multiple large records.
*/
public MutableRecordReader(InputGate inputGate, String[] tmpDirectories) {
super(inputGate, tmpDirectories);
}
@Override
public boolean next(final T target) throws IOException, InterruptedException {
return getNextRecord(target);
}
@Override
public void clearBuffers() {
super.clearBuffers();
}
}
//RecordWriter：记录写入器
public class RecordWriter<T extends IOReadableWritable> {
private static final Logger LOG = LoggerFactory.getLogger(RecordWriter.class);
private final ResultPartitionWriter targetPartition;
private final ChannelSelector<T> channelSelector;
private final int numberOfChannels;
private final int[] broadcastChannels;
private final RecordSerializer<T> serializer;
private final Optional<BufferBuilder>[] bufferBuilders;
private final Random rng = new XORShiftRandom();
private Counter numBytesOut = new SimpleCounter();
private Counter numBuffersOut = new SimpleCounter();
private final boolean flushAlways;
/** Default name for teh output flush thread, if no name with a task reference is given. */
private static final String DEFAULT_OUTPUT_FLUSH_THREAD_NAME = "OutputFlusher";
/** The thread that periodically flushes the output, to give an upper latency bound. */
private final Optional<OutputFlusher> outputFlusher;
/** To avoid synchronization overhead on the critical path, best-effort error tracking is enough here.*/
private Throwable flusherException;
public RecordWriter(ResultPartitionWriter writer) {
this(writer, new RoundRobinChannelSelector<T>(), -1, null);
}
public RecordWriter(
ResultPartitionWriter writer,
ChannelSelector<T> channelSelector,
long timeout,
String taskName) {
this.targetPartition = writer;
this.channelSelector = channelSelector;
this.numberOfChannels = writer.getNumberOfSubpartitions();
this.channelSelector.setup(numberOfChannels);
this.serializer = new SpanningRecordSerializer<T>();
this.bufferBuilders = new Optional[numberOfChannels];
this.broadcastChannels = new int[numberOfChannels];
for (int i = 0; i < numberOfChannels; i++) {
broadcastChannels[i] = i;
bufferBuilders[i] = Optional.empty();
}
checkArgument(timeout >= -1);
this.flushAlways = (timeout == 0);
if (timeout == -1 || timeout == 0) {
outputFlusher = Optional.empty();
} else {
String threadName = taskName == null ?
DEFAULT_OUTPUT_FLUSH_THREAD_NAME :
DEFAULT_OUTPUT_FLUSH_THREAD_NAME + " for " + taskName;
outputFlusher = Optional.of(new OutputFlusher(threadName, timeout));
outputFlusher.get().start();
}
}
public void emit(T record) throws IOException, InterruptedException {
checkErroneous();
emit(record, channelSelector.selectChannel(record));
}
/**
* This is used to broadcast Streaming Watermarks in-band with records. This ignores
* the {@link ChannelSelector}.
*/
public void broadcastEmit(T record) throws IOException, InterruptedException {
checkErroneous();
serializer.serializeRecord(record);
boolean pruneAfterCopying = false;
for (int channel : broadcastChannels) {
if (copyFromSerializerToTargetChannel(channel)) {
pruneAfterCopying = true;
}
}
// Make sure we don't hold onto the large intermediate serialization buffer for too long
if (pruneAfterCopying) {
serializer.prune();
}
}
......
}
//ResultPartitionWriter：分区结果写入器
public interface ResultPartitionWriter {
BufferProvider getBufferProvider();
ResultPartitionID getPartitionId();
int getNumberOfSubpartitions();
int getNumTargetKeyGroups();
void addBufferConsumer(BufferConsumer bufferConsumer, int subpartitionIndex) throws IOException;
/**
* Manually trigger consumption from enqueued {@link BufferConsumer BufferConsumers} in all subpartitions.
*/
void flushAll();
/**
* Manually trigger consumption from enqueued {@link BufferConsumer BufferConsumers} in one specified subpartition.
*/
void flush(int subpartitionIndex);
}
@ThreadSafe
public abstract class AbstractCollectingResultPartitionWriter implements ResultPartitionWriter {
private final BufferProvider bufferProvider;
private final ArrayDeque<BufferConsumer> bufferConsumers = new ArrayDeque<>();
public AbstractCollectingResultPartitionWriter(BufferProvider bufferProvider) {
this.bufferProvider = checkNotNull(bufferProvider);
}
@Override
public BufferProvider getBufferProvider() {
return bufferProvider;
}
@Override
public ResultPartitionID getPartitionId() {
return new ResultPartitionID();
}
@Override
public int getNumberOfSubpartitions() {
return 1;
}
@Override
public int getNumTargetKeyGroups() {
return 1;
}
@Override
public synchronized void addBufferConsumer(BufferConsumer bufferConsumer, int targetChannel) throws IOException {
checkState(targetChannel < getNumberOfSubpartitions());
bufferConsumers.add(bufferConsumer);
processBufferConsumers();
}
private void processBufferConsumers() throws IOException {
while (!bufferConsumers.isEmpty()) {
BufferConsumer bufferConsumer = bufferConsumers.peek();
Buffer buffer = bufferConsumer.build();
try {
deserializeBuffer(buffer);
if (!bufferConsumer.isFinished()) {
break;
}
bufferConsumers.pop().close();
}
finally {
buffer.recycleBuffer();
}
}
}
@Override
public synchronized void flushAll() {
try {
processBufferConsumers();
} catch (IOException e) {
throw new RuntimeException(e);
}
}
@Override
public void flush(int subpartitionIndex) {
flushAll();
}
protected abstract void deserializeBuffer(Buffer buffer) throws IOException;
}
通过这几个类的互相组合，就可以完成数据的读写操作。其实应该画几个类图和流程图来看可能会更方便清楚一些，不过那个太占地儿了，而且在Idea中可以自动生成，就不再传上来了。
五、总结
通过上述的分析，对Flink中的分布式通信框架就有一个整体上的宏观认识，注重细节，并不是说沉溺于细节，整体把握，具体分析。这才是学习别人的优秀的设计思想的好的办法。
