Spark 内存管理之 MemoryConsumer

MemoryConsumer 是内存的使用者，如 sort, shuffle。MemoryConsumer 的 spill 方法是抽象方法，由子类提供。
MemoryConsumer 的字段和构造方法如下：

public abstract class MemoryConsumer {

  protected final TaskMemoryManager taskMemoryManager;
  private final long pageSize;
  private final MemoryMode mode;
  protected long used;

  protected MemoryConsumer(TaskMemoryManager taskMemoryManager, long pageSize, MemoryMode mode) {
    this.taskMemoryManager = taskMemoryManager;
    this.pageSize = pageSize;
    this.mode = mode;
  }

spill

释放内存

  /*
   * @param size the amount of memory should be released
   * @param trigger the MemoryConsumer that trigger this spilling
   * @return the amount of released memory in bytes
   */
public abstract long spill(long size, MemoryConsumer trigger) throws IOException;

分配内存

分配内存有3种，allocateArray 和 allocatePage 包括具体的占用这些内存的对象。acquireMemory 仅分配指定大小的内存。

public LongArray allocateArray(long size) {
    long required = size * 8L;
    MemoryBlock page = taskMemoryManager.allocatePage(required, this);
    if (page == null || page.size() < required) {
      throwOom(page, required);
    }
    used += required;
    return new LongArray(page);
  }

释放内存

同理，释放内存有 freeArray, freePage 和 freeMemory。

MemoryConsumer 的子类

BytesToBytesMap
HybridRowQueue
LongToUnsafeRowMap