我的原则:先会用再说,内部慢慢来。
学以致用,根据场景学源码
文章目录
- 一、DirectByteBuffer 架构
- 1.1 代码UML
- 1.2 申请内存Flow图
- 二、DirectByteBuffer 实战 Demo
- 2.1 使用 ByteBuffer.allocateDirect 申请堆外内存
- 2.2 加上 -XX:+DisableExplicitGC 后
- 三、DirectByteBuffer 源码剖析
- 3.1 allocateDirect 方法
- 3.2 构造方法 DirectByteBuffer(int)
- 3.3 reserveMemory 方法
- 3.3.1 tryReserveMemory 方法
- 3.3.2 getJavaLangRefAccess 方法
- 3.3.3 tryHandlePendingReference 方法
- 3.3.4 System.gc()
- 3.3.5 获取内存的9次尝试
- 3.4 allocateMemory 方法
- 3.4.1 Unsafe类
- 3.5 unreserveMemory 方法
- 3.6 构造 Cleaner对象
- 3.6.1 Deallocator 对象
- 3.6.2 freeMemory 方法
- 四、总结
- 4.1 tryHandlePendingReference 的调用场景
- 4.2 堆外缓存的特点
- 4.3 使用堆外内存的原因
- 4.4 对外内存的使用场景
- 五、番外篇
// @VM args:-XX:MaxDirectMemorySize=40m
public class _07_00_TestDirectByteBuffer {
public static void main(String[] args) {
while(true) {
ByteBuffer buffer = ByteBuffer.allocateDirect(1 * 1024 * 1024);
}
}
}
结果: 设置堆内存最大40m,但是代码完全不停歇,毫无对内存满的事情发生。
...
[Full GC (System.gc()) [PSYoungGen: 64K->0K(38400K)] [ParOldGen: 716K->716K(87552K)] 780K->716K(125952K), [Metaspace: 3145K->3145K(1056768K)], 0.0041255 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
[GC (System.gc()) [PSYoungGen: 665K->64K(38400K)] 1382K->780K(125952K), 0.0009704 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
[Full GC (System.gc()) [PSYoungGen: 64K->0K(38400K)] [ParOldGen: 716K->716K(87552K)] 780K->716K(125952K), [Metaspace: 3145K->3145K(1056768K)], 0.0042594 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
[GC (System.gc()) [PSYoungGen: 665K->96K(38400K)] 1382K->812K(125952K), 0.0008948 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
[Full GC (System.gc()) [PSYoungGen: 96K->0K(38400K)] [ParOldGen: 716K->716K(87552K)] 812K->716K(125952K), [Metaspace: 3145K->3145K(1056768K)], 0.0033712 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
[GC (System.gc()) [PSYoungGen: 665K->64K(38400K)] 1382K->780K(125952K), 0.0014349 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
[Full GC (System.gc()) [PSYoungGen: 64K->0K(38400K)] [ParOldGen: 716K->716K(87552K)] 780K->716K(125952K), [Metaspace: 3145K->3145K(1056768K)], 0.0147563 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
Heap
PSYoungGen total 38400K, used 1996K [0x0000000795580000, 0x0000000798000000, 0x00000007c0000000)
eden space 33280K, 6% used [0x0000000795580000,0x0000000795773370,0x0000000797600000)
from space 5120K, 0% used [0x0000000797600000,0x0000000797600000,0x0000000797b00000)
to space 5120K, 0% used [0x0000000797b00000,0x0000000797b00000,0x0000000798000000)
ParOldGen total 87552K, used 716K [0x0000000740000000, 0x0000000745580000, 0x0000000795580000)
object space 87552K, 0% used [0x0000000740000000,0x00000007400b3350,0x0000000745580000)
Metaspace used 3154K, capacity 4496K, committed 4864K, reserved 1056768K
class space used 341K, capacity 388K, committed 512K, reserved 1048576K
输出:
Exception in thread "main" java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:694)
at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at indi.sword.util.basic.reference._07_00_TestDirectByteBuffer.main(_07_00_TestDirectByteBuffer.java:12)
Heap
PSYoungGen total 38400K, used 6017K [0x0000000795580000, 0x0000000798000000, 0x00000007c0000000)
eden space 33280K, 18% used [0x0000000795580000,0x0000000795b60680,0x0000000797600000)
from space 5120K, 0% used [0x0000000797b00000,0x0000000797b00000,0x0000000798000000)
to space 5120K, 0% used [0x0000000797600000,0x0000000797600000,0x0000000797b00000)
ParOldGen total 87552K, used 0K [0x0000000740000000, 0x0000000745580000, 0x0000000795580000)
object space 87552K, 0% used [0x0000000740000000,0x0000000740000000,0x0000000745580000)
Metaspace used 3154K, capacity 4496K, committed 4864K, reserved 1056768K
class space used 343K, capacity 388K, committed 512K, reserved 1048576K
=== 点击查看top目录 ===
public static ByteBuffer allocateDirect(int capacity) {
return new DirectByteBuffer(capacity);
}
=== 点击查看top目录 ===
DirectByteBuffer(int cap) { // package-private
super(-1, 0, cap, cap);
// 计算需要分配的内存大小
boolean pa = VM.isDirectMemoryPageAligned();
int ps = Bits.pageSize();
long size = Math.max(1L, (long)cap + (pa ? ps : 0));
//=== 3.3 告诉内存管理器要分配内存
Bits.reserveMemory(size, cap);
long base = 0;
try {
// 3.4 分配直接内存
base = unsafe.allocateMemory(size);
} catch (OutOfMemoryError x) {
// 3.5 通知 bits 释放内存
Bits.unreserveMemory(size, cap);
throw x;
}
unsafe.setMemory(base, size, (byte) 0);
// 计算内存的地址
if (pa && (base % ps != 0)) {
// Round up to page boundary
address = base + ps - (base & (ps - 1));
} else {
address = base;
}
// 3.6 创建Cleaner!!!! 重点讲这个
cleaner = Cleaner.create(this, new Deallocator(base, size, cap));
att = null;
}
=== 点击查看top目录 ===
static void reserveMemory(long size, int cap) {
if (!memoryLimitSet && VM.isBooted()) {
// 初始化maxMemory,就是使用-XX:MaxDirectMemorySize指定的最大直接内存大小
maxMemory = VM.maxDirectMemory();
memoryLimitSet = true;
}
//=== 3.3.1 第一次先采取最乐观的方式直接尝试告诉Bits要分配内存
if (tryReserveMemory(size, cap)) {
return;
}
// 内存获取失败 往下走。。。
// === 3.3.2 获取 JavaLangRefAccess
final JavaLangRefAccess jlra = SharedSecrets.getJavaLangRefAccess();
// retry while helping enqueue pending Reference objects
// which includes executing pending Cleaner(s) which includes
// Cleaner(s) that free direct buffer memory
/*
tryHandlePendingReference方法:消耗 pending 队列,1. 丢到 Enqueue队列,2. 调用 cleaner.clean() 方法释放内存。
=== 3.3.3 尝试多次获取
失败:tryHandlePendingReference方法的 pending 队列完尽
成功:释放了空间,tryReserveMemory 成功
*/
while (jlra.tryHandlePendingReference()) { // 这个地方返回 false ,也就是 pending 队列完尽,就返回 false
if (tryReserveMemory(size, cap)) {
return;
}
}
// trigger VM's Reference processing
//=== 3.3.4 Full GC 看到没有!!!! System.gc()在这
System.gc();
// a retry loop with exponential back-off delays
// (this gives VM some time to do it's job)
// === 3.3.5 9次循环,不断延迟要求分配内存
boolean interrupted = false;
try {
long sleepTime = 1;
int sleeps = 0;
// 不断*2,按照1ms,2ms,4ms,...,256ms的等待间隔尝试9次分配内存
while (true) {
if (tryReserveMemory(size, cap)) {
return;
}
if (sleeps >= MAX_SLEEPS) { //最多循环9次, final int MAX_SLEEPS = 9;
break;
}
if (!jlra.tryHandlePendingReference()) {
try {
Thread.sleep(sleepTime);
sleepTime <<= 1; // 左边移动一位,也就 * 2
sleeps++;
} catch (InterruptedException e) {
interrupted = true;
}
}
}
// no luck
throw new OutOfMemoryError("Direct buffer memory");
} finally {
if (interrupted) {
// don't swallow interrupts
Thread.currentThread().interrupt();
}
}
}
=== 点击查看top目录 ===
// -XX:MaxDirectMemorySize限制的是总cap,而不是真实的内存使用量,(在页对齐的情况下,真实内存使用量和总cap是不同的)
private static boolean tryReserveMemory(long size, int cap) {
// -XX:MaxDirectMemorySize limits the total capacity rather than the
// actual memory usage, which will differ when buffers are page
// aligned.
long totalCap;
while (cap <= maxMemory - (totalCap = totalCapacity.get())) {
if (totalCapacity.compareAndSet(totalCap, totalCap + cap)) {
reservedMemory.addAndGet(size);
count.incrementAndGet();
return true;
}
}
return false;
}
=== 点击查看top目录 ===
public static JavaLangRefAccess getJavaLangRefAccess() {
return javaLangRefAccess;
}
看下 set 方法的引用(Idea中 Fn + Option + 7 )
原来是存在与 Reference 类的 static 块里面
看下:【JAVA Reference】ReferenceQueue 与 Reference 源码剖析(二)# Reference的static代码块
package sun.misc;
public interface JavaLangRefAccess {
/**
* Help ReferenceHandler thread process next pending
* {@link java.lang.ref.Reference}
*
* @return {@code true} if there was a pending reference and it
* was enqueue-ed or {@code false} if there was no
* pending reference
*/
boolean tryHandlePendingReference();
}
while (jlra.tryHandlePendingReference()) { // 这个地方返回 false ,也就是 pending 队列没有了,就返回 false
if (tryReserveMemory(size, cap)) {
return;
}
}
while 能够把当前全部 pending 队列中的 reference 都消化掉,要么Enqueue,要么Cleaner去进行 clean() 操作。
=== 3.3.3 while 死循环尝试申请内存
tryHandlePendingReference方法:消耗 pending 队列,1. 丢到 Enqueue队列,2. 调用 cleaner.clean() 方法释放内存。
失败:tryHandlePendingReference方法的 pending 队列完尽
成功:释放了空间,tryReserveMemory 成功
只有这样消耗光了 pending,才会往下走 === 3.3.4 System.gc() == ;
=== 点击查看top目录 ===
=== 点击查看top目录 ===
=== 点击查看top目录 ===
public native long allocateMemory(long bytes);
class DirectByteBuffer extends MappedByteBuffer implements DirectBuffer{
// Cached unsafe-access object
protected static final Unsafe unsafe = Bits.unsafe();
}
class Bits {
private static final Unsafe unsafe = Unsafe.getUnsafe();
static Unsafe unsafe() {
return unsafe;
}
}
@CallerSensitive
public static Unsafe getUnsafe() {
Class<?> caller = Reflection.getCallerClass();
if (!VM.isSystemDomainLoader(caller.getClassLoader()))
throw new SecurityException("Unsafe");
return theUnsafe;
}
Java提供了Unsafe类用来进行直接内存的分配与释放
public native long allocateMemory(long var1);
public native void freeMemory(long var1);
=== 点击查看top目录 ===
static void unreserveMemory(long size, int cap) {
long cnt = count.decrementAndGet();
long reservedMem = reservedMemory.addAndGet(-size);
long totalCap = totalCapacity.addAndGet(-cap);
assert cnt >= 0 && reservedMem >= 0 && totalCap >= 0;
}
=== 点击查看top目录 ===
cleaner = Cleaner.create(this, new Deallocator(base, size, cap));
这个create静态方法提供给我们来实例化Cleaner对象,需要两个参数:
Cleaner#create方法「【JAVA Reference】Cleaner 源码剖析(三)」
=== 点击查看top目录 ===
private static class Deallocator
implements Runnable
{
private static Unsafe unsafe = Unsafe.getUnsafe();
private long address;
private long size;
private int capacity;
private Deallocator(long address, long size, int capacity) {
assert (address != 0);
this.address = address;
this.size = size;
this.capacity = capacity;
}
public void run() {
// 堆外内存已经被释放了
if (address == 0) {
// Paranoia
return;
}
// 3.6.2 调用C++代码释放堆外内存
unsafe.freeMemory(address);
address = 0; // 设置为0,表示已经释放了
// 刚刚的 3.5 释放后,标记资源
Bits.unreserveMemory(size, capacity);
}
}
=== 点击查看top目录 ===
/**
* Disposes of a block of native memory, as obtained from {@link
* #allocateMemory} or {@link #reallocateMemory}. The address passed to
* this method may be null, in which case no action is taken.
*
* @see #allocateMemory
*/
public native void freeMemory(long address);
=== 点击查看top目录 ===
同时,还可以使用池+堆外内存 的组合方式,来对生命周期较短,但涉及到I/O操作的对象进行堆外内存的再使用。( Netty中就使用了该方式 )
下一章节:【JAVA Reference】Finalizer 剖析 (六)
上一章节:【JAVA Reference】Cleaner 对比 finalize 对比 AutoCloseable(四)