最近在工作中遇到GC问题,发现GC日志中存在很多[GC pause (GCLocker Initiated GC) (young)的日志,另外还发现特别多的大对象分配:
56123.146: [G1Ergonomics (Concurrent Cycles) request concurrent cycle initiation, reason: occupancy higher than threshold, occupancy: 2415919104 bytes, allocation request: 3718600 bytes, threshold: 2415919095 bytes (45.00 %), source: concurrent humongous allocation]
在排查过程中用到了些排查工具,特记录下备忘;
snappy
项目中使用了snappy做压缩和解压缩,系统的负载很高:
JNIEXPORT jint JNICALL Java_org_xerial_snappy_SnappyNative_rawCompress__Ljava_lang_Object_2IILjava_lang_Object_2I
(JNIEnv * env, jobject self, jobject input, jint inputOffset, jint inputLen, jobject output, jint outputOffset)
{
char* in = (char*) env->GetPrimitiveArrayCritical((jarray) input, 0);
char* out = (char*) env->GetPrimitiveArrayCritical((jarray) output, 0);
if(in == 0 || out == 0) {
// out of memory
if(in != 0) {
env->ReleasePrimitiveArrayCritical((jarray) input, in, 0);
}
if(out != 0) {
env->ReleasePrimitiveArrayCritical((jarray) output, out, 0);
}
throw_exception(env, self, 4);
return 0;
}
size_t compressedLength;
snappy::RawCompress(in + inputOffset, (size_t) inputLen, out + outputOffset, &compressedLength);
env->ReleasePrimitiveArrayCritical((jarray) input, in, 0);
env->ReleasePrimitiveArrayCritical((jarray) output, out, 0);
return (jint) compressedLength;
}
JNI_ENTRY(void*, jni_GetPrimitiveArrayCritical(JNIEnv *env, jarray array, jboolean *isCopy))
JNIWrapper("GetPrimitiveArrayCritical");
GC_locker::lock_critical(thread);
if (isCopy != NULL) {
*isCopy = JNI_FALSE;
}
oop a = JNIHandles::resolve_non_null(array);
BasicType type;
if (a->is_objArray()) {
type = T_OBJECT;
} else {
type = TypeArrayKlass::cast(a->klass())->element_type();
}
void* ret = arrayOop(a)->base(type);
return ret;
JNI_END
JNI_ENTRY(void, jni_ReleasePrimitiveArrayCritical(JNIEnv *env, jarray array, void *carray, jint mode))
JNIWrapper("ReleasePrimitiveArrayCritical");
// The array, carray and mode arguments are ignored
GC_locker::unlock_critical(thread);
JNI_END
可以看到snappy最终使用了GC_locker::lock_critical(thread)和GC_locker::unlock_critical(thread)来进出JNI临界区;
JNI相关日志参数
- verbose:jni 打印JNI调用日志
- PrintJNIGCStalls: 打印线程进出JNI临界区日志
async-profiler
项目地址:https://github.com/jvm-profiling-tools/async-profiler
常用命令:
- ./profiler.sh -d 20 -f 2449-alloc.svg -e alloc 2449
- ./profiler.sh -d 10 -f humongous.jfr -e alloc 2449
对于jfr文件,可以用java自带的jfr命令来进行分析
jfr print humongous.jfr
jfr summary humongous.jfr
jfr print --events jdk.ObjectAllocationOutsideTLAB humongous.jfr