jmeter优化记录文档
文章目录
- jmeter优化记录文档
- jvm 优化参考文档
- jmeter 优化
- 系统相关信息:
- 系统环境:
- 硬件信息:
- 系统参数调整
- jmeter内存溢出
- 错误日志如下
- 情况分析
- 解决方案
- jmeter第二次内存溢出
- 错误日志信息
- 日志分析
- 解决方案
- jmeter第三次内存溢出
- 错误日志信息
- jvm参数调整
- jmeter第四次内存溢出
- 错误日志信息
- jvm参数调整
- jmeter脚本优化
- jmeter第五次内存溢出
- 错误日志信息
- 情况分析
- jvm参数调整
- jmeter第六次报错, 不是内存溢出
- 错误日志信息
- 分析
- 解决方案
- jmeter第七次报错, 栈内存至少228k
- 错误日志信息
- 分析
- jvm调整
- jmeter第八次报错, 栈内存不足
- 错误日志信息
- 日志分析
- 解决方案
- 系统内存设置为16G, 最大线程数还是32732问题
- 错误日志信息
- 错误分析
- 解决方案
- 系统内存16G, 跑出来的最大线程数量: 426551
- 系统内存8G, 跑出来的最大线程数量: 221308
- jmeter跑4w长连接, 运行数据整理
- 其它问题
- 线程栈大小是260k, 与-Xss256k不一致, 多了4k
- 线程栈内存的实际大小计算
jmeter优化记录文档
jvm 优化参考文档
JVM调优总结
jmeter 优化
系统相关信息:
系统环境:
[root@ts ~]# cat /etc/redhat-release
CentOS Linux release 7.6.1810 (Core)
硬件信息:
cpu信息:
cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c
4 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
内存: 8G
磁盘:500G
系统参数调整
jmeter内存溢出
错误日志如下
jmeter控制台日志, 告诉你OutOfMemoryError, 详细情况请看 hs_err_pid42012.log
Creating summariser
Created the tree successfully using iotserver.jmx
Starting the test @ Mon Jul 22 18:23:56 CST 2019 (1563791036039)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
summary + 2450 in 00:00:03 = 730.5/s Avg: 1 Min: 0 Max: 50 Err: 0 (0.00%) Active: 2452 Started: 2452 Finished: 0
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to protect stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Uncaught Exception java.lang.OutOfMemoryError: unable to create new native thread. See log file for details.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f6165bc0000, 65536, 1) failed; error='无法分配内存' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /data/apache-jmeter-5.1.1/bin/hs_err_pid42012.log
#
# Compiler replay data is saved as:
# /data/apache-jmeter-5.1.1/bin/replay_pid42012.log
jvm输出日志: hs_err_pid42012.log部分内容如下:
这个log里头的数据贼多, 包含进程, 线程, jvm堆栈信息, system信息, 动态链接库(Dynamic libraries, 这是干嘛用的…)
嗯, 非常的详细
[root@localhost bin]# cat hs_err_pid42012.log | more
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2743), pid=42012, tid=0x00007f6161311700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_191-b12) (build 1.8.0_191-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode linux-amd64 compressed oops)
# Core dump written. Default location: /data/apache-jmeter-5.1.1/bin/core or core.42012
#
--------------- T H R E A D ---------------
Current thread (0x00007f6174180000): JavaThread "C2 CompilerThread0" daemon [_thread_in_vm, id=42022, stack(0x00007f6161212000,0x00007f6161312000)]
Stack: [0x00007f6161212000,0x00007f6161312000], sp=0x00007f616130d620, free space=1005k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xace425] VMError::report_and_die()+0x2c5
V [libjvm.so+0x4deb77] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x67
V [libjvm.so+0x90c570] os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x100
V [libjvm.so+0x903eaf] os::commit_memory(char*, unsigned long, unsigned long, bool)+0x1f
V [libjvm.so+0xacaec0] VirtualSpace::expand_by(unsigned long, bool)+0x140
V [libjvm.so+0x608fdd] CodeHeap::expand_by(unsigned long)+0x6d
V [libjvm.so+0x459806] CodeCache::allocate(int, bool)+0x46
V [libjvm.so+0x45351a] BufferBlob::create(char const*, int)+0x8a
V [libjvm.so+0x457eeb] CodeBuffer::initialize(int, int)+0x1b
V [libjvm.so+0x924977] Compile::init_buffer(unsigned int*)+0x1c7
V [libjvm.so+0x929a72] Compile::Output()+0x312
V [libjvm.so+0x47e200] Compile::Code_Gen()+0x2d0
V [libjvm.so+0x47f7f1] Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool)+0xd91
V [libjvm.so+0x3c461a] C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0xaa
V [libjvm.so+0x48ad7c] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x73c
V [libjvm.so+0x48c998] CompileBroker::compiler_thread_loop()+0x6d8
V [libjvm.so+0xa76f9b] JavaThread::thread_main_inner()+0xdb
V [libjvm.so+0xa7722f] JavaThread::run()+0x25f
V [libjvm.so+0x909c92] java_start(Thread*)+0x102
C [libpthread.so.0+0x7dd5] start_thread+0xc5
Current CompileTask:
C2: 19736 3292 4 org.apache.logging.log4j.core.pattern.ExtendedThrowablePatternConverter::format (117 bytes)
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00007f58e4472000 JavaThread "线程组2 2-12624" [_thread_blocked, id=74670, stack(0x00007f58c28eb000,0x00007f58c29ec000)]
0x00007f58e4470000 JavaThread "线程组2 2-12623" [_thread_blocked, id=74669, stack(0x00007f58c29ec000,0x00007f58c2aed000)]
0x00007f58e446e000 JavaThread "线程组2 2-12622" [_thread_blocked, id=74668, stack(0x00007f58c2aed000,0x00007f58c2bee000)]
0x00007f58e446c000 JavaThread "线程组2 2-12621" [_thread_blocked, id=74667, stack(0x00007f58c2bee000,0x00007f58c2cef000)]
0x00007f58e4469800 JavaThread "线程组2 2-12620" [_thread_blocked, id=74666, stack(0x00007f58c2cef000,0x00007f58c2df0000)]
0x00007f58e4467800 JavaThread "线程组2 2-12619" [_thread_blocked, id=74665, stack(0x00007f58c2df0000,0x00007f58c2ef1000)]
0x00007f58e4465800 JavaThread "线程组2 2-12618" [_thread_blocked, id=74664, stack(0x00007f58c2ef1000,0x00007f58c2ff2000)]
0x00007f58e4463800 JavaThread "线程组2 2-12617" [_thread_blocked, id=74663, stack(0x00007f58c2ff2000,0x00007f58c30f3000)]
0x00007f58e4461800 JavaThread "线程组2 2-12616" [_thread_blocked, id=74662, stack(0x00007f58c30f3000,0x00007f58c31f4000)]
0x00007f58e445f800 JavaThread "线程组2 2-12615" [_thread_blocked, id=74661, stack(0x00007f58c31f4000,0x00007f58c32f5000)]
0x00007f58e445d800 JavaThread "线程组2 2-12614" [_thread_blocked, id=74660, stack(0x00007f58c32f5000,0x00007f58c33f6000)]
0x00007f58e445b800 JavaThread "线程组2 2-12613" [_thread_blocked, id=74659, stack(0x00007f58c33f6000,0x00007f58c34f7000)]
0x00007f58e4459800 JavaThread "线程组2 2-12612" [_thread_blocked, id=74658, stack(0x00007f58c34f7000,0x00007f58c35f8000)]
0x00007f58e4457800 JavaThread "线程组2 2-12611" [_thread_blocked, id=74657, stack(0x00007f58c35f8000,0x00007f58c36f9000)]
...
...
...
下面的这段gc日志, 应该是jvm内存溢出时的详细情况:
Other Threads:
0x00007f617413f000 VMThread [stack: 0x00007f6161616000,0x00007f6161716000] [id=42018]
0x00007f6174194800 WatcherThread [stack: 0x00007f6160e0e000,0x00007f6160f0e000] [id=42026]
VM state:synchronizing (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00007f6174006570] CodeCache_lock - owner thread: 0x00007f6174180000
[0x00007f61740071f0] Threads_lock - owner thread: 0x00007f617413f000
Heap:
PSYoungGen total 1835008K, used 1101341K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 1572864K, 53% used [0x0000000740000000,0x000000077338f438,0x00000007a0000000)
from space 262144K, 99% used [0x00000007a0000000,0x00000007afff8010,0x00000007b0000000)
to space 262144K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007c0000000)
ParOldGen total 4194304K, used 364094K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000)
object space 4194304K, 8% used [0x0000000640000000,0x000000065638fab0,0x0000000740000000)
Metaspace used 22383K, capacity 23006K, committed 23344K, reserved 1069056K
class space used 2714K, capacity 2996K, committed 3072K, reserved 1048576K
Card table byte_map: [0x00007f61643ff000,0x00007f6165000000] byte_map_base: 0x00007f61611ff000
Marking Bits: (ParMarkBitMap*) 0x00007f617a5e5d40
Begin Bits: [0x00007f6140000000, 0x00007f6146000000)
End Bits: [0x00007f6146000000, 0x00007f614c000000)
Polling page: 0x00007f617b227000
CodeCache: size=245760Kb used=11969Kb max_used=11973Kb free=233790Kb
bounds [0x00007f6165000000, 0x00007f6165bc0000, 0x00007f6174000000]
total_blobs=3738 nmethods=3270 adapters=379
compilation: enabled
Compilation events (10 events):
Event: 16.467 Thread 0x00007f6174184800 nmethod 3287 0x00007f6165ba2d50 code [0x00007f6165ba2f00, 0x00007f6165ba3418]
Event: 16.510 Thread 0x00007f6174184800 3288 3 sun.nio.cs.StreamEncoder::writeBytes (132 bytes)
Event: 16.511 Thread 0x00007f6174184800 nmethod 3288 0x00007f6165baf790 code [0x00007f6165baf9a0, 0x00007f6165bb0618]
Event: 16.733 Thread 0x00007f6174184800 3289 s 3 java.io.BufferedOutputStream::write (67 bytes)
Event: 16.735 Thread 0x00007f6174184800 nmethod 3289 0x00007f6165ba1fd0 code [0x00007f6165ba21a0, 0x00007f6165ba29f8]
Event: 18.274 Thread 0x00007f6174183000 3290 4 java.nio.ByteBuffer::arrayOffset (35 bytes)
Event: 18.275 Thread 0x00007f6174183000 nmethod 3290 0x00007f6165acf7d0 code [0x00007f6165acf920, 0x00007f6165acf9b8]
Event: 19.543 Thread 0x00007f6174184800 3291 3 org.apache.logging.log4j.core.impl.MutableLogEvent::getThrownProxy (34 bytes)
Event: 19.544 Thread 0x00007f6174184800 nmethod 3291 0x00007f6165baefd0 code [0x00007f6165baf140, 0x00007f6165baf448]
Event: 19.568 Thread 0x00007f6174180000 3292 4 org.apache.logging.log4j.core.pattern.ExtendedThrowablePatternConverter::format (117 bytes)
GC Heap History (10 events):
Event: 1.983 GC heap before
{Heap before GC invocations=3 (full 1):
PSYoungGen total 1835008K, used 94371K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 1572864K, 6% used [0x0000000740000000,0x0000000745c28f78,0x00000007a0000000)
from space 262144K, 0% used [0x00000007a0000000,0x00000007a0000000,0x00000007b0000000)
to space 262144K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007c0000000)
ParOldGen total 4194304K, used 13439K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000)
object space 4194304K, 0% used [0x0000000640000000,0x0000000640d1ff28,0x0000000740000000)
Metaspace used 21128K, capacity 21708K, committed 21936K, reserved 1069056K
class space used 2656K, capacity 2893K, committed 2944K, reserved 1048576K
Event: 1.986 GC heap after
Heap after GC invocations=3 (full 1):
PSYoungGen total 1835008K, used 762K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 1572864K, 0% used [0x0000000740000000,0x0000000740000000,0x00000007a0000000)
from space 262144K, 0% used [0x00000007b0000000,0x00000007b00be820,0x00000007c0000000)
to space 262144K, 0% used [0x00000007a0000000,0x00000007a0000000,0x00000007b0000000)
ParOldGen total 4194304K, used 13439K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000)
object space 4194304K, 0% used [0x0000000640000000,0x0000000640d1ff28,0x0000000740000000)
Metaspace used 21128K, capacity 21708K, committed 21936K, reserved 1069056K
class space used 2656K, capacity 2893K, committed 2944K, reserved 1048576K
}
Event: 1.986 GC heap before
{Heap before GC invocations=4 (full 2):
PSYoungGen total 1835008K, used 762K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 1572864K, 0% used [0x0000000740000000,0x0000000740000000,0x00000007a0000000)
from space 262144K, 0% used [0x00000007b0000000,0x00000007b00be820,0x00000007c0000000)
to space 262144K, 0% used [0x00000007a0000000,0x00000007a0000000,0x00000007b0000000)
ParOldGen total 4194304K, used 13439K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000)
object space 4194304K, 0% used [0x0000000640000000,0x0000000640d1ff28,0x0000000740000000)
Metaspace used 21128K, capacity 21708K, committed 21936K, reserved 1069056K
class space used 2656K, capacity 2893K, committed 2944K, reserved 1048576K
Event: 2.064 GC heap after
Heap after GC invocations=4 (full 2):
PSYoungGen total 1835008K, used 0K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 1572864K, 0% used [0x0000000740000000,0x0000000740000000,0x00000007a0000000)
from space 262144K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007c0000000)
to space 262144K, 0% used [0x00000007a0000000,0x00000007a0000000,0x00000007b0000000)
ParOldGen total 4194304K, used 6103K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000)
object space 4194304K, 0% used [0x0000000640000000,0x00000006405f5c20,0x0000000740000000)
Metaspace used 21128K, capacity 21708K, committed 21936K, reserved 1069056K
class space used 2656K, capacity 2893K, committed 2944K, reserved 1048576K
}
Event: 2.378 GC heap before
{Heap before GC invocations=5 (full 2):
PSYoungGen total 1835008K, used 1572864K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 1572864K, 100% used [0x0000000740000000,0x00000007a0000000,0x00000007a0000000)
from space 262144K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007c0000000)
to space 262144K, 0% used [0x00000007a0000000,0x00000007a0000000,0x00000007b0000000)
ParOldGen total 4194304K, used 6103K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000)
object space 4194304K, 0% used [0x0000000640000000,0x00000006405f5c20,0x0000000740000000)
Metaspace used 21828K, capacity 22454K, committed 22832K, reserved 1069056K
class space used 2707K, capacity 2964K, committed 3072K, reserved 1048576K
Event: 2.386 GC heap after
Heap after GC invocations=5 (full 2):
PSYoungGen total 1835008K, used 4977K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 1572864K, 0% used [0x0000000740000000,0x0000000740000000,0x00000007a0000000)
from space 262144K, 1% used [0x00000007a0000000,0x00000007a04dc5a0,0x00000007b0000000)
to space 262144K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007c0000000)
ParOldGen total 4194304K, used 6111K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000)
object space 4194304K, 0% used [0x0000000640000000,0x00000006405f7c20,0x0000000740000000)
Metaspace used 21828K, capacity 22454K, committed 22832K, reserved 1069056K
class space used 2707K, capacity 2964K, committed 3072K, reserved 1048576K
}
Event: 5.465 GC heap before
{Heap before GC invocations=6 (full 2):
PSYoungGen total 1835008K, used 1577841K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 1572864K, 100% used [0x0000000740000000,0x00000007a0000000,0x00000007a0000000)
from space 262144K, 1% used [0x00000007a0000000,0x00000007a04dc5a0,0x00000007b0000000)
to space 262144K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007c0000000)
ParOldGen total 4194304K, used 6111K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000)
object space 4194304K, 0% used [0x0000000640000000,0x00000006405f7c20,0x0000000740000000)
Metaspace used 22335K, capacity 22974K, committed 23344K, reserved 1069056K
class space used 2707K, capacity 2964K, committed 3072K, reserved 1048576K
Event: 5.533 GC heap after
Heap after GC invocations=6 (full 2):
PSYoungGen total 1835008K, used 82272K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 1572864K, 0% used [0x0000000740000000,0x0000000740000000,0x00000007a0000000)
from space 262144K, 31% used [0x00000007b0000000,0x00000007b5058020,0x00000007c0000000)
to space 262144K, 0% used [0x00000007a0000000,0x00000007a0000000,0x00000007b0000000)
ParOldGen total 4194304K, used 6119K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000)
object space 4194304K, 0% used [0x0000000640000000,0x00000006405f9c20,0x0000000740000000)
Metaspace used 22335K, capacity 22974K, committed 23344K, reserved 1069056K
class space used 2707K, capacity 2964K, committed 3072K, reserved 1048576K
}
Event: 14.387 GC heap before
{Heap before GC invocations=7 (full 2):
PSYoungGen total 1835008K, used 1655136K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 1572864K, 100% used [0x0000000740000000,0x00000007a0000000,0x00000007a0000000)
from space 262144K, 31% used [0x00000007b0000000,0x00000007b5058020,0x00000007c0000000)
to space 262144K, 0% used [0x00000007a0000000,0x00000007a0000000,0x00000007b0000000)
ParOldGen total 4194304K, used 6119K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000)
object space 4194304K, 0% used [0x0000000640000000,0x00000006405f9c20,0x0000000740000000)
Metaspace used 22364K, capacity 22974K, committed 23344K, reserved 1069056K
class space used 2712K, capacity 2964K, committed 3072K, reserved 1048576K
Event: 15.201 GC heap after
Heap after GC invocations=7 (full 2):
PSYoungGen total 1835008K, used 262112K [0x0000000740000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 1572864K, 0% used [0x0000000740000000,0x0000000740000000,0x00000007a0000000)
from space 262144K, 99% used [0x00000007a0000000,0x00000007afff8010,0x00000007b0000000)
to space 262144K, 0% used [0x00000007b0000000,0x00000007b0000000,0x00000007c0000000)
ParOldGen total 4194304K, used 364094K [0x0000000640000000, 0x0000000740000000, 0x0000000740000000)
object space 4194304K, 8% used [0x0000000640000000,0x000000065638fab0,0x0000000740000000)
Metaspace used 22364K, capacity 22974K, committed 23344K, reserved 1069056K
class space used 2712K, capacity 2964K, committed 3072K, reserved 1048576K
}
...
...
...
VM Arguments:
jvm_args: -Xms6g -Xmx6g -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -Xloggc:../log/gc-%p.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=10G
java_command: ./ApacheJMeter.jar -n -t iotserver.jmx -l is-log.jtl
java_class_path (initial): ./ApacheJMeter.jar
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/usr/local/java/jdk1.8
CLASSPATH=.:/data/apache-jmeter-5.1.1/lib/ext/ApacheJMeter_core.jar:/data/apache-jmeter-5.1.1/lib/jorphan.jar
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/java/jdk1.8/bin:/data/apache-jmeter-5.1.1/bin:/root/bin
SHELL=/bin/bash
Signal Handlers:
SIGSEGV: [libjvm.so+0xaced60], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xaced60], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0x907ca0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0x907ca0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0x907ca0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0x907ca0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR1: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x907b70], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: [libjvm.so+0x908190], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGINT: [libjvm.so+0x908190], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGTERM: [libjvm.so+0x908190], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.so+0x908190], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
--------------- S Y S T E M ---------------
OS:CentOS Linux release 7.6.1810 (Core)
uname:Linux 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64
libc:glibc 2.17 NPTL 2.17
rlimit: STACK 8192k, CORE infinity, NPROC 1048576, NOFILE 1048576, AS infinity
load average:0.57 0.22 0.17
/proc/meminfo:
MemTotal: 8009248 kB
MemFree: 2596300 kB
MemAvailable: 2665208 kB
Buffers: 3140 kB
Cached: 272720 kB
SwapCached: 0 kB
Active: 4032420 kB
Inactive: 145268 kB
Active(anon): 3902272 kB
Inactive(anon): 8660 kB
Active(file): 130148 kB
Inactive(file): 136608 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 8257532 kB
SwapFree: 8257532 kB
Dirty: 17044 kB
Writeback: 0 kB
AnonPages: 3901932 kB
Mapped: 38752 kB
Shmem: 9064 kB
Slab: 378928 kB
SReclaimable: 47160 kB
SUnreclaim: 331768 kB
KernelStack: 525584 kB
PageTables: 75400 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 12262156 kB
Committed_AS: 40427472 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 160740 kB
VmallocChunk: 34359341052 kB
HardwareCorrupted: 0 kB
AnonHugePages: 2328576 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 96192 kB
DirectMap2M: 8292352 kB
container (cgroup) information:
container_type: cgroupv1
cpu_cpuset_cpus: 0-3
cpu_memory_nodes: 0
active_processor_count: 4
cpu_quota: -1
cpu_period: 100000
cpu_shares: -1
memory_limit_in_bytes: -1
memory_and_swap_limit_in_bytes: -1
memory_soft_limit_in_bytes: -1
memory_usage_in_bytes: 4278185984
memory_max_usage_in_bytes: 0
CPU:total 4 (initial active 4) (1 cores per cpu, 1 threads per core) family 6 model 45 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, tsc, tscinvbit, tscinv
/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
stepping : 7
microcode : 0x70d
cpu MHz : 2300.000
cache size : 15360 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 cx16 pcid s
se4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm epb tsc_adjust dtherm arat pln pts
bogomips : 4600.00
clflush size : 64
cache_alignment : 64
address sizes : 43 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
stepping : 7
microcode : 0x70d
cpu MHz : 2300.000
cache size : 15360 KB
physical id : 2
siblings : 1
core id : 0
cpu cores : 1
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 cx16 pcid s
se4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm epb tsc_adjust dtherm arat pln pts
bogomips : 4600.00
clflush size : 64
cache_alignment : 64
address sizes : 43 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
stepping : 7
microcode : 0x70d
cpu MHz : 2300.000
cache size : 15360 KB
physical id : 4
siblings : 1
core id : 0
cpu cores : 1
apicid : 4
initial apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 cx16 pcid s
se4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm epb tsc_adjust dtherm arat pln pts
bogomips : 4600.00
clflush size : 64
cache_alignment : 64
address sizes : 43 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
stepping : 7
microcode : 0x70d
cpu MHz : 2300.000
cache size : 15360 KB
physical id : 6
siblings : 1
core id : 0
cpu cores : 1
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 cx16 pcid s
se4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx hypervisor lahf_lm epb tsc_adjust dtherm arat pln pts
bogomips : 4600.00
clflush size : 64
cache_alignment : 64
address sizes : 43 bits physical, 48 bits virtual
power management:
Memory: 4k page, physical 8009248k(2596300k free), swap 8257532k(8257532k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct 6 2018 05:43:09 by "java_re" with gcc 7.3.0
time: Mon Jul 22 18:24:15 2019
elapsed time: 21 seconds (0d 0h 0m 21s)
情况分析
主要是分析jvm 堆栈日志
- jvm给年轻代分了2G内存, 年老代分了4G内存, 持久代分了1G
- jmeter创建了大量的临时对象, 在gc时被清理掉,如: (eden space 1572864K, 100%) --> (from space 262144K, 31%), 从1.5G 优化到 79M
- 年轻代中的Eden 与两个Survivor(from space, to space) 内存比例是3 : 1, Eden(1,572,864K)占75%, 两个Survivor(262144K * 2)占25%, 一个Survivor占1/8
- 报出OutOfMemoryError时, 年轻代中Eden(1,572,864K)使用53%, from space(262144K)使用99%, 年老代(4194304K)使用8%
- 报出OutOfMemoryError可能是因为from space没有足够的内存
- 年老代中生命周期比较长的对象较少
下面的使用目前使用的jvm参数
jvm_args: -Xms6g -Xmx6g -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -Xloggc:../log/gc-%p.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=100 -XX:GCLogFileSize=10G
这是jmeter的启动命令: ./ApacheJMeter.jar -n -t iotserver.jmx -l is-log.jtl
解决方案
主要是根据上面的情况分析, 对jvm参数做优化
- 增加年轻代内存大小.
-XX:NewRatio=1: 设置年轻代（包括Eden和两个Survivor区）与年老代的比值（除去持久代）. 设置为1, 表示 1:1 - 增加标记次数, 减少年轻代对象进入年老代的数量
-XX:MaxTenuringThreshold=4: 设置垃圾最大年龄。如果设置为0的话，则年轻代对象不经过Survivor区，直接进入年老代。设置为4, 表示要被标记4次, 第5次才能进入年老代 - 修改持久代为64M
-XX:MaxPermSize=64m
优化之后的jvm参数
jvm_args:
-Xms6g
-Xmx6g
-XX:NewRatio=1
-XX:MaxTenuringThreshold=4
-XX:MaxPermSize=64m
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-Xloggc:../log/gc-%p.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=100
-XX:GCLogFileSize=10G
这是jmeter的启动命令: ./ApacheJMeter.jar -n -t iotserver.jmx -l is-log.jtl
jmeter第二次内存溢出
错误日志信息
jmeter控制台日志:
[root@localhost bin]# ./jmeter.sh -n -t iotserver.jmx -l is-log.jtl
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=64m; support was removed in 8.0
Creating summariser
Created the tree successfully using iotserver.jmx
Starting the test @ Tue Jul 23 14:39:46 CST 2019 (1563863986691)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
summary + 15878 in 00:00:13 = 1238.9/s Avg: 0 Min: 0 Max: 184 Err: 0 (0.00%) Active: 15893 Started: 15893 Finished: 0
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to protect stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to protect stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f22aa325000, 12288, 0) failed; error='无法分配内存' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /data/apache-jmeter-5.1.1/bin/hs_err_pid75390.log
Uncaught Exception java.lang.OutOfMemoryError: unable to create new native thread. See log file for details.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
jvm输出日志: hs_err_pid75390.log
内容如下:(捡重点)
内存溢出时的jvm堆栈情况如下:
VM state:synchronizing (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00007f2b5c007210] Safepoint_lock - owner thread: 0x00007f2b5c13f000
[0x00007f2b5c007290] Threads_lock - owner thread: 0x00007f2b5c13f000
Heap:
PSYoungGen total 2752512K, used 1322036K [0x0000000700000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 2359296K, 39% used [0x0000000700000000,0x0000000738b0d328,0x0000000790000000)
from space 393216K, 100% used [0x0000000790000000,0x00000007a8000000,0x00000007a8000000)
to space 393216K, 0% used [0x00000007a8000000,0x00000007a8000000,0x00000007c0000000)
ParOldGen total 3145728K, used 169567K [0x0000000640000000, 0x0000000700000000, 0x0000000700000000)
object space 3145728K, 5% used [0x0000000640000000,0x000000064a597e10,0x0000000700000000)
Metaspace used 22366K, capacity 22942K, committed 23344K, reserved 1069056K
class space used 2714K, capacity 2996K, committed 3072K, reserved 1048576K
Card table byte_map: [0x00007f2b4c03f000,0x00007f2b4cc40000] byte_map_base: 0x00007f2b48e3f000
Marking Bits: (ParMarkBitMap*) 0x00007f2b61f92d40
Begin Bits: [0x00007f2b28000000, 0x00007f2b2e000000)
End Bits: [0x00007f2b2e000000, 0x00007f2b34000000)
Polling page: 0x00007f2b62bd4000
CodeCache: size=245760Kb used=12507Kb max_used=12525Kb free=233252Kb
bounds [0x00007f2b4d000000, 0x00007f2b4dc50000, 0x00007f2b5c000000]
total_blobs=3773 nmethods=3307 adapters=379
compilation: enabled
...
...
jvm输出的gc日志: gc-pid75390.log.0.current
Java HotSpot(TM) 64-Bit Server VM (25.191-b12) for linux-amd64 JRE (1.8.0_191-b12), built on Oct 6 2018 05:43:09 by "java_re" with gcc 7.3.0
Memory: 4k page, physical 8009248k(7534804k free), swap 8257532k(8257532k free)
CommandLine flags: -XX:GCLogFileSize=10737418240 -XX:InitialHeapSize=6442450944 -XX:InitialTenuringThreshold=4 -XX:MaxHeapSize=6442450944 -XX:MaxTenuringThreshold=4 -XX:NewRatio=1 -XX:NumberOfGCLogFiles=100 -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+Pr
intHeapAtGC -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseGCLogFileRotation -XX:+UseParallelGC
{Heap before GC invocations=1 (full 0):
PSYoungGen total 2752512K, used 377488K [0x0000000700000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 2359296K, 16% used [0x0000000700000000,0x00000007170a4010,0x0000000790000000)
from space 393216K, 0% used [0x00000007a8000000,0x00000007a8000000,0x00000007c0000000)
to space 393216K, 0% used [0x0000000790000000,0x0000000790000000,0x00000007a8000000)
ParOldGen total 3145728K, used 0K [0x0000000640000000, 0x0000000700000000, 0x0000000700000000)
object space 3145728K, 0% used [0x0000000640000000,0x0000000640000000,0x0000000700000000)
Metaspace used 20647K, capacity 21098K, committed 21296K, reserved 1069056K
class space used 2564K, capacity 2784K, committed 2816K, reserved 1048576K
1.727: [GC (Metadata GC Threshold) [PSYoungGen: 377488K->14034K(2752512K)] 377488K->14122K(5898240K), 0.0206407 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
Heap after GC invocations=1 (full 0):
PSYoungGen total 2752512K, used 14034K [0x0000000700000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 2359296K, 0% used [0x0000000700000000,0x0000000700000000,0x0000000790000000)
from space 393216K, 3% used [0x0000000790000000,0x0000000790db4a80,0x00000007a8000000)
to space 393216K, 0% used [0x00000007a8000000,0x00000007a8000000,0x00000007c0000000)
ParOldGen total 3145728K, used 88K [0x0000000640000000, 0x0000000700000000, 0x0000000700000000)
object space 3145728K, 0% used [0x0000000640000000,0x0000000640016028,0x0000000700000000)
Metaspace used 20647K, capacity 21098K, committed 21296K, reserved 1069056K
class space used 2564K, capacity 2784K, committed 2816K, reserved 1048576K
}
{Heap before GC invocations=2 (full 1):
PSYoungGen total 2752512K, used 14034K [0x0000000700000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 2359296K, 0% used [0x0000000700000000,0x0000000700000000,0x0000000790000000)
from space 393216K, 3% used [0x0000000790000000,0x0000000790db4a80,0x00000007a8000000)
to space 393216K, 0% used [0x00000007a8000000,0x00000007a8000000,0x00000007c0000000)
ParOldGen total 3145728K, used 88K [0x0000000640000000, 0x0000000700000000, 0x0000000700000000)
object space 3145728K, 0% used [0x0000000640000000,0x0000000640016028,0x0000000700000000)
Metaspace used 20647K, capacity 21098K, committed 21296K, reserved 1069056K
class space used 2564K, capacity 2784K, committed 2816K, reserved 1048576K
1.748: [Full GC (Metadata GC Threshold) [PSYoungGen: 14034K->0K(2752512K)] [ParOldGen: 88K->13460K(3145728K)] 14122K->13460K(5898240K), [Metaspace: 20647K->20647K(1069056K)], 0.0436125 secs] [Times: user=0.10 sys=0.01, real=0.04 secs]
Heap after GC invocations=2 (full 1):
PSYoungGen total 2752512K, used 0K [0x0000000700000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 2359296K, 0% used [0x0000000700000000,0x0000000700000000,0x0000000790000000)
from space 393216K, 0% used [0x0000000790000000,0x0000000790000000,0x00000007a8000000)
to space 393216K, 0% used [0x00000007a8000000,0x00000007a8000000,0x00000007c0000000)
ParOldGen total 3145728K, used 13460K [0x0000000640000000, 0x0000000700000000, 0x0000000700000000)
object space 3145728K, 0% used [0x0000000640000000,0x0000000640d250b8,0x0000000700000000)
Metaspace used 20647K, capacity 21098K, committed 21296K, reserved 1069056K
class space used 2564K, capacity 2784K, committed 2816K, reserved 1048576K
}
{Heap before GC invocations=3 (full 1):
PSYoungGen total 2752512K, used 141557K [0x0000000700000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 2359296K, 6% used [0x0000000700000000,0x0000000708a3d780,0x0000000790000000)
from space 393216K, 0% used [0x0000000790000000,0x0000000790000000,0x00000007a8000000)
to space 393216K, 0% used [0x00000007a8000000,0x00000007a8000000,0x00000007c0000000)
ParOldGen total 3145728K, used 13460K [0x0000000640000000, 0x0000000700000000, 0x0000000700000000)
object space 3145728K, 0% used [0x0000000640000000,0x0000000640d250b8,0x0000000700000000)
Metaspace used 21114K, capacity 21708K, committed 21936K, reserved 1069056K
class space used 2656K, capacity 2893K, committed 2944K, reserved 1048576K
1.875: [GC (System.gc()) [PSYoungGen: 141557K->794K(2752512K)] 155018K->14254K(5898240K), 0.0029754 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
Heap after GC invocations=3 (full 1):
PSYoungGen total 2752512K, used 794K [0x0000000700000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 2359296K, 0% used [0x0000000700000000,0x0000000700000000,0x0000000790000000)
from space 393216K, 0% used [0x00000007a8000000,0x00000007a80c6820,0x00000007c0000000)
to space 393216K, 0% used [0x0000000790000000,0x0000000790000000,0x00000007a8000000)
ParOldGen total 3145728K, used 13460K [0x0000000640000000, 0x0000000700000000, 0x0000000700000000)
object space 3145728K, 0% used [0x0000000640000000,0x0000000640d250b8,0x0000000700000000)
Metaspace used 21114K, capacity 21708K, committed 21936K, reserved 1069056K
class space used 2656K, capacity 2893K, committed 2944K, reserved 1048576K
}
{Heap before GC invocations=4 (full 2):
PSYoungGen total 2752512K, used 794K [0x0000000700000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 2359296K, 0% used [0x0000000700000000,0x0000000700000000,0x0000000790000000)
from space 393216K, 0% used [0x00000007a8000000,0x00000007a80c6820,0x00000007c0000000)
to space 393216K, 0% used [0x0000000790000000,0x0000000790000000,0x00000007a8000000)
ParOldGen total 3145728K, used 13460K [0x0000000640000000, 0x0000000700000000, 0x0000000700000000)
object space 3145728K, 0% used [0x0000000640000000,0x0000000640d250b8,0x0000000700000000)
Metaspace used 21114K, capacity 21708K, committed 21936K, reserved 1069056K
class space used 2656K, capacity 2893K, committed 2944K, reserved 1048576K
1.878: [Full GC (System.gc()) [PSYoungGen: 794K->0K(2752512K)] [ParOldGen: 13460K->6097K(3145728K)] 14254K->6097K(5898240K), [Metaspace: 21114K->21114K(1069056K)], 0.0729254 secs] [Times: user=0.22 sys=0.01, real=0.07 secs]
Heap after GC invocations=4 (full 2):
PSYoungGen total 2752512K, used 0K [0x0000000700000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 2359296K, 0% used [0x0000000700000000,0x0000000700000000,0x0000000790000000)
from space 393216K, 0% used [0x00000007a8000000,0x00000007a8000000,0x00000007c0000000)
to space 393216K, 0% used [0x0000000790000000,0x0000000790000000,0x00000007a8000000)
ParOldGen total 3145728K, used 6097K [0x0000000640000000, 0x0000000700000000, 0x0000000700000000)
object space 3145728K, 0% used [0x0000000640000000,0x00000006405f4460,0x0000000700000000)
Metaspace used 21114K, capacity 21708K, committed 21936K, reserved 1069056K
class space used 2656K, capacity 2893K, committed 2944K, reserved 1048576K
}
{Heap before GC invocations=5 (full 2):
PSYoungGen total 2752512K, used 2359296K [0x0000000700000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 2359296K, 100% used [0x0000000700000000,0x0000000790000000,0x0000000790000000)
from space 393216K, 0% used [0x00000007a8000000,0x00000007a8000000,0x00000007c0000000)
to space 393216K, 0% used [0x0000000790000000,0x0000000790000000,0x00000007a8000000)
ParOldGen total 3145728K, used 6097K [0x0000000640000000, 0x0000000700000000, 0x0000000700000000)
object space 3145728K, 0% used [0x0000000640000000,0x00000006405f4460,0x0000000700000000)
Metaspace used 21797K, capacity 22390K, committed 22832K, reserved 1069056K
class space used 2707K, capacity 2964K, committed 3072K, reserved 1048576K
2.228: [GC (Allocation Failure) [PSYoungGen: 2359296K->5745K(2752512K)] 2365393K->11850K(5898240K), 0.0084773 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
Heap after GC invocations=5 (full 2):
PSYoungGen total 2752512K, used 5745K [0x0000000700000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 2359296K, 0% used [0x0000000700000000,0x0000000700000000,0x0000000790000000)
from space 393216K, 1% used [0x0000000790000000,0x000000079059c5a0,0x00000007a8000000)
to space 393216K, 0% used [0x00000007a8000000,0x00000007a8000000,0x00000007c0000000)
ParOldGen total 3145728K, used 6105K [0x0000000640000000, 0x0000000700000000, 0x0000000700000000)
object space 3145728K, 0% used [0x0000000640000000,0x00000006405f6460,0x0000000700000000)
Metaspace used 21797K, capacity 22390K, committed 22832K, reserved 1069056K
class space used 2707K, capacity 2964K, committed 3072K, reserved 1048576K
}
{Heap before GC invocations=6 (full 2):
PSYoungGen total 2752512K, used 2365041K [0x0000000700000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 2359296K, 100% used [0x0000000700000000,0x0000000790000000,0x0000000790000000)
from space 393216K, 1% used [0x0000000790000000,0x000000079059c5a0,0x00000007a8000000)
to space 393216K, 0% used [0x00000007a8000000,0x00000007a8000000,0x00000007c0000000)
ParOldGen total 3145728K, used 6105K [0x0000000640000000, 0x0000000700000000, 0x0000000700000000)
object space 3145728K, 0% used [0x0000000640000000,0x00000006405f6460,0x0000000700000000)
Metaspace used 22314K, capacity 22910K, committed 23344K, reserved 1069056K
class space used 2707K, capacity 2964K, committed 3072K, reserved 1048576K
3.993: [GC (Allocation Failure) [PSYoungGen: 2365041K->45728K(2752512K)] 2371146K->51841K(5898240K), 0.0373071 secs] [Times: user=0.13 sys=0.02, real=0.04 secs]
Heap after GC invocations=6 (full 2):
PSYoungGen total 2752512K, used 45728K [0x0000000700000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 2359296K, 0% used [0x0000000700000000,0x0000000700000000,0x0000000790000000)
from space 393216K, 11% used [0x00000007a8000000,0x00000007aaca8000,0x00000007c0000000)
to space 393216K, 0% used [0x0000000790000000,0x0000000790000000,0x00000007a8000000)
ParOldGen total 3145728K, used 6113K [0x0000000640000000, 0x0000000700000000, 0x0000000700000000)
object space 3145728K, 0% used [0x0000000640000000,0x00000006405f8460,0x0000000700000000)
Metaspace used 22314K, capacity 22910K, committed 23344K, reserved 1069056K
class space used 2707K, capacity 2964K, committed 3072K, reserved 1048576K
}
{Heap before GC invocations=7 (full 2):
PSYoungGen total 2752512K, used 2405024K [0x0000000700000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 2359296K, 100% used [0x0000000700000000,0x0000000790000000,0x0000000790000000)
from space 393216K, 11% used [0x00000007a8000000,0x00000007aaca8000,0x00000007c0000000)
to space 393216K, 0% used [0x0000000790000000,0x0000000790000000,0x00000007a8000000)
ParOldGen total 3145728K, used 6113K [0x0000000640000000, 0x0000000700000000, 0x0000000700000000)
object space 3145728K, 0% used [0x0000000640000000,0x00000006405f8460,0x0000000700000000)
Metaspace used 22343K, capacity 22910K, committed 23344K, reserved 1069056K
class space used 2711K, capacity 2964K, committed 3072K, reserved 1048576K
13.523: [GC (Allocation Failure) [PSYoungGen: 2405024K->393216K(2752512K)] 2411137K->562783K(5898240K), 0.5209667 secs] [Times: user=1.23 sys=0.42, real=0.53 secs]
Heap after GC invocations=7 (full 2):
PSYoungGen total 2752512K, used 393216K [0x0000000700000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 2359296K, 0% used [0x0000000700000000,0x0000000700000000,0x0000000790000000)
from space 393216K, 100% used [0x0000000790000000,0x00000007a8000000,0x00000007a8000000)
to space 393216K, 0% used [0x00000007a8000000,0x00000007a8000000,0x00000007c0000000)
ParOldGen total 3145728K, used 169567K [0x0000000640000000, 0x0000000700000000, 0x0000000700000000)
object space 3145728K, 5% used [0x0000000640000000,0x000000064a597e10,0x0000000700000000)
Metaspace used 22343K, capacity 22910K, committed 23344K, reserved 1069056K
class space used 2711K, capacity 2964K, committed 3072K, reserved 1048576K
}
日志分析
jvm日志分析如下:
- 持久代内存大小还是1069056K, 修改失败
- 年轻代内存3G(eden space + from space * 2 = 3145728K), 年老代内存3G, 比例是1:1
- Eden与两个Survivor比值 3:1, 一个Survivor占 1/8
- 报出OutOfMemoryError时, 年轻代中Eden(1,572,864K)使用39%, from space(262144K)使用100%, 年老代(4194304K)使用5%
- 报出OutOfMemoryError可能是因为from space没有足够的内存, 导致这种情况发生,是因为form space内存不够用
增加标记次数, 会增加from space中对象的数量, 增大Survivor的压力
解决方案
对jvm参数优化
- 不做任何标记, 年轻代对象不经过Survivor区，直接进入年老代
-XX:MaxTenuringThreshold=0: 设置垃圾最大年龄。如果设置为0的话，则年轻代对象不经过Survivor区，直接进入年老代。 - 因为不经过Survivor,所以把Survivor尽可能的调小
-XX:SurvivorRatio=16: 设置年轻代中Eden区与Survivor区的大小比值。设置为16，则两个Survivor区与一个Eden区的比值为2:16，一个Survivor区占整个年轻代的1/18 - Metaspace(元空间)设置为128M
Java8中持久代(永久区)这个内存空间被完全的移除, JVM参数PermSize 和 MaxPermSize会被忽略，当前在启动时会有警告信息(这段日志居然被我忽略了): Java HotSpot™ 64-Bit Server VM warning: ignoring option MaxPermSize=64m; support was removed in 8.0
元空间（Metaspace）：一个新的内存空间的诞生.与 Oracle JRockit 和 IBM JVM类似，JDK 8.HotSpot JVM开始使用本地化的内存存放类的元数据，这个空间叫做元空间（Metaspace）。
元空间的容量 默认情况下，类元数据分配受到可用的本机内存容量的限制（容量依然取决于你使用32位JVM还是64位操作系统的虚拟内存的可用性）。一个新的参数 (MaxMetaspaceSize)可以使用。允许你来限制用于类元数据的本地内存。如果没有特别指定，元空间将会根据应用程序在运行时的需求动态设置大小。
优化之后的jvm参数
jvm_args:
-Xms6g
-Xmx6g
-XX:NewRatio=1
-XX:MaxTenuringThreshold=0
-XX:SurvivorRatio=16
-XX:MaxMetaspaceSize=128m
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-Xloggc:../log/gc-%p.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=100
-XX:GCLogFileSize=10G
这是jmeter的启动命令: ./ApacheJMeter.jar -n -t iotserver.jmx -l is-log.jtl
jmeter第三次内存溢出
错误日志信息
摘要, jmeter打印的日志, jvm打印的日志
jvm输出日志:
Heap:
PSYoungGen total 2971136K, used 713856K [0x0000000738800000, 0x00000007f8800000, 0x00000007f8800000)
eden space 2796544K, 25% used [0x0000000738800000,0x00000007641202d8,0x00000007e3300000)
from space 174592K, 0% used [0x00000007e3300000,0x00000007e3300000,0x00000007edd80000)
to space 174592K, 0% used [0x00000007edd80000,0x00000007edd80000,0x00000007f8800000)
ParOldGen total 3145728K, used 689937K [0x0000000678800000, 0x0000000738800000, 0x0000000738800000)
object space 3145728K, 21% used [0x0000000678800000,0x00000006a29c4788,0x0000000738800000)
Metaspace used 22377K, capacity 23006K, committed 23344K, reserved 1069056K
class space used 2713K, capacity 2996K, committed 3072K, reserved 1048576K
jvm gc日志:
{Heap before GC invocations=6 (full 2):
PSYoungGen total 2971136K, used 2796544K [0x0000000738800000, 0x00000007f8800000, 0x00000007f8800000)
eden space 2796544K, 100% used [0x0000000738800000,0x00000007e3300000,0x00000007e3300000)
from space 174592K, 0% used [0x00000007e3300000,0x00000007e3300000,0x00000007edd80000)
to space 174592K, 0% used [0x00000007edd80000,0x00000007edd80000,0x00000007f8800000)
ParOldGen total 3145728K, used 11523K [0x0000000678800000, 0x0000000738800000, 0x0000000738800000)
object space 3145728K, 0% used [0x0000000678800000,0x0000000679340ed8,0x0000000738800000)
Metaspace used 22337K, capacity 22974K, committed 23344K, reserved 1069056K
class space used 2707K, capacity 2964K, committed 3072K, reserved 1048576K
4.715: [GC (Allocation Failure) [PSYoungGen: 2796544K->0K(2971136K)] 2808067K->68223K(6116864K), 0.0573879 secs] [Times: user=0.17 sys=0.04, real=0.06 secs]
Heap after GC invocations=6 (full 2):
PSYoungGen total 2971136K, used 0K [0x0000000738800000, 0x00000007f8800000, 0x00000007f8800000)
eden space 2796544K, 0% used [0x0000000738800000,0x0000000738800000,0x00000007e3300000)
from space 174592K, 0% used [0x00000007edd80000,0x00000007edd80000,0x00000007f8800000)
to space 174592K, 0% used [0x00000007e3300000,0x00000007e3300000,0x00000007edd80000)
ParOldGen total 3145728K, used 68223K [0x0000000678800000, 0x0000000738800000, 0x0000000738800000)
object space 3145728K, 2% used [0x0000000678800000,0x000000067ca9fe98,0x0000000738800000)
Metaspace used 22337K, capacity 22974K, committed 23344K, reserved 1069056K
class space used 2707K, capacity 2964K, committed 3072K, reserved 1048576K
}
{Heap before GC invocations=7 (full 2):
PSYoungGen total 2971136K, used 2796544K [0x0000000738800000, 0x00000007f8800000, 0x00000007f8800000)
eden space 2796544K, 100% used [0x0000000738800000,0x00000007e3300000,0x00000007e3300000)
from space 174592K, 0% used [0x00000007edd80000,0x00000007edd80000,0x00000007f8800000)
to space 174592K, 0% used [0x00000007e3300000,0x00000007e3300000,0x00000007edd80000)
ParOldGen total 3145728K, used 68223K [0x0000000678800000, 0x0000000738800000, 0x0000000738800000)
object space 3145728K, 2% used [0x0000000678800000,0x000000067ca9fe98,0x0000000738800000)
Metaspace used 22366K, capacity 22974K, committed 23344K, reserved 1069056K
class space used 2712K, capacity 2964K, committed 3072K, reserved 1048576K
13.797: [GC (Allocation Failure) [PSYoungGen: 2796544K->0K(2971136K)] 2864767K->689937K(6116864K), 0.7301310 secs] [Times: user=1.88 sys=0.66, real=0.74 secs]
Heap after GC invocations=7 (full 2):
PSYoungGen total 2971136K, used 0K [0x0000000738800000, 0x00000007f8800000, 0x00000007f8800000)
eden space 2796544K, 0% used [0x0000000738800000,0x0000000738800000,0x00000007e3300000)
from space 174592K, 0% used [0x00000007e3300000,0x00000007e3300000,0x00000007edd80000)
to space 174592K, 0% used [0x00000007edd80000,0x00000007edd80000,0x00000007f8800000)
ParOldGen total 3145728K, used 689937K [0x0000000678800000, 0x0000000738800000, 0x0000000738800000)
object space 3145728K, 21% used [0x0000000678800000,0x00000006a29c4788,0x0000000738800000)
Metaspace used 22366K, capacity 22974K, committed 23344K, reserved 1069056K
class space used 2712K, capacity 2964K, committed 3072K, reserved 1048576K
}
jvm参数调整
JVM_ARGS="-Xms6g
-Xmx6g
-XX:NewRatio=1
-XX:MaxTenuringThreshold=1
-XX:SurvivorRatio=4
-XX:MaxMetaspaceSize=128m
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-Xloggc:../log/gc-%p.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=100
-XX:GCLogFileSize=10G"
jmeter第四次内存溢出
错误日志信息
摘要, jmeter打印的日志, jvm打印的日志
jvm输出日志:
Heap:
PSYoungGen total 2621440K, used 1177433K [0x0000000738800000, 0x00000007f8800000, 0x00000007f8800000)
eden space 2097152K, 31% used [0x0000000738800000,0x00000007605de518,0x00000007b8800000)
from space 524288K, 99% used [0x00000007b8800000,0x00000007d87f8010,0x00000007d8800000)
to space 524288K, 0% used [0x00000007d8800000,0x00000007d8800000,0x00000007f8800000)
ParOldGen total 3145728K, used 113176K [0x0000000678800000, 0x0000000738800000, 0x0000000738800000)
object space 3145728K, 3% used [0x0000000678800000,0x000000067f6862b0,0x0000000738800000)
Metaspace used 22389K, capacity 23006K, committed 23344K, reserved 1069056K
class space used 2714K, capacity 2996K, committed 3072K, reserved 1048576K
jvm gc日志:
{Heap before GC invocations=6 (full 2):
PSYoungGen total 2621440K, used 2102161K [0x0000000738800000, 0x00000007f8800000, 0x00000007f8800000)
eden space 2097152K, 100% used [0x0000000738800000,0x00000007b8800000,0x00000007b8800000)
from space 524288K, 0% used [0x00000007b8800000,0x00000007b8ce45a0,0x00000007d8800000)
to space 524288K, 0% used [0x00000007d8800000,0x00000007d8800000,0x00000007f8800000)
ParOldGen total 3145728K, used 6121K [0x0000000678800000, 0x0000000738800000, 0x0000000738800000)
object space 3145728K, 0% used [0x0000000678800000,0x0000000678dfa780,0x0000000738800000)
Metaspace used 22338K, capacity 22974K, committed 23344K, reserved 1069056K
class space used 2707K, capacity 2964K, committed 3072K, reserved 1048576K
4.778: [GC (Allocation Failure) [PSYoungGen: 2102161K->75088K(2621440K)] 2108283K->85932K(5767168K), 0.0607136 secs] [Times: user=0.18 sys=0.05, real=0.06 secs]
Heap after GC invocations=6 (full 2):
PSYoungGen total 2621440K, used 75088K [0x0000000738800000, 0x00000007f8800000, 0x00000007f8800000)
eden space 2097152K, 0% used [0x0000000738800000,0x0000000738800000,0x00000007b8800000)
from space 524288K, 14% used [0x00000007d8800000,0x00000007dd154010,0x00000007f8800000)
to space 524288K, 0% used [0x00000007b8800000,0x00000007b8800000,0x00000007d8800000)
ParOldGen total 3145728K, used 10844K [0x0000000678800000, 0x0000000738800000, 0x0000000738800000)
object space 3145728K, 0% used [0x0000000678800000,0x0000000679297090,0x0000000738800000)
Metaspace used 22338K, capacity 22974K, committed 23344K, reserved 1069056K
class space used 2707K, capacity 2964K, committed 3072K, reserved 1048576K
}
{Heap before GC invocations=7 (full 2):
PSYoungGen total 2621440K, used 2172240K [0x0000000738800000, 0x00000007f8800000, 0x00000007f8800000)
eden space 2097152K, 100% used [0x0000000738800000,0x00000007b8800000,0x00000007b8800000)
from space 524288K, 14% used [0x00000007d8800000,0x00000007dd154010,0x00000007f8800000)
to space 524288K, 0% used [0x00000007b8800000,0x00000007b8800000,0x00000007d8800000)
ParOldGen total 3145728K, used 10844K [0x0000000678800000, 0x0000000738800000, 0x0000000738800000)
object space 3145728K, 0% used [0x0000000678800000,0x0000000679297090,0x0000000738800000)
Metaspace used 22367K, capacity 22974K, committed 23344K, reserved 1069056K
class space used 2711K, capacity 2964K, committed 3072K, reserved 1048576K
13.108: [GC (Allocation Failure) [PSYoungGen: 2172240K->524256K(2621440K)] 2183084K->637432K(5767168K), 0.5645814 secs] [Times: user=1.34 sys=0.54, real=0.57 secs]
Heap after GC invocations=7 (full 2):
PSYoungGen total 2621440K, used 524256K [0x0000000738800000, 0x00000007f8800000, 0x00000007f8800000)
eden space 2097152K, 0% used [0x0000000738800000,0x0000000738800000,0x00000007b8800000)
from space 524288K, 99% used [0x00000007b8800000,0x00000007d87f8010,0x00000007d8800000)
to space 524288K, 0% used [0x00000007d8800000,0x00000007d8800000,0x00000007f8800000)
ParOldGen total 3145728K, used 113176K [0x0000000678800000, 0x0000000738800000, 0x0000000738800000)
object space 3145728K, 3% used [0x0000000678800000,0x000000067f6862b0,0x0000000738800000)
Metaspace used 22367K, capacity 22974K, committed 23344K, reserved 1069056K
class space used 2711K, capacity 2964K, committed 3072K, reserved 1048576K
}
jvm参数调整
JVM_ARGS="-Xms6g
-Xmx6g
-Xss256k
-XX:+UseParallelGC
-XX:+UseParallelOldGC
-XX:ParallelGCThreads=16
-XX:+UseAdaptiveSizePolicy
-XX:NewRatio=2
-XX:MaxTenuringThreshold=0
-XX:SurvivorRatio=32
-XX:MaxMetaspaceSize=128m
-XX:TargetSurvivorRatio=90
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-Xloggc:../log/gc-%p.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=100
-XX:GCLogFileSize=10G"
jmeter脚本优化
使用并发线程组插件: Concurrency Thread Group
jmeter第五次内存溢出
错误日志信息
摘要, jmeter打印的日志, jvm打印的日志
控制台日志:
Creating summariser
Created the tree successfully using iotserver2.jmx
Starting the test @ Wed Jul 24 10:06:39 CST 2019 (1563933999131)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f10061cd000, 12288, 0) failed; error='无法分配内存' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /data/apache-jmeter-5.1.1/bin/hs_err_pid455847.log
Uncaught Exception java.lang.OutOfMemoryError: unable to create new native thread. See log file for details.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
jvm输出日志:
Heap:
PSYoungGen total 2096640K, used 107457K [0x0000000778800000, 0x00000007f8800000, 0x00000007f8800000)
eden space 2096128K, 5% used [0x0000000778800000,0x000000077f0f06a8,0x00000007f8700000)
from space 512K, 0% used [0x00000007f8780000,0x00000007f8780000,0x00000007f8800000)
to space 512K, 0% used [0x00000007f8700000,0x00000007f8700000,0x00000007f8780000)
ParOldGen total 4194304K, used 1091832K [0x0000000678800000, 0x0000000778800000, 0x0000000778800000)
object space 4194304K, 26% used [0x0000000678800000,0x00000006bb23e2c8,0x0000000778800000)
Metaspace used 22555K, capacity 23210K, committed 23552K, reserved 1069056K
class space used 2738K, capacity 2998K, committed 3072K, reserved 1048576K
jvm gc日志:
{Heap before GC invocations=9 (full 2):
PSYoungGen total 2096640K, used 2035200K [0x0000000778800000, 0x00000007f8800000, 0x00000007f8800000)
eden space 2035200K, 100% used [0x0000000778800000,0x00000007f4b80000,0x00000007f4b80000)
from space 61440K, 0% used [0x00000007f4c00000,0x00000007f4c00000,0x00000007f8800000)
to space 512K, 0% used [0x00000007f4b80000,0x00000007f4b80000,0x00000007f4c00000)
ParOldGen total 4194304K, used 408701K [0x0000000678800000, 0x0000000778800000, 0x0000000778800000)
object space 4194304K, 9% used [0x0000000678800000,0x000000069171f7c8,0x0000000778800000)
Metaspace used 22541K, capacity 23198K, committed 23552K, reserved 1069056K
class space used 2737K, capacity 2996K, committed 3072K, reserved 1048576K
153.068: [GC (Allocation Failure) [PSYoungGen: 2035200K->0K(2035712K)] 2443901K->699624K(6230016K), 0.3116885 secs] [Times: user=0.83 sys=0.20, real=0.31 secs]
Heap after GC invocations=9 (full 2):
PSYoungGen total 2035712K, used 0K [0x0000000778800000, 0x00000007f8800000, 0x00000007f8800000)
eden space 2035200K, 0% used [0x0000000778800000,0x0000000778800000,0x00000007f4b80000)
from space 512K, 0% used [0x00000007f4b80000,0x00000007f4b80000,0x00000007f4c00000)
to space 512K, 0% used [0x00000007f8780000,0x00000007f8780000,0x00000007f8800000)
ParOldGen total 4194304K, used 699624K [0x0000000678800000, 0x0000000778800000, 0x0000000778800000)
object space 4194304K, 16% used [0x0000000678800000,0x00000006a333a008,0x0000000778800000)
Metaspace used 22541K, capacity 23198K, committed 23552K, reserved 1069056K
class space used 2737K, capacity 2996K, committed 3072K, reserved 1048576K
}
{Heap before GC invocations=10 (full 2):
PSYoungGen total 2035712K, used 2035200K [0x0000000778800000, 0x00000007f8800000, 0x00000007f8800000)
eden space 2035200K, 100% used [0x0000000778800000,0x00000007f4b80000,0x00000007f4b80000)
from space 512K, 0% used [0x00000007f4b80000,0x00000007f4b80000,0x00000007f4c00000)
to space 512K, 0% used [0x00000007f8780000,0x00000007f8780000,0x00000007f8800000)
ParOldGen total 4194304K, used 699624K [0x0000000678800000, 0x0000000778800000, 0x0000000778800000)
object space 4194304K, 16% used [0x0000000678800000,0x00000006a333a008,0x0000000778800000)
Metaspace used 22541K, capacity 23198K, committed 23552K, reserved 1069056K
class space used 2737K, capacity 2996K, committed 3072K, reserved 1048576K
228.261: [GC (Allocation Failure) [PSYoungGen: 2035200K->0K(2096640K)] 2734824K->1091832K(6290944K), 0.5029285 secs] [Times: user=1.34 sys=0.43, real=0.51 secs]
Heap after GC invocations=10 (full 2):
PSYoungGen total 2096640K, used 0K [0x0000000778800000, 0x00000007f8800000, 0x00000007f8800000)
eden space 2096128K, 0% used [0x0000000778800000,0x0000000778800000,0x00000007f8700000)
from space 512K, 0% used [0x00000007f8780000,0x00000007f8780000,0x00000007f8800000)
to space 512K, 0% used [0x00000007f8700000,0x00000007f8700000,0x00000007f8780000)
ParOldGen total 4194304K, used 1091832K [0x0000000678800000, 0x0000000778800000, 0x0000000778800000)
object space 4194304K, 26% used [0x0000000678800000,0x00000006bb23e2c8,0x0000000778800000)
Metaspace used 22541K, capacity 23198K, committed 23552K, reserved 1069056K
class space used 2737K, capacity 2996K, committed 3072K, reserved 1048576K
}
情况分析
参考文档: Java HotSpot™ 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000794500000, 576716800, 0)
分析: 服务器物理内存不够, 因为给jvm分了太多内存, 导致最后jvm创建线程, 给线程分配栈内存时, 内存不足; jvm创建线程,分配栈内存, 使用的都是系统内存, 如果给堆内存分配太多, 就会导致栈内存不够.
嗯, 走了好多弯路, 慢悠悠的还是找到了问题的原因.
还有就是, 控制台日志, 很重要…
jvm的基础知识也很重要…, 不然,不会一直在堆内存上找问题; 结果发现是 栈内存的问题
我TMD想静静…
jvm参数调整
JVM_ARGS="-Xms4g
-Xmx4g
-Xss256k
-XX:+UseParallelGC
-XX:+UseParallelOldGC
-XX:ParallelGCThreads=16
-XX:NewRatio=2
-XX:MaxTenuringThreshold=0
-XX:SurvivorRatio=64
-XX:MaxMetaspaceSize=128m
-XX:TargetSurvivorRatio=90
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-Xloggc:../log/gc-%p.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=100
-XX:GCLogFileSize=10G"
jmeter第六次报错, 不是内存溢出
错误日志信息
摘要, jmeter打印的日志, jvm打印的日志
控制台日志:
Creating summariser
Created the tree successfully using iotserver2.jmx
Starting the test @ Wed Jul 24 10:25:28 CST 2019 (1563935128173)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
summary + 769 in 00:00:01 = 649.5/s Avg: 1 Min: 0 Max: 55 Err: 0 (0.00%) Active: 771 Started: 771 Finished: 0
summary + 10000 in 00:00:30 = 333.3/s Avg: 14 Min: 0 Max: 1008 Err: 0 (0.00%) Active: 10000 Started: 10000 Finished: 0
summary = 10769 in 00:00:31 = 345.3/s Avg: 13 Min: 0 Max: 1008 Err: 0 (0.00%)
summary + 10000 in 00:00:30 = 333.3/s Avg: 0 Min: 0 Max: 51 Err: 1 (0.01%) Active: 10000 Started: 10000 Finished: 0
summary = 20769 in 00:01:01 = 339.4/s Avg: 7 Min: 0 Max: 1008 Err: 1 (0.00%)
summary + 19999 in 00:00:30 = 666.8/s Avg: 5 Min: 0 Max: 3008 Err: 0 (0.00%) Active: 20000 Started: 20000 Finished: 0
summary = 40768 in 00:01:31 = 447.1/s Avg: 6 Min: 0 Max: 3008 Err: 1 (0.00%)
summary + 20000 in 00:00:30 = 666.6/s Avg: 0 Min: 0 Max: 1 Err: 0 (0.00%) Active: 20000 Started: 20000 Finished: 0
summary = 60768 in 00:02:01 = 501.5/s Avg: 4 Min: 0 Max: 3008 Err: 1 (0.00%)
summary + 20368 in 00:00:30 = 679.0/s Avg: 0 Min: 0 Max: 2 Err: 1 (0.00%) Active: 20373 Started: 20373 Finished: 0
summary = 81136 in 00:02:31 = 536.7/s Avg: 3 Min: 0 Max: 3008 Err: 2 (0.00%)
summary + 30000 in 00:00:30 = 1000.0/s Avg: 34 Min: 0 Max: 3007 Err: 0 (0.00%) Active: 30000 Started: 30000 Finished: 0
summary = 111136 in 00:03:01 = 613.4/s Avg: 11 Min: 0 Max: 3008 Err: 2 (0.00%)
summary + 29998 in 00:00:30 = 1000.0/s Avg: 0 Min: 0 Max: 1 Err: 0 (0.00%) Active: 30000 Started: 30000 Finished: 0
summary = 141134 in 00:03:31 = 668.3/s Avg: 9 Min: 0 Max: 3008 Err: 2 (0.00%)
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to protect stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to protect stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Uncaught Exception java.lang.OutOfMemoryError: unable to create new native thread. See log file for details.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32744 bytes for ChunkPool::allocate
# An error report file with more information is saved as:
# /data/apache-jmeter-5.1.1/bin/hs_err_pid498659.log
[thread 140,458,524,440,320 also had an error]
[thread 140,458,388,125,440 also had an error]
[thread 140,458,711,607,040 also had an error]
[thread 140,458,575,292,160 also had an error]
jvm输出日志:
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32744 bytes for ChunkPool::allocate
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (allocation.cpp:273), pid=498659, tid=0x0000 7fbf 171a 6700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_191-b12) (build 1.8.0_191-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
--------------- T H R E A D ---------------
Current thread (0x00007fbf393a0000): JavaThread "bzm - Concurrency Thread Group-ThreadStarter 1-10426" [_thread_in_vm, id=509120, stack(0x00007fbf17166000,0x00007fbf171a7000)]
Stack: [0x00007fbf17166000,0x00007fbf171a7000], sp=0x00007fbf171a41b0, free space=248k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xace425] VMError::report_and_die()+0x2c5
V [libjvm.so+0x4deb77] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x67
V [libjvm.so+0x2c6d50] ChunkPool::allocate(unsigned long, AllocFailStrategy::AllocFailEnum)+0xe0
V [libjvm.so+0x2c6532] Arena::grow(unsigned long, AllocFailStrategy::AllocFailEnum)+0x42
V [libjvm.so+0xabd5dd] vframe::new_vframe(frame const*, RegisterMap const*, JavaThread*) [clone .part.29]+0x3d
V [libjvm.so+0x4ed6fb] Deoptimization::revoke_biases_of_monitors(JavaThread*, frame, RegisterMap*)+0x14b
V [libjvm.so+0x4eedb4] Deoptimization::uncommon_trap_inner(JavaThread*, int)+0x1a4
V [libjvm.so+0x4f39e1] Deoptimization::uncommon_trap(JavaThread*, int)+0x11
v blob 0x00007fc0510456b9
C 0x00000000002d6ec0
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v blob 0x00007fc0510456b9
J 3276 C2 org.apache.jmeter.threads.JMeterThread.run()V (851 bytes) @ 0x00007fc051bdf028 [0x00007fc051bdd620+0x1a08]
J 3200 C2 java.lang.Thread.run()V (17 bytes) @ 0x00007fc051b00868 [0x00007fc051b00820+0x48]
v ~StubRoutines::call_stub
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
0x00007fbdb44a9000 JavaThread "bzm - Concurrency Thread Group-ThreadStarter 1-32612" [_thread_in_native, id=531307, stack(0x0000 7fbd aaf8 7000,0x00007fbdaafc8000)]
0x00007fbdb44a7000 JavaThread "bzm - Concurrency Thread Group-ThreadStarter 1-32611" [_thread_in_native, id=531306, stack(0x00007fbdaafc8000,0x00007fbdab009000)]
0x00007fbdb44a5000 JavaThread "bzm - Concurrency Thread Group-ThreadStarter 1-32610" [_thread_in_native, id=531305, stack(0x00007fbdab009000,0x00007fbdab04a000)]
...
...
...
0x00007fbfbc005800 JavaThread "bzm - Concurrency Thread Group-ThreadStarter 1-2" [_thread_blocked, id=498693, stack(0x00007fc001ed2000,0x00007fc001f13000)]
0x00007fbfbc004000 JavaThread "NanoOffset" daemon [_thread_blocked, id=498692, stack(0x00007fc001f13000,0x00007fc001f54000)]
0x00007fbfbc002000 JavaThread "bzm - Concurrency Thread Group-ThreadStarter 1-1" [_thread_blocked, id=498691, stack(0x00007fc001f54000,0x00007fc001f95000)]
0x00007fc06000a000 JavaThread "DestroyJavaVM" [_thread_blocked, id=498660, stack(0x00007fc067b4b000,0x00007fc067b8b000)]
0x00007fc060cdd800 JavaThread "UDP Listener" daemon [_thread_in_native, id=498688, stack(0x00007fc0023d6000,0x00007fc002417000)]
0x00007fc060cdb000 JavaThread "StandardJMeterEngine" [_thread_blocked, id=498687, stack(0x00007fc002417000,0x00007fc002458000)]
0x00007fc060b67000 JavaThread "Timer-0" daemon [_thread_blocked, id=498686, stack(0x00007fc00285d000,0x00007fc00289e000)]
0x00007fc06033b800 JavaThread "Service Thread" daemon [_thread_blocked, id=498684, stack(0x00007fc0036b6000,0x00007fc0036f7000)]
0x00007fc06031e000 JavaThread "C1 CompilerThread2" daemon [_thread_in_vm, id=498683, stack(0x00007fc0036f8000,0x00007fc0037f8000)]
0x00007fc06031c800 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=498682, stack(0x00007fc0037f9000,0x00007fc0038f9000)]
0x00007fc060319800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=498681, stack(0x00007fc0038fa000,0x00007fc0039fa000)]
0x00007fc060317800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=498680, stack(0x00007fc010036000,0x00007fc010077000)]
0x00007fc0602e5000 JavaThread "Finalizer" daemon [_thread_blocked, id=498679, stack(0x00007fc010077000,0x00007fc0100b8000)]
0x00007fc0602e2000 JavaThread "Reference Handler" daemon [_thread_blocked, id=498678, stack(0x00007fc0100b8000,0x00007fc0100f9000)]
Other Threads:
0x00007fc0602d8800 VMThread [stack: 0x00007fc0039fb000,0x00007fc003afb000] [id=498677]
0x00007fc060350000 WatcherThread [stack: 0x00007fc0035b6000,0x00007fc0036b6000] [id=498685]
VM state:synchronizing (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00007fc060007380] Safepoint_lock - owner thread: 0x00007fc0602d8800
[0x00007fc060007400] Threads_lock - owner thread: 0x00007fc0602d8800
Heap:
PSYoungGen total 1397248K, used 1359479K [0x00000007a3300000, 0x00000007f8800000, 0x00000007f8800000)
eden space 1396736K, 97% used [0x00000007a3300000,0x00000007f629dec8,0x00000007f8700000)
from space 512K, 0% used [0x00000007f8780000,0x00000007f8780000,0x00000007f8800000)
to space 512K, 0% used [0x00000007f8700000,0x00000007f8700000,0x00000007f8780000)
ParOldGen total 2796544K, used 925660K [0x00000006f8800000, 0x00000007a3300000, 0x00000007a3300000)
object space 2796544K, 33% used [0x00000006f8800000,0x0000000730ff7110,0x00000007a3300000)
Metaspace used 22567K, capacity 23204K, committed 23552K, reserved 1069056K
class space used 2740K, capacity 2999K, committed 3072K, reserved 1048576K
Card table byte_map: [0x00007fc064658000,0x00007fc064e59000] byte_map_base: 0x00007fc060e94000
Marking Bits: (ParMarkBitMap*) 0x00007fc066f56d40
Begin Bits: [0x00007fc004000000, 0x00007fc008000000)
End Bits: [0x00007fc008000000, 0x00007fc00c000000)
Polling page: 0x00007fc067b98000
CodeCache: size=245760Kb used=12723Kb max_used=12785Kb free=233036Kb
bounds [0x00007fc051000000, 0x00007fc051c90000, 0x00007fc060000000]
total_blobs=3820 nmethods=3354 adapters=379
compilation: enabled
jvm gc日志:
{Heap before GC invocations=9 (full 2):
PSYoungGen total 1397248K, used 1376256K [0x00000007a3300000, 0x00000007f8800000, 0x00000007f8800000)
eden space 1376256K, 100% used [0x00000007a3300000,0x00000007f7300000,0x00000007f7300000)
from space 20992K, 0% used [0x00000007f7380000,0x00000007f7380000,0x00000007f8800000)
to space 512K, 0% used [0x00000007f7300000,0x00000007f7300000,0x00000007f7380000)
ParOldGen total 2796544K, used 357823K [0x00000006f8800000, 0x00000007a3300000, 0x00000007a3300000)
object space 2796544K, 12% used [0x00000006f8800000,0x000000070e56fe80,0x00000007a3300000)
Metaspace used 22537K, capacity 23198K, committed 23552K, reserved 1069056K
class space used 2737K, capacity 2996K, committed 3072K, reserved 1048576K
94.677: [GC (Allocation Failure) [PSYoungGen: 1376256K->0K(1376768K)] 1734079K->676513K(4173312K), 0.3133553 secs] [Times: user=0.86 sys=0.22, real=0.32 secs]
Heap after GC invocations=9 (full 2):
PSYoungGen total 1376768K, used 0K [0x00000007a3300000, 0x00000007f8800000, 0x00000007f8800000)
eden space 1376256K, 0% used [0x00000007a3300000,0x00000007a3300000,0x00000007f7300000)
from space 512K, 0% used [0x00000007f7300000,0x00000007f7300000,0x00000007f7380000)
to space 512K, 0% used [0x00000007f8780000,0x00000007f8780000,0x00000007f8800000)
ParOldGen total 2796544K, used 676513K [0x00000006f8800000, 0x00000007a3300000, 0x00000007a3300000)
object space 2796544K, 24% used [0x00000006f8800000,0x0000000721ca87a0,0x00000007a3300000)
Metaspace used 22537K, capacity 23198K, committed 23552K, reserved 1069056K
class space used 2737K, capacity 2996K, committed 3072K, reserved 1048576K
}
{Heap before GC invocations=10 (full 2):
PSYoungGen total 1376768K, used 1376256K [0x00000007a3300000, 0x00000007f8800000, 0x00000007f8800000)
eden space 1376256K, 100% used [0x00000007a3300000,0x00000007f7300000,0x00000007f7300000)
from space 512K, 0% used [0x00000007f7300000,0x00000007f7300000,0x00000007f7380000)
to space 512K, 0% used [0x00000007f8780000,0x00000007f8780000,0x00000007f8800000)
ParOldGen total 2796544K, used 676513K [0x00000006f8800000, 0x00000007a3300000, 0x00000007a3300000)
object space 2796544K, 24% used [0x00000006f8800000,0x0000000721ca87a0,0x00000007a3300000)
Metaspace used 22554K, capacity 23198K, committed 23552K, reserved 1069056K
class space used 2738K, capacity 2996K, committed 3072K, reserved 1048576K
155.615: [GC (Allocation Failure) [PSYoungGen: 1376256K->0K(1397248K)] 2052769K->925660K(4193792K), 0.4225358 secs] [Times: user=1.03 sys=0.32, real=0.43 secs]
Heap after GC invocations=10 (full 2):
PSYoungGen total 1397248K, used 0K [0x00000007a3300000, 0x00000007f8800000, 0x00000007f8800000)
eden space 1396736K, 0% used [0x00000007a3300000,0x00000007a3300000,0x00000007f8700000)
from space 512K, 0% used [0x00000007f8780000,0x00000007f8780000,0x00000007f8800000)
to space 512K, 0% used [0x00000007f8700000,0x00000007f8700000,0x00000007f8780000)
ParOldGen total 2796544K, used 925660K [0x00000006f8800000, 0x00000007a3300000, 0x00000007a3300000)
object space 2796544K, 33% used [0x00000006f8800000,0x0000000730ff7110,0x00000007a3300000)
Metaspace used 22554K, capacity 23198K, committed 23552K, reserved 1069056K
class space used 2738K, capacity 2996K, committed 3072K, reserved 1048576K
}
分析
- jvm堆内存还剩挺多的
- 查看jmeter控制台日志, 应该是系统内存不够了, 无法创建新的线程, 无法分配栈内存
解决方案
- 把堆内存调小点, 让系统有更多的内存创建线程
- 栈的大小-Xss128k
JVM_ARGS="-Xms2g
-Xmx2g
-Xss128k
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-Xloggc:../log/gc-%p.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=100
-XX:GCLogFileSize=10G"
jmeter第七次报错, 栈内存至少228k
错误日志信息
摘要, jmeter打印的日志, jvm打印的日志
控制台日志:
The stack size specified is too small, Specify at least 228k
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
分析
- 栈内存至少228k
jvm调整
- 栈的大小的调整,应该都按照8的倍数来吧… 我觉得是这样的, 所有调整为 -Xss232k
JVM_ARGS="-Xms2g
-Xmx2g
-Xss232k
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-Xloggc:../log/gc-%p.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=100
-XX:GCLogFileSize=10G"
jmeter第八次报错, 栈内存不足
错误日志信息
摘要, jmeter打印的日志, jvm打印的日志
jmeter控制台日志:(使用tac打印, 请倒着看)
[thread 139915517572864 also had an error][thread 139915516622592 also had an error]
[thread 139915515909888 also had an error]
[thread 139915517335296 also had an error]
[thread 139915517810432 also had an error]
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Uncaught Exception java.lang.OutOfMemoryError: unable to create new native thread. See log file for details.
# /data/apache-jmeter-5.1.1/bin/hs_err_pid531418.log
# An error report file with more information is saved as:
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# There is insufficient memory for the Java Runtime Environment to continue.
#
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f3f5a186000, 12288, 0) failed; error='无法分配内存' (errno=12)
summary = 149079 in 00:03:33 = 699.0/s Avg: 6 Min: 0 Max: 2030 Err: 2 (0.00%)
summary + 29998 in 00:00:30 = 999.9/s Avg: 0 Min: 0 Max: 1 Err: 0 (0.00%) Active: 30000 Started: 30000 Finished: 0
summary = 119081 in 00:03:03 = 649.7/s Avg: 8 Min: 0 Max: 2030 Err: 2 (0.00%)
summary + 29999 in 00:00:30 = 1000.0/s Avg: 14 Min: 0 Max: 2030 Err: 0 (0.00%) Active: 30000 Started: 30000 Finished: 0
summary = 89082 in 00:02:33 = 581.1/s Avg: 6 Min: 0 Max: 1474 Err: 2 (0.00%)
summary + 25766 in 00:00:30 = 858.9/s Avg: 6 Min: 0 Max: 1474 Err: 1 (0.00%) Active: 26163 Started: 26163 Finished: 0
summary = 63316 in 00:02:03 = 513.6/s Avg: 6 Min: 0 Max: 1125 Err: 1 (0.00%)
summary + 20000 in 00:00:30 = 666.6/s Avg: 0 Min: 0 Max: 1 Err: 0 (0.00%) Active: 20000 Started: 20000 Finished: 0
summary = 43316 in 00:01:33 = 464.3/s Avg: 9 Min: 0 Max: 1125 Err: 1 (0.00%)
summary + 20000 in 00:00:30 = 666.6/s Avg: 21 Min: 0 Max: 1125 Err: 0 (0.00%) Active: 20000 Started: 20000 Finished: 0
summary = 23316 in 00:01:03 = 368.4/s Avg: 0 Min: 0 Max: 145 Err: 1 (0.00%)
summary + 9999 in 00:00:30 = 333.3/s Avg: 0 Min: 0 Max: 145 Err: 1 (0.01%) Active: 10000 Started: 10000 Finished: 0
summary = 13317 in 00:00:33 = 400.1/s Avg: 0 Min: 0 Max: 73 Err: 0 (0.00%)
summary + 9999 in 00:00:30 = 333.3/s Avg: 0 Min: 0 Max: 20 Err: 0 (0.00%) Active: 10000 Started: 10000 Finished: 0
summary + 3318 in 00:00:03 = 1009.7/s Avg: 0 Min: 0 Max: 73 Err: 0 (0.00%) Active: 3319 Started: 3319 Finished: 0
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
Starting the test @ Wed Jul 24 11:26:26 CST 2019 (1563938786237)
Created the tree successfully using iotserver2.jmx
Creating summariser
jvm输出日志: hs_err_pid531418.log
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2743), pid=531418, tid=0x00007f3f5a1bf700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_191-b12) (build 1.8.0_191-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.191-b12 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
--------------- T H R E A D ---------------
Current thread (0x00007f3f60487000): JavaThread "bzm - Concurrency Thread Group-ThreadStarter 1-32623" [_thread_new, id=564065, stack(0x00007f3f5a186000,0x00007f3f5a1c0000)]
Stack: [0x00007f3f5a186000,0x00007f3f5a1c0000], sp=0x00007f3f5a1bead0, free space=226k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xace425] VMError::report_and_die()+0x2c5
V [libjvm.so+0x4deb77] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*)+0x67
V [libjvm.so+0x90c166] os::pd_commit_memory(char*, unsigned long, bool)+0xf6
V [libjvm.so+0x903d3f] os::commit_memory(char*, unsigned long, bool)+0x1f
V [libjvm.so+0x90d6a8] os::pd_create_stack_guard_pages(char*, unsigned long)+0x48
V [libjvm.so+0xa772cc] JavaThread::run()+0x2fc
V [libjvm.so+0x909c92] java_start(Thread*)+0x102
C [libpthread.so.0+0x7dd5] start_thread+0xc5
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
=>0x00007f3f60487000 JavaThread "bzm - Concurrency Thread Group-ThreadStarter 1-32623" [_thread_new, id=564065, stack(0x00007f3f5a186000,0x00007f3f5a1c0000)]
0x00007f3f60485000 JavaThread "bzm - Concurrency Thread Group-ThreadStarter 1-32622" [_thread_blocked, id=564064, stack(0x00007f3f5a1c0000,0x00007f3f5a1fa000)]
0x00007f3f60483000 JavaThread "bzm - Concurrency Thread Group-ThreadStarter 1-32621" [_thread_blocked, id=564063, stack(0x00007f3f5a1fa000,0x00007f3f5a234000)]
0x00007f3f60481000 JavaThread "bzm - Concurrency Thread Group-ThreadStarter 1-32620" [_thread_blocked, id=564062, stack(0x00007f3f5a234000,0x00007f3f5a26e000)]
...
...
...
0x00007f4164005800 JavaThread "bzm - Concurrency Thread Group-ThreadStarter 1-2" [_thread_blocked, id=531440, stack(0x00007f41a9689000,0x00007f41a96c3000)]
0x00007f4164004000 JavaThread "NanoOffset" daemon [_thread_blocked, id=531439, stack(0x00007f41a96c3000,0x00007f41a96fd000)]
0x00007f4164002000 JavaThread "bzm - Concurrency Thread Group-ThreadStarter 1-1" [_thread_blocked, id=531438, stack(0x00007f41a96fd000,0x00007f41a9737000)]
0x00007f41d0009800 JavaThread "DestroyJavaVM" [_thread_blocked, id=531419, stack(0x00007f41d77b2000,0x00007f41d77ec000)]
0x00007f41d0b42000 JavaThread "UDP Listener" daemon [_thread_in_native, id=531435, stack(0x00007f41a9b71000,0x00007f41a9bab000)]
0x00007f41d0b3f800 JavaThread "StandardJMeterEngine" [_thread_blocked, id=531434, stack(0x00007f41a9bab000,0x00007f41a9be5000)]
0x00007f41d09cd800 JavaThread "Timer-0" daemon [_thread_blocked, id=531433, stack(0x00007f41a9fea000,0x00007f41aa024000)]
0x00007f41d0190000 JavaThread "Service Thread" daemon [_thread_blocked, id=531431, stack(0x00007f41aae41000,0x00007f41aae7b000)]
0x00007f41d0184800 JavaThread "C1 CompilerThread2" daemon [_thread_in_vm, id=531430, stack(0x00007f41aae7c000,0x00007f41aaf7c000)]
0x00007f41d0183000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=531429, stack(0x00007f41aaf7d000,0x00007f41ab07d000)]
0x00007f41d0180000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=531428, stack(0x00007f41ab07e000,0x00007f41ab17e000)]
0x00007f41d017e800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=531427, stack(0x00007f41ab17e000,0x00007f41ab1b8000)]
0x00007f41d014b800 JavaThread "Finalizer" daemon [_thread_blocked, id=531426, stack(0x00007f41ab1b8000,0x00007f41ab1f2000)]
0x00007f41d0148800 JavaThread "Reference Handler" daemon [_thread_blocked, id=531425, stack(0x00007f41d400c000,0x00007f41d4046000)]
Other Threads:
0x00007f41d013f000 VMThread [stack: 0x00007f41ab1f3000,0x00007f41ab2f3000] [id=531424]
0x00007f41d0194800 WatcherThread [stack: 0x00007f41aad41000,0x00007f41aae41000] [id=531432]
VM state:synchronizing (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00007f41d0006670] CodeCache_lock - owner thread: 0x00007f41d0184800
[0x00007f41d0007270] Safepoint_lock - owner thread: 0x00007f41d013f000
[0x00007f41d00072f0] Threads_lock - owner thread: 0x00007f41d013f000
[0x00007f41d0007770] AdapterHandlerLibrary_lock - owner thread: 0x00007f40c15e4800
[0x00007f41d0007cf0] Compile_lock - owner thread: 0x00007f41d0184800
[0x00007f41d0007df0] MethodCompileQueue_lock - owner thread: 0x00007f41d0184800
Heap:
PSYoungGen total 465920K, used 149184K [0x00000000d5580000, 0x0000000100000000, 0x0000000100000000)
eden space 232960K, 31% used [0x00000000d5580000,0x00000000d9ca93b8,0x00000000e3900000)
from space 232960K, 32% used [0x00000000f1c80000,0x00000000f6706c60,0x0000000100000000)
to space 232960K, 0% used [0x00000000e3900000,0x00000000e3900000,0x00000000f1c80000)
ParOldGen total 1398272K, used 1063374K [0x0000000080000000, 0x00000000d5580000, 0x00000000d5580000)
object space 1398272K, 76% used [0x0000000080000000,0x00000000c0e73858,0x00000000d5580000)
Metaspace used 22565K, capacity 23214K, committed 23552K, reserved 1069056K
class space used 2740K, capacity 3000K, committed 3072K, reserved 1048576K
Card table byte_map: [0x00007f41d46b9000,0x00007f41d4aba000] byte_map_base: 0x00007f41d42b9000
72217.6
jvm gc日志
{Heap before GC invocations=21 (full 2):
PSYoungGen total 465920K, used 247637K [0x00000000d5580000, 0x0000000100000000, 0x0000000100000000)
eden space 232960K, 100% used [0x00000000d5580000,0x00000000e3900000,0x00000000e3900000)
from space 232960K, 6% used [0x00000000f1c80000,0x00000000f2ad54d0,0x0000000100000000)
to space 232960K, 0% used [0x00000000e3900000,0x00000000e3900000,0x00000000f1c80000)
ParOldGen total 1398272K, used 1035646K [0x0000000080000000, 0x00000000d5580000, 0x00000000d5580000)
object space 1398272K, 74% used [0x0000000080000000,0x00000000bf35f858,0x00000000d5580000)
Metaspace used 22552K, capacity 23198K, committed 23552K, reserved 1069056K
class space used 2737K, capacity 2996K, committed 3072K, reserved 1048576K
213.760: [GC (Allocation Failure) [PSYoungGen: 247637K->29813K(465920K)] 1283283K->1074915K(1864192K), 0.1786738 secs] [Times: user=0.54 sys=0.04, real=0.18 secs]
Heap after GC invocations=21 (full 2):
PSYoungGen total 465920K, used 29813K [0x00000000d5580000, 0x0000000100000000, 0x0000000100000000)
eden space 232960K, 0% used [0x00000000d5580000,0x00000000d5580000,0x00000000e3900000)
from space 232960K, 12% used [0x00000000e3900000,0x00000000e561d4d0,0x00000000f1c80000)
to space 232960K, 0% used [0x00000000f1c80000,0x00000000f1c80000,0x0000000100000000)
ParOldGen total 1398272K, used 1045102K [0x0000000080000000, 0x00000000d5580000, 0x00000000d5580000)
object space 1398272K, 74% used [0x0000000080000000,0x00000000bfc9b858,0x00000000d5580000)
Metaspace used 22552K, capacity 23198K, committed 23552K, reserved 1069056K
class space used 2737K, capacity 2996K, committed 3072K, reserved 1048576K
}
{Heap before GC invocations=22 (full 2):
PSYoungGen total 465920K, used 262773K [0x00000000d5580000, 0x0000000100000000, 0x0000000100000000)
eden space 232960K, 100% used [0x00000000d5580000,0x00000000e3900000,0x00000000e3900000)
from space 232960K, 12% used [0x00000000e3900000,0x00000000e561d4d0,0x00000000f1c80000)
to space 232960K, 0% used [0x00000000f1c80000,0x00000000f1c80000,0x0000000100000000)
ParOldGen total 1398272K, used 1045102K [0x0000000080000000, 0x00000000d5580000, 0x00000000d5580000)
object space 1398272K, 74% used [0x0000000080000000,0x00000000bfc9b858,0x00000000d5580000)
Metaspace used 22552K, capacity 23198K, committed 23552K, reserved 1069056K
class space used 2737K, capacity 2996K, committed 3072K, reserved 1048576K
227.662: [GC (Allocation Failure) [PSYoungGen: 262773K->76315K(465920K)] 1307875K->1139689K(1864192K), 0.1487917 secs] [Times: user=0.50 sys=0.02, real=0.15 secs]
Heap after GC invocations=22 (full 2):
PSYoungGen total 465920K, used 76315K [0x00000000d5580000, 0x0000000100000000, 0x0000000100000000)
eden space 232960K, 0% used [0x00000000d5580000,0x00000000d5580000,0x00000000e3900000)
from space 232960K, 32% used [0x00000000f1c80000,0x00000000f6706c60,0x0000000100000000)
to space 232960K, 0% used [0x00000000e3900000,0x00000000e3900000,0x00000000f1c80000)
ParOldGen total 1398272K, used 1063374K [0x0000000080000000, 0x00000000d5580000, 0x00000000d5580000)
object space 1398272K, 76% used [0x0000000080000000,0x00000000c0e73858,0x00000000d5580000)
Metaspace used 22552K, capacity 23198K, committed 23552K, reserved 1069056K
class space used 2737K, capacity 2996K, committed 3072K, reserved 1048576K
}
free -m内容:
[root@localhost ~]# free -m
total used free shared buff/cache available
Mem: 7821 3104 3679 9 1037 4247
Swap: 8063 0 8063
[root@localhost ~]# free -m
total used free shared buff/cache available
Mem: 7821 3610 3114 9 1097 3688
Swap: 8063 0 8063
[root@localhost ~]# free -m
total used free shared buff/cache available
Mem: 7821 4269 2396 9 1156 2981
Swap: 8063 0 8063
[root@localhost ~]# free -m
total used free shared buff/cache available
Mem: 7821 4269 2396 9 1156 2980
Swap: 8063 0 8063
[root@localhost ~]# free -m
total used free shared buff/cache available
Mem: 7821 4280 2384 9 1156 2970
Swap: 8063 0 8063
[root@localhost ~]# free -m
total used free shared buff/cache available
Mem: 7821 253 6610 9 957 7201
Swap: 8063 0 8063
test程序, 测试当前系统最大能够创建的线程数:
i = 32731
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to protect stack guard pages failed.
i = 32732
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to protect stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at com.nes.test.TestNativeOutOfMemoryError.main(TestNativeOutOfMemoryError.java:17)
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fddf1474000, 12288, 0) failed; error='无法分配内存' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /root/hs_err_pid695126.log
top 内容:
日志分析
- 问题还是出在 栈内存不足
- 栈内存计算: 232 * 32623 + 232 * 12 = 7568536 ~= 7.2G, 跟jvm实际使用内存, 应该有偏差
jmeter输出日志中, process信息, “bzm - Concurrency Thread Group-ThreadStarter 1” 线程数量为: 32623, 其它线程: 12
每个线程的栈内存为: 232k - 堆内存计算: 年老代: 1063374K, 年轻代: 149184K, 总共: 1212558K ~= 1.15G, 跟jvm实际使用内存, 应该有偏差
- free -m显示, 用户使用内存 4280M ~= 4.2G
- free -m显示, 系统总内存Mem: 7821M, Swap: 8063M
- 测试程序跑出的最大线程数: 32732, 跟jmeter的线程数比较符合 记一次内存溢出java.lang.OutOfMemoryError: unable to create new native thread
解决方案
- 加内存, 然后用测试程序跑最大线程数
系统内存设置为16G, 最大线程数还是32732问题
错误日志信息
test程序, 测试当前系统最大能够创建的线程数:
i = 32731
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to protect stack guard pages failed.
i = 32732
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to protect stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at com.nes.test.TestNativeOutOfMemoryError.main(TestNativeOutOfMemoryError.java:17)
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00007fddf1474000, 12288, 0) failed; error='无法分配内存' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 12288 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /root/hs_err_pid695126.log
系统参数:
[root@localhost ~]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31212
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1048576
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[root@localhost ~]# cat /proc/sys/kernel/pid_max
1048576
[root@localhost ~]# cat /proc/sys/kernel/threads-max
1048576
[root@localhost ~]# cat /proc/sys/vm/max_map_count
65530
错误分析
- 猜测是jvm限制, 因为系统的最大线程数量限制已经放开了: 1048576
- 操作系统限制: 系统最大可开线程数, 主要受下面几个参数影响: pid_max, threads-max, max_map_count, max_user_process
- max_map_count = 65530 / 2 = 32765, 与目前测试程序抛出的结果一致, 就是这个参数的影响
- 参考文档: JVM最大线程数
解决方案
- 修改max_map_count = 1048576, reboot重启系统
配置文件如下:
[root@localhost bin]# cat /etc/sysctl.conf
# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
fs.file-max = 1048576
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_mem = 524288 1048576 1572864
net.ipv4.tcp_rmem = 4096 4096 8388608
net.ipv4.tcp_wmem = 4096 4096 8388608
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
kernel.pid_max=1048576
kernel.threads-max=1048576
vm.max_map_count=1048576
系统内存16G, 跑出来的最大线程数量: 426551
系统内存16G, 跑出来的结果 (系统直接跑死了, kill -9 都没反应):
i = 426548
i = 426549
i = 426550
i = 426551
查看系统内存:
[root@localhost ~]# free -m
total used free shared buff/cache available
Mem: 15885 12667 156 1 3061 5
Swap: 8063 5797 2266
[root@localhost ~]# top
top - 15:58:27 up 10 min, 4 users, load average: 1.29, 1.45, 0.72
Tasks: 192 total, 2 running, 190 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 12.1 sy, 0.0 ni, 62.5 id, 24.0 wa, 0.0 hi, 1.4 si, 0.0 st
KiB Mem : 16266748 total, 155128 free, 12984668 used, 3126952 buff/cache
KiB Swap: 8257532 total, 2328316 free, 5929216 used. 81040 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
18 root rt 0 0 0 0 S 42.7 0.0 0:02.16 migration/2
433481 root 20 0 162116 1992 1208 R 7.2 0.0 0:01.77 top
45 root 20 0 0 0 0 S 3.2 0.0 0:38.15 kswapd0
6914 root 20 0 428.9g 2.4g 68 S 3.2 15.6 2:36.68 java
6876 root 20 0 157276 400 308 S 0.2 0.0 0:00.05 sshd
1 root 20 0 191348 444 236 S 0.0 0.0 0:01.25 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:01.81 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 0:00.18 rcu_sched
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain
11 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
12 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
13 root rt 0 0 0 0 S 0.0 0.0 0:01.90 migration/1
14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/1
16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
17 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
19 root 20 0 0 0 0 S 0.0 0.0 0:00.10 ksoftirqd/2
20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0
21 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H
22 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/3
23 root rt 0 0 0 0 S 0.0 0.0 0:01.54 migration/3
24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3
25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0
26 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0H
28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
29 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
30 root 20 0 0 0 0 S 0.0 0.0 0:00.17 khungtaskd
31 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback
32 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd
33 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
34 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
35 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
36 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd
37 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md
38 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 edac-poller
39 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 watchdogd
40 root 20 0 0 0 0 S 0.0 0.0 0:00.91 kworker/2:1
46 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
47 root 39 19 0 0 0 S 0.0 0.0 0:00.00 khugepaged
48 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto
56 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld
58 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kmpath_rdacd
59 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kaluad
用户层使用内存: 12.4G
用户层使用硬盘交换分区大小: 5.65G
jvm 信息:
年轻代可以占用的最大内存: 1.2G
年老代可以占用的最大内存: 2.5G
元空间可以占用的最大内存: 1G
jvm 具体信息:
[root@localhost ~]# jps
6914 jar
433483 Jps
[root@localhost ~]# jstat -gcoldcapacity 6914
OGCMN OGCMX OGC OC YGC FGC FGCT GCT
171008.0 2711552.0 345600.0 345600.0 7 1 1.476 3.518
[root@localhost ~]# jstat -gcmetacapacity 6914
MCMN MCMX MC CCSMN CCSMX CCSC YGC FGC FGCT GCT
0.0 1056768.0 4864.0 0.0 1048576.0 512.0 7 1 1.476 3.518
[root@localhost ~]# jstat -gcnewcapacity 6914
NGCMN NGCMX NGC S0CMX S0C S1CMX S1C ECMX EC YGC FGC
84992.0 1355776.0 809984.0 451584.0 187392.0 451584.0 119808.0 1354752.0 435200.0 7 1
每个线程栈的内存大小:
(12984668 + 5929216) / 426550 ~= 44k
(top中用户层使用内存 + top中用户使用硬盘交换分区大小) / 线程数量 = 线程栈大小
系统内存8G, 跑出来的最大线程数量: 221308
系统内存8G, 跑出来的结果:
i = 221306
i = 221307
i = 221308
查看系统内存:
[root@localhost ~]# free -m
total used free shared buff/cache available
Mem: 7821 6067 128 1 1624 14
Swap: 8063 3708 4355
[root@localhost ~]# top
top - 16:42:47 up 2 min, 4 users, load average: 1.77, 1.03, 0.41
Tasks: 226 total, 2 running, 224 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 21.8 sy, 0.0 ni, 52.0 id, 25.2 wa, 0.0 hi, 1.0 si, 0.0 st
KiB Mem : 8009252 total, 126180 free, 6229436 used, 1653636 buff/cache
KiB Swap: 8257532 total, 4623656 free, 3633876 used. 3528 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
227981 root 20 0 162144 2356 1564 R 76.7 0.0 0:00.65 top
45 root 20 0 0 0 0 R 2.7 0.0 0:18.66 kswapd0
6655 root 20 0 224.0g 713852 0 S 2.7 8.9 1:11.57 java
1 root 20 0 191244 368 168 S 0.0 0.0 0:01.21 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
6 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u8:0
7 root rt 0 0 0 0 S 0.0 0.0 0:00.01 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 0:00.08 rcu_sched
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain
11 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
12 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
13 root rt 0 0 0 0 S 0.0 0.0 0:00.48 migration/1
14 root 20 0 0 0 0 S 0.0 0.0 0:00.02 ksoftirqd/1
15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0
16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
17 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
18 root rt 0 0 0 0 S 0.0 0.0 0:00.03 migration/2
19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/2
20 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0
21 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H
22 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/3
23 root rt 0 0 0 0 S 0.0 0.0 0:00.01 migration/3
24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/3
25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0
26 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0H
28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
29 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
30 root 20 0 0 0 0 S 0.0 0.0 0:00.03 khungtaskd
31 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback
32 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd
33 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
34 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
35 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
36 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd
37 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md
38 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 edac-poller
39 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 watchdogd
40 root 20 0 0 0 0 S 0.0 0.0 0:00.59 kworker/2:1
46 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
47 root 39 19 0 0 0 S 0.0 0.0 0:00.00 khugepaged
48 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto
56 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld
57 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u8:1
用户层使用内存: 5.84G
用户层使用硬盘交换分区大小: 3.46G
jvm 信息:
年轻代可以占用的最大内存: 652M
年老代可以占用的最大内存: 1.27G
元空间可以占用的最大内存: 1G
jvm 具体信息:
[root@localhost ~]# jps
227983 Jps
6655 jar
[root@localhost ~]# jstat -gcoldcapacity 6655
OGCMN OGCMX OGC OC YGC FGC FGCT GCT
84992.0 1335296.0 178176.0 178176.0 7 1 0.846 2.021
[root@localhost ~]# jstat -gcmetacapacity 6655
MCMN MCMX MC CCSMN CCSMX CCSC YGC FGC FGCT GCT
0.0 1056768.0 4864.0 0.0 1048576.0 512.0 7 1 0.846 2.021
[root@localhost ~]# jstat -gcnewcapacity 6655
NGCMN NGCMX NGC S0CMX S0C S1CMX S1C ECMX EC YGC FGC
41984.0 667648.0 414720.0 222208.0 92672.0 222208.0 59392.0 666624.0 228864.0 7 1
每个线程栈的内存大小:
(12984668 + 3633876) / 221308 ~= 44k
(top中用户层使用内存 + top中用户使用硬盘交换分区大小) / 线程数量 = 线程栈大小
又得重启系统
jmeter跑4w长连接, 运行数据整理
jmeter控制台日志:
Creating summariser
Created the tree successfully using iotserver2.jmx
Starting the test @ Wed Jul 24 16:54:04 CST 2019 (1563958444484)
Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445
summary + 1 in 00:00:00 = 6.5/s Avg: 14 Min: 14 Max: 14 Err: 0 (0.00%) Active: 23 Started: 23 Finished: 0
summary + 10000 in 00:00:30 = 333.3/s Avg: 0 Min: 0 Max: 18 Err: 0 (0.00%) Active: 10000 Started: 10000 Finished: 0
summary = 10001 in 00:00:30 = 331.7/s Avg: 0 Min: 0 Max: 18 Err: 0 (0.00%)
summary + 10000 in 00:00:30 = 333.3/s Avg: 0 Min: 0 Max: 22 Err: 0 (0.00%) Active: 10000 Started: 10000 Finished: 0
summary = 20001 in 00:01:00 = 332.5/s Avg: 0 Min: 0 Max: 22 Err: 0 (0.00%)
summary + 20000 in 00:00:30 = 666.6/s Avg: 20 Min: 0 Max: 1004 Err: 1 (0.01%) Active: 20000 Started: 20000 Finished: 0
summary = 40001 in 00:01:30 = 443.7/s Avg: 10 Min: 0 Max: 1004 Err: 1 (0.00%)
summary + 20000 in 00:00:30 = 666.7/s Avg: 0 Min: 0 Max: 1 Err: 0 (0.00%) Active: 20000 Started: 20000 Finished: 0
summary = 60001 in 00:02:00 = 499.4/s Avg: 6 Min: 0 Max: 1004 Err: 1 (0.00%)
summary + 20000 in 00:00:30 = 666.6/s Avg: 0 Min: 0 Max: 1 Err: 0 (0.00%) Active: 20000 Started: 20000 Finished: 0
summary = 80001 in 00:02:30 = 532.8/s Avg: 5 Min: 0 Max: 1004 Err: 1 (0.00%)
summary + 30000 in 00:00:30 = 1000.0/s Avg: 0 Min: 0 Max: 1003 Err: 1 (0.00%) Active: 30000 Started: 30000 Finished: 0
summary = 110001 in 00:03:00 = 610.6/s Avg: 3 Min: 0 Max: 1004 Err: 2 (0.00%)
summary + 30000 in 00:00:30 = 1000.0/s Avg: 0 Min: 0 Max: 1 Err: 0 (0.00%) Active: 30000 Started: 30000 Finished: 0
summary = 140001 in 00:03:30 = 666.2/s Avg: 3 Min: 0 Max: 1004 Err: 2 (0.00%)
summary + 40000 in 00:00:30 = 1333.3/s Avg: 47 Min: 0 Max: 4795 Err: 0 (0.00%) Active: 40000 Started: 40000 Finished: 0
summary = 180001 in 00:04:00 = 749.5/s Avg: 13 Min: 0 Max: 4795 Err: 2 (0.00%)
summary + 40000 in 00:00:30 = 1333.3/s Avg: 0 Min: 0 Max: 3 Err: 1 (0.00%) Active: 40000 Started: 40000 Finished: 0
summary = 220001 in 00:04:30 = 814.3/s Avg: 10 Min: 0 Max: 4795 Err: 3 (0.00%)
summary + 40000 in 00:00:30 = 1333.3/s Avg: 0 Min: 0 Max: 2 Err: 0 (0.00%) Active: 40000 Started: 40000 Finished: 0
summary = 260001 in 00:05:00 = 866.2/s Avg: 9 Min: 0 Max: 4795 Err: 3 (0.00%)
summary + 40000 in 00:00:30 = 1333.3/s Avg: 0 Min: 0 Max: 1 Err: 0 (0.00%) Active: 40000 Started: 40000 Finished: 0
summary = 300001 in 00:05:30 = 908.7/s Avg: 7 Min: 0 Max: 4795 Err: 3 (0.00%)
summary + 40000 in 00:00:30 = 1333.3/s Avg: 0 Min: 0 Max: 1 Err: 1 (0.00%) Active: 40000 Started: 40000 Finished: 0
summary = 340001 in 00:06:00 = 944.0/s Avg: 6 Min: 0 Max: 4795 Err: 4 (0.00%)
可以看见, Active: 40000 Started: 40000, 跑了4w个长连接
jvm gc日志:
{Heap before GC invocations=60 (full 36):
PSYoungGen total 469504K, used 237568K [0x00000000d5580000, 0x0000000100000000, 0x0000000100000000)
eden space 237568K, 100% used [0x00000000d5580000,0x00000000e3d80000,0x00000000e3d80000)
from space 231936K, 0% used [0x00000000f1d80000,0x00000000f1d80000,0x0000000100000000)
to space 229376K, 0% used [0x00000000e3d80000,0x00000000e3d80000,0x00000000f1d80000)
ParOldGen total 1398272K, used 1291652K [0x0000000080000000, 0x00000000d5580000, 0x00000000d5580000)
object space 1398272K, 92% used [0x0000000080000000,0x00000000ced612d8,0x00000000d5580000)
Metaspace used 22383K, capacity 22892K, committed 23552K, reserved 1069056K
class space used 2665K, capacity 2861K, committed 3072K, reserved 1048576K
1053.413: [Full GC (Ergonomics) [PSYoungGen: 237568K->0K(469504K)] [ParOldGen: 1291652K->1291652K(1398272K)] 1529220K->1291652K(1867776K), [Metaspace: 22383K->22354K(1069056K)], 1.6052809 secs] [Times: user=5.09 sys=0.10, real=1.61 secs]
Heap after GC invocations=60 (full 36):
PSYoungGen total 469504K, used 0K [0x00000000d5580000, 0x0000000100000000, 0x0000000100000000)
eden space 237568K, 0% used [0x00000000d5580000,0x00000000d5580000,0x00000000e3d80000)
from space 231936K, 0% used [0x00000000f1d80000,0x00000000f1d80000,0x0000000100000000)
to space 229376K, 0% used [0x00000000e3d80000,0x00000000e3d80000,0x00000000f1d80000)
ParOldGen total 1398272K, used 1291652K [0x0000000080000000, 0x00000000d5580000, 0x00000000d5580000)
object space 1398272K, 92% used [0x0000000080000000,0x00000000ced613a0,0x00000000d5580000)
Metaspace used 22354K, capacity 22850K, committed 23552K, reserved 1069056K
class space used 2661K, capacity 2854K, committed 3072K, reserved 1048576K
}
{Heap before GC invocations=61 (full 37):
PSYoungGen total 469504K, used 237568K [0x00000000d5580000, 0x0000000100000000, 0x0000000100000000)
eden space 237568K, 100% used [0x00000000d5580000,0x00000000e3d80000,0x00000000e3d80000)
from space 231936K, 0% used [0x00000000f1d80000,0x00000000f1d80000,0x0000000100000000)
to space 229376K, 0% used [0x00000000e3d80000,0x00000000e3d80000,0x00000000f1d80000)
ParOldGen total 1398272K, used 1291652K [0x0000000080000000, 0x00000000d5580000, 0x00000000d5580000)
object space 1398272K, 92% used [0x0000000080000000,0x00000000ced613a0,0x00000000d5580000)
Metaspace used 22358K, capacity 22850K, committed 23552K, reserved 1069056K
class space used 2661K, capacity 2854K, committed 3072K, reserved 1048576K
1079.574: [Full GC (Ergonomics) [PSYoungGen: 237568K->0K(469504K)] [ParOldGen: 1291652K->1293336K(1398272K)] 1529220K->1293336K(1867776K), [Metaspace: 22358K->22353K(1069056K)], 1.5909306 secs] [Times: user=5.14 sys=0.04, real=1.60 secs]
堆内存使用: 1293336K ~= 1.2G
堆内存最大大小: 1867776K = 1.78125G
元空间大小: 1G
jvm参数
JVM_ARGS="-Xms2g
-Xmx2g
-Xss232k
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-Xloggc:../log/gc-%p.log
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=100
-XX:GCLogFileSize=10G"
系统信息:
top
top - 17:23:03 up 30 min, 4 users, load average: 0.92, 0.50, 0.39
Tasks: 196 total, 1 running, 195 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.3 us, 0.3 sy, 0.0 ni, 99.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 0.7 us, 0.3 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.3 us, 0.3 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu3 : 0.3 us, 0.7 sy, 0.0 ni, 99.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 8009248 total, 1952972 free, 5058548 used, 997728 buff/cache
KiB Swap: 8257532 total, 8257532 free, 0 used. 2262988 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6700 root 20 0 14.6g 2.8g 16376 S 3.3 36.3 12:13.33 java
46742 root 20 0 162120 2400 1608 R 0.7 0.0 0:14.31 top
1 root 20 0 191244 4184 2564 S 0.0 0.1 0:01.21 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:00.09 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 0:00.29 rcu_sched
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain
11 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
12 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
13 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
14 root 20 0 0 0 0 S 0.0 0.0 0:00.09 ksoftirqd/1
16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
17 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/2
18 root rt 0 0 0 0 S 0.0 0.0 0:00.03 migration/2
19 root 20 0 0 0 0 S 0.0 0.0 0:00.08 ksoftirqd/2
21 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H
22 root rt 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/3
23 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/3
24 root 20 0 0 0 0 S 0.0 0.0 0:00.09 ksoftirqd/3
26 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0H
28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs
29 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
30 root 20 0 0 0 0 S 0.0 0.0 0:00.08 khungtaskd
31 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback
32 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd
33 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
34 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
35 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
36 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd
37 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md
38 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 edac-poller
39 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 watchdogd
45 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kswapd0
46 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
47 root 39 19 0 0 0 S 0.0 0.0 0:00.40 khugepaged
48 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto
56 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld
58 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kmpath_rdacd
59 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kaluad
60 root 20 0 0 0 0 S 0.0 0.0 0:00.46 kworker/3:1
61 root 20 0 0 0 0 S 0.0 0.0 0:00.24 kworker/0:1
free -m
[root@localhost bin]# free -m
total used free shared buff/cache available
Mem: 7821 4941 1879 8 1000 2207
Swap: 8063 0 8063
iftop
19.1Mb 38.1Mb 57.2Mb 76.3Mb 95.4Mb
mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqvqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
192.168.1.17 => 192.168.1.16 885Kb 2.60Mb 2.74Mb
<= 684Kb 0.98Mb 1.05Mb
192.168.1.17 => 192.168.1.11 2.48Kb 35.2Kb 27.4Kb
<= 160b 416b 360b
192.168.1.255 => 192.168.1.156 0b 0b 0b
<= 2.27Kb 2.27Kb 2.18Kb
192.168.1.255 => 192.168.1.110 0b 0b 0b
<= 0b 2.07Kb 2.16Kb
192.168.1.17 => public1.114dns.com 296b 401b 335b
<= 436b 510b 461b
255.255.255.255 => 0.0.0.0 0b 0b 0b
<= 0b 278b 116b
192.168.1.255 => 192.168.1.130 0b 0b 0b
<= 0b 188b 78b
192.168.1.255 => 192.168.1.111 0b 0b 0b
<= 0b 187b 78b
192.168.1.255 => 192.168.1.185 0b 0b 0b
<= 0b 187b 78b
192.168.1.17 => 192.168.1.110 0b 78b 32b
<= 0b 71b 30b
255.255.255.255 => 192.168.1.60 0b 0b 0b
<= 0b 141b 59b
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
TX: cum: 8.29MB peak: 6.09Mb rates: 888Kb 2.63Mb 2.76Mb
RX: 3.17MB 1.90Mb 687Kb 0.98Mb 1.06Mb
TOTAL: 11.5MB 7.99Mb 1.54Mb 3.62Mb 3.82Mb
其它问题
线程栈大小是260k, 与-Xss256k不一致, 多了4k
jmeter输出日志, 线程中的stack(0x00007fbdab009000,0x00007fbdab04a000)
0x00007fbdab04a000 - 0x00007fbdab009000 = 0x41000
0x41000 = 266240(10进制)
266240 / 1024 = 160K
0x00007fbdb44a5000 JavaThread "bzm - Concurrency Thread Group-ThreadStarter 1-32610" [_thread_in_native, id=531305, stack(0x00007fbdab009000,0x00007fbdab04a000)]
这是什么原因造成的?
线程栈内存的实际大小计算
- 栈内存计算: 232 * 32623 + 232 * 12 = 7568536 ~= 7.2G, 跟jvm实际使用内存, 应该有偏差
jmeter输出日志中, process信息, “bzm - Concurrency Thread Group-ThreadStarter 1” 线程数量为: 32623, 其它线程: 12
每个线程的栈内存为: 232k - 堆内存计算: 年老代: 1063374K, 年轻代: 149184K, 总共: 1212558K ~= 1.15G, 跟jvm实际使用内存, 应该有偏差
- free -m显示, 用户使用内存 4280M ~= 4.2G
- free -m显示, 系统总内存Mem: 7821M, Swap: 8063M
如何计算实际大小?