#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 2863333376 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2756), pid=19116, tid=139792166455040
#
# JRE version: (7.0_79-b15) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.79-b02 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
--------------- T H R E A D ---------------
Current thread (0x00007f23dc00a000): JavaThread "Unknown thread" [_thread_in_vm, id=19117, stack(0x00007f23e02c8000,0x00007f23e66c9000)]
Stack: [0x00007f23e02c8000,0x00007f23e66c9000], sp=0x00007f23e66c7330, free space=102396k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x9a32da] VMError::report_and_die()+0x2ea
V [libjvm.so+0x497f7b] report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b
V [libjvm.so+0x81fcce] os::Linux::commit_memory_impl(char*, unsigned long, bool)+0xfe
V [libjvm.so+0x820219] os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
V [libjvm.so+0x819faa] os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
V [libjvm.so+0x99eae9] VirtualSpace::expand_by(unsigned long, bool)+0x1c9
V [libjvm.so+0x99ec6d] VirtualSpace::initialize(ReservedSpace, unsigned long)+0xcd
V [libjvm.so+0x57962f] CardGeneration::CardGeneration(ReservedSpace, unsigned long, int, GenRemSet*)+0x11f
V [libjvm.so+0x9534e3] TenuredGeneration::TenuredGeneration(ReservedSpace, unsigned long, int, GenRemSet*)+0x83
V [libjvm.so+0x57abbd] GenerationSpec::init(ReservedSpace, int, GenRemSet*)+0x3bd
V [libjvm.so+0x56afe4] GenCollectedHeap::initialize()+0x344
V [libjvm.so+0x9751aa] Universe::initialize_heap()+0xca
V [libjvm.so+0x976379] universe_init()+0x79
V [libjvm.so+0x5b1d25] init_globals()+0x65
V [libjvm.so+0x95dc6d] Threads::create_vm(JavaVMInitArgs*, bool*)+0x1ed
V [libjvm.so+0x639fe4] JNI_CreateJavaVM+0x74
C [libjli.so+0x2f8e] JavaMain+0x9e
--------------- P R O C E S S ---------------
Java Threads: ( => current thread )
Other Threads:
=>0x00007f23dc00a000 (exited) JavaThread "Unknown thread" [_thread_in_vm, id=19117, stack(0x00007f23e02c8000,0x00007f23e66c9000)]
VM state:not at safepoint (not fully initialized)
VM Mutex/Monitor currently owned by a thread: None
GC Heap History (0 events):
No events
Deoptimization events (0 events):
No events
Internal exceptions (0 events):
No events
Events (0 events):
No events
Dynamic libraries:
00400000-00401000 r-xp 00000000 ca:01 657555 /alidata/java/jdk1.7.0_79/bin/java
00600000-00601000 rw-p 00000000 ca:01 657555 /alidata/java/jdk1.7.0_79/bin/java
0190a000-0192b000 rw-p 00000000 00:00 0 [heap]
680000000-6d5550000 rw-p 00000000 00:00 0
780000000-800000000 rw-p 00000000 00:00 0
7f23d7402000-7f23d781a000 rw-p 00000000 00:00 0
7f23d781a000-7f23d816f000 rw-p 00000000 00:00 0
7f23d816f000-7f23d8170000 rw-p 00000000 00:00 0
7f23d8170000-7f23d83e0000 rwxp 00000000 00:00 0
7f23d83e0000-7f23db170000 rw-p 00000000 00:00 0
7f23db170000-7f23db18a000 r-xp 00000000 ca:01 655575 /alidata/java/jdk1.7.0_79/jre/lib/amd64/libzip.so
7f23db18a000-7f23db38a000 ---p 0001a000 ca:01 655575 /alidata/java/jdk1.7.0_79/jre/lib/amd64/libzip.so
7f23db38a000-7f23db38b000 rw-p 0001a000 ca:01 655575 /alidata/java/jdk1.7.0_79/jre/lib/amd64/libzip.so
7f23db38b000-7f23db396000 r-xp 00000000 ca:01 786730 /lib/x86_64-linux-gnu/libnss_files-2.19.so
7f23db396000-7f23db595000 ---p 0000b000 ca:01 786730 /lib/x86_64-linux-gnu/libnss_files-2.19.so
7f23db595000-7f23db596000 r--p 0000a000 ca:01 786730 /lib/x86_64-linux-gnu/libnss_files-2.19.so
7f23db596000-7f23db597000 rw-p 0000b000 ca:01 786730 /lib/x86_64-linux-gnu/libnss_files-2.19.so
7f23db597000-7f23db5a2000 r-xp 00000000 ca:01 786774 /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7f23db5a2000-7f23db7a1000 ---p 0000b000 ca:01 786774 /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7f23db7a1000-7f23db7a2000 r--p 0000a000 ca:01 786774 /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7f23db7a2000-7f23db7a3000 rw-p 0000b000 ca:01 786774 /lib/x86_64-linux-gnu/libnss_nis-2.19.so
7f23db7a3000-7f23db7ba000 r-xp 00000000 ca:01 786762 /lib/x86_64-linux-gnu/libnsl-2.19.so
7f23db7ba000-7f23db9b9000 ---p 00017000 ca:01 786762 /lib/x86_64-linux-gnu/libnsl-2.19.so
7f23db9b9000-7f23db9ba000 r--p 00016000 ca:01 786762 /lib/x86_64-linux-gnu/libnsl-2.19.so
7f23db9ba000-7f23db9bb000 rw-p 00017000 ca:01 786762 /lib/x86_64-linux-gnu/libnsl-2.19.so
7f23db9bb000-7f23db9bd000 rw-p 00000000 00:00 0
7f23db9bd000-7f23db9c6000 r-xp 00000000 ca:01 786760 /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7f23db9c6000-7f23dbbc5000 ---p 00009000 ca:01 786760 /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7f23dbbc5000-7f23dbbc6000 r--p 00008000 ca:01 786760 /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7f23dbbc6000-7f23dbbc7000 rw-p 00009000 ca:01 786760 /lib/x86_64-linux-gnu/libnss_compat-2.19.so
7f23dbbc7000-7f23dbbf0000 r-xp 00000000 ca:01 655541 /alidata/java/jdk1.7.0_79/jre/lib/amd64/libjava.so
7f23dbbf0000-7f23dbdf0000 ---p 00029000 ca:01 655541 /alidata/java/jdk1.7.0_79/jre/lib/amd64/libjava.so
7f23dbdf0000-7f23dbdf2000 rw-p 00029000 ca:01 655541 /alidata/java/jdk1.7.0_79/jre/lib/amd64/libjava.so
7f23dbdf2000-7f23dbdff000 r-xp 00000000 ca:01 655568 /alidata/java/jdk1.7.0_79/jre/lib/amd64/libverify.so
7f23dbdff000-7f23dbffe000 ---p 0000d000 ca:01 655568 /alidata/java/jdk1.7.0_79/jre/lib/amd64/libverify.so
7f23dbffe000-7f23dc000000 rw-p 0000c000 ca:01 655568 /alidata/java/jdk1.7.0_79/jre/lib/amd64/libverify.so
7f23dc000000-7f23dc027000 rw-p 00000000 00:00 0
7f23dc027000-7f23e0000000 ---p 00000000 00:00 0
7f23e00c0000-7f23e00c7000 r-xp 00000000 ca:01 786771 /lib/x86_64-linux-gnu/librt-2.19.so
7f23e00c7000-7f23e02c6000 ---p 00007000 ca:01 786771 /lib/x86_64-linux-gnu/librt-2.19.so
7f23e02c6000-7f23e02c7000 r--p 00006000 ca:01 786771 /lib/x86_64-linux-gnu/librt-2.19.so
7f23e02c7000-7f23e02c8000 rw-p 00007000 ca:01 786771 /lib/x86_64-linux-gnu/librt-2.19.so
7f23e02c8000-7f23e02cb000 ---p 00000000 00:00 0
7f23e02cb000-7f23e66c9000 rw-p 00000000 00:00 0 [stack:19117]
7f23e66c9000-7f23e67ce000 r-xp 00000000 ca:01 786531 /lib/x86_64-linux-gnu/libm-2.19.so
7f23e67ce000-7f23e69cd000 ---p 00105000 ca:01 786531 /lib/x86_64-linux-gnu/libm-2.19.so
7f23e69cd000-7f23e69ce000 r--p 00104000 ca:01 786531 /lib/x86_64-linux-gnu/libm-2.19.so
7f23e69ce000-7f23e69cf000 rw-p 00105000 ca:01 786531 /lib/x86_64-linux-gnu/libm-2.19.so
7f23e69cf000-7f23e7544000 r-xp 00000000 ca:01 655595 /alidata/java/jdk1.7.0_79/jre/lib/amd64/server/libjvm.so
7f23e7544000-7f23e7744000 ---p 00b75000 ca:01 655595 /alidata/java/jdk1.7.0_79/jre/lib/amd64/server/libjvm.so
7f23e7744000-7f23e7808000 rw-p 00b75000 ca:01 655595 /alidata/java/jdk1.7.0_79/jre/lib/amd64/server/libjvm.so
7f23e7808000-7f23e7849000 rw-p 00000000 00:00 0
7f23e7849000-7f23e7a04000 r-xp 00000000 ca:01 786768 /lib/x86_64-linux-gnu/libc-2.19.so
7f23e7a04000-7f23e7c03000 ---p 001bb000 ca:01 786768 /lib/x86_64-linux-gnu/libc-2.19.so
7f23e7c03000-7f23e7c07000 r--p 001ba000 ca:01 786768 /lib/x86_64-linux-gnu/libc-2.19.so
7f23e7c07000-7f23e7c09000 rw-p 001be000 ca:01 786768 /lib/x86_64-linux-gnu/libc-2.19.so
7f23e7c09000-7f23e7c0e000 rw-p 00000000 00:00 0
7f23e7c0e000-7f23e7c11000 r-xp 00000000 ca:01 786750 /lib/x86_64-linux-gnu/libdl-2.19.so
7f23e7c11000-7f23e7e10000 ---p 00003000 ca:01 786750 /lib/x86_64-linux-gnu/libdl-2.19.so
7f23e7e10000-7f23e7e11000 r--p 00002000 ca:01 786750 /lib/x86_64-linux-gnu/libdl-2.19.so
7f23e7e11000-7f23e7e12000 rw-p 00003000 ca:01 786750 /lib/x86_64-linux-gnu/libdl-2.19.so
7f23e7e12000-7f23e7e28000 r-xp 00000000 ca:01 656276 /alidata/java/jdk1.7.0_79/lib/amd64/jli/libjli.so
7f23e7e28000-7f23e8028000 ---p 00016000 ca:01 656276 /alidata/java/jdk1.7.0_79/lib/amd64/jli/libjli.so
7f23e8028000-7f23e8029000 rw-p 00016000 ca:01 656276 /alidata/java/jdk1.7.0_79/lib/amd64/jli/libjli.so
7f23e8029000-7f23e8042000 r-xp 00000000 ca:01 786769 /lib/x86_64-linux-gnu/libpthread-2.19.so
7f23e8042000-7f23e8241000 ---p 00019000 ca:01 786769 /lib/x86_64-linux-gnu/libpthread-2.19.so
7f23e8241000-7f23e8242000 r--p 00018000 ca:01 786769 /lib/x86_64-linux-gnu/libpthread-2.19.so
7f23e8242000-7f23e8243000 rw-p 00019000 ca:01 786769 /lib/x86_64-linux-gnu/libpthread-2.19.so
7f23e8243000-7f23e8247000 rw-p 00000000 00:00 0
7f23e8247000-7f23e826a000 r-xp 00000000 ca:01 786764 /lib/x86_64-linux-gnu/ld-2.19.so
7f23e82ba000-7f23e839f000 rw-p 00000000 00:00 0
7f23e839f000-7f23e8455000 rw-p 00000000 00:00 0
7f23e8455000-7f23e845d000 rw-s 00000000 ca:01 1048600 /tmp/hsperfdata_root/19116
7f23e845d000-7f23e8461000 rw-p 00000000 00:00 0
7f23e8463000-7f23e8466000 rw-p 00000000 00:00 0
7f23e8466000-7f23e8467000 r--p 00000000 00:00 0
7f23e8467000-7f23e8469000 rw-p 00000000 00:00 0
7f23e8469000-7f23e846a000 r--p 00022000 ca:01 786764 /lib/x86_64-linux-gnu/ld-2.19.so
7f23e846a000-7f23e846b000 rw-p 00023000 ca:01 786764 /lib/x86_64-linux-gnu/ld-2.19.so
7f23e846b000-7f23e846c000 rw-p 00000000 00:00 0
7fff664a2000-7fff664c3000 rw-p 00000000 00:00 0 [stack]
7fff665fe000-7fff66600000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
VM Arguments:
jvm_args: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms4096m -Xmx4096m -Xss102400K -XX:PermSize=1024m -XX:MaxPermSize=2048m -Djava.endorsed.dirs=/alidata/server/apache-tomcat-7.0.62/endorsed -Dcatalina.base=/alidata/server/apache-tomcat-7.0.62 -Dcatalina.home=/alidata/server/apache-tomcat-7.0.62 -Djava.io.tmpdir=/alidata/server/apache-tomcat-7.0.62/temp
java_command: org.apache.catalina.startup.Bootstrap stop
Launcher Type: SUN_STANDARD
Environment Variables:
JAVA_HOME=/alidata/java/jdk1.7.0_79
PATH=/alidata/server/apache-tomcat-7.0.62/bin:/alidata/java/jdk1.7.0_79/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
SHELL=/bin/bash
Signal Handlers:
SIGSEGV: [libjvm.so+0x9a3bf0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x9a3bf0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x81cb60], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: [libjvm.so+0x81cb60], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGXFSZ: [libjvm.so+0x81cb60], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGILL: [libjvm.so+0x81cb60], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x81e360], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGINT: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
--------------- S Y S T E M ---------------
OS:jessie/sid
uname:Linux 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64
libc:glibc 2.19 NPTL 2.19
rlimit: STACK 8192k, CORE 0k, NPROC 31445, NOFILE 65535, AS infinity
load average:0.00 0.01 0.05
/proc/meminfo:
MemTotal: 4046788 kB
MemFree: 1792160 kB
Buffers: 191356 kB
Cached: 327532 kB
SwapCached: 0 kB
Active: 1988480 kB
Inactive: 119512 kB
Active(anon): 1589220 kB
Inactive(anon): 308 kB
Active(file): 399260 kB
Inactive(file): 119204 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 72 kB
Writeback: 0 kB
AnonPages: 1589124 kB
Mapped: 31780 kB
Shmem: 404 kB
Slab: 106724 kB
SReclaimable: 94344 kB
SUnreclaim: 12380 kB
KernelStack: 1760 kB
PageTables: 7308 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 2023392 kB
Committed_AS: 12751176 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 7616 kB
VmallocChunk: 34359727531 kB
HardwareCorrupted: 0 kB
AnonHugePages: 1513472 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 26624 kB
DirectMap2M: 4298752 kB
CPU:total 1 (32 cores per cpu, 2 threads per core) family 6 model 45 stepping 7, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, aes, ht, tsc
/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
stepping : 7
microcode : 0x70d
cpu MHz : 2300.056
cache size : 15360 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl pni ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm
bogomips : 4600.11
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
Memory: 4k page, physical 4046788k(1792160k free), swap 0k(0k free)
vm_info: Java HotSpot(TM) 64-Bit Server VM (24.79-b02) for linux-amd64 JRE (1.7.0_79-b15), built on Apr 10 2015 11:34:48 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
time: Fri Jun 3 10:36:47 2016
elapsed time: 0 seconds
今天线上 报出这个错误 ,tomcat down 掉无法重启 查看 catalina.sh 发现配置不合理
JAVA_OPTS="-Xms4096m -Xmx4096m -Xss102400K -XX:PermSize=1024m -XX:MaxPermSize=2048m"
我们服务器降级过 目前内存4g
JAVA_OPTS="-Xms3072m -Xmx3072m -Xss1024K -XX:PermSize=128m -XX:MaxPermSize=256m"
问题解决
另外 这个问题也可能是程序自身内存溢出导致