JVM GC 中的问题 想请教一下大家
我的网站现在是 NGINX+TOMCAT(2) 一个nginx 2个tomcat的负载
这几天总是出现垃圾回收问题 莫名其妙的突然的一两秒的时间吃满了老年代 整个业务停顿 两个tomcat基本上是同时的
我们这几天调整了数据库的连接数已经优化了一下代码中容易出现内存泄漏的地方
目前没有出现 秒级别的内存暴增 但是出现了一个新的问题
新生代GC 和FULL GC过于频繁
这是 jstat 中查看到的 GC 状态
明显看到年轻态GC和FULL GC的频率非常 都是几秒就一次 虽然一次的时间非常短 网站访问上基本看不出来
不过我在GC的日志上没有看到任何关于 FULL GC的记录
[root@app-a ~]# cat /alidata1/tomcat7/logs/gc.log2015-12-10-12-44 | grep FULL [root@app-a ~]# cat /alidata1/tomcat7/logs/gc.log2015-12-10-12-44 | grep full [root@app-a ~]#
这是我们JVM的配置信息
JAVA_OPTS="-server -XX:PermSize=128m -XX:MaxPermSize=256m -server -Xms512m -Xmx4096m -Xss1024K -XX:-UseGCOverheadLimit -XX:+UseConcMarkSweepGC -Xloggc:${CATALINA_HOME}/logs/gc.log`date +%Y-%m-%d-%H-%M` -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCTimeStamps -XX:CMSInitiatingOccupancyFraction=40 -XX:ConcGCThreads=4 "
jmap导出的内存信息 这里面我也没看出什么问题会导致密集的FULL GC
[root@app-a ~]# jmap -heap 12722 Attaching to process ID 12722, please wait... Debugger attached successfully. Server compiler detected. JVM version is 24.79-b02 using parallel threads in the new generation. using thread-local object allocation. Concurrent Mark-Sweep GC Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 4294967296 (4096.0MB) NewSize = 348913664 (332.75MB) MaxNewSize = 348913664 (332.75MB) OldSize = 697892864 (665.5625MB) NewRatio = 2 SurvivorRatio = 8 PermSize = 134217728 (128.0MB) MaxPermSize = 268435456 (256.0MB) G1HeapRegionSize = 0 (0.0MB) Heap Usage: New Generation (Eden + 1 Survivor Space): capacity = 314048512 (299.5MB) used = 220341176 (210.13372039794922MB) free = 93707336 (89.36627960205078MB) 70.16150931484114% used Eden Space: capacity = 279183360 (266.25MB) used = 216671904 (206.63442993164062MB) free = 62511456 (59.615570068359375MB) 77.60917556117958% used From Space: capacity = 34865152 (33.25MB) used = 3669272 (3.4992904663085938MB) free = 31195880 (29.750709533691406MB) 10.52418185355968% used To Space: capacity = 34865152 (33.25MB) used = 0 (0.0MB) free = 34865152 (33.25MB) 0.0% used concurrent mark-sweep generation: capacity = 308662272 (294.36328125MB) used = 184305296 (175.76722717285156MB) free = 124356976 (118.59605407714844MB) 59.710989232917974% used Perm Generation: capacity = 152252416 (145.19921875MB) used = 91325336 (87.0946273803711MB) free = 60927080 (58.104591369628906MB) 59.982848482351834% used 64250 interned Strings occupying 7626600 bytes.
贴一段现在 GC的日志
[root@app-a ~]# tail -f /alidata1/tomcat7/logs/gc.log2015-12-10-12-44 18058.549: [CMS-concurrent-sweep: 0.095/0.096 secs] [Times: user=0.13 sys=0.01, real=0.09 secs] 18058.550: [CMS-concurrent-reset-start] 18058.559: [CMS-concurrent-reset: 0.009/0.009 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 18059.067: [GC [1 CMS-initial-mark: 180402K(304880K)] 226932K(611568K), 0.0459380 secs] [Times: user=0.04 sys=0.00, real=0.05 secs] 18059.113: [CMS-concurrent-mark-start] 18059.249: [CMS-concurrent-mark: 0.136/0.136 secs] [Times: user=0.54 sys=0.00, real=0.13 secs] 18059.249: [CMS-concurrent-preclean-start] 18059.252: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 18059.252: [CMS-concurrent-abortable-preclean-start] 18063.021: [GC18063.021: [ParNew: 276297K->3999K(306688K), 0.0095410 secs] 456700K->184426K(611568K), 0.0098160 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 18064.334: [CMS-concurrent-abortable-preclean: 1.537/5.082 secs] [Times: user=2.22 sys=0.06, real=5.08 secs] 18064.335: [GC[YG occupancy: 163511 K (306688 K)]18064.335: [Rescan (parallel) , 0.0537150 secs]18064.389: [weak refs processing, 0.0000370 secs]18064.389: [scrub string table, 0.0029290 secs] [1 CMS-remark: 180427K(304880K)] 343939K(611568K), 0.0569440 secs] [Times: user=0.22 sys=0.01, real=0.06 secs] 18064.392: [CMS-concurrent-sweep-start] 18064.485: [CMS-concurrent-sweep: 0.093/0.093 secs] [Times: user=0.09 sys=0.00, real=0.09 secs] 18064.485: [CMS-concurrent-reset-start] 18064.495: [CMS-concurrent-reset: 0.009/0.009 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 18064.960: [GC [1 CMS-initial-mark: 180169K(304880K)] 358925K(611568K), 0.1598060 secs] [Times: user=0.16 sys=0.00, real=0.16 secs] 18065.120: [CMS-concurrent-mark-start] 18065.252: [CMS-concurrent-mark: 0.132/0.132 secs] [Times: user=0.52 sys=0.00, real=0.13 secs] 18065.252: [CMS-concurrent-preclean-start] 18065.255: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] 18065.255: [CMS-concurrent-abortable-preclean-start] 18068.401: [GC18068.401: [ParNew: 276639K->3932K(306688K), 0.0079560 secs] 456808K->184332K(611568K), 0.0082070 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] CMS: abort preclean due to time 18070.276: [CMS-concurrent-abortable-preclean: 1.496/5.022 secs] [Times: user=1.92 sys=0.04, real=5.02 secs] 18070.277: [GC[YG occupancy: 117489 K (306688 K)]18070.277: [Rescan (parallel) , 0.0366510 secs]18070.314: [weak refs processing, 0.0000350 secs]18070.314: [scrub string table, 0.0028720 secs] [1 CMS-remark: 180400K(304880K)] 297889K(611568K), 0.0397790 secs] [Times: user=0.14 sys=0.00, real=0.04 secs] 18070.318: [CMS-concurrent-sweep-start] 18070.409: [CMS-concurrent-sweep: 0.091/0.092 secs] [Times: user=0.10 sys=0.00, real=0.09 secs] 18070.410: [CMS-concurrent-reset-start] 18070.419: [CMS-concurrent-reset: 0.009/0.009 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 18071.001: [GC [1 CMS-initial-mark: 180351K(304880K)] 367570K(611568K), 0.1737200 secs] [Times: user=0.18 sys=0.00, real=0.18 secs] 18071.175: [CMS-concurrent-mark-start] 18071.314: [CMS-concurrent-mark: 0.139/0.139 secs] [Times: user=0.55 sys=0.00, real=0.14 secs] 18071.314: [CMS-concurrent-preclean-start] 18071.317: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 18071.317: [CMS-concurrent-abortable-preclean-start] 18071.991: [GC18071.991: [ParNew: 276572K->3957K(306688K), 0.0111020 secs] 456923K->184443K(611568K), 0.0113350 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 18073.935: [CMS-concurrent-abortable-preclean: 0.757/2.618 secs] [Times: user=1.15 sys=0.03, real=2.62 secs] 18073.935: [GC[YG occupancy: 154441 K (306688 K)]18073.935: [Rescan (parallel) , 0.0680420 secs]18074.004: [weak refs processing, 0.0000590 secs]18074.004: [scrub string table, 0.0030090 secs] [1 CMS-remark: 180485K(304880K)] 334927K(611568K), 0.0714420 secs] [Times: user=0.21 sys=0.00, real=0.07 secs] 18074.007: [CMS-concurrent-sweep-start] 18074.105: [CMS-concurrent-sweep: 0.097/0.098 secs] [Times: user=0.11 sys=0.00, real=0.10 secs] 18074.106: [CMS-concurrent-reset-start] 18074.115: [CMS-concurrent-reset: 0.009/0.009 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
GC的日志中基本上都是正常的 CMS GC 不过这种CMS GC感觉也比以前的要多了很多
以前我们的GC的日志中基本上都是 新生代的记录
[root@app-a ~]# tail -n 100 /alidata1/tomcat7/logs/gc.log2015-12-09-15-55 40362.760: [GC40362.760: [ParNew: 275335K->2655K(306688K), 0.0117540 secs] 2688676K->2416017K(3853640K), 0.0119800 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] 40368.978: [GC40368.979: [ParNew: 275295K->2538K(306688K), 0.0168210 secs] 2688657K->2415918K(3853640K), 0.0170130 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 40378.913: [GC40378.913: [ParNew: 275178K->2823K(306688K), 0.0157030 secs] 2688558K->2416313K(3853640K), 0.0159070 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 40387.111: [GC40387.112: [ParNew: 275463K->2643K(306688K), 0.0164300 secs] 2688953K->2416185K(3853832K), 0.0166460 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] 40394.449: [GC40394.449: [ParNew: 275283K->2281K(306688K), 0.0129860 secs] 2688825K->2415853K(3853832K), 0.0132070 secs] [Times: user=0.04 sys=0.00, real=0.02 secs] 40404.618: [GC40404.618: [ParNew: 274921K->2720K(306688K), 0.0122430 secs] 2688493K->2416543K(3854024K), 0.0124570 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 40415.475: [GC40415.476: [ParNew: 275360K->2434K(306688K), 0.0118850 secs] 2689183K->2416296K(3854024K), 0.0121560 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 40426.315: [GC40426.315: [ParNew: 275074K->2907K(306688K), 0.0135730 secs] 2688936K->2416783K(3854024K), 0.0138070 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]