JVM调优-剖析ygc越来越慢

cpu就那么多,吞吐量和延时只能依靠调整jvm取最优。想让马儿跑,还想马儿不吃草?——加cpu。

前言

系统服务的核心指标即cpu,mem,disk,net。在稳定的滴滴容器云上,以及结合一般的web服务属性,mem,disk,net往往不会成为我们的瓶颈。最难搞的就是cpu。先抛出来一个大图,大家感受下通过jvm调优对cpu的影响


image.png

从图中可看,大致有三个阶段:
生不如死阶段:经常听到内存泄露,内存溢出,没错,今天见到了cpu泄露。这个阶段,cpu随着时间,会慢慢跌到0点。

苟延残喘阶段:调小jvm年轻代之后,就进入了我们的第二阶段,锯齿型cpu使用率,伴随着fgc的发生都会恢复cpu的使用

假药停阶段:因为考虑我们是web型高并发服务,所有极大部分对象都是朝夕生死,所以我调大了年轻代的内存,调小了老年代的内存,观察cpu发现很稳定。。其实不然,此处发现fgc频率和ygc频率一致了。。。

gc信息

理论分析

对于web服务,高并发往往是我们追求的目标,而高的并发量,自然代表会有更多的对象创建和销毁,这个阶段基本上发生在年轻代。所以年轻代需要尽可能的大。而上文说的cpu掉底,正是发生在年轻代的垃圾收集导致的。

ygc步骤

此时,对于使用标记-复制算法的ygc,我们一般有以下几个步骤:

1.从GC Roots开始进行可达性分析,分析当前处于年轻代对象是否存活(极有可能出现问题)

2.将eden区和survivor from区的对象拷贝到to区(耗时稳定,和内存成正比)

3.清理eden区和from区(耗时稳定,和内存成正比)

而针对于ygc越来越慢的问题的,我们其实需要去分析的就是上边的几步骤的耗时。

GC-ROOT

能够成为gc-root的对象有以下这些:

所有JNI Handles JNIHandles
所有线程的栈
所有被Synchronize锁持有的对象
VM内实现的MBean所持有的对象
JVMTI所持有的对象
(可选)所有已加载的类 或 所有已加载的系统类
(可选)所有驻留字符串(StringTable)
(可选)代码缓存(CodeCache)
(可选)PermGen的remember set所记录的存在跨代引用的区域

实操过程

而如果使用parnew和cms垃圾回收器,官方并没有提供ygc在各阶段耗时的方法,,万幸发现AliJdk中,提供了该功能,通过 PrintGCRootsTraceTime能打印出 ParNewCMS的详细耗时。

  • 从Alibaba Dragonwell的Github页面下载二进制tar包,链接 https://github.com/alibaba/dragonwell8/releases
  • 将下载下来的tar包解压到目标安装目录即可
  • 将jdk启动路径替换为新安装jdk路径即可,记得添加参数 -XX:+PrintGCRootsTraceTime.

ygc对比:

采样1:
2020-05-22T10:08:41.685+0800: 44736.891: [GC (Allocation Failure) 2020-05-22T10:08:41.685+0800: 44736.891: [ParNew      [GC Worker Start (ms):  44736891.4  44736891.5  44736891.5  44736891.5  44736891.6  44736891.6  44736891.7  44736891.7
       Min: 44736891.4, Avg: 44736891.6, Max: 44736891.7, Diff: 0.2]
      [Root Processing (ms):  75.5  75.5  75.4  74.8  75.4  75.5  74.9  75.3
       Min: 74.8, Avg: 75.3, Max: 75.5, Diff: 0.7, Sum: 602.2]
         [Thread Roots (ms):  0.0  0.6  0.6  0.6  0.5  0.5  0.5  0.4
          Min: 0.0, Avg: 0.5, Max: 0.6, Diff: 0.6, Sum: 3.7]
         [StringTable Roots (ms):  59.3  67.2  67.4  67.1  67.3  67.4  67.2  67.3
          Min: 59.3, Avg: 66.3, Max: 67.4, Diff: 8.0, Sum: 530.2]
         [Universe Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [JNI Handles Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [ObjectSynchronizer Roots (ms):  0.0  0.0  0.0  0.3  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.3, Diff: 0.3, Sum: 0.3]
         [FlatProfiler Roots (ms):  0.0  0.0  0.0  0.0  0.1  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.1]
         [Management Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [SystemDictionary Roots (ms):  0.0  0.1  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.1]
         [CLDG Roots (ms):  8.7  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 1.1, Max: 8.7, Diff: 8.7, Sum: 8.7]
         [CodeCache Roots (ms):  0.0  0.2  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.2, Sum: 0.2]
         [JVMTI Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [older-gen scanning (ms):  7.5  7.5  7.4  6.8  7.5  7.6  7.2  7.6
          Min: 6.8, Avg: 7.4, Max: 7.6, Diff: 0.8, Sum: 59.0]
      [GC Worker Other (ms):  0.1  0.1  0.1  0.8  0.1  0.0  0.5  0.1
       Min: 0.0, Avg: 0.2, Max: 0.8, Diff: 0.8, Sum: 1.9]
      [GC Worker Total (ms):  75.6  75.6  75.6  75.5  75.5  75.5  75.4  75.4
       Min: 75.4, Avg: 75.5, Max: 75.6, Diff: 0.2, Sum: 604.1]
      [GC Worker End (ms):  44736967.1  44736967.1  44736967.1  44736967.1  44736967.1  44736967.1  44736967.1  44736967.1
       Min: 44736967.1, Avg: 44736967.1, Max: 44736967.1, Diff: 0.0]
: 1743479K->94992K(1887488K), 0.0804062 secs] 2133152K->490104K(3984640K), 0.0806634 secs] [Times: user=0.54 sys=0.07, real=0.08 secs]
采样2:
2020-05-22T10:13:29.342+0800: 45024.548: [GC (Allocation Failure) 2020-05-22T10:13:29.342+0800: 45024.548: [ParNew      [GC Worker Start (ms):  45024548.5  45024548.6  45024548.6  45024548.6  45024549.7  45024552.0  45024552.0  45024554.2
       Min: 45024548.5, Avg: 45024550.3, Max: 45024554.2, Diff: 5.6]
      [Root Processing (ms):  113.7  113.6  113.4  113.2  112.3  110.0  109.8  107.7
       Min: 107.7, Avg: 111.7, Max: 113.7, Diff: 6.0, Sum: 893.8]
         [Thread Roots (ms):  0.0  2.9  1.3  2.8  0.2  0.0  0.0  0.0
          Min: 0.0, Avg: 0.9, Max: 2.9, Diff: 2.9, Sum: 7.2]
         [StringTable Roots (ms):  100.5  104.0  105.5  104.3  105.6  103.6  103.4  101.3
          Min: 100.5, Avg: 103.5, Max: 105.6, Diff: 5.1, Sum: 828.1]
         [Universe Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [JNI Handles Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [ObjectSynchronizer Roots (ms):  0.0  0.0  0.2  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.2, Sum: 0.2]
         [FlatProfiler Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [Management Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [SystemDictionary Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [CLDG Roots (ms):  6.5  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.8, Max: 6.5, Diff: 6.5, Sum: 6.5]
         [CodeCache Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [JVMTI Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [older-gen scanning (ms):  6.7  6.8  6.4  6.2  6.4  6.4  6.4  6.4
          Min: 6.2, Avg: 6.5, Max: 6.8, Diff: 0.6, Sum: 51.7]
      [GC Worker Other (ms):  0.0  0.1  0.3  0.4  0.3  0.3  0.5  0.4
       Min: 0.0, Avg: 0.3, Max: 0.5, Diff: 0.4, Sum: 2.3]
      [GC Worker Total (ms):  113.7  113.7  113.7  113.6  112.6  110.3  110.3  108.1
       Min: 108.1, Avg: 112.0, Max: 113.7, Diff: 5.6, Sum: 896.1]
      [GC Worker End (ms):  45024662.3  45024662.3  45024662.3  45024662.3  45024662.3  45024662.3  45024662.3  45024662.3
       Min: 45024662.3, Avg: 45024662.3, Max: 45024662.3, Diff: 0.0]
: 1802020K->124247K(1887488K), 0.1179329 secs] 2406433K->734108K(3984640K), 0.1181915 secs] [Times: user=0.79 sys=0.10, real=0.12 secs]

如上对比可知,ygc在做root processing阶段时,耗时出现了明显的增高:

[Root Processing (ms):  75.5  75.5  75.4  74.8  75.4  75.5  74.9  75.3 
[Root Processing (ms):  113.7  113.6  113.4  113.2  112.3  110.0  109.8  107.7

而其中正是在扫描 StringTable Roots 耗时出现了明显的增高

[StringTable Roots (ms):  59.3  67.2  67.4  67.1  67.3  67.4  67.2  67.3
[StringTable Roots (ms):  100.5  104.0  105.5  104.3  105.6  103.6  103.4  101.3

以上信息我们只能观察到ygc耗时是越来越高的,只能间接推断出来影响到我们的cpu占用,所以我在另外一个f服务又复现了ygc耗时高和gc线程占用cpu比例高的现象:

服务启动1分钟后:其中一个gc占用的cpu比例为0.7%,StringTable扫描平均耗时在4.4ms,看下图

gc线程

gc日志

随着时间的推移,服务启动10分钟后:其中一个gc占用的cpu比例为1.0-2.3%徘徊,StringTable扫描平均耗时在26.7ms,看下图

gc日志

随着时间的推移,服务启动15分钟后:其中一个gc占用的cpu比例为1.7-4.0%徘徊,StringTable扫描平均耗时在39.6ms

gc日志

以上信息,我们可以得知,ygc的StringTable扫描对于CPU的占用是有影响的。

什么是StringTable

那么问题就出现在这个StringTable Roots上。那么接下来分析一波这个玩意是干啥的。

StringTable的数据结构是一个HashTable,代码调用String.intern()时,会先去该StringTable中查找是否存在相同的字符串。以此达到缓存的作用。

JVM字符串常量池在JDK8 中是将字符串对象放在堆中(jdk7是放到perm区,但是那样会经常导致永久代内存溢出,所以在jdk8中将它移到了heap区),通过StringTable来维护引用,StringTable 结构是类HashMap的实现,如果数组长度过短(String Table (JDK8)默认长度时60013 ),hash碰撞频繁就会导致链表过长,影响查找效率,耗时过长。

优化过程

寻找StringTable触发点

找到了问题的根因,那么我们只需要分析出来究竟是谁引起的StringTable越来越大即可。查阅资料后发现:
jackson内部使用了String.intern对key进行了缓存优化,故优化了两个地方

  1. JsonFormat工具,使用的是jackson工具
  2. SpringBoot web 默认使用的是jackson工具来序列化参数
优化方式如下:
private static JsonFactory  factory = new JsonFactory().disable(JsonFactory.Feature.INTERN_FIELD_NAMES);
private static ObjectMapper objectMapper = new ObjectMapper(factory).configure(
    DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
    .configure(DeserializationFeature.FAIL_ON_IGNORED_PROPERTIES, false)
    .configure(DeserializationFeature.FAIL_ON_INVALID_SUBTYPE, false)
    .configure(SerializationFeature.FAIL_ON_EMPTY_BEANS, false);

但是优化之后,发现StringTable仍然是在慢慢变大(没有之前变大的那么快了),所以肯定是还有其他工具使用了String.intern方法了。
故有个思路就是运行期检测String,intern方法的调用栈,以此来发现究竟是谁调用了它。

  1. 使用btrace动态监测方法执行:工具是很强大。。但是奈何intern是一个native方法,该工具无法作用到native方法。
  2. 在寻找其他方法中(排查问题可能是个持续过程,很多隐藏的问题不是一时半会就能解决的,如果暂时没有思路,不妨跳出当前的思考范式,换个角度先把问题缓解或解决一部分)
  3. 目前正在尝试使用openjdk,修改StringTable的c++源码,通过这种方式来打印当前方法的调用栈,以此来确认是谁调用了string.intern方法。
StringTableSize优化

在寻找StringTable的触发点的同时,我们还是要保证线上服务的稳定运行的。所以对StringTable大小进行了优化。
调大StringTableSize=600131,参数命令: -XX:StringTableSize=600131 以此先缓解StringTable的问题,如图,在单机250qps读+250qps写的压力下,我们可以看到系统cpu idle在75-85之间徘徊,延时稳定,基本满足我们的要求。


image.png

image.png
控制变量优化大法

虽说通过调整StringTableSize能缓解这个问题,但是随着时间的增加,接口延时越来越高的问题,同样不是我们业务能接受的。所以这里,我采取了控制变量法,来排除具体是哪个地方导致的StringTable增大。

  • 第一步,因为我之前是同时压读写两个接口,所以不能定位到具体哪个接口导致的问题,所以我分别对两个接口(特征查询,特征上报)进行了单独的压测。 最终发现,特征查询接口对StringTable的影响很大。

  • 第二步,定位到特征查询接口对StringTable有直接的影响,那么我就分析是不是这个接口的代码逻辑里边有涉及到String.intern()相关的逻辑。所以我将这个接口的逻辑拆分为多步,然后依次减少这个接口的操作步数,这样就能够精确的定位到是哪一步的代码存在问题了。

  • 第三步,通过上一步的多次验证,发现我的接口不加任何逻辑,就是请求过来,直接返回结果,StringTable仍然是处于增长的状态。所以答案渐渐浮出水面,我自己写的代码逻辑其实不存在问题。问题发生在代码逻辑执行之前。

  • 第四步,所以这个时候,能想到的就是对于web请求,我们使用了webutil工具,对于请求数据和返回数据做了通用的处理。顺势一想,确实webutil工具肯定会有反序列化数据的逻辑,分析了一下webutil的代码,发现确实有使用jackson工具的地方


    image.png

所以将该工具进行了,、都禁用了字符串缓存功能。效果不错,以下是对比:

优化webutil之前,随着时间的推移,StringTable的查询耗时在25.5ms左右,

2020-06-04T12:05:03.332+0800: 145.688: [GC (Allocation Failure) 2020-06-04T12:05:03.332+0800: 145.688: [ParNew      [GC Worker Start (ms):  145688.5  145688.6  145688.7  145688.7  145688.7  145688.7  145688.7  145688.7  145688.8  145688.8  145688.8  145688.8  145688.8
       Min: 145688.5, Avg: 145688.7, Max: 145688.8, Diff: 0.3]
      [Root Processing (ms):  31.1  36.5  30.4  29.6  30.4  30.6  30.8  34.1  29.9  29.8  29.7  29.3  29.7
       Min: 29.3, Avg: 30.9, Max: 36.5, Diff: 7.2, Sum: 401.9]
         [Thread Roots (ms):  0.0  0.5  0.5  0.5  0.5  0.5  0.4  0.5  0.4  0.4  0.4  0.2  0.3
          Min: 0.0, Avg: 0.4, Max: 0.5, Diff: 0.5, Sum: 5.0]
         [StringTable Roots (ms):  21.3  26.1  25.9  25.8  25.9  25.7  25.8  25.7  25.8  25.7  25.8  26.0  25.8
          Min: 21.3, Avg: 25.5, Max: 26.1, Diff: 4.8, Sum: 331.2]
         [Universe Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [JNI Handles Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [ObjectSynchronizer Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [FlatProfiler Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [Management Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [SystemDictionary Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [CLDG Roots (ms):  5.2  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.4, Max: 5.2, Diff: 5.2, Sum: 5.2]
         [CodeCache Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [JVMTI Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [older-gen scanning (ms):  4.6  10.0  4.1  3.3  4.1  4.4  4.6  7.9  3.7  3.6  3.5  3.1  3.6
          Min: 3.1, Avg: 4.7, Max: 10.0, Diff: 6.9, Sum: 60.5]
      [GC Worker Other (ms):  5.6  0.0  6.1  6.9  6.1  5.9  5.6  2.4  6.9  6.6  6.7  7.0  6.6
       Min: 0.0, Avg: 5.6, Max: 7.0, Diff: 7.0, Sum: 72.4]
      [GC Worker Total (ms):  36.7  36.5  36.5  36.5  36.5  36.4  36.4  36.4  36.9  36.4  36.4  36.4  36.4
       Min: 36.4, Avg: 36.5, Max: 36.9, Diff: 0.5, Sum: 474.3]
      [GC Worker End (ms):  145725.2  145725.2  145725.2  145725.2  145725.2  145725.2  145725.2  145725.2  145725.6  145725.2  145725.2  145725.2  145725.2
       Min: 145725.2, Avg: 145725.2, Max: 145725.6, Diff: 0.5]

优化webutil之后,随着时间的推移,StringTable的查询耗时在0.4ms左右,

2020-06-04T12:00:29.907+0800: 801.182: [GC (Allocation Failure) 2020-06-04T12:00:29.907+0800: 801.183: [ParNew      [GC Worker Start (ms):  801182.7  801185.0  801185.1  801185.1  801185.8  801187.4  801187.5  801187.6  801188.5  801188.5  801189.5  801189.5  801189.5
       Min: 801182.7, Avg: 801187.0, Max: 801189.5, Diff: 6.8]
      [Root Processing (ms):  10.9  8.2  10.0  8.6  7.3  5.6  5.6  5.8  5.5  5.6  3.9  3.5  3.5
       Min: 3.5, Avg: 6.5, Max: 10.9, Diff: 7.5, Sum: 84.0]
         [Thread Roots (ms):  0.0  1.3  0.6  0.6  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.2, Max: 1.3, Diff: 1.3, Sum: 2.6]
         [StringTable Roots (ms):  0.0  0.8  1.3  1.3  1.2  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.4, Max: 1.3, Diff: 1.3, Sum: 4.6]
         [Universe Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [JNI Handles Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [ObjectSynchronizer Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [FlatProfiler Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [Management Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [SystemDictionary Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [CLDG Roots (ms):  9.2  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.7, Max: 9.2, Diff: 9.2, Sum: 9.2]
         [CodeCache Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [JVMTI Roots (ms):  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
          Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0, Sum: 0.0]
         [older-gen scanning (ms):  1.7  6.1  8.0  6.6  6.0  5.6  5.6  5.7  5.5  5.6  3.9  3.5  3.5
          Min: 1.7, Avg: 5.2, Max: 8.0, Diff: 6.4, Sum: 67.5]
      [GC Worker Other (ms):  1.5  2.0  0.0  1.4  2.0  2.1  2.0  1.8  1.1  1.0  1.7  2.1  2.1
       Min: 0.0, Avg: 1.6, Max: 2.1, Diff: 2.1, Sum: 20.8]
      [GC Worker Total (ms):  12.4  10.2  10.0  10.0  9.3  7.7  7.6  7.6  6.6  6.6  5.7  5.6  5.6
       Min: 5.6, Avg: 8.1, Max: 12.4, Diff: 6.8, Sum: 104.9]
      [GC Worker End (ms):  801195.1  801195.1  801195.1  801195.1  801195.1  801195.1  801195.1  801195.1  801195.1  801195.1  801195.1  801195.1  801195.1
       Min: 801195.1, Avg: 801195.1, Max: 801195.1, Diff: 0.0]
: 840630K->2352K(943744K), 0.0164835 secs] 892655K->54377K(4089472K), 0.0166493 secs] [Times: user=0.08 sys=0.03, real=0.02 secs]

参考文献:

jvm优化
蚂蚁消息中间件 (MsgBroker) 在 YGC 优化上的探索
JVM源码分析之String.intern()导致的YGC不断变长
Jackson的一个坑
深入解析String#intern
深入 JVM 彻底剖析 ygc 越来越慢的原因(下)

你可能感兴趣的:(JVM调优-剖析ygc越来越慢)