Netty Memory Leak Error

1. Problem

1.1 Exception

java.lang.OutOfMemoryError: GC overhead limit exceeded
        at com.jcraft.jzlib.InfBlocks.<init>(InfBlocks.java:114)
        at com.jcraft.jzlib.Inflate.inflateInit(Inflate.java:181)
        at com.jcraft.jzlib.Inflater.init(Inflater.java:128)
        at com.jcraft.jzlib.Inflater.init(Inflater.java:114)
        at com.jcraft.jzlib.Inflater.init(Inflater.java:98)
        at io.netty.handler.codec.compression.JZlibDecoder.<init>(JZlibDecoder.java:50)
        at io.netty.handler.codec.compression.ZlibCodecFactory.newZlibDecoder(ZlibCodecFactory.java:111)
        at com.**.ClientHandler.messageReceived(ClientHandler.java:96)
        at com.**.ClientHandler.messageReceived(ClientHandler.java:1)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)
        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
        at io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:254)
        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)
        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
        at io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)
        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
        at io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:243)
        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)
        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
        at io.netty.channel.PausableChannelEventExecutor.invokeChannelRead(PausableChannelEventExecutor.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:389)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:243)
        at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:84)
        at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)

2. Analysis

2.1 Dump Analysis

2. 2 One Client Monitoring

memory leak caused by thread leak? 

noteworthy:

(1) heap size getting smaller

(2) threads number getting larger

(3) some threads in event loop group are in WAITING status.

2.3 Two clients monitoring

Noteworthy: after the second client started, threads number quickly rose up, so the root cause is clear, there are race conditions which leads to dead lock and WAITING threads.

two clients overnight:

 

Reference:

[1] Netty版本升级血泪史之线程篇. http://www.infoq.com/cn/articles/netty-version-upgrade-history-thread-part/