Netty 解决粘包半包 解码器ByteToMessageDecoder

本文主要分析 ByteToMessageDecoder,它是用来解决粘包半包处理主要处理类。

ByteToMessageDecoder

什么是粘包半包?
tcp是基于报文的协议,当使用netty传输数据时,可能一次读数据不是一个完整的数据,可能读取的是一半或者多余具体应用层数据的数据。

如何解决这样的问题呢?
可以用一个全局缓存池,每次解码器进行解码时,只能解码一个完整数据,不够或者多余则放到下次解码。
ByteToMessageDecoder 就是这样解决这样的问题的。

解码器原理

在netty读取数据时,会尝试一次性最多进行16次IO读取,每次读取都会调用 pipeline.fireChannelRead(byteBuf) 方法:
AbstractNioByteChannel$NioByteUnsaferead 方法:

                do {
                    byteBuf = allocHandle.allocate(allocator);
                    allocHandle.lastBytesRead(doReadBytes(byteBuf));
                    if (allocHandle.lastBytesRead() <= 0) {
                        // nothing was read. release the buffer.
                        byteBuf.release();
                        byteBuf = null;
                        close = allocHandle.lastBytesRead() < 0;
                        if (close) {
                            // There is nothing left to read as we received an EOF.
                            readPending = false;
                        }
                        break;
                    }

                    allocHandle.incMessagesRead(1);
                    readPending = false;
                    pipeline.fireChannelRead(byteBuf);
                    byteBuf = null;
                } while (allocHandle.continueReading());

这样,经过pipeline传递,如果有配置 ByteToMessageDecoder 的子类作为handler处理器,则会进入到ByteToMessageDecoderchannelRead:方法:

    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
        if (msg instanceof ByteBuf) {
            CodecOutputList out = CodecOutputList.newInstance();
            try {
                first = cumulation == null;
                cumulation = cumulator.cumulate(ctx.alloc(),
                        first ? Unpooled.EMPTY_BUFFER : cumulation, (ByteBuf) msg);
                callDecode(ctx, cumulation, out);
            } catch (DecoderException e) {
                throw e;
            } catch (Exception e) {
                throw new DecoderException(e);
            } finally {
                if (cumulation != null && !cumulation.isReadable()) {
                    numReads = 0;
                    cumulation.release();
                    cumulation = null;
                } else if (++ numReads >= discardAfterReads) {
                    // We did enough reads already try to discard some bytes so we not risk to see a OOME.
                    // See https://github.com/netty/netty/issues/4275
                    numReads = 0;
                    discardSomeReadBytes();
                }

                int size = out.size();
                firedChannelRead |= out.insertSinceRecycled();
                fireChannelRead(ctx, out, size);
                out.recycle();
            }
        } else {
            ctx.fireChannelRead(msg);
        }
    }
  1. 只有msg 类型为ByteBuf,才会进行处理。
  2. 构造一个 CodecOutputList,CodecOutputList是一个Netty自定义的 List,继承自 AbstractList,每次解码后,如果满足要求,会将解码后数据放入CodecOutputList 中。。
  3. Cumulator是其内部定义的字节合并容器,对于字节数据合并,有两种实现,一种是使用字节拷贝方式,将一个里面所有数据,拷贝到另一个ByteBuf中,另一种是通过CompositeByteBuf 这种特殊的ByteBuf进行逻辑上的合并。
                cumulation = cumulator.cumulate(ctx.alloc(),
                        first ? Unpooled.EMPTY_BUFFER : cumulation, (ByteBuf) msg);

当 是第一次解析,则使用空ByteBuf和msg进行合并,否则就使用上一次没有用完的cumulation进行合并。
4. 调用 callDecode 对cumulation进行解码。
5. 接完码后,会在finally块中,调用fireChannelRead 进行数据传递。

callDecode方法

ByteToMessageDecodercallDecode 方法:

    protected void callDecode(ChannelHandlerContext ctx, ByteBuf in, List out) {
        try {
            while (in.isReadable()) {
                int outSize = out.size();
                if (outSize > 0) {
                    fireChannelRead(ctx, out, outSize);
                    out.clear();
                    // Check if this handler was removed before continuing with decoding.
                    // If it was removed, it is not safe to continue to operate on the buffer.
                    //
                    // See:
                    // - https://github.com/netty/netty/issues/4635
                    if (ctx.isRemoved()) {
                        break;
                    }
                    outSize = 0;
                }
                int oldInputLength = in.readableBytes();
                decodeRemovalReentryProtection(ctx, in, out);
                // Check if this handler was removed before continuing the loop.
                // If it was removed, it is not safe to continue to operate on the buffer.
                //
                // See https://github.com/netty/netty/issues/1664
                if (ctx.isRemoved()) {
                    break;
                }
                if (outSize == out.size()) {
                    if (oldInputLength == in.readableBytes()) {
                        break;
                    } else {
                        continue;
                    }
                }
                if (oldInputLength == in.readableBytes()) {
                    throw new DecoderException(
                            StringUtil.simpleClassName(getClass()) +
                                    ".decode() did not read anything but decoded a message.");
                }
                if (isSingleDecode()) {
                    break;
                }
            }
        } catch (DecoderException e) {
            throw e;
        } catch (Exception cause) {
            throw new DecoderException(cause);
        }
    }
 
  
  1. 如果有读取的数据,则一直循环读下去。
    有下面的情况将会退出:
  • 当前ChannelHandlerContext被清除了(remove)
  • 没有读取到任何数据
  • singleDecode 为true,即只配置一次。
  1. 记录当前解码的结果,如果有结果,则调用 fireChannelRead(ctx, out, outSize); 往下一个handler传递。清空out数组。
  2. 记录当前字节数,而后调用 decodeRemovalReentryProtection 进行具体解码操作。

ByteToMessageDecoderdecodeRemovalReentryProtection 方法:

    final void decodeRemovalReentryProtection(ChannelHandlerContext ctx, ByteBuf in, List out)
            throws Exception {
        decodeState = STATE_CALLING_CHILD_DECODE;
        try {
            decode(ctx, in, out);
        } finally {
            boolean removePending = decodeState == STATE_HANDLER_REMOVED_PENDING;
            decodeState = STATE_INIT;
            if (removePending) {
                fireChannelRead(ctx, out, out.size());
                out.clear();
                handlerRemoved(ctx);
            }
        }
    }
 
  
  1. 记录当前状态,而后调用子类 decode 方法进行具体解码操作。
  2. 如果是 STATE_HANDLER_REMOVED_PENDING 状态,并调用对应时间

总体的 ByteToMessageDecoder 解码器就是到这里了,它并没有给出具体的解码方法,只是收集读取到的字节数据,并调用子类具体decode方法进行解码。

具体解码器分析

接下来以几个netty自带解码器框架来进行分析。

LineBasedFrameDecoder

基于行的解码器,即以 \n 作为分隔符,即一行就是一次解码后的内容。
LineBasedFrameDecoderdecode 方法:

    @Override
    protected final void decode(ChannelHandlerContext ctx, ByteBuf in, List out) throws Exception {
        Object decoded = decode(ctx, in);
        if (decoded != null) {
            out.add(decoded);
        }
    }
 
  

关键代码:

                if (stripDelimiter) {
                    frame = buffer.readRetainedSlice(length);
                    buffer.skipBytes(delimLength);
                } else {
                    frame = buffer.readRetainedSlice(length + delimLength);
                }

                return frame;
  1. 根据分隔符的到开始的长度读取,并返回一个slice,并且增加原ByteBuf的引用计数。
  2. 最终返回的,仍然是一个ByteBuf,但是这个ByteBuf已经是完整的netty数据包了。
DelimiterBasedFrameDecoder

基于分隔符的解码器,和 LineBasedFrameDecoder 类似,只是LineBasedFrameDecoder可以配置多个分隔符,大师decode解码时,每次回以长度帧最小的分隔符进行解码。

FixedLengthFrameDecoder

固定长度解码器,即配置长度,每次只读取这么多数据。
FixedLengthFrameDecoderdecode 方法:

    @Override
    protected final void decode(ChannelHandlerContext ctx, ByteBuf in, List out) throws Exception {
        Object decoded = decode(ctx, in);
        if (decoded != null) {
            out.add(decoded);
        }
    }

    protected Object decode(
            @SuppressWarnings("UnusedParameters") ChannelHandlerContext ctx, ByteBuf in) throws Exception {
        if (in.readableBytes() < frameLength) {
            return null;
        } else {
            return in.readRetainedSlice(frameLength);
        }
    }
 
  
LengthFieldBasedFrameDecoder

可以用单独的header指定body大小的decoder,这个decoder最强大。
LengthFieldBasedFrameDecoder 构造方法为:

    public LengthFieldBasedFrameDecoder(
            ByteOrder byteOrder, int maxFrameLength, int lengthFieldOffset, int lengthFieldLength,
            int lengthAdjustment, int initialBytesToStrip, boolean failFast) {
        this.byteOrder = checkNotNull(byteOrder, "byteOrder");
        checkPositive(maxFrameLength, "maxFrameLength");
        checkPositiveOrZero(lengthFieldOffset, "lengthFieldOffset");
        checkPositiveOrZero(initialBytesToStrip, "initialBytesToStrip");
        if (lengthFieldOffset > maxFrameLength - lengthFieldLength) {
            throw new IllegalArgumentException(
                    "maxFrameLength (" + maxFrameLength + ") " +
                    "must be equal to or greater than " +
                    "lengthFieldOffset (" + lengthFieldOffset + ") + " +
                    "lengthFieldLength (" + lengthFieldLength + ").");
        }
        this.maxFrameLength = maxFrameLength;
        this.lengthFieldOffset = lengthFieldOffset;
        this.lengthFieldLength = lengthFieldLength;
        this.lengthAdjustment = lengthAdjustment;
        this.lengthFieldEndOffset = lengthFieldOffset + lengthFieldLength;
        this.initialBytesToStrip = initialBytesToStrip;
        this.failFast = failFast;
    }

前面几个为检查参数正确性,后面为基本赋值:

  1. maxFrameLength :最大帧长度,即整个解析出来的帧大小不能超过这个长度。
  2. lengthFieldOffset :header偏移量
  3. lengthFieldLength :header 长度
  4. lengthAdjustment :调整量,frameLength += lengthAdjustment + lengthFieldEndOffset;
  5. lengthFieldEndOffset :初始读取量
  6. initialBytesToStrip :跳过的字节数,跳过之后,actualFrameLength = frameLengthInt - initialBytesToStrip;

从Java doc注释几个例子来理解下整个decoder流程:

  1. 当参数为:
lengthFieldOffset=0  // 
lengthFieldLength=2  // 
lengthAdjustment=0   // 
initialBytesToStrip=0   // 

这样如果解析前的ByteBuf为左边,经过decoder解析出来完整的包为右边

 BEFORE DECODE (14 bytes)         AFTER DECODE (14 bytes)
 +--------+----------------+      +--------+----------------+
 | Length | Actual Content |----->| Length | Actual Content |
 | 0x000C | "HELLO, WORLD" |      | 0x000C | "HELLO, WORLD" |
 +--------+----------------+      +--------+----------------+
  1. 当参数为:
lengthFieldOffset=0  // 
lengthFieldLength=2  // 
lengthAdjustment=0   // 
initialBytesToStrip=2   // 

最后的初始字节进位,则说明是跳过部分字节开始:

 BEFORE DECODE (14 bytes)         AFTER DECODE (12 bytes)
 +--------+----------------+      +----------------+
 | Length | Actual Content |----->| Actual Content |
 | 0x000C | "HELLO, WORLD" |      | "HELLO, WORLD" |
 +--------+----------------+      +----------------+
  1. 当参数为:
lengthFieldOffset=0  // 
lengthFieldLength=2  // 
lengthAdjustment=-2   // 
initialBytesToStrip=0   // 

此时 lengthAdjustment,解密后为:

 +--------+----------------+      +--------+----------------+
 | Length | Actual Content |----->| Length | Actual Content |
 | 0x000E | "HELLO, WORLD" |      | 0x000E | "HELLO, WORLD" |
 +--------+----------------+      +--------+----------------+

LengthFieldBasedFrameDecoderdecode 方法:

    @Override
    protected final void decode(ChannelHandlerContext ctx, ByteBuf in, List out) throws Exception {
        Object decoded = decode(ctx, in);
        if (decoded != null) {
            out.add(decoded);
        }
    }
 
  

仍然是decode方法:
LengthFieldBasedFrameDecoderdecode 方法:

   protected Object decode(ChannelHandlerContext ctx, ByteBuf in) throws Exception {
        if (discardingTooLongFrame) {
            // 如果上一次处理,超出了帧大小,则将多余的字节丢弃。
            discardingTooLongFrame(in);
        }
        if (in.readableBytes() < lengthFieldEndOffset) {
        	// 如果读取字节数,比长度字段偏移量都少,说明是个半包,直接退出
            return null;
        }
		// actualLengthFieldOffset 表示长度字段偏移量
        int actualLengthFieldOffset = in.readerIndex() + lengthFieldOffset;
        // 获取帧长度
        long frameLength = getUnadjustedFrameLength(in, actualLengthFieldOffset, lengthFieldLength, byteOrder);
		// 如果长度小于0,则抛出异常
        if (frameLength < 0) {
            failOnNegativeLengthField(in, frameLength, lengthFieldEndOffset);
        }
		// 此时 frameLength 为 偏移量+调整量
        frameLength += lengthAdjustment + lengthFieldEndOffset;

        if (frameLength < lengthFieldEndOffset) {
        	// 长度调整完后,比偏移量还少,肯定有问题
            failOnFrameLengthLessThanLengthFieldEndOffset(in, frameLength, lengthFieldEndOffset);
        }

        if (frameLength > maxFrameLength) {
        	// 大于最大长度
            exceededFrameLength(in, frameLength);
            return null;
        }

        // never overflows because it's less than maxFrameLength
        int frameLengthInt = (int) frameLength;
        if (in.readableBytes() < frameLengthInt) {
            // 解析完之后,发现不够一个帧长度,说明是半包
            return null;
        }

        if (initialBytesToStrip > frameLengthInt) {
        	// 跳过的字节数大于帧长度,直接报错
            failOnFrameLengthLessThanInitialBytesToStrip(in, frameLength, initialBytesToStrip);
        }
        // 跳过帧长度
        in.skipBytes(initialBytesToStrip);

        // extract frame
        int readerIndex = in.readerIndex();
        // 实际长度
        int actualFrameLength = frameLengthInt - initialBytesToStrip;
        // 解析出帧
        ByteBuf frame = extractFrame(ctx, in, readerIndex, actualFrameLength);
        // 重置reader index
        in.readerIndex(readerIndex + actualFrameLength);
        return frame;
    }

总体流程由上面注释。

总结

本文从源码上,对 Netty 粘包半包原理,从总体思路,到具体 ByteToMessageDecoder 子类进行处理过程中decode方法分析。

关注博主公众号: 六点A君。
哈哈哈,一起研究Netty:
在这里插入图片描述

你可能感兴趣的:(Java-Netty)