网络篇的这几篇文章都在谈理论,这篇文章我将带大家来分析一个实战例子:基于 netty 的高并发安全聊天客户端。这是我工作中的一个项目,这篇文章将带大家了解 IM 的实现逻辑。
目录:
1. netty 介绍
Netty 是一个高性能、异步事件驱动的 NIO 框架,它提供了对 TCP、UDP 和文件传输的支持。作为一个异步 NIO 框架,Netty 的全部 IO 操作都是异步非堵塞的,通过 Future-Listener 机制,用户能够方便的主动获取或者通过通知机制获得 IO 操作结果。
作为当前最流行的 NIO 框架。Netty 在互联网领域、大数据分布式计算领域、游戏行业、通信行业等获得了广泛的应用,一些业界著名的开源组件也基于 Netty 的 NIO 框架构建。
网络传输方式问题:传统的 RPC 框架或者基于 RMI 等方式的远程服务(过程)调用采用了同步堵塞 IO。当 client 的并发压力或者网络时延增大之后,同步堵塞 IO 会因为频繁的 wait 导致 IO 线程经常性的堵塞。因为线程无法高效的工作,IO 处理能力自然下降。
采用 BIO 通信模型的服务端,通常由一个独立的 Acceptor 线程负责监听 client 的连接。接收到 client 连接之后为 client 连接创建一个新的线程处理请求消息,处理完毕之后,返回应答消息给 client,线程销毁,这就是典型的一请求一应答模型。该架构最大的问题就是不具备弹性伸缩能力,当并发访问量添加后,服务端的线程个数和并发访问数成线性正比,因为线程是 Java 虚拟机很宝贵的系统资源,当线程数膨胀之后,系统的性能急剧下降。随着并发量的继续添加,可能会发生句柄溢出、线程堆栈溢出等问题,并导致 server 宕机。
序列化方式问题:Java 序列化存在例如以下几个典型问题:
线程模型问题:因为采用同步堵塞 IO,这会导致每一个 TCP 连接都占用1个线程,因为线程资源是 JVM 虚拟机很宝贵的资源,当 IO 读写堵塞导致线程无法及时释放时,会导致系统性能急剧下降,严重的甚至会导致虚拟机无法创建新的线程。
(1) 异步非堵塞通信
在 IO 编程过程中,当须要同一时候处理多个 client 接入请求时,能够利用多线程或者 IO 多路复用技术进行处理。IO 多路复用技术通过把多个 IO 的堵塞复用到同一个 select 的堵塞上,从而使得系统在单线程的情况下能够同一时候处理多个 client 请求。
与传统的多线程/多进程模型比,I/O 多路复用的最大优势是系统开销小。系统不须要创建新的额外进程或者线程,也不须要维护这些进程和线程的执行,减少了系统的维护工作量,节省了系统资源。JDK1.4 提供了对非堵塞IO(NIO)的支持。JDK1.5_update10 版本使用 epoll 替代了传统的 select/poll,极大的提升了 NIO 通信的性能。
Netty 采用了异步通信模式,一个 IO 线程能够并发处理N个 client 连接和读写操作,这从根本上攻克了传统同步堵塞 IO 一连接一线程模型,架构的性能、弹性伸缩能力和可靠性都得到了极大的提升。
(2) 零拷贝
Netty 的零拷贝主要体如今例如以下三个方面:
(3) 内存池
随着 JVM 虚拟机和 JIT 即时编译技术的发展,对象的分配和回收是个很轻量级的工作。可是对于缓冲区 Buffer,情况却少有不同,特别是对于堆外直接内存的分配和回收,是一件耗时的操作。为了尽量重用缓冲区,Netty 提供了基于内存池的缓冲区重用机制。
2. 数据库设计
接下来看看客户端消息相关的数据库设计。
会话表:
消息表:
群组表:
用户表:
3. 聊天的 JNI 封包解包
Java 层通过该类进行封包解包的数据传输。
public final class GdpPack implements NoProGuard, Serializable {
private int len;
private short seq;
private int opCode;
private int bodyLen;
private byte messageType;
private String bodyBuffer;
private byte[] buffer;
static {
loadLibrary("ucgpac");
}
public GdpPack() {
}
public GdpPack(int len) {
this.len = len;
}
public GdpPack(short seq, int opCode, String bodyBuffer) {
this.seq = seq;
this.opCode = opCode;
this.bodyBuffer = bodyBuffer;
this.bodyLen = bodyBuffer.length();
// 在默认情况下,messageType是请求
this.messageType = 1;
}
public GdpPack(short seq, int opCode, byte messageType, String bodyBuffer) {
this.seq = seq;
this.opCode = opCode;
this.messageType = messageType;
this.bodyBuffer = bodyBuffer;
this.bodyLen = bodyBuffer.length();
}
public void parse(byte[] buffer, byte[] randomKey) {
if (buffer != null && randomKey != null) {
parse(buffer, randomKey, this);
}
}
public byte[] fill(byte[] randomKey) {
return randomKey != null ? fill(randomKey, this) : null;
}
private native void parse(byte[] buffer, byte[] randomKey, GdpPack gdpPack);
private native byte[] fill(byte[] randomKey, GdpPack gdpPack);
get/set functions ...
}
变量说明:
JNI 函数:
private native void parse(byte[] buffer, byte[] randomKey, GdpPack gdpPack);
将 buffer 字节数据和 randomKey 解析并填充到 GdpPack 对象。
private native byte[] fill(byte[] randomKey, GdpPack gdpPack);
将 randomKey 填充到 GdpPack 对象。
extern "C" {
JNIEXPORT void JNICALL Java_waterhole_im_GdpPack_parse(JNIEnv *env, jobject jobj, jbyteArray buffer, jbyteArray randomKey, jobject gdpPack) {
GdpPack * pack = new GdpPack();
char * tmpBuffer = jByteArray2chars(env, buffer);
char * tmpRandom = jByteArray2chars(env, randomKey);
pack->parse(tmpBuffer, tmpRandom);
delete tmpBuffer;
delete tmpRandom;
jclass cls = env->FindClass("waterhole/im/GdpPack");
jfieldID bodyBufferF = env->GetFieldID(cls, "bodyBuffer", "Ljava/lang/String;");
char * resTemp = new char[pack->getBodyLen() * sizeof(char)];
memset(resTemp, 0x00, pack->getBodyLen() * sizeof(char));
memcpy(resTemp, pack->getBodyBuffer(), pack->getBodyLen());
env->SetObjectField(gdpPack, bodyBufferF, string2Jstring(env, resTemp));
delete resTemp;
jfieldID bodyLenF = env->GetFieldID(cls, "bodyLen", "I");
env->SetIntField(gdpPack, bodyLenF, pack->getBodyLen());
jfieldID seqF = env->GetFieldID(cls, "seq", "S");
env->SetShortField(gdpPack, seqF, pack->getSeq());
jfieldID opCodeF = env->GetFieldID(cls, "opCode", "I");
env->SetIntField(gdpPack, opCodeF, pack->getOpCode());
env->DeleteLocalRef(cls);
//delete pack;
}
JNIEXPORT jbyteArray JNICALL Java_waterhole_im_GdpPack_fill(JNIEnv *env, jobject jobc, jbyteArray randomKey, jobject gdpPack) {
jclass cls = env->FindClass("waterhole/im/GdpPack");
jfieldID bodyBufferF = env->GetFieldID(cls, "bodyBuffer", "Ljava/lang/String;");
char* bodyBuffer = jstring2String(env, (jstring) env->GetObjectField(gdpPack, bodyBufferF));
jfieldID seqF = env->GetFieldID(cls, "seq", "S");
short seq = env->GetShortField(gdpPack, seqF);
jfieldID opCodeF = env->GetFieldID(cls, "opCode", "I");
int opCode = env->GetIntField(gdpPack, opCodeF);
jfieldID msgTypeF = env->GetFieldID(cls, "messageType", "B");
char msgType = env->GetByteField(gdpPack, msgTypeF);
GdpPack * pack = new GdpPack(seq, opCode, msgType, bodyBuffer, strlen(bodyBuffer));
char * tmpRandom = jByteArray2chars(env, randomKey);
char * resStr = pack->fill(tmpRandom);
delete bodyBuffer;
delete tmpRandom;
jbyteArray ret = chars2jByteArray(env, resStr, pack->getLen());
env->DeleteLocalRef(cls);
//delete pack;
return ret;
}
}
JNI 设置对象参数,核心在于 parse() 和 fill() 实现:
void GdpPack::parse(char * buffer, char * key)
{
_buffer = buffer;
_len = readInt(_buffer);
_stackVersion = readShort(_buffer + 8);
_encryptType = readChar(_buffer + 10);
_compressType = readChar(_buffer + 11);
_messageType = readChar(_buffer + 12);
_seq = readShort(_buffer + 13);
_bodyFormat = readChar(_buffer + 15);;
_bodyModelVersion = readShort(_buffer + 16);
_time = readLong(_buffer + 18);
_statusCode = readShort(_buffer + 26);
_opCode = readInt(_buffer + 28);
_bodyLen = readInt(_buffer + 32);
if(_bodyLen > 0)
_bodyBuffer = readChars(_buffer + 36, _bodyLen);
if(_encryptType > 0)
{
decrypt(_bodyBuffer, _bodyLen, key);
}
}
char* GdpPack::fill(char * key)
{
_buffer = new char[_len + 4];//85
memset(_buffer, 0x00, _len+4);
writeInt(_len, _buffer);
writeChars(_label, _buffer + 4, 4);
writeShort(_stackVersion, _buffer + 8);
writeChar(_encryptType, _buffer + 10);
writeChar(_compressType, _buffer + 11);
writeChar(_messageType, _buffer + 12);
writeShort(_seq, _buffer + 13);
writeChar(_bodyFormat, _buffer + 15);
writeShort(_bodyModelVersion, _buffer + 16);
writeLong(_time, _buffer + 18);
writeShort(_statusCode, _buffer + 26);
writeInt(_opCode, _buffer + 28);
writeInt(_bodyLen, _buffer + 32);
if(_encryptType > 0)
{
encrypt(_bodyBuffer, _bodyLen, key);
}
writeChars(_bodyBuffer, _buffer + 36, _bodyLen);
return _buffer;
}
就是将字节按位取参数,加密和解密封装成 GdpPack 对象,这边的 bodyBuffer 做了数据加密,加解密算法这边就不说了。这一层也保证了数据安全,位于表示层。
4. 长连接的实现
这边只讲讲连接的实现逻辑。
数据封包:
public final class GdpPackageEncoder extends OneToOneEncoder {
private static final String TAG = "GdpPackageEncoder";
private final SocketManager mSocketManager = SocketManager.instance();
@Override
protected Object encode(ChannelHandlerContext channelHandlerContext, Channel channel, Object o)
throws Exception {
if (!(o instanceof GdpPack)) {
return o;
}
final GdpPack packet = (GdpPack) o;
final ChannelBuffer buffer;
// 在确定总长度前,如果body为空则总长度设置为0不再发送后续协议头部
if (packet.getBodyBuffer() == null || packet.getBodyBuffer().length() == 0) {
buffer = wrappedBuffer(new byte[]{0, 0, 0, 0});
} else {
buffer = wrappedBuffer(packet.fill(mSocketManager.getRandomKey()));
}
// e是为了日志醒目
error(TAG,
"@GdpPackageEncoder " + packet.getBodyBuffer() + "-OpCode:" + packet.getOpCode());
return buffer;
}
}
调用 JNI 填充方法。
数据解包:
public final class GdpPackageDecoder extends FrameDecoder {
private static final String TAG = "GdpPackageDecoder";
// 跳过的字节长度
private static final short DECODER_SKIP_LEN = 4;
// RandomKey的字节长度
private static final short DECODER_RANDOM_KEY_LEN = 36;
// GdpPack包最小字节长度
private static final short DECODER_MIN_PACKET_LEN = 32;
@Override
protected Object decode(ChannelHandlerContext channelHandlerContext, Channel channel,
ChannelBuffer channelBuffer) throws Exception {
try {
info(TAG, "Decode just in len:" + channelBuffer.readableBytes());
// 错误的信息
if (channelBuffer.readableBytes() < DECODER_SKIP_LEN) {
return null;
}
final int totalLen = channelBuffer.getInt(channelBuffer.readerIndex());
// 还没有解码成完整的包
if (channelBuffer.readableBytes() < totalLen + DECODER_SKIP_LEN) {
return null;
}
switch (totalLen) {
case 0:
// 是否是心跳包或randomKey包
readHeartbeat(channelBuffer);
return null;
case DECODER_RANDOM_KEY_LEN:
// Random Key String
channelBuffer.readInt();
byte[] randomKeyBytes = new byte[DECODER_RANDOM_KEY_LEN];
channelBuffer.readBytes(randomKeyBytes);
error(TAG, "decoder RandomKey:" + new String(randomKeyBytes));
// 保存本次的randomKey
mSocketManager.setRandomKey(randomKeyBytes);
mSocketManager.setRandomNumber(GdpPack.auth(randomKeyBytes, randomKeyBytes.length));
// 更新session状态,抛给上层去处理
EventBus.getDefault().post(POST_UPDATE_SESSION);
return null;
default:
break;
}
// 最终拿到的是完整的GdpPack包,将数据通过jni拼装成GdpPack对象
return isTokenUnsafely(channelBuffer, totalLen) ? null : assemeGdpPack(channelBuffer, totalLen);
} catch (Exception e) {
return null;
}
}
/**
* 为了保障token的安全 在连接成功后 会发送一些混淆包
* 这些包的长度会小于 DECODER_MIN_PACKET_LEN ,而且不等于 DECODER_RANDOM_KEY_LEN
* 对这些包不用进行任何的处理
*/
private boolean isTokenUnsafely(ChannelBuffer channelBuffer, int totalLen) {
if (totalLen <= DECODER_MIN_PACKET_LEN) {
channelBuffer.readInt();
channelBuffer.readBytes(totalLen);
return true;
}
return false;
}
private final SocketManager mSocketManager = SocketManager.instance();
@NonNull
private GdpPack assemeGdpPack(ChannelBuffer channelBuffer, int totalLen) {
GdpPack packet = new GdpPack();
byte[] bufferBytes = new byte[totalLen + DECODER_SKIP_LEN];
channelBuffer.readBytes(bufferBytes);
packet.parse(bufferBytes, mSocketManager.getRandomKey());
/**
* Below is a patch
* In certain occasion,the json string returned from server is followed by
* few strange characters.not find the solution so add this patch to prevent
* that from happening
*/
if (packet.getBodyLen() != packet.getBodyBuffer().length()) {
packet.setBodyBuffer(packet.getBodyBuffer().substring(0, packet.getBodyLen()));
}
error(TAG, "@GdpPackageDecoder:" + packet.getBodyBuffer() + "-opCode:" + packet.getOpCode());
return packet;
}
/**
* 数据读完后,判断是不是心跳包或返回randomKey的包
* 如果是心跳包,则将心跳包的回调从队列中移除掉,如果是randomKey的包,保存这次的randomKey,
* 并且拿这个randomKey是服务器刷新session,session 刷新成功后才是有效的连接
*
* @see #readHeartbeat(ChannelBuffer)
*/
private void readHeartbeat(ChannelBuffer channelBuffer) {
// Heart beat
channelBuffer.readInt();
// 异常心跳包回调
mSocketManager.popHeartbeat(0);
}
}
解析数据包,分为无效包、心跳包、randomKey 包、消息收发数据包的解析填充。
4.2 连接
public final class SocketThread extends Thread {
private static final String TAG = "SocketThread";
private ClientBootstrap mClientBootstrap = null;
private ChannelFuture mChannelFuture = null;
private Channel mChannel = null;
private String mStrHost;
private final int mPort;
private final SocketManager mSocketManager = SocketManager.instance();
public SocketThread(String strHost, int nPort, SimpleChannelUpstreamHandler handler) {
mStrHost = strHost;
mPort = nPort;
init(handler);
}
@Override
public void run() {
doConnect();
}
private void init(final SimpleChannelUpstreamHandler handler) {
try {
// only one IO thread
ChannelFactory channelFactory = new NioClientSocketChannelFactory(
newCachedThreadPool(), newCachedThreadPool());
mClientBootstrap = new ClientBootstrap(channelFactory);
mClientBootstrap.setOption("connectTimeoutMillis", CONNECT_TIMEOUT);
mClientBootstrap.setPipelineFactory(new ChannelPipelineFactory() {
public ChannelPipeline getPipeline() throws Exception {
ChannelPipeline pipeline = Channels.pipeline();
SSLEngine engine = getClientContext().createSSLEngine();
engine.setUseClientMode(true);
SslHandler sslHandler = new SslHandler(engine, getDefaultBufferPool(),
false, new HashedWheelTimer(), HAND_SHAKE_TIMEOUT);
sslHandler.setCloseOnSSLException(true);
pipeline.addFirst("tls", sslHandler);
pipeline.addLast("decoder", new GdpPackageDecoder());
pipeline.addLast("encoder", new GdpPackageEncoder());
pipeline.addLast("handler", handler);
return pipeline;
}
});
mClientBootstrap.setOption("tcpNoDelay", true);
mClientBootstrap.setOption("keepAlive", true);
} catch (OutOfMemoryError e) {
mSocketManager.onMsgServerDisconnect();
System.gc();
} catch (ChannelException e) {
mSocketManager.onMsgServerDisconnect();
}
}
private boolean doConnect() {
if (mClientBootstrap == null) {
mSocketManager.onMsgServerDisconnect();
return false;
}
// 优先IPV4,因为IPV6在一些机器上有问题
try {
for (InetAddress addr : InetAddress.getAllByName(mStrHost)) {
if (addr instanceof Inet4Address) {
mStrHost = addr.getHostAddress();
break;
}
}
} catch (UnknownHostException e) {
error(TAG, e.getMessage());
}
mClientBootstrap.setOption("remoteAddress", new InetSocketAddress(mStrHost, mPort));
try {
InetSocketAddress address;
if ((null == mChannel || !mChannel.isConnected()) && null != mStrHost && mPort > 0) {
// Start the connection attempt
address = mClientBootstrap == null ? null : (InetSocketAddress)
mClientBootstrap.getOption("remoteAddress");
mChannelFuture = mClientBootstrap.connect(address);
// Wait until the connection attempt succeeds or fails
mChannel = mChannelFuture.awaitUninterruptibly().getChannel();
if (!mChannelFuture.isSuccess()) {
mClientBootstrap.releaseExternalResources();
mSocketManager.onMsgServerDisconnect();
return false;
}
}
if (this.mChannel != null && this.mChannel.isConnected()) {
mSocketManager.initSocketSuccess();
} else {
mSocketManager.onMsgServerDisconnect();
}
// Wait until the connection is closed or the connection attemp fails
mChannelFuture.getChannel().getCloseFuture().awaitUninterruptibly();
mClientBootstrap.releaseExternalResources();
return true;
} catch (Throwable e) {
// 交换域名备用
swapServerAddress();
mSocketManager.onMsgServerDisconnect();
return false;
}
}
public void close() {
if (mChannelFuture != null) {
try {
if (mChannelFuture.getChannel() != null) {
mChannelFuture.getChannel().close();
}
mChannelFuture.cancel();
} catch (Exception e) {
error(TAG, e.getMessage());
}
}
}
@Deprecated
public boolean isClose() {
return mChannelFuture == null || mChannelFuture.getChannel() == null ||
!mChannelFuture.getChannel().isConnected() ||
!mChannelFuture.getChannel().isWritable();
}
public boolean sendReqPacket(GdpPack gdpPack) {
if (mChannelFuture != null && null != mChannelFuture.getChannel()) {
Channel currentChannel = mChannelFuture.getChannel();
if (!(currentChannel.isWritable() && currentChannel.isConnected())) {
mSocketManager.onMsgServerDisconnect();
return false;
}
mChannelFuture.getChannel().write(gdpPack);
return true;
} else {
return false;
}
}
}
这个就是连接的核心代码,一个线程就能维护整个长连接,进行包的收发。里面涉及到连接,SSL 配置。
public final class IMClientHandler extends SimpleChannelUpstreamHandler {
// 上下文对象
private final Context mContext = ContextWrapper.getInstance().obtainContext();
@Override
public void channelConnected(final ChannelHandlerContext ctx, final ChannelStateEvent e) throws Exception {
info(TAG, "channelConnected");
SslHandler sslHandler = ctx.getPipeline().get(SslHandler.class);
sslHandler.handshake().addListener(new ChannelFutureListener() {
public void operationComplete(ChannelFuture future) throws Exception {
info(TAG, "future.isSuccess():" + future.isSuccess());
if (!future.isSuccess()) {
swapServerAddress();
// java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
try {
future.getChannel().close();
} catch (Exception e1) {
SocketManager.instance().onMsgServerDisconnect();
}
} else {
SocketManager.instance().onMsgServerConnected();
}
}
});
}
@Override
public void channelDisconnected(ChannelHandlerContext ctx, ChannelStateEvent e) throws Exception {
super.channelDisconnected(ctx, e);
info(TAG, "channelDisconnected");
SocketManager.instance().onMsgServerDisconnect();
HeartBeatManager.instance().onMsgServerDisconnect();
ConnectManager.instance().onMsgServerDisconnect();
}
@Override
public void messageReceived(ChannelHandlerContext ctx, final MessageEvent e) throws Exception {
final GdpPack pack = (GdpPack) e.getMessage();
if (pack != null) {
AsyncTaskAssistant.executeOnThreadPool(new Runnable() {
@Override
public void run() {
try {
SocketManager.instance().packetDispatch(pack);
} catch (Exception e1) {
error(TAG, e1.getMessage());
}
}
});
}
}
@Override
public void exceptionCaught(final ChannelHandlerContext ctx, ExceptionEvent e) {
try {
super.exceptionCaught(ctx, e);
if (e != null) {
Throwable throwable = e.getCause();
if (throwable != null) {
final String errorMsg = throwable.getMessage();
if (!TextUtils.isEmpty(errorMsg)) {
error(TAG, errorMsg);
MobclickAgent.reportError(mContext, errorMsg);
if (errorMsg.contains("timed out")) {
swapServerAddress();
}
}
}
// 异常时手动关闭channel, @see http://netty.io/3.8/guide/
if (e.getChannel() != null) {
e.getChannel().close();
}
}
} catch (Exception e1) {
error(TAG, e1.getMessage());
}
}
}
连接成功、连接断开、连接异常和收到消息的回调。这边因为使用了 SSL 加密,所以必须判断握手成功再刷新 Session,此时才是真正连接成功,能进行消息收发。
5. 消息异常处理
当重新连接成功后,这个聊天对话可能在服务器堆积了大量的未读消息,此时应该用本地存储的最后一条消息 id,去服务器获取未读。因为聊天都会滑动到最新的消息,所以先去获取最新的20条消息。拉取成功后插入数据库,然后显示给用户,同时显示所有未读消息数。当用户下拉旧的消息时,从数据库获取之前的消息,如果之前的消息 id 存在断层,则从服务器获取并插入显示,如果没有,则直接用数据库的记录显示。
客户端这边是以服务器最终确定成功的消息 id 作为排序,所以如果之前发送失败的消息,即使客户端发送时间再早,最后都会以服务器接收成功的时间为准 (服务接收成功会给一个唯一的消息 id,按顺序排列)。