单机模拟高并发进行压力测试
本文主要介绍如何使用netty实现C100K级别的并发长连接
Socket五元组
理论上，客户端IP与端口，服务端IP与端口以及通讯协议（TCP|UDP）构成了一个连接的五元组。改变其中任何一个元素都可以新建一个连接。如果想要单台机器或者两台机器模拟数十万的连接数，仅仅靠系统分配的端口数目是不够的，我们必须要改变其中的IP元素。本文采取的方式是利用linux的secondary ip机制，为服务端分配多个IP，理论上最多可以分配(2^32-1)个IP，连接数目可达2^16 * IP数目，远远超出我们的目标连接数。
调整系统参数限制
端口范围检查
lg@lenove:~$ sudo sysctl --all | grep port_range
net.ipv4.ip_local_port_range = 10000 60000 //有5w可用tcp设置检查
lg@lenove:~$ sudo sysctl --all | grep tw_reuse
net.ipv4.tcp_tw_reuse = 1 //允许将TIME-WAIT sockets重新用于新的TCP连接，优化客户端处理能力检查系统文件句柄限制
lg@lenove:~$ ulimit -Hn
1000000 //100w左右互博：单机模拟100k并发
配置secondary ip， 这里配了20个，理论上可以支持100W个TCP连接（20个IP*5w个端口）
lg@lenove:~$ seq 20 | xargs -i sudo ip addr add 127.0.1.{}/8 dev lo
lg@lenove:~$ ip addr show
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 127.0.1.1/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.2/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.3/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.4/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.5/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.6/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.7/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.8/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.9/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.10/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.11/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.12/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.13/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.14/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.15/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.16/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.17/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.18/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.19/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet 127.0.1.20/8 scope host secondary lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
红蓝对抗：双机直连模拟100k并发
双机模拟和单机模拟原理类似，不过在路由设置上麻烦一些。
首先，两台主机需要用双绞线直连，以规避路由器的连接数目限制，同时获得带宽保证。
服务器端 IP设置
lg@lenove:~$ sudo ip addr add 192.168.255.1/30 brd + label enp4s0:server dev enp4s0
lg@lenove:~$ seq 1 20 | xargs -i sudo ip addr add 192.168.0.{}/24 dev enp4s0
lg@lenove:~$ ip addr show
2: enp4s0: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether c8:5b:76:b2:50:33 brd ff:ff:ff:ff:ff:ff
inet 192.168.16.102/24 brd 192.168.16.255 scope global enp4s0
valid_lft forever preferred_lft forever
inet 192.168.255.1/30 brd 192.168.255.3 scope global enp4s0:server
valid_lft forever preferred_lft forever
inet 192.168.0.1/24 scope global enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.2/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.3/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.4/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.5/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.6/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.7/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.8/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.9/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.10/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.11/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.12/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.13/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.14/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.15/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.16/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.17/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.18/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.19/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet 192.168.0.20/24 scope global secondary enp4s0
valid_lft forever preferred_lft forever
inet6 fe80::b684:f2df:7ecc:a317/64 scope link
valid_lft forever preferred_lft forever
客户端IP设置
lg@lenove:~$ sudo ip addr add 192.168.255.2/30 brd + label enp2s0:client dev enp2s0
lg@lenove:~$ sudo ip route add 192.168.0.0/24 via 192.168.255.1
lg@lenove:~$ ip route show
192.168.0.0/24 via 192.168.255.1 dev enp2s0注意，服务器地址为192.168.255.1/30,客户端地址为192.168.255.2/30,两者处于同一个网段，并且，客户端将192.168.0.0/24网段的网关指向了192.168.255.1。
带宽测速，将近一千兆：
lg@lenove:~$ iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 4.00 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.255.1 port 5001 connected with 192.168.255.2 port 38656
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1.10 GBytes 940 Mbits/sec客户端代码实现
因项目涉及到公司业务，不便上传源码。现摘取关键实现部分：
构造虚拟主机列表
private final static int V_HOST_SIZE = 20;
private static Stream hostStream() {
return IntStream.rangeClosed(1, V_HOST_SIZE)
.mapToObj(GatewayServerTest::ip);
}
private static String ip(int index) {
return String.format("127.0.1.%d", index);
// return String.format("192.168.0.%d", index);
} 阻塞，等待连接完成
private static void sync(ChannelFuture channelFuture) {
handleException(channelFuture::sync);
}
private static T handleException(SupplierWithException supplier) {
try {
return supplier.get();
} catch (Exception e) {
throw new RuntimeException(e.getMessage());
}
} 创建连接，非阻塞模式
private static Stream connect(Bootstrap bootstrap, String host, int n) {
return Stream.generate(() -> connect(bootstrap, host)).limit(n);
}
private static ChannelFuture connect(Bootstrap bootstrap, String host) {
return handleException(() -> bootstrap.connect(host, port));
} main方法,主要目的是，每次发起DELTA个连接，平均分配给V_HOST_SIZE个虚拟主机，直到连接数目达到CONN_SIZE
private final static int CONN_SIZE = 100000;
private final static int DELTA = 100;
public static void main(String[] args) {
EventLoopGroup ioGroup = new NioEventLoopGroup(4);
final Bootstrap bootstrap = new Bootstrap();
bootstrap.group(ioGroup)
.channel(NioSocketChannel.class)
.option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT)
.handler(new ChannelInitializer() {
@Override
public void initChannel(final SocketChannel ch) throws Exception {
ch.pipeline().addLast(
new CustomHandler()
);
}
});
final AtomicInteger started = new AtomicInteger(0);
while (started.get() < CONN_SIZE) {
final List futures = hostStream()
.flatMap(host -> connect(bootstrap, host, DELTA / V_HOST_SIZE))
.collect(Collectors.toList());
futures.stream()
.peek(future -> started.incrementAndGet())
.forEach(GatewayServerTest::sync);
try {
Thread.sleep(1000 * 60 * 60 * 24);
} catch (InterruptedException e) {
e.printStackTrace();
} finally {
ioGroup.shutdownGracefully();
}
} CustomHandler的作用主要是定时向服务器发送GPS数据，一秒一次
public class CustomHandler extends ChannelInboundHandlerAdapter {
@Override
public void channelActive(ChannelHandlerContext ctx) throws Exception {
ctx.executor().execute(new GpsTask(ctx));
}
private final static class GpsTask implements Runnable {
private ChannelHandlerContext context;
public GpsTask(ChannelHandlerContext context) {
this.context = context;
}
public void run() {
context.writeAndFlush("116.40, 39.91");
context.executor().schedule(this, 1, TimeUnit.SECONDS);
}
}
}在笔者的Thinkpad E470上面，左右互博模式的并发连接数可以达到20W，30w时出现了机器卡死现象；红蓝对抗模式下，可以达到30w以上的并发连接数，但会出现丢失连接的现象。
附录–相关异常
- NoRouteToHostException: 可能是端口耗尽所致