Linux下使用libevent库实现服务器端编程
Linux下使用libevent库实现服务器端编程
- 一、背景
-
- CentOS7 安装注意事项（先安装openssl开发库）
- Ubuntu 安装注意事项（先安装openssl开发库）
- 阿里云CentOS服务器端测试例程
- 二、相关知识
-
- 2.1 事件驱动（I/O复用）
- 2.2 Libevent库介绍
- 三、服务端编程实例
- 四、测试与分析
- 五、参考文章
- linux高性能服务器开发十大必须掌握的核心技术
原文链接：https://blog.csdn.net/stayneckwind2/article/details/71374439
一、背景
TCP服务器端一般用到非阻塞Socket、IO复用，Linux可以支持epoll、select，而Windows支持select、IOCP，考虑平台适用性，需要对IO事件进行封装兼容；
CentOS7 安装注意事项（先安装openssl开发库）
注意：在阿里云服务器CentOS上，先要进行如下操作：
把libevent添加到系统动态链接库，同时刷新系统动态链接库。
yum install openssl-devel.x86_64
cd /usr/local/lib
echo $(pwd) >> /etc/ld.so.conf
ldconfig
Ubuntu 安装注意事项（先安装openssl开发库）
https://blog.csdn.net/m0_46577050/article/details/122978664?spm=1001.2014.3001.5502
编译程序过程，别忘了加个 -levent 把libevent库引入，如涉及多线程处理还需要加个-levent_pthreads
阿里云CentOS服务器端测试例程
#include 编译源代码时：
gcc server1.c -o server1 -levent
二、相关知识
2.1 事件驱动（I/O复用）
服务端常用到的 select、poll、epoll都属于I/O复用模型，常用于单线程上处理多个网络连接；
引用Stevens UNP书中的图，I/O复用表示把应用程序中所有连接事件都交给内核监控，当某个连接出现可写、可读事件后，内核通知应用去处理；
另外的一种模型是多线程（进程）处理，每个线程（进程）处理一个连接请求；
相比较下[2]：多线程的处理逻辑简单、开销大容易达到系统限制；I/O复用逻辑处理要求高、但可以处理大量网络连接；
2.2 Libevent库介绍
libevent是一个轻量级的高性能的网络库，具备了如下特性[1]：
事件驱动，高性能；
轻量级，专注于网络(相对于ACE)；
开放源码，代码相当精炼、易读；
跨平台，支持Windows、Linux、BSD和Mac OS；
支持多种I/O多路复用技术（epoll、poll、dev/poll、select和kqueue等），在不同的操作系统下，做了多路复用模型的抽象，可以选择使用不同的模型，通过事件函数提供服务；
支持I/O，定时器和信号等事件；
采用Reactor模式；
2.2 Libevent常用函数
在Libevent中，信号处理、Socket处理、定时器处理都被视作事件，由内部的dispatcher进行统一处理；
首先是先介绍 libevent 实例的创建 event_base_new、循环分发 event_base_dispatch、退出分发 event_base_loopbreak、释放函数 event_base_free (event.h)：
/**
* Create and return a new event_base to use with the rest of Libevent.
*
* @return a new event_base on success, or NULL on failure.
*
* @see event_base_free(), event_base_new_with_config()
*/
struct event_base *event_base_new(void);
/**
Event dispatching loop
This loop will run the event base until either there are no more added
events, or until something calls event_base_loopbreak() or
event_base_loopexit().
@param base the event_base structure returned by event_base_new() or
event_base_new_with_config()
@return 0 if successful, -1 if an error occurred, or 1 if no events were
registered.
@see event_base_loop()
*/
int event_base_dispatch(struct event_base *);
/**
Deallocate all memory associated with an event_base, and free the base.
Note that this function will not close any fds or free any memory passed
to event_new as the argument to callback.
@param eb an event_base to be freed
*/
void event_base_free(struct event_base *);
/**
Abort the active event_base_loop() immediately.
event_base_loop() will abort the loop after the next event is completed;
event_base_loopbreak() is typically invoked from this event's callback.
This behavior is analogous to the "break;" statement.
Subsequent invocations of event_loop() will proceed normally.
@param eb the event_base structure returned by event_init()
@return 0 if successful, or -1 if an error occurred
@see event_base_loopexit()
*/
int event_base_loopbreak(struct event_base *);
然后是事件的创建 event_new、释放 event_free、加入到监控 event_add、从监控中移除 event_del
/**
Get the kernel event notification mechanism used by Libevent.
The EV_PERSIST flag can also be passed in the events argument: it makes
event_add() persistent until event_del() is called.
The EV_ET flag is compatible with EV_READ and EV_WRITE, and supported
only by certain backends. It tells Libevent to use edge-triggered
events.
The EV_TIMEOUT flag has no effect here.
It is okay to have multiple events all listening on the same fds; but
they must either all be edge-triggered, or all not be edge triggerd.
When the event becomes active, the event loop will run the provided
callbuck function, with three arguments. The first will be the provided
fd value. The second will be a bitfield of the events that triggered:
EV_READ, EV_WRITE, or EV_SIGNAL. Here the EV_TIMEOUT flag indicates
that a timeout occurred, and EV_ET indicates that an edge-triggered
event occurred. The third event will be the callback_arg pointer that
you provide.
@param base the event base to which the event should be attached.
@param fd the file descriptor or signal to be monitored, or -1.
@param events desired events to monitor: bitfield of EV_READ, EV_WRITE,
EV_SIGNAL, EV_PERSIST, EV_ET.
@param callback callback function to be invoked when the event occurs
@param callback_arg an argument to be passed to the callback function
@return a newly allocated struct event that must later be freed with
event_free().
@see event_free(), event_add(), event_del(), event_assign()
*/
struct event *event_new(struct event_base *, evutil_socket_t, short, event_callback_fn, void *);
/**
Deallocate a struct event * returned by event_new().
If the event is pending or active, first make it non-pending and
non-active.
*/
void event_free(struct event *);
/**
Add an event to the set of pending events.
The function event_add() schedules the execution of the ev event when the
event specified in event_assign()/event_new() occurs, or when the time
specified in timeout has elapesed. If atimeout is NULL, no timeout
occurs and the function will only be
called if a matching event occurs. The event in the
ev argument must be already initialized by event_assign() or event_new()
and may not be used
in calls to event_assign() until it is no longer pending.
If the event in the ev argument already has a scheduled timeout, calling
event_add() replaces the old timeout with the new one, or clears the old
timeout if the timeout argument is NULL.
@param ev an event struct initialized via event_set()
@param timeout the maximum amount of time to wait for the event, or NULL
to wait forever
@return 0 if successful, or -1 if an error occurred
@see event_del(), event_assign(), event_new()
*/
int event_add(struct event *ev, const struct timeval *timeout);
/**
Remove an event from the set of monitored events.
The function event_del() will cancel the event in the argument ev. If the
event has already executed or has never been added the call will have no
effect.
@param ev an event struct to be removed from the working set
@return 0 if successful, or -1 if an error occurred
@see event_add()
*/
int event_del(struct event *);
最后再介绍下几个 libevent的缓冲管理的接口evbuffer，服务端的缓冲处理一直都是非常重要的；
evbuffer实现了一个动态调整的连续内存空间，用于维护应用数据与网络设备之间的读入、写出操作；
evbuffer的创建与销毁函数（buffer.h）：
/**
Allocate storage for a new evbuffer.
@return a pointer to a newly allocated evbuffer struct, or NULL if an error
occurred
*/
struct evbuffer *evbuffer_new(void);
/**
Deallocate storage for an evbuffer.
@param buf pointer to the evbuffer to be freed
*/
void evbuffer_free(struct evbuffer *buf);
evbuffer支持socket读写，如从网络中读取数据evbuffer_read、解析处理数据后进行移除evbuffer_remove
/**
Read from a file descriptor and store the result in an evbuffer.
@param buffer the evbuffer to store the result
@param fd the file descriptor to read from
@param howmuch the number of bytes to be read
@return the number of bytes read, or -1 if an error occurred
@see evbuffer_write()
*/
int evbuffer_read(struct evbuffer *buffer, evutil_socket_t fd, int howmuch);
/**
Read data from an evbuffer and drain the bytes read.
If more bytes are requested than are available in the evbuffer, we
only extract as many bytes as were available.
@param buf the evbuffer to be read from
@param data the destination buffer to store the result
@param datlen the maximum size of the destination buffer
@return the number of bytes read, or -1 if we can't drain the buffer.
*/
int evbuffer_remove(struct evbuffer *buf, void *data, size_t datlen);
写入数据 evbuffer_add、并通过网络发送出去 evbuffer_write，类似的还是有 evbuffer_add_file、evbuffer_add_printf
/**
Append data to the end of an evbuffer.
@param buf the evbuffer to be appended to
@param data pointer to the beginning of the data buffer
@param datlen the number of bytes to be copied from the data buffer
@return 0 on success, -1 on failure.
*/
int evbuffer_add(struct evbuffer *buf, const void *data, size_t datlen);
/**
Write the contents of an evbuffer to a file descriptor.
The evbuffer will be drained after the bytes have been successfully written.
@param buffer the evbuffer to be written and drained
@param fd the file descriptor to be written to
@return the number of bytes written, or -1 if an error occurred
@see evbuffer_read()
*/
int evbuffer_write(struct evbuffer *buffer, evutil_socket_t fd);
三、服务端编程实例
需求：编写一个程序对指定地址、端口开启TCP服务，并无限接收客户端流量（无需解析）完成吞吐量的测试；
主函数的步骤就是获取输入、创建libevent实例、注册开启监听服务、注册信号事件、进入分发循环；
int main(int argc, char *argv[])
{
int opt = 0;
instance_t inst = {0};
if ( argc < 2 ) {
usage(argv[0]);
goto _E1;
}
/* Initalize config */
inst.sfd = -1;
inst.enable = 1;
inst.cfg.tcp_port = 5001;
inst.cfg.tcp_addr = strdup("127.0.0.1");
while ( (opt = getopt(argc, argv, "s:p:")) != FAILURE ) {
switch ( opt ) {
case 's':
FREE_POINTER(inst.cfg.tcp_addr);
inst.cfg.tcp_addr = strdup(optarg);
break;
case 'p':
inst.cfg.tcp_port = (u16)atoi(optarg);
break;
default:
usage(argv[0]);
goto _E1;
}
}
/* Initalize the event library */
inst.base = event_base_new();
/* Initalize events */
inst.ev_signal = evsignal_new(inst.base, SIGTERM, __on_signal, &inst);
assert(inst.ev_signal);
event_add(inst.ev_signal, NULL);
__do_listen(&inst);
event_base_dispatch(inst.base);
evconnlistener_free(inst.ev_listener);
event_free(inst.ev_signal);
event_base_free(inst.base);
_E1:
evutil_closesocket(inst.sfd);
FREE_POINTER(inst.cfg.tcp_addr);
return EXIT_SUCCESS;
}
在信号处理函数方面，被例监控了 SIGTERM函数，即收到中断信号退出分发循环：
static void __on_signal(evutil_socket_t fd, short event, void *arg)
{
instance_t *pinst = (instance_t *)arg;
event_base_loopbreak(pinst->base);
printf("SIGTERM Breakout...\n");
}
启用监听时，使用了libevent 的utils、listener工具集，主要是 evconnlistener_new_bind 函数
大概看了一下 evconnlistener_new_bind 内部的实现，就是在根据地址端口进行监听、绑定，epoll上对 listen socket进行可读的事件监控，
有新连接到来时触发 listen socket的可读，然后循环进行 accept 操作（socket是非阻塞）；
static void __on_accept(struct evconnlistener *ev_listener, evutil_socket_t cfd,
struct sockaddr *paddr, int socklen, void *args)
{
struct sockaddr_in *pcaddr = (struct sockaddr_in *)paddr;
__do_alloc((instance_t *)args, cfd);
printf("[L] Setup new connection %s:%hu, fd: %d\n",
inet_ntoa(pcaddr->sin_addr), ntohs(pcaddr->sin_port), cfd);
}
static void __do_listen(instance_t *pinst)
{
struct sockaddr_in saddr = {0};
saddr.sin_family = AF_INET;
saddr.sin_port = htons(pinst->cfg.tcp_port);
inet_aton(pinst->cfg.tcp_addr, &saddr.sin_addr);
pinst->ev_listener = evconnlistener_new_bind(pinst->base, __on_accept, pinst,
LEV_OPT_REUSEABLE | LEV_OPT_CLOSE_ON_FREE, -1, (struct sockaddr *)&saddr, sizeof(saddr));
assert(pinst->ev_listener);
}
补充一下结构体说明，instance_t为整个程序的实例结构、item_t为每个连接的上下文结构
typedef struct instance
{
u8 enable;
evutil_socket_t sfd;
struct config {
u16 tcp_port;
char *tcp_addr;
} cfg;
struct event *ev_signal;
struct evconnlistener *ev_listener;
struct event_base *base;
} instance_t;
typedef struct item
{
instance_t *pinst;
struct event *ev_client;
struct evbuffer *evbuf_req; /* request */
struct evbuffer *evbuf_rsp; /* response */
u64 total_read;
u64 total_send;
} item_t;
继续刚才的思路，新建连接后会调用__on_accept回调函数，内部又调用了 __do_alloc 函数，
在这个函数中需要初始化连接的 item_t 结构、创建evbuffer、并将新连接可读事件进行监控；
static void __do_alloc(instance_t *pinst, evutil_socket_t fd)
{
item_t *pitem = (item_t *)calloc(1, sizeof(item_t));
assert(pitem);
pitem->pinst = pinst;
pitem->evbuf_req = evbuffer_new();
pitem->evbuf_rsp = evbuffer_new();
assert(pitem->evbuf_req);
assert(pitem->evbuf_rsp);
pitem->ev_client = event_new(pinst->base, fd, EV_READ | EV_PERSIST,
__on_request, pitem);
event_add(pitem->ev_client, NULL);
assert(SUCCESS == evutil_make_socket_nonblocking(fd));
}
这样，客户端发送数据时，这边将触发socket可读事件，调用函数 __on_request
在__on_request 中，为了简单起见，跳过了数据解析处理的流程，直接缓冲区超过10M后进行清空处理；
static void __on_request(evutil_socket_t fd, short event, void *arg)
{
item_t *pitem = (item_t *)arg;
int read_byte = evbuffer_read(pitem->evbuf_req, fd, -1);
if ( read_byte <= 0 ) {
printf("[%2d] Close, read cnt: %llu\n", fd, pitem->total_read);
__do_close(fd, pitem);
return;
}
pitem->total_read += read_byte;
if ( evbuffer_get_length(pitem->evbuf_req) > 10 * 1024 * 1024 ) {
assert(0 == evbuffer_drain(pitem->evbuf_req, 10 * 1024 * 1024));
printf("[%2d] Reset, read cnt: %llu\n", fd, pitem->total_read);
}
}
同时，客户端关闭连接或连接异常时，evbuffer_read将返回小于等于零，进行 __do_close 关闭连接、释放空间；
static void __do_close(evutil_socket_t fd, item_t *pitem)
{
evutil_closesocket(fd);
evbuffer_free(pitem->evbuf_req);
evbuffer_free(pitem->evbuf_rsp);
event_del(pitem->ev_client);
event_free(pitem->ev_client);
pitem->evbuf_req = NULL;
pitem->evbuf_rsp = NULL;
pitem->ev_client = NULL;
free(pitem);
}
四、测试与分析
编译程序过程，别忘了加个 -levent 把libevent库引入，如涉及多线程处理还需要加个-levent_pthreads
测试，使用 iperf 充当客户端进行发包处理：
iperf -c 127.0.0.1 -i 1 -n 11M -P 10
------------------------------------------------------------
Client connecting to 127.0.0.1, TCP port 5001
TCP window size: 648 KByte (default)
------------------------------------------------------------
[ 9] local 127.0.0.1 port 44384 connected with 127.0.0.1 port 5001
[ 3] local 127.0.0.1 port 44377 connected with 127.0.0.1 port 5001
[ 4] local 127.0.0.1 port 44378 connected with 127.0.0.1 port 5001
[ 5] local 127.0.0.1 port 44379 connected with 127.0.0.1 port 5001
[ 6] local 127.0.0.1 port 44380 connected with 127.0.0.1 port 5001
[ 8] local 127.0.0.1 port 44382 connected with 127.0.0.1 port 5001
[ 7] local 127.0.0.1 port 44381 connected with 127.0.0.1 port 5001
[ 11] local 127.0.0.1 port 44385 connected with 127.0.0.1 port 5001
[ 12] local 127.0.0.1 port 44386 connected with 127.0.0.1 port 5001
[ 10] local 127.0.0.1 port 44383 connected with 127.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 9] 0.0- 0.3 sec 11.0 MBytes 325 Mbits/sec
[ 3] 0.0- 0.3 sec 11.0 MBytes 328 Mbits/sec
[ 4] 0.0- 0.3 sec 11.0 MBytes 265 Mbits/sec
[ 6] 0.0- 0.3 sec 11.0 MBytes 266 Mbits/sec
[ 7] 0.0- 0.3 sec 11.0 MBytes 265 Mbits/sec
[ 10] 0.0- 0.4 sec 11.0 MBytes 263 Mbits/sec
[ 5] 0.0- 0.4 sec 11.0 MBytes 263 Mbits/sec
[ 8] 0.0- 0.4 sec 11.0 MBytes 263 Mbits/sec
[ 11] 0.0- 0.4 sec 11.0 MBytes 259 Mbits/sec
[ 12] 0.0- 0.4 sec 11.0 MBytes 263 Mbits/sec
[SUM] 0.0- 0.4 sec 110 MBytes 2.59 Gbits/sec
./tcp_server -s 127.0.0.1 -p 5001
[L] Setup new connection 127.0.0.1:44377, fd: 8
[L] Setup new connection 127.0.0.1:44378, fd: 9
[L] Setup new connection 127.0.0.1:44379, fd: 10
[L] Setup new connection 127.0.0.1:44380, fd: 11
[L] Setup new connection 127.0.0.1:44381, fd: 12
[L] Setup new connection 127.0.0.1:44382, fd: 13
[L] Setup new connection 127.0.0.1:44383, fd: 14
[L] Setup new connection 127.0.0.1:44384, fd: 15
[L] Setup new connection 127.0.0.1:44385, fd: 16
[L] Setup new connection 127.0.0.1:44386, fd: 17
[15] Reset, read cnt: 10488600
[ 8] Reset, read cnt: 10485784
[ 9] Reset, read cnt: 10485784
[10] Reset, read cnt: 10485784
[11] Reset, read cnt: 10485784
[13] Reset, read cnt: 10485784
[15] Close, read cnt: 11534360
[12] Reset, read cnt: 10485784
[16] Reset, read cnt: 10485784
[17] Reset, read cnt: 10485784
[14] Reset, read cnt: 10485784
[ 8] Close, read cnt: 11534360
[ 9] Close, read cnt: 11534360
[10] Close, read cnt: 11534360
[11] Close, read cnt: 11534360
[13] Close, read cnt: 11534360
[12] Close, read cnt: 11534360
[16] Close, read cnt: 11534360
[17] Close, read cnt: 11534360
[14] Close, read cnt: 11534360
使用libevent 进行服务端编程非常方便，无须过多干涉网络编程的细节，不过还得真正去理解网络处理的原理；
同时，evbuffer内部的缓冲处理也为上层应用提供了不少便利，能够给项目开发带来可观的效率提升；
五、参考文章
[1] http://blog.csdn.net/majianfei1023/article/details/46485705
[2] http://blog.csdn.net/historyasamirror/article/details/5778378
[3] http://blog.csdn.net/yusiguyuan/article/details/20458565
linux高性能服务器开发十大必须掌握的核心技术
https://blog.csdn.net/qq_40989769/article/details/120434980