[杂烩]Windows IOCP与Linux的epoll机制对比

看到一些资料，说Windows的IOCP后发制人，比Linux的epoll优越很多。那么优势到底在哪里？如何选择Linux或Windows作为服务器呢？
也看到有资料说，同样在Linux上，使用reactor机制的网络库性能比使用Proactor机制的Asio性能好接近1/5，这个例子可能与题目无关，不过我在文中可能会做一些相关的分析。

---

系统I/O模型 可分为三类：

• 阻塞型(blocking model)，
• 非阻塞同步型(non-blocking model): "wait until any socket is available to read or write from/to buffer, then call non blocking socket function which returns immediately."
• 以及非阻塞异步型(asynchronous aka. overlapping model): "call a socket function which returns immediately, then wait for its completion, then access the result data object"

IOCP基于非阻塞异步模型，而epoll基于非阻塞同步模型。

参考Hyunjik Bae, A game programmer since 1995说的：

Written 7 Dec 2014
Linux affords epoll which is similar to IOCP. Here are the main differences:

1. IOCP is based on asynchronous model, but epoll is based on non-blocking model. Asynchronous aka. overlapping model is "call a socket function which returns immediately, then wait for its completion, then access the result data object". Meanwhile, non blocking model is "wait until any socket is available to read or write from/to buffer, then call non blocking socket function which returns immediately." At last, epoll cannot allow app developers to write best code for zero-copy from/to socket buffer. Accessing much memory read/write is expensive in modern hardware, so this can be a issue.
2. IOCP allows multiple thread to wait for completion event of the same IOCP port handle. However, epoll allows only one thread. Of course, calling the wait function (epoll_wait) from several threads works, but it may cause multiple threads awake by the same socket, which is not we want to get. One workaround way is to create epoll handles as many as threads and distribute many socket handles to them.
3. As epoll works not infringing any existing non blocking socket functions (and file functions), using epoll and socket together is easier and more intuitive. However, IOCP and especially TCP connecting and accepting functions are tricky and somewhat awkward: check out ConnectEx and AcceptEx in MSDN.

---

Linux下开发

参考[4]比较了Windows IOCP和Linux epoll的性能，结论是如果使用Linux，应该使用支持RSS(multi-queue)的NIC，这样可以达到与IOCP类似的性能。

Linux下Reactor模式和Proactor模式

Boost.Asio为了兼容Windows和Linux，在Linux上用epoll和select去模拟proactor模式，影响了它的效率和实现复杂度。其效率不及使用原生I/O机制的其它实现为Reactor模式的网络库。

---

结论

引用来自参考[3]的话:

Both epoll and IOCP are suitable for, and typically used to write high performance networking servers handling a large number of concurrent connections. However those technologies differ significantly enough to require different event processing code. This difference most likely will make a common implementation of connection/socket class meaningless, as the amount of duplicated code would be minimal. In several implementation I have done an attempt to unify the code resulted in a much less maintainable code comparing to separate implementations, and was always rejected.

Also when porting, it is usually easier to port the IOCP-based code to use epoll than vice versa.

So my suggestion:

• If you need to develop the cross-platform networking server, you should focus on Windows and start with IOCP support. Once it is done, it would be easy to add epoll-based backend.
• Usually it is futile( [ˈfju:taɪl], 无用的) to implement the single Connection and ConnectionMgr classes. You will end up not only with a whole lot of #ifdef’s but also with different logic. Better create the base ConnectionMgr class and inherit from it. This way you can keep any shared code in the base class, if there’s any.
• Watch out for the scope of your Connection, and make sure you do not delete the object which has read and/or write operations pending.

---

参考资料

[1] Why doesn't Linux have a system like IOCP or Rio does?, 2014.
[2] 两种高性能I/O设计模式(Reactor/Proactor)的比较 - 文章 - 伯乐在线
[3] Practical difference between epoll and Windows IO Completion Ports (IOCP)
[4] Windows IOCP vs Linux EPOLL Performance Comparison