socket send latency (from email)

今天发信问了一个大牛关于tcp socket send 时间的问题.没想到很快就收到回复, 而且解决了心里纠结很久的一个事情.

-------------------------------------------------------------------------------------------------------

Hi Richard,

Thanks for your time, I'm currently measuring the network latency. I found that the duration of socket send(timer start before the socket send , and stop just after send) is like 20 microsecond on my linux box. How can i check how many times it copies data in kernel mode? and how can I reduce latency here?

Thank you very much.

--------------------------------------------------------------------------------------------------------

Richard Croucher

Date: 9/22/2010

Subject: RE: InfiniBand does not sacrifice reliability or in-order delivery, It has simply moved it into the hardware, so you get all the...

20uS sounds about right. The current Linux TCP stack already does a reasonable job of minimizing memory copies. It will copy it from your buffer into the process socket and from their into the shared data_skb. It will then get DMA'ed from there by the NIC. However, there is a lot of protocol processing. Around 30-30 different functions get called as it progresses down the stack. These appear to consume most of the time.

You can instrument the kernel with Systemtap to find out exactly which ones are called but there is nothing you can really do to reduce it without major kernel rewrites and potential interoperability issues.

I'm bypassing TCP altogether and can achieve a similar send in about 5uS across InfiniBand.

If you want to continue with TCP, then Openonload does improve it to about 10uS but is restricted to Solarflare and SMC Tiger NICs. There is a zero copy implementation for Linux. However ,the standard implementation only works for sending files (the http server use-case). Some embedded systems have implemented it for memory buffers but it is problematical since there is no way of finding out when the buffer can be re-used (the kernel may need to access it for a re-transmit). Consequently, this has never been included in the stock kernels.