romandion

Topics in High-Performance Messaging

-by-

Robert A. Van Valzah
Todd L. Montgomery
Eric Bowden

March 2011

Table of Contents

1. Introduction
2. TCP Latency
3. Group Rate Control
4. Ethernet Flow Control
5. Packet Loss Myths
6. Monitoring Messaging Systems
7. UDP Buffering Background
8. UDP Buffer Sizing
9. Multicast Loopback
10. Sending Multicast on Multiple Interfaces
11. TTL=0 to Keep Multicast Local

12. TTL=1 and Cisco CPU Usage
13. Intermittent Network Multicast Loss
14. Multicast Address Assignment
15. Multicast Retransmissions
16. Messaging Latency Budget
17. Sources of Latency
18. Latency from Interrupt Coalescing
19. Latency Measurement Overview
20. Measuring CPU Scheduling Latency
21. Measuring CPU Contention Latency

Abstract

We have worked together in the field of high-performance messaging for manyyears, and in that time, have seen some messaging systems that worked well and some thatdidn't. Successful deployment of a messaging system requires background information thatis not easily available; most of what we know, we had to learn in the school of hard knocks.To save others a knock or two, we have collected here the essential background information andcommentary on some of the issues involved in successful deployments. This information isorganized as a series of topics around which there seems to be confusion or uncertainty.Please contact us ifyou have questions or comments.

1. Introduction

In the field of high-performance messaging systems, performance tends to be thedominant factor in making design decisions. In this context, "performance" can indicatehigh message rates, high payload data transfer rates, low latency, high scalability, highefficiency, or all of the above. Such factors tend to be important in applications likefinancial market data, satellite, telemetry, and military command & control.

Successful deployment of high-performance messaging systems requirescross-disciplinary knowledge. System-level issues must be considered including network,OS, and host hardware issues.

2. TCP Latency

TCP can be used for latency-sensitive applications, but several factors must beconsidered when choosing it over other protocols that may be better suited tolatency-sensitive applications.

TCP provides low-latency delivery only if:

All receivers can always keep up with the sender
The network is never congested

This essentially boils down to "TCP is OK for latency-sensitive applications ifnothing ever goes wrong."

A little historical perspective helps in understanding TCP. It's easy for one user ofa network to think about TCP only from the point of view of someone who needs to reliablytransfer data though the network. But network architects and administrators often haveother goals. They want to make sure that all users of a network can share the available bandwidthequally and to assure that the network is always available. If TCP allowed all users tosend data whenever they wanted, bandwidth would not be shared equally and networkthroughput might collapse in response to the load. Hence the focus of TCP is decidingwhen to allow a user to send data.The protocol architects who created TCP named it Transmission Control Protocol to reflect this focus. Said differently, TCPadds latency whenever necessary tomaintain equal bandwidth sharing and network stability. Latency-sensitive applicationstypically don't want a transport protocol deciding when they can send.

2.1. TCPLatency Behavior

Broadly speaking, the way that TCP does rate control makes it unsuitable forlatency-sensitive applications. (See Section 3 for acontrast with other ways of doing rate control.) A TCP receiver will add latency whenever packet loss or networkrouting causes packets to arrive out of order. A TCP sender will add latency when going faster would cause networkcongestion or when it would be sending faster than the receiver can process the incomingdata.

2.2. TCP Receiver-Side Latency

TCP only supports one delivery model: in-order delivery. This means that a TCPreceiver must add latency whenever data arrives out of order so as to put the data backin order. TCP also often unnecessarily retransmits data that was already successfullyreceived after out-of-order data is received.

There are two main causes of out-of-order data reception. The most frequent cause ispacket loss, either at the physical or network-layer. Another, less frequent cause ofout-of-order data reception is that packets can take different paths through the network;one path have more latency than another. In either case, TCP inserts latency to put thepackets back in order, as illustrated in Figure 1.

Figure 1. In-Order Delivery

Note that the receiving application cannot receive packet 3 until (the retransmitted)packet 2 has arrived. The receiving TCP stack adds latency while waiting for thesuccessful arrival of packet 2 before packet 3 can be delivered.

Contrast this with the case where a transport with an arrival-order delivery model isused as shown in Figure 2.

Figure 2. Arrival-Order Delivery

Note that packet 3 is delivered to the application layer as soon as it arrives.

TCP cannot provide arrival order delivery, but UDP can. However, simple UDP is awkwardfor many applications because it provides no reliability in delivery. In the aboveexample, it becomes the application's responsibility to detect packet 2's loss andrequest its retransmission.

With this in mind, we created areliable multicast protocol called LBT-RM. It offers lower latency than TCP since it usesUDP and arrival order delivery, but it can also provide reliability through the built-inloss detection and retransmission logic. Of course, each message delivered by LBT-RM hasa sequence number so that the application has message sequencing information available ifneeded.

If you're curious about how much latency is being added by the TCP in-order deliverymodel, you can often get a hint by looking at output from the netstat-s command. Look for statistics from TCP. In particular, look for out-of-orderpackets received and total packets received. Divide these two numbers to get a percentageof packets that are being delayed by TCP in-order delivery.

For example, consider this output from netstat -s.

tcp:
. . .
      2854 packets received
. . .
          172 out-of-order packets (151915 bytes)

The above example statistics show that about 6% of all incoming TCP packets are beingdelayed due the requirement of in-order delivery (this is pretty high; we hope that youare not experiencing this degree of packet loss). Be aware that netstat output tends to vary quite a bit from operating system tooperating system, so your output may look quite different from the above.

2.3.TCP Sender-Side Latency

TCP is designed to send as fast as possible while maintaining equal bandwidth sharingwith all other TCP streams and reliable delivery. A TCP sender may slow down for tworeasons:

Going faster would cause network congestion.
Going faster would send data faster than the receiver can process it.

If you were to liken TCP to an automobile driver, you'd have to call TCP the mostlead-footed yet considerate driver on the road. As long as the road ahead is clear, TCPwill drive as fast as possible. But as soon as other traffic appears on the road, TCPwill slow down so that other drivers (i.e. other TCP streams) have a equal shot at usingthe road too. The technical term for this behavior is congestion control. When a network becomes congested, the TCPalgorithms insure that the pain is felt equally by all active TCP streams, which isexactly the behavior you want if your primary concern is bandwidth sharing and networkstability. However, if you have a latency-sensitive application, you would prefer that itget priority, leaving the other less-critical applications to divide up the remainingbandwidth. That is, you would like to be able to add flashing lights and a siren to yourlatency-sensitive applications.

It's important to note that TCP uses two different mechanisms to measure congestion:round-trip time (RTT) and packet loss. (See Myth: AllPacket Loss is Bad for details.) Sticking with the automobile analogy, an increase inRTT causes TCP to take its foot off the gas to coast while packet loss causes it to stepon the brakes. TCP cuts its transmission rate by 1/2 every time it detects loss. This canadd additional latency after the original loss. Even small amounts of loss candrastically cut TCP throughput.

TCP also makes sure that the sender does not send data faster than the receiver canprocess it. A certain amount of buffer space is allocated on the receiver, and when thatbuffer fills, the sender is blocked from sending more. The technical term for thisbehavior is flow control. TCP'sphilosophy is "better late than never". Many latency-sensitive applications prefer the"better never than late" philosophy. At the very least, a slow, possibly problematicreceiver should not cause unnecessary latency with other receivers. Obviously, thereneeds to be enough buffer space to handle normal periods of traffic bursts and temporaryreceiver slow-downs, but if/when those buffers do fill up, it's better to drop data thanblock the sender.

2.4. TCP Latency Recommendations

It's probably best to avoid using TCP for latency-sensitive applications. As analternative, consider protocols that slow down a sender only when required for networkstability. For example, the Ultra Messaging Streaming Edition (UMS) product offers protocols like LBT-RM that have been designedspecifically for latency-sensitive applications. Such protocols allow reliable yetlatency-bounded delivery without sacrificing network stability.

When TCP cannot be avoided, it's probably best to:

Use clever buffering and non-blocking sockets with TCP to drop data for congested TCPconnections when data loss is preferable over latency. UMS uses such buffering for its latency-bounded TCPfeature.
Consider disabling Nagle's algorithm with the TCP_NODELAYoption to setsockopt(). Although this may produce decreasedlatency, it often does so at the expense of efficiency.
Route latency-sensitive TCP traffic over a dedicated network where congestion isminimized or eliminated.
Keep kernel socket buffers to a minimum so that latency is not added as data is storedin such buffers. TCP windows substantially larger than the bandwidth-delayproduct (BDP) for the network will not increase TCP throughput but they can addlatency.

3. Group RateControl

Any application seeking to deliver the same data stream to a group of receivers faceschallenges in dealing with slow receivers. Group members that can keep up with the sender may be inconvenienced or perhapseven harmed by those that can'tkeep up. At the very least, slow receivers cause the sender to use memory for bufferingthat could perhaps be put to other uses. Buffering adds latency, at least for the slowreceiver and perhaps for all receivers. Note that rate control issues are present forgroups using either multicast or unicast addressing.

3.1. The CrybabyReceiver Problem

Often, the whole group suffers due to the problems of one member. In extreme cases,the throughput for the group falls to zero because resources that the group shares arededicated to the needs of one or a few members. Examples of shared resources include thesender's CPU, memory, and network bandwidth. This phenomenon is sometimes called the"crybaby receiver problem" because the cries (e.g. retransmission requests) from onereceiver dominate the attention of the parent (the sender).

The chance of encountering a crybaby receiver problem increases as the number ofreceivers in the group increases. Odds are that at least one receiver will be havingdifficulty keeping up if the group is large enough.

As long as all receivers in the group are best served by the sender running at thesame speed, there is no conflict within the group. Scenarios such as crybaby receiversoften present a conflict between what's best for a few members and what's best for themajority. Robust systems will have policies for dealing with the apparent conflict andconsistently resolving it.

3.2. Group Rate Control Policies

There are three rate control policies that a sender can use for dealing with suchconflict within a group. Two are extremes and the third is a middle ground.

3.2.1. Policy Extreme1

The sender slows down to the rate of the slowest receiver. If the sender cannotcontrol the rate at which new data arrives for transmission to the group, it must eitherbuffer the data until the slowest receiver is ready or drop it. All receivers in the group then experience latency or lostdata.

3.2.2. Policy Extreme2

The sender sends as fast as is convenient for it. This is often the rate at which newdata arrives or is generated. It is often possible that this rate is too fast for eventhe fastest receivers. All receivers that can't keep up with the rate most convenient forthe sender will experience lost data. This is often called "uncontrolled" since there isno mechanism to regulate the rate used by the sender.

3.2.3. MiddleGround

The sender operates within a set of boundaries established a system administrator orarchitect. The sender's goal is to minimize data loss and latency in the receiver groupwhile staying within the configured limits.

3.3. Group Rate Control Consequences

The extreme policies have potentially dire consequences for many applications. Forexample, neither is ideal for market data and other types of latency-sensitive data.

Extreme 2 is the policy most often used. Successful use of it requires that networksand receivers be provisioned to keep up with the fastest rate that might be convenientfor the sender. Such policies often leave little bandwidth for TCP traffic and can bevulnerable to "NAK storms" and other maladies which can destabilize the entirenetwork.

Extreme 1 is appropriate only for transactional applications where it's more importantfor the group to stay in sync than for the group to have low latency.

The middle ground policy is ideal for many latency-sensitive applications such astransport of financial market data. It allows for low-latency reliable delivery whilemaintaining the stability of the network. No amount of overload can cause a "NAK storm"or other network outage when the policy is established with knowledge of the capabilitiesof the network.

3.4. PolicySelection

The need to establish a group rate control policy is often not apparent to thoseaccustomed to dealing with two-party communication (e.g. TCP). When there are only twoparties communicating, one policy is commonly used: the sender goes as fast as it canwithout going faster than the receiver or being unfair to others on the network. (See Section 2 for details.) This is the only sensible policy forapplications that can withstand some latency and cannot withstand data loss. Withtwo-party communication using TCP, it's the only policy choice you have. However, groupcommunication with one sender and many receivers opens up all of the policy possibilitiesmentioned above.

Some messaging systems support only one policy and hence require no policyconfiguration. Others may allow a choice of policy. Any middle ground policy will need tobe configured to establish the boundaries within which it should operate.

The best results are obtained when the specific needs of a messaging application havebeen considered and the messaging system has been configured to reflect them. A messagingsystem has no means to automatically select from among the available group rate controlpolicies. Human judgment is required.

3.5. Group Rate ControlRecommendations

A group rate control policy should be chosen to match the needs of the applicationserving the group. The choice is often made by weighing the benefits of low latencyagainst reliable delivery. These benefits have to be considered for individuals withinthe group and for the group as a whole.

In some applications, members of the group benefit from the presence of other members.In these applications, the group benefit from reliable reception among all receivers mayoutweigh the pain of added latency or limited group throughput. Consider a group ofservers that share the load from a common set of clients. Assume that the servers have tostay synchronized with a stream of messages to be able to answer client queries. If oneserver leaves the group because it lost some messages, the clients it was serving wouldmove to the remaining servers. This could lead to a domino effect where a traffic burstcaused the slowest server to drop from the group which in turn increased the load on theother servers causing them to fail in turn as well. Clearly in a situation like this,it's better for the whole group of servers to slow down a bit during a traffic peak sothat even the slowest among them can keep up without loss. The appropriate group ratecontrol policy for such an application is Extreme 1 (see Section 3.2.1).

Other applications see no incremental benefit for the group if all members experiencereliable reception. Consider a group of independent traders who all subscribe to a marketdata stream. Traders who can keep up with the rate convenient for the sender do not wantthe sender to slow down for those who can't. Extreme 2 (see Section 3.2.2) is probably the appropriate policy for anapplication like this. However, care must be taken to prevent the sender from goingfaster than even the fastest trader as that often leads to NAK storms from which there isno recovery.

The Ultra Messaging team recommends a carefulanalysis of the policies that are best for the group using an application and forindividual members of the group. The rate control policy best suited to your applicationwill generally emerge from such an analysis. We have found that it is possible to buildstable, low-latency messaging systems with careful network design and a messaging layerlike UMS that supports a middle ground policy through the use ofrate controls.

3.6. Group Rate Control and TransportProtocols

Once you've chosen a group rate control policy appropriate for your application, it'simportant to chose a transport protocol that can implement your chosen policy. Sometransport protocols offer only the extreme policies while others allow parameters to beset to implement a middle ground policy.

UDP provides no rate control at all, so it follows group policy Extreme 2 above (seeSection 3.2.2).

TCP doesn't operate naturally as a group communication protocol since it only supportsunicast addressing. However, when TCP is used to send copies of the same data stream tomore than one receiver, all of the group rate control issues discussed above are present.TCP's inherent flow control feature follows group policy Extreme 1 described above (seeSection 3.2.1). If the sender is willing to usenon-blocking I/O and manage buffering, then Extreme 2 and middle ground policies can beimplemented to some degree. For example UMS supports middle ground policies over TCPwith its latency-bounded TCP feature.

Some reliable multicast transport protocols provide no rate control at all (e.g. TIBCORendezvous), some provide only a fixed maximum rate limit (e.g. PGM), and some provideseparate rate controls for initial data transmission and retransmission (e.g. LBT-RM fromInformatica). See Section 15 for details.

4.Ethernet Flow Control

In the process of testing the LBT-RM reliable multicast protocol used in the InformaticaUMS messaging product, we noticed that some Ethernet switchesand NICs now implement a form of flow control that can cause unexpected results. This iscommonly seen in mixed speed networks (e.g. 10/100 Mbps or 100/1000 Mbps). We have seenapparently strange results such as two machines being able to send TCP at 100 Mbps butbeing limited to 10 Mbps for multicast using LBT-RM.

The cause of this seems to be Ethernet switches and NICs that implement the IEEE802.3x Ethernet flow control standard. It seems to be most commonly implemented indesktop-grade switches. Infrastructure-grade equipment makers seem to have identified the potentialharm that Ethernet flow control can cause and decided to implement it carefully if atall.

The intent of Ethernet flow control is to prevent loss in switches by providing backpressure to the sending NIC on ports that are going too fast to avoid loss. While thissounds like a good idea on the surface, higher-layer protocols like TCP were designed torely on loss as a signal that they should send more slowly. Hence, at the very least,Ethernet flow control duplicates the flow control mechanism already built into TCP. Atworst, it unnecessarily slows senders when there's no need.

Problems can happen with multicast on mixed-speed networks if a switch with Ethernetflow control applies back pressure on a multicast sender when it tries to go faster thanthe slowest active port on the switch. Consider the scenario where two machines X and Yare connected to a switch at 100 Mbps while a third machine Z is connected to the sameswitch at 10 Mbps. X can send TCP traffic to Y at 100 Mbps since the switch sees nocongestion delivering data to Y's switch port. But if X starts sending multicast, theswitch tries to forward the multicast to Z's switch port as well as to Y's. Since Z islimited to 10 Mbps, the switch senses congestion and prevents X from sending faster than10 Mbps.

We have tested switches that behave this way even if there are no multicast receiverson the slow ports. That is, these switches did not have an IGMP snooping feature or anequivalent that would prevent multicast traffic from being forwarded out ports wherethere were no interested receivers. It seems best to avoid such switches when usingLBT-RM. Any production system with switches like this may be vulnerable to the problemwhere the network runs fine until one day somebody plugs an old laptop into a conferenceroom jack and inadvertently limits all the multicast sources to 10 Mbps.

It may also be possible to configure hosts to ignore flow control signals fromswitches. For example, on Linux, the /sbin/ethtool command maybe able to configure a host to ignore pause signals that are sent from a switch sensingcongestion. Module configuration parameters or registry settings might also beappropriate.

5. Packet LossMyths

Packet loss is a common occurrence that is often misunderstood. Many applications canrun successfully with little or no attention paid to packet loss. However,latency-sensitive applications can often benefit from improved performance if packet lossis better understood. We have encountered many myths surrounding packet lossin our UMS development.The following paragraphs address these myths and provide links toadditional helpful information.

Myth--Most loss is caused by transmission errors, gamma rays,etc.
Myth--There is no loss in networks that are operatingproperly.
Myth--Loss only happens in networks (i.e. not inhosts).
Myth--All packet loss is bad.
Myth--Unicast and multicast loss always go together.

Myth--Most loss is caused by transmissionerrors, gamma rays, etc.

Reality--Our experienceand anecdotal evidence from others indicates that buffer overflow is the most commoncause of packet loss. These buffers may be in network hardware (e.g. switches androuters) or it may be in operating systems. See Section 7 for background information.

Myth--There is no loss in networks thatare operating properly.

Reality--The normaloperation of TCP congestion control may cause loss due to queue overflow. See thisreport for more information. Loss rates of several percent were common under heavycongestion.

Myth--Loss only happens in networks(i.e. not in hosts).

Reality--The flowcontrol mechanism of TCP should prevent packet loss due to host buffer overflows.However, UDP contains no flow-control mechanism leaving the possibility that UDP receiverbuffers will overflow. Hosts receiving high-volume UDP traffic often experience internalpacket loss due to UDP buffer overflow. See Section7.6 for more on the contrast between TCP buffering and UDP buffering. See Section 8.9 for advice on detecting UDP buffer overflow ina host.

Myth--All packet loss is bad.

Reality--Packet lossplays at least two important, beneficial roles:

Implicit signaling of congestion: TCP uses loss to discover contention with other TCPstreams. Each TCP stream passing through a congestion point must dynamically discover theexistence of other streams sharing the congestion point in order to fairly share theavailable bandwidth. They do this by monitoring 1) the SRTT (Smoothed Round-Trip Time) asan indication of queue depth, and 2) packet loss as an indication of queue overflow.Hence TCP relies upon packet loss as an implicit signal of network congestion. See Section 2.3 for discussion of the impact this canhave on latency.
Efficient discarding of stale, latency-sensitive data: See Section 8.1 for more information.

Myth--Unicast and multicast loss always go together.

Reality--Unicast andmulticast traffic often follows different forwarding paths in networks and hosts. Justbecause unicast (TCP) reports no loss, you can't assume that there is no multicastloss.

6. Monitoring Messaging Systems

Production deployments of messaging systems often employ real-time monitoring andrapid human intervention if something goes wrong. Urgent reaction to problems detected byreal-time monitoring can be required to prevent the messaging system from becomingunstable. A common example of this is using real-time monitoring to discover crybabyreceivers and repair or remove them from the network. (See Section 3.1 for details.) The presence of a crybaby receivercan starve lossless receivers of bandwidth needed for new messages in what is commonlycalled a NAK storm.

UMS is designed so that real-time monitoring and urgent response are notrequired. The design of UMS encourages stable operation by allowing you to pre-configurehow UMS will use resources under all traffic and network conditions. Hence manualintervention is not required when those conditions occur.

Monitoring UMS still fills important roles other than maintaining stable operation.Chiefly among these are capacity planning and gaining a better understanding the latencythat UMS adds to recover from loss. Collecting accumulated statistics from all sourcesand all receivers once per day is generally adequate for these purposes.

7.UDP Buffering Background

Understanding UDP buffering is critical to leveraging the maximum possible performancefrom the reliable multicast (LBT-RM) and reliable unicast (LBT-RU) protocols withinInformatica'sUMSmessaging product. Our reliable unicast and reliable multicastprotocols use UDP to achieve control of transport latency that would be impossible withTCP.

Much of what we've learned deploying our UDP-based protocols should be applicable toother high-performance work with UDP. We have collected here some of the backgroundinformation that we've found helpful in understanding issues related to UDPbuffering.

Although UDP is buffered on both the send and receive side, we've seldom seen a needto be concerned with send-side UDP buffering. For brevity, we'll use simply "UDPbuffering" to refer to receive-side UDP buffering.

7.1. Functionof UDP Buffering

UDP packets may arrive in bursts because they were sent rapidly or because they werebunched together by the normal buffering action of network switches and routers.

Similarly, UDP packets may be consumed rapidly when CPU time is available to run theconsuming application. Or they may be consumed slowly because CPU time is being used torun other processes.

UDP receive buffering serves to match the arrival rate of UDP packets (or "datagrams")with their consumption rate by an application program. Of course, buffering cannot helpcases where the long-term average send rate exceeds the average receive rate.

7.2.Importance of UDP Buffering

UDP receive buffering is done in the operating system kernel. Typically the kernelallocates a fixed-size buffer for each socket receiving UDP. Buffer space is consumed forevery UDP packet that has arrived but has not yet been delivered to the consumingapplication. Unused space is generally unavailable for other purposes because it must bereadily available for the possible arrival of more packets. See Section 7.4 for a further explanation.

If a UDP packet arrives for a socket with a full buffer, it is discarded by the kerneland a counter is incremented. See Section 8.9 forinformation on detecting UDP loss. A common myth is that all UDP loss is bad. (See Myth: All Packet Loss is Bad.) Even if it's not allbad, UDP loss does have its consequences. (See Section8.3.)

7.3. Kernel and UserMode Roles in UDP Buffering

The memory required for the kernel to do UDP buffering is a scarce resource that thekernel tries to allocate wisely. (See Section7.4 for more on the rationale.) Applications expecting low-volume UDP traffic orthose expecting low CPU scheduling latency (see Section 17.6) need not consume very much ofthis scarce resource. Applications expecting high-volume UDP traffic or those expecting ahigh CPU scheduling latency may be justified in consuming more UDP buffer space thanothers.

Typically, the kernel allocates a modest-size buffer when a UDP socket is created.This is generally adequate for less-demanding applications. Applications requiring alarger UDP receive buffer can request it with the system call setsockopt(...SO_RCVBUF...).

Explicit configuration of the application may be required before it will request alarger UDP buffer. (For UMS, this is the context option transport_lbtrm_receiver_socket_buffer.)

The kernel configuration will allow such requests to succeed only up to a set sizelimit. This limit may often be increased by changing the kernel configuration. See Section 8.8 for information on settingkernel UDP buffer limits.

Hence two steps are often required to get adequate UDP buffer space:

Change the kernel configuration to increase the limit on the largest UDP bufferallocation that it will allow.
Change the application to request a larger UDP buffer.

7.4. Unix Kernel UDP Buffer Limit Rationale

It may seem to some that the default maximum UDP buffer size on many Unix kernels is abit stingy. Understanding the rationale behind these limits may help.

UDP is typically used for low-volume query/response work (e.g. DNS, NTP, etc.). Thekernel default limits assume that UDP will be used in this way.

UDP kernel buffer space is allocated from physical memory for the exclusive use of oneprocess. The kernel tries to make sure that one process can't starve others for physicalmemory by allocating large UDP buffers that exhaust all the physical memory on amachine.

To some degree, the meager default limits are a legacy from the days when 4 MB was alot of physical memory. In these days when several gigabytes of physical memory space iscommon, such small default limits seem particularly stingy.

7.5. UDPBuffering and CPU Scheduling Latency

For the lowest-possible latency, the operating system would run a process wishing toreceive a UDP packet as soon as the packet arrives. In practice, the operating system mayallow other processes to finish using their CPU time slice first. It may also seek toimprove efficiency by accumulating several UDP packets before running the application. Wewill call the time that elapses between when a UDP packet arrives and when the consumingapplication gets to run on a CPU the CPU scheduling latency. See Section 17.6 for more information. UDP bufferspace fills during CPU scheduling latency and empties when the consuming process runs ona CPU. CPU scheduling latency plays a key role optimal UDP buffer sizing. See Section 8.1 for more information.

7.6.TCP Receive Buffering vs. UDP Receive Buffering

The operating system kernel automatically allocates TCP receive buffer space based onpolicy settings, available memory, and other factors. TCP in the sending kernelcontinuously monitors available receive buffer space in the receiving kernel. When a TCPreceive buffer fills up, the sending kernel prevents the sending application from usingCPU time to generate any more data for the connection. This behavior is called "flowcontrol." In a nutshell, it prevents the receiver buffer space from overflowing by addinglatency at the sender. We say that the speed of a TCP sender is "receiver-paced" becausethe sending application is prevented from sending when data cannot be delivered to thereceiving application.

The OS kernel also allocates UDP receive buffer space based on policy settings.However, UDP senders do not monitor available UDP receive buffer space in the receivingkernel. UDP receivers simply discard incoming packets once all available buffer space isexhausted. We say that the speed of a UDP sender is "sender-paced" because the sendingapplication can send whenever it wants without regard to available buffer space in thereceiving kernel.

The default TCP buffer settings are generally adequate and usually require adjustmentonly for unusual network parameters or performance goals.

The appropriate size for an application's UDP receive buffer is influenced by factorsthat cannot be known when operating system default policies are established. Further,there is nothing the operating system can do to automatically discover the appropriatesize. An application may know that it would benefit from a UDP receive buffer larger orsmaller than the default used by the operating system. It can request a non-default UDPbuffer size from the operating system, but the request may not be granted due to a policylimit. See Section 7.4 for reasons behind suchpolicy limits. Operating system policies may have to be adjusted to successfully runhigh-performance UDP applications like UMS. See Section 8.8 to change operating systempolicies.

8. UDP BufferSizing

There are many questions surrounding UDP buffer sizing. What is the optimal size? Whatare the consequences of an improperly sized UDP buffer? What are the equations needed tocompute an appropriate size for a UDP buffer? What default limit will the OS kernel placeon UDP buffer size and how can I change it? How can I tell if I'm having UDP lossproblems due to buffers that are too small? Answers to these questions and more are givenin the following sections.

8.1.Optimal UDP Buffer Sizing

UDP buffer sizes should be large enough to allow an application to endure the normalvariance in CPU scheduling latency without suffering packet loss. They should also besmall enough to prevent the application from having to read through excessively old datafollowing an unusual spike in CPU scheduling latency.

8.2. UDP Buffer Space Too Small:Consequences

Too little UDP buffer space causes the operating system kernel to discard UDP packets.The resulting packet loss has consequences described below.

The kernel often keeps counts of UDP packets received and lost. See Section 8.9 for information on detecting UDP loss due toUDP buffer space overflow. A common myth is that all UDP loss is bad (see Myth: All Packet Loss is Bad).

8.3. UDPLoss: Consequences

In most cases, it's the secondary effects of UDP loss that matter most. That is, it'sthe reaction to the loss that hasmaterial consequences more so than the loss itself. Note that the consequences discussedhere are independent of the causeof the loss. Inadequate UDP receive buffering is just one of the more common causes we'veencountered deploying UMS.

Consider these areas when assessing the consequences of UDP loss:

Latency--The time that passesbetween the initial transmission of a UDP packet and the eventual successful reception ofa retransmission is latency that could have been avoided were it not for the interveningloss.
Bandwidth--UDP loss usuallyresults in requests for retransmission, unless more up-to-date information is expectedsoon (e.g. in the case of stock quote updates). Bandwidth used for retransmissions maybecome significant, especially in cases where there is a large amount of loss or a largenumber of receivers experiencing loss. See Section15 for more information on multicast retransmissions.
CPU Time--UDP loss causes thereceiver to use CPU time to detect the loss, request one or more retransmissions, andperform the repair. Note that efficiently dealing with loss among a group of receiversrequires the use of many timers, often of short-duration. Scheduling and processing suchtimers generally requires CPU time in both the operating system kernel ("system time")and in the application receiving UDP ("user time"). Additional CPU time is required toswitch between kernel and user modes.

On the sender, CPU time is used to process retransmission requests and to sendretransmissions as appropriate. As on the receiver, many timers are required forefficient retransmission processing, thus requiring many switches between kernel and usermodes.
Memory--UDP receivers that canonly process data in the order that it was initially sent must allocate memory whilewaiting for retransmissions to arrive. UDP loss causes such receivers to receive data inan order different than that used by the sender. Memory is used to restore the order inwhich it was initially sent.

Even UDP receivers that canprocess UDP packets in the order they arrive may not be able to tolerate duplication ofpackets. Such receivers must allocate memory to track which packets have beensuccessfully processed and which have not.

UDP senders interested in reliable reception by their receivers must allocate memoryto retain UDP packets after their initial transmission. Retained packets are used to fillretransmission requests.

8.4. UDP Buffer Space Too Large:Consequences

Even though too little UDP buffer space is definitely bad and more is generallybetter, it is still possible to have too much of a good thing. Perhaps the two mostsignificant consequences of too much UDP buffer space are slower recovery from loss andphysical memory usage. Each of these is discussed in turn below.

Slower Recovery--To bestunderstand the consequences of too much UDP buffer space, consider a stream of packetsthat regularly updates the current value of a rapidly-changing variable in every tenthpacket. Why buffer more than ten packets? Doing so would only increase the number ofstale packets that must be discarded at the application layer. Given a data stream likethis, it's generally better to configure a ten-packet buffer in the kernel so that nomore than ten stale packets have to be read by the application before a return to freshones from the stream.

It's often counter-intuitive, but excessive UDP buffering can actually increase therecovery time following a large packet loss event. UDP receive buffers should be sized tomatch the latency budget allocated for CPU scheduling latency with knowledge of expecteddata rates. See Section 16 for more informationon latency budgets. See Section 8.6 for a UDPbuffer sizing equation.
Physical Memory Usage--It ispossible to exhaust available physical memory with UDP buffer space. Requesting a UDPreceive buffer of 32 MB and then invoking ten receiver applications uses 320 MB ofphysical memory. See Section 7.4 for moreinformation.

8.5. UDP Buffer Size Equations: Latency

Assuming that an average rate is known for a UDP data stream, the amount of latencythat would be added by a full UDP receive buffer can be computed as:

Max Latency = Buffer Size / Average Rate

Note: Take care to watch for different units in buffer size and average rate(e.g. kilobytes vs. megabits per second).

8.6.UDP Buffer Size Equations: Buffer Size

Assuming that an average rate is known for a UDP data stream, the buffer size neededto avoid loss a given worst case CPU scheduling latency can be computed as:

Buffer Size = Max Latency * Average Rate

Note: Since data rates are often measured in bits per second while buffers are often allocated in bytes, careful conversion may benecessary.

8.7. UDP Buffer Kernel Defaults

The kernel variable that limits the maximum size allowed for a UDP receive buffer hasdifferent names and default values by kernel given in the following table:

Kernel	Variable	Default Value
Linux	`net.core.rmem_max`	131071
Solaris	`udp_max_buf`	262144
FreeBSD, Darwin	`kern.ipc.maxsockbuf`	262144
AIX	`sb_max`	1048576
Windows	None we know of	Seems to grant all reasonable requests

8.8. Setting Kernel UDP Buffer Limits

The examples in this table give the commands needed to set the kernel UDP buffer limitto 8 MB. Root privilege is required to execute these commands.

Kernel	Command
Linux	`sysctl -w net.core.rmem_max=8388608`
Solaris	`ndd -set /dev/udp udp_max_buf 8388608`
FreeBSD, Darwin	`sysctl -w kern.ipc.maxsockbuf=8388608`
AIX	`no -o sb_max=8388608` (note: AIX only permits sizes of1048576, 4194304 or 8388608)

8.8.1. Making Changes Survive Reboot

The AIX command given above will change the current value and automatically modify /etc/tunables/nextboot so that the change will survive rebooting.Other platforms require additional work described below to make changes survive areboot.

For Linux and FreeBSD, simply add the sysctl variable setting given above to /etc/sysctl.conf leaving off the sysctl -wpart.

We haven't found a convention for Solaris, but would love to hear about it if we'vemissed something. We've had success just adding the ndd commandgiven above to the end of /etc/rc2.d/S20sysetup.

8.9. DetectingUDP Loss

Interpreting the output of netstat is important in detectingUDP loss. Unfortunately, the output varies considerably from one flavor of Unix toanother. Hence we can't give one set of instructions that will work with all flavors.

For each Unix flavor, we tested under normal conditions and then under conditionsforcing UDP loss while keeping a close eye on the output of netstat-s before and after the tests. This revealed the statistics that appeared to have arelationship with UDP packet loss. Output from Solaris and FreeBSD netstat was the most intuitive; Linux and AIX much less so.Following sections give the command we used and highlight the important output fordetecting UDP loss.

8.9.1. Detecting Solaris UDP Loss

Use netstat -s. Look for udpInOverflows. It will be in the IPv4section, not in the UDP section as you might expect. Forexample:

IPv4:
      udpInOverflows      = 82427

8.9.2. Detecting Linux UDP Loss

Use netstat -su. Look for packet receiveerrors in the Udp section. For example:

Udp:
      38799 packet receive errors

8.9.3. Detecting Windows UDP Loss

The command, netstat -s, doesn't work the same in Microsoft® Windows® as itdoes in other operating systems. Therefore, unfortunately, there is no way to detect UDPbuffer overflow in Microsoft Windows.

8.9.4.Detecting AIX UDP Loss

Use netstat -s. Look for fragmentsdropped (dup or out of space) in the ip section. Forexample:

ip:
      77070 fragments dropped (dup or out of space)

8.9.5. Detecting FreeBSD and Darwin UDP Loss

Use netstat -s. Look for dropped due tofull socket buffers in the udp section. For example:

udp:
      6343 dropped due to full socket buffers

9. MulticastLoopback

It's common practice for two processes running on the same machine to use the loopbacknetwork pseudo-interface -- IP address 127.0.0.1 -- to communicate even when noreal network interface is present. The loopback interface neatly solves the problem ofproviding a usable destination address when there are no other interfaces or when thoseinterfaces are unusable because they are not active. An example of an inactive interfacemight be an Ethernet interface with no link integrity (or carrier detect) signal. Hencean unplugged laptop may be used to develop unicast applications simply by configuringthose applications to use the loopback interface's address of 127.0.0.1.

Unfortunately, the kernel's loopback interface is not used for multicast, and is thereforeincapable of forwarding multicast from application to application on the samemachine. Ethernet interfaces arecapable of forwarding multicast, but only when they are connected to a network. This is seldom anissue on desktops that are generally connected, but it does come up with laptops thattend to spend some of their time unplugged.

To enable development of Informatica UMS applications even on an unplugged laptop,you may configure UMS to use only unicast.

A simpler solution: loopback the Ethernet interface so that the kernel sees itonce again as being capable of forwarding multicast between applications. An inexpensivecommercial product will do the trick.

Note that there may be a time delay between when Ethernet loopback is established andwhen the interface becomes usable for multicast loopback. The interface won't be usabletill it has an IP address. It may take a minute or so for a DHCP client to give up ontrying to find a DHCP server. At least on Windows, after DHCP times out, the interfaceshould get a 169.254.x.y "link local" address if the machine doesn't have a cachedDHCP lease that is still valid.

10. Sending Multicast on MultipleInterfaces

Informatica UMS customers, who are often concerned about a "single point offailure", often ask if UMS can simultaneously send one message on two differentinterfaces. Unfortunately, there is no single thing an application can do throughthe socket interface to cause one packet to be sent on two interfaces.

It may seem odd that you can write a single system call that will accept a TCPconnection on all of the interfaces on a machine, yet you can't write a single systemcall that will send a packet out on all interfaces. We think this behavior can be tracedback to the presumption that all interfaces will be part of the same internet (or perhapsthe same "Autonomous System" or "AS" to use BGP parlance). You generally wouldn't want to send the samepacket into your internet more than once. Clearly, this doesn't account for the casewhere the interfaces on a machine straddle AS boundaries or cases where you actuallydo want the same packet sent twice(say for redundancy).

11. TTL=0 to Keep Multicast Local

Some users of UMS wish to limit the scope of multicast tothe local machine. This can save some NIC transmit bandwidth for cases where it is knownthat there are no interested receivers reachable by network connections.

Setting TTL to 1 is a well-defined way of limiting traffic to the sending LAN, but itcan have undesired consequences (see Section12). We haven't yet found a definition for the expected behavior when TTL is set to0, but limiting the scope to the local machine would seem to be the obvious meaning. Weknow that multicast is not copied to local receivers through the loopback interface usedfor local unicast receivers. We believe that it may be copied closer to the device driverlevel in some kernels. Hence the particular NIC driver in use may have an impact on TTL 0behavior. It's always best to test for traffic leakage before assuming that TTL 0 willkeep it local to the sending machine.

We have run tests in our Ultra Messaging® lab andfound that TTL 0 will keep multicast traffic local on Linux 2.4 kernels, Solaris 8 and10, Windows XP, AIX, and FreeBSD.

12. TTL=1 and Cisco CPU Usage

We have seen cases where multicast packets with a TTL of 1 caused high CPU usage onsome Cisco routers. The most direct way to diagnose the problem is to see the symptoms goaway when the TTL is increased beyond the network diameter. We have changed the defaultTTL used by our UMS product to 16 in the hope of avoiding theproblem. The remainder of this section describes the cause of the problem and suggestsother ways to avoid it.

The problem is known to happen on Cisco Catalyst 65xx and 67xx switches using asupervisor 720 module. Multicast traffic is normally forwarded at wire speed in hardwareon these switches. However, when a multicast packet arrives for switching with TTL=1, thehardware passes it up to be process switched instead of forwarding it in hardware.(Unicast packets with TTL=1 require this behavior so that an ICMP TTL expired message canbe generated in the CPU.)

There are several ways of dealing with the problem outlined in following sections.

12.1.Solution: Increase TTL

At the risk of stating the obvious, we have to say that one potential solution is toconfigure all multicast sources so that they will always have a TTL larger than thenetwork diameter and hence will never hit this problem. However, this is best thought ofas voluntary compliance by multicast sources rather than an administrative prohibition. Acommon goal is to establish multicast connectivity within one group of networks whilepreventing it outside those nets. Source TTL settings alone are often inadequate formeeting such goals so we present other methods below.

12.2. Solution: IP MulticastTTL-Threshold

The Cisco IOS command ip multicast ttl-threshold looks likeit might help, but it forces all multicast traffic leaving the interface to be routed viathe mcache. This may be better than having the TTL expire and the traffic be processswitched, but ip multicast boundary is probably a bettersolution (see Section 12.3).

12.3. Solution: IP Multicast Boundary

The Cisco IOS command ip multicast boundary can be used toestablish a layer-2 boundary that multicast will not cross. This example configurationshows how traffic outside the range 239.192.0.0/16 can be prevented from ever passingpassing up to layer-3 processing:

interface Vlan13
 ip multicast boundary 57
. . .
access-list 57 permit 239.192.0.0 0.0.255.255
access-list 57 deny any

12.4. Solution: Hardware Rate Limit

Recent versions (PFC3B or later) of the Supervisor 720 module have hardware that canbe used to limit how often the CPU will be interrupted by packets with expiring TTL. Thisexample shows how to enable the hardware rate limiter and check its operation:

Router#mls rate-limit all ttl-failure 500 100
Router#show mls rate-limit
  Sharing Codes: S - static, D - dynamic
  Codes dynamic sharing: H - owner (head) of the group, g - guest of the group

    Rate Limiter Type       Status     Packets/s   Burst  Sharing
  ---------------------   ----------   ---------   -----  -------
. . .
            TTL FAILURE   On                 500     100  Not sharing
. . .
Router#

13. Intermittent Network MulticastLoss

We have diagnosed some tricky cases of intermittent multicast loss in thenetwork while deploying the Informatica UMS messaging product. Such problems can manifest themselvesas spikes in latency while UMS repairs the loss (see Section 17.3).

Troubleshooting intermittent loss is a different type of problem than establishingbasic multicast connectivity. Network hardware vendors such as Cisco providetroubleshooting advice for connectivity problems, but don't seem to offer much advice fortroubleshooting intermittent problems. Notes on our experiences follow.

The root cause of intermittent multicast loss is generally a failing in a mechanismdesigned to limit the flow of multicast traffic so that it reaches only interestedreceivers. Examples of such mechanisms include IGMP snooping, CGMP, and PIM. Althoughthey operate in different ways and at different layers, they all depend on receivingmulticast interest information from receivers.

IGMP is used by multicast routers to maintain interest information for hosts onnetworks to which they are directly connected. (See the Wikipedia article on IGMP formore details.) PIM-SM or another multicast routing protocol must proxy interestinformation for receivers reachable only through other multicast routers. Such mechanismsuse timers to clean up the interests of receivers that leave the network withoutexplicitly revoking their interest. Problems with the settings of these timers can causeintermittent multicast loss. Similarly, interoperability problems between differentimplementations of these mechanisms can cause temporary lapses in multicastconnectivity.

Simple tests can help to diagnose such problems and localize the cause. It's best tobegin by adding multicast receivers at key points in the network topology. A receiverprocess added on the same machine as the multicast source can make sure that the sourceis at least attempting to send thedata. It may also be helpful to add a simple hub or other network tap before themulticast source reaches the first switch in the network to confirm that packets aren'tbeing lost before they leave the NIC. Logging multicast traffic with accurate time stampsand comparing logs across different monitoring points on the network can help isolate thenetwork component that's causing the intermittent loss.

We used such techniques to identify an IGMP snooping interoperability problem between3Com, Dell, and Cisco switches on a customer network. The customer chose to work aroundthe problem by simply disabling IGMP snooping on all the switches. This was a reasonablechoice since the receiver density across switch ports was fairly high and their multicastrates were quite low so that IGMP snooping wasn't adding much value on their network.

We've also seen a case where an operating system mysteriously stopped answering IGMPmembership queries for interested processes running on the machine. It did continue tosend an initial join message when a process started and a leave message when a processexited. An IGMP snooping switch heard the initial join message, started forwardingtraffic, but later stopped forwarding it after a timeout had lapsed without hearingadditional membership reports from the port. Rebooting the operating system fixed theproblem.

Similar symptoms may appear if IGMP snooping switches are used without a multicastrouter or other source of IGMP membership query messages. Many modern switches that arecapable of IGMP snooping can also act as an IGMP querier. Such a feature must be used ifa LAN doesn't have a multicast router and you want the selective forwarding benefit ofIGMP snooping. If IGMP snooping is enabled without a querier on the LAN, thenintermittent multicast connectivity is likely to result.

14. Multicast Address Assignment

There are several subtle points that often deserve consideration when assigningmulticast addresses. We've collected these as advice and rationale here.

Avoid 224.0.0.x--Traffic toaddresses of the form 224.0.0.x isoften flooded to all switch ports. This address range is reserved for link-local uses.Many routing protocols assume that all traffic within this range will be received by allrouters on the network. Hence (at least all Cisco) switches flood traffic within thisrange. The flooding behavior overrides the normal selective forwarding behavior of amulticast-aware switch (e.g. IGMP snooping, CGMP, etc.).
Watch for 32:1 overlap--32non-contiguous IP multicast addresses are mapped onto each Ethernet multicast address. Areceiver that joins a single IP multicast group implicitly joins 31 others due to thisoverlap. Of course, filtering in the operating system discards undesired multicasttraffic from applications, but NIC bandwidth and CPU resources are nonetheless consumeddiscarding it. The overlap occurs in the 5 high-order bits, so it's best to use the 23low-order bits to make distinct multicast streams unique. For example, IP multicastaddresses in the range 239.0.0.0 to 239.127.255.255 all map to unique Ethernet multicastaddresses. However, IP multicast address 239.128.0.0 maps to the same Ethernet multicast address as239.0.0.0, 239.128.0.1 maps to the same Ethernetmulticast address as 239.0.0.1,etc.
Avoid x.0.0.y andx.128.0.y--Combining the above two considerations, it's best to avoid using IPmulticast addresses of the form x.0.0.yand x.128.0.y since they all map onto the range ofEthernet multicast addresses that are flooded to all switch ports.
Watch for address assignmentconflicts--IANA administersInternetmulticast addresses. Potential conflicts with Internet multicast address assignmentscan be avoided by using GLOPaddressing ( AS required) or administratively scoped addresses. Such addresses can be safely used ona network connected to the Internet without fear of conflict with multicast sourcesoriginating on the Internet. Administratively scoped addresses are roughly analogous tothe unicast address space forprivate internets. Site-local multicast addresses are of the form 239.255.x.y, but can grow down to 239.252.x.y ifneeded. Organization-local multicast addresses are of the form 239.192-251.x.y, but can grow down to 239.x.y.z ifneeded.

That's the condensed version of our advice. For a more detailed treatment (57 pages!),see Cisco's Guidelines for Enterprise IP Multicast Address Allocation paper.

15. Multicast Retransmissions

It is quite likely that members of a multicast group will experience different losspatterns. Group members who experienced loss will be interested in retransmission of thelost data while members who have already received it will not. In many ways, thispresents conflicts similar to those involved in choosing the best sending rate for thegroup (see Section 3). The conflicts are particularlypronounced when loss is experienced by one or a small group of receivers (see Section 3.1).

15.1. Retransmission Cost/Benefit

It's important to note how small group loss differs from the case where the packetloss is experienced by many receivers in the group. When loss is widespread, there iswidespread benefit from the consequent multicast retransmissions. When loss is isolatedto one or a few receivers, they benefit from the retransmissions, but the other receiversmay experience latency while the sender retransmits. In the common case where there is alimited amount of network bandwidth between the sender and receivers, the bandwidthneeded for retransmissions must be subtracted from that available for new datatransmission. For example, the LBT-RM reliable multicast protocol withinInformatica UMSalways prioritizes retransmissions ahead of new data,so they add a small amount of latency when there there is new data waiting to be sent toall members of the group.

15.2. Multicast Retransmission ControlTechniques

Reliable delivery to a multicast group in the face of loss involves a trade offbetween throughput/latency and reliable reception for all members. Ideally,administrative policy should establish the boundaries within which reliability will bemaintained. A simple and effective way to establish a boundary is to limit the amount ofbandwidth that will be used for retransmissions. Establishing such a boundary caneffectively defend a group of receivers against an "attack" from a crybaby receiver. Nomatter how much loss is experienced across the receiver set, the sender will limit theretransmission rate to be within the boundary set by administrative policy.

Note that limiting the retransmission request rate on receivers might be better than doing nothing,but it's not as effective as limiting the bandwidth available for retransmission. Forexample, if a large number of receivers experience loss, then the combined retransmissionrequest rate could be unacceptably high, even if each individual receiver limits its ownretransmission request rate.

15.3. Avoiding Multicast Receiver Loss

Even if retransmission rates have been limited, it is still important to identify thecause of isolated receiver loss problems and repair them. Usually, such loss is caused byoverrunning NIC buffers or UDP socket buffers. Assuming that the sender cannot be sloweddown, receiver loss can generally avoided by one or more of these means:

Increasing NIC ringbuffer size (e.g. use a brand name, server-class NIC instead of a generic,workstation-class NIC, also see Section18.3)
Decreasing the OS NIC interrupt service latency (e.g. decrease CPU workload or addmore CPUs to a multi-CPU machine)
Increasing UDP socket buffer size (see Section 8.8)
Decreasing the OS process context switching time (e.g. decrease CPU workload or addmore CPUs to a multi-CPU machine)

Establishing a bandwidth limit on retransmissions will not help an isolated receiverexperiencing loss, but it can be a critical factor in ensuring that one receiver does nottake down the whole group with excessive retransmission requests. Retransmission ratelimits are likely to increase the number of unrecoverable losses on receiversexperiencing loss. Still, it's generally best for the group to first establish a defenseagainst future crybaby receivers before working to fix any individual receiverproblems.

16.Messaging Latency Budget

The latency of a messaging application is the sum of the latencies of its parts.Often, the primary concern is the application latency, not the latencies of the parts. Intheory, this allows some parts of the application more than average latency for a givenmessage if other parts use less than average.

In practice, it seldom works out this way. The same factors that make one part takelonger often make other parts take longer. Increasing message size is the obviousexample. Another example is a message traffic burst. It can cause CPU contention therebyadding latency throughout a messaging application. Network traffic bursts can sharplyincrease latency in network queues. See Section 17.8 for details.

The challenge is building a messaging application that consistently meets its latencygoals for all workloads. Ideally, failures are logged for later analysis if latency goalscannot be met. Similar challenges are faced by the builders of VoIP telephony systems andothers concerned with real-time performance. We have seen successful messagingapplications designed by borrowing the concept of a latency budget from other real-time systems.

A latency budget is best applied by first establishing a total latency budget for themessaging application. Then identify all the sources of latency in the application andallocate a portion of the application budget to each source. (See Section 17 for a list of commonly-encountered latencysources.) Each latency source is aware of its budget and monitors its performance. Itlogs any cases where it can't complete its work within the budgeted time.

Most "managed" pieces of network hardware already do this. Routers and managedswitches have counters that are incremented every time a packet cannot be queued becausea queue is already full. This isn't so much a time budget as it is a queue space budget.But space can be easily converted to time by dividing by the speed of the interface.

TheInformatica UMS messaging system supports latency boundaries in messagebatching, latency-bounded TCP, and in our reliable unicast and multicast transportprotocols. Messaging applications are notified whenever loss-free delivery cannot bemaintained within the latency budget.

One of the big benefits we've seen from establishing a latency budget is that it helpsto guide the work and reduce "finger pointing" in large organizations. A messagingapplication often can't meet its latency goals without the cooperation of many groupswithin a large organization. The team that maintains the network infrastructure must keepqueuing delays to a minimum. The team that administers operating systems must optimizetuning and limit non-essential load on the OS. The team that budgets for hardwarepurchases must make sure adequate CPU and networking hardware is purchased. The team thatadministers the messaging system must configure it to make efficient use of available CPUand network resources. An overall application latency budget that is subdivided overpotential latency sources is a good tool for identifying the root cause of latencyproblems when they occur. If each latency source logs cases where it exceeded its budgetor dropped a message, it's much easier to take corrective action.

17. Sources ofLatency

Our work with messaging systems has given usmany opportunities to investigate sources of latency in messaging. This section lists thesources we most frequently encounter. It contains links to other sections where latencysources are discussed in more detail. Sources are listed in approximate order ofdecreasing contribution to worst-case system latency. Of course, the amount of latencydue to each source in each system will be different.

Some sources of latency impact every message sent while others impact only somemessages. Some sources have separate fixed and variable components. The fixed componentforms a lower bound on latency for all messages while the variable component contributesto the variance in latency between messages. Where possible, latency sources will becharacterized by when they occur and their variability.

17.1. Intermediaries

Many messaging systems contain components that add little value to the overall system.The time taken for communication between these components often adds up to a substantialfraction of the total system latency. To eliminate this extra and unnecessary overhead, andtherefore provide the best possible performance (both throughput and latency) in any given network and application environment, Informaticaadopted a "no daemons" design philosophy in itsUMSproduct.

17.2. Garbage Collection

One of the compelling benefits of managed languages like Java and C# is thatprogrammers need not worry about tracking reference counts and freeing memory occupied byunused objects. This pushes the burden of reference counting to the language run-timecode (Java's JVM or C#'s CLR). A common source of intermittent but significant latencyfor messaging systems involving managed languages is garbage collection in the languagerun-time code.

The latency happens when the run-time systems stops execution of all user code whilecounting references. More modern run-time systems allow execution even while garbagecollection is happening (e.g. mark and sweep techniques).

Garbage collection latency generally happens infrequently, but depending on thesystem, it can be significant when it does happen. It impacts all messages that arriveduring collection and those that are queued as a result of the latency.

17.3. Retransmissions

It is common for messaging applications to expect loss-free message delivery even ifthe network layer drops packets. This implies that either the messaging layer itself or atransport layer protocol under it must repair the loss. The time taken to discover theloss, request retransmission, and the arrival of the retransmission all contribute toretransmission latency. See Section 15 for moreinformation on multicast retransmissions.

Retransmission latency typically only happens when physical- or network-layer lossrequires retransmission. This is infrequent in well-managed networks. In the event thatretransmission is requested by a receiver detecting loss, the fixed component ofretransmission latency is the RTT from source to receiver while the remainder isvariable.

17.4. Reordering

Messages may arrive out of order at a receiver due to network loss or simply becausethe network has multiple paths allowing some messages faster paths than others. When amessage arrives ahead of some that were sent before it, it can be held until itspredecessors also arrive or may be delivered immediately. The holding process allowsapplications to receive messages in order even when they arrive out of order, but it addsa reordering latency. InformaticaUMS offers an arrival-order delivery feature that avoids thisdelay, but TCP offers no such option. See Section 2 for adeeper discussion of reordering latency in TCP.

Note that in the case where a lost message cannot be recovered, the reordering latencycan become very large. Following loss, TCP uses an exponential retransmission algorithmwhich can lead to latencies measured in minutes. Bounded-reliability protocols likeLBT-RM and LBT-RU allow control over the timers that detect unrecoverable loss. Thesetimers can be set to move on much more quickly following unrecoverable loss.

Reordering latency only happens when messages arrive out of order. It is entirelyvariable with no fixed component. However, in cases where loss is the cause of reorderinglatency, reordering latency is the lesser of the retransmission latency or unrecoverableloss detection threshold.

17.5. Batching

Achieving low latency is almost always at odds with efficient use of resources. Thereis a fixed overhead associated with every network packet generated and every interruptserviced. CPU time is required to generate and decode the packet. Network bandwidth isrequired for physical, network, and transport layer protocols. When message payloads aresmall relative to the network MTU, this overhead can be amortized over many messages bybatching them together into a single packet. This provides a significant efficiencyimprovement over the simple, low-latency alternative of sending one message perpacket.

Similarly, in the world of NIC hardware, a NIC can often be configured to interruptthe CPU for each network packet as it is received. This provides the lowest possiblelatency, but doesn't get very much work done for the fixed cost of servicing theinterrupt. Many modern Gigabit Ethernet NICs have a feature that allows the NIC to delayinterrupting the CPU until several packets have arrived, thus amortizing the fixed costof servicing the interrupt over all of the packets serviced. This adds latency in theinterest of efficiency and performance under load. See Section 18 for more information.

Batching latency may be small or insignificant if messages are sent very quickly. Themost difficult trade offs have to be made when it can't be known in advance when the nextmessage will be sent. Trade offs may be specified as a maximum latency that would beadded before sending a network packet. Additionally, a minimum size required to triggertransmission of a network packet could be specified. UMSoffers applications control over these batching parameters so that they can optimize thetrade off between efficiency and latency.

Batching latency is variable depending on the batching control parameters and the timebetween messages.

Batching can happen in the transport layer as well as in the messaging layer. Nagle'salgorithm is widely used in TCP to improve efficiency by delaying packet transmissions.See Section 2.4 for more information.

17.6. CPU Scheduling

One of the critical resources a messaging system needs is CPU time. There is oftensome latency between when messaging code is ready to run and when it actually gets a CPUscheduled.

There are generally both fixed and variable components to CPU scheduling latency. Thefixed component is the latency when a CPU is idle and can be immediately scheduled tomessaging code following a device interrupt that makes it ready to run. See Section 20 for more background andmeasurements we've taken.

The variable component of CPU scheduling latency is often due to contention over CPUresources. If no idle CPUs are available when messaging code becomes ready to run, thenthe CPU scheduling latency will also include CPU contention latency. See Section 21 for more background andmeasurements we've taken.

17.7. Socket Buffers

Socket buffers are present in the OS kernel on both the sending and receiving ends.These buffers help to smooth out fluctuations in message generation and consumptionrates. However, like any buffering, socket buffers can add latency to a messaging system.The worst-case latency can be computed by dividing the size of the socket buffer by thedata rate flowing through the buffer.

Socket buffer latency can vary from near zero to the maximum computed above. Thefaster messages are produced or the more slowly they are consumed, the more likely theyare to be delayed by socket buffering.

17.8. Network Queuing

Switches and routers buffer partial or complete packets on queues before forwardingthem. Such buffering helps to smooth out peaks in packet entrance rates that may brieflyexceed the wire speed of an exit interface.

Desktop and consumer-grade switches tend to have little memory for buffering and henceare generally only willing to queue a packet or two for each exit interface. However,routers and infrastructure-grade switches have enough memory to buffer several dozenpackets per exit interface. Each queued packet adds latency equal to the time needed toserialize it (see Section 17.10).

Network queuing latency is present whenever switches and routers are used. However, itis highly variable. On an idle network with cut-through switching, it could be just theserialization latency for the portion of a packet needed to determine its destination. Ona congested routed network, it could be many dozen times the serialization latency of anMTU-sized packet.

17.9. Network Access Control

Sometimes also called network admission control, this latency source is a combinationof factors that may add latency before a packet is sent. The simplest case is wheremessages are being sent faster than the wire speed of the network. Latency may also beadded before a packet is sent if there is contention for network bandwidth. (It is acommon misconception that switched networks eliminate such contention. Switches do help,but the contention point often becomes getting packets out of the switch rather than intoit.) See Section 4 for a possible cause of networkaccess control latency.

Transport-layer protocols like LBT-RM may also add latency if an application attemptsto send messages faster than the network can safely deliver them. The rate controls thatimpose this latency promise stable network operation in return. Indeed, most forms ofnetwork access control latency end up being beneficial by preventing unfair use orcongestive collapse of the network.

Network access control latency should happen infrequently on modern networks wheremessage generation rates are matched to the ability of the network to safely carry them.It can be highly variable and difficult to estimate because it may arise from the actionsof others on the network.

17.10. Serialization

Networks employ various techniques for serializing data so that it can be movedconveniently. Typically, a fixed-frequency clock coordinates the action of a sender andreceiver. One bit is often transmitted for each beat of the clock. Serialization latencyis due to the fact that a receiver cannot use a packet until its last bit hasarrived.

The common DS0 communications line operates at 56 Kbps and hence adds 214 ms ofserialization latency to a 1500-byte packet. A DS1 line ("T1") operates at 1.5 Mbps,adding 8 ms of latency. Even a 100 Mbps Ethernet adds 120 μs of latency on a 1500-bytepacket.

Serialization latency should be constant for a given clock rate. It should beconsistent across all messages.

Note: Serialization latency and the speed of light (discussed in the nextsection) can be easily visualized and contrasted using a Java applet. The linked web page uses the term "transmission delay"where we use the term "serialization latency." Similarly, it uses the term "propagationdelay" where we use "speed of light."

17.11. Speed of Light

186,000 miles per second--it's not just a good idea, it's the law!

Although often insignificant under one roof or even around a campus, latency due tothe speed of light can be a significant issue in WAN communication. Signals travelthrough copper wires and optical fibers at only about 60% of their speed in a vacuum.Hence the 3,000 km trip from Chicago to San Francisco takes about 15 ms while the 6,400km trip from Chicago to London takes about 33 ms.

As far as we know, the speed of light is a constant of the universe so we'd expect thelatency it adds over a fixed path to also be constant.

17.12. AddressResolution Protocol (ARP)

Address Resolution Protocol, or ARP, is invoked the first time a unicast message issent to an IP address.

ARP on the sending host sends a broadcast message to every host on the subnet, asking"Who has IP x.x.x.x?"
The host with IP address "x.x.x.x" responds with its Ethernet address in an ARPreply.
The sending host can now resolve the IP address to an Ethernet address, and sendmessages to it.

ARP will cache the Ethernet address for a specific period of time and reuse thatinformation for future message send operations. After the ARP cache timeout period, theaddress is typically dropped from the ARP cache, adding application latency to the nextmessage sent from that host to that IP address because of the overhead of the addressresolution process. A complicating factor is that different operating systems havedifferent defaults for this timeout, for example, Linux often uses 1 minute, whereasSolaris often uses 5 minutes.

Another potential source of latency is an application that is sending unicast messagesvery fast while ARP resolution is holding up the sending process. These messages must bebuffered until the ARP reply message is received, which causes latency, and may lead toloss, adding additional latency for the recovery of those messages.

Note that ARP latency is the total roundtrip cost of the originating host sending the"Who has..." message and waiting for the response. Users on 1Gbe networks may estimate aminimum of 80-100 usecs.

All of the above applies to unicast addressing only. With multicast addressing, ARP isnot an issue, because the network router takes care of managing the list of interestednetwork ports.

One way to eliminate ARP latency completely is to use a static ARP cache, with thehosts configured to know about each other. Of course, this is a very labor-intensivesolution, and maintaining such configurations can be onerous and error-prone.

Yet another solution is to configure the ARP cache timeout on all hosts on the networkto be much longer, say, 12 hours. This may also require devoting more memory to the ARPcache itself. If an initial heartbeat message is built into every app to absorb thelatency hit on a meaningless message before the start of the business day, there shouldbe no ARP-induced latency during business hours. However, one downside to this approachis if a host's IP address or MAC address were to change during this time, a manual ARPflush would be required.

18. Latency from Interrupt Coalescing

ManyInformaticacustomers usingUMSare concerned about latency. We have helped them troubleshootlatency problems and have sometimes found a significant cause to be interrupt coalescingin Gigabit Ethernet NIC hardware. Fortunately, the behavior of interrupt coalescing isconfigurable and can generally be adjusted to the particular needs of an application.

As mentioned in Section 17.5, interruptcoalescing represents a trade-off between latency and throughput. Coalescing interruptsalways adds latency to arriving messages, but the resulting efficiency gains may bedesirable where high throughput is desired over low latency.

The default for some NICs or drivers is an "adaptive" or "dynamic" interruptcoalescing setting that seems to significantly favor high throughput over low latency.The advice in this section is generally aimed toward changing the default to favor lowlatency, perhaps at the expense of high throughput.

The details of configuring interrupt coalescing behavior will vary depending on theoperating system and perhaps even the type of NIC in use. We have had specific experiencewith Linux using Intel and Broadcom NICs, and with Windows using Intel NICs.

18.1. Linux Interrupt Coalescing

On Linux, the conventional way to configure interrupt coalescing seems to be the ethtool command. However, some NIC drivers seem to requireconfiguration in other ways. In our experience, if ethtool -ceth0 did not work, then another method was available.

For example, we have seen that with some Intel NICs on Linux, interrupt coalescing iscontrolled through the modprobe method of configuring loadabledrivers. /etc/modprobe.conf might look like this:

options e1000 InterruptThrottleRate=8000

Some default configurations set InterruptThrottleRateto 1 which selects dynamic interrupt coalescing. We have seen significant reductions inlatency by changing this to a fixed value (e.g. 8,000) as suggested in Intel ApplicationNote AP-450. Using a value of 8,000 would limit receive latency to 1second/8000 = 125 μs worst case.

For NICs and drivers configured with ethtool, make sure theadaptive-rx parameter is off forthe lowest possible latency. Also check the settings of all of the rx-usecs and rx-framesparameters.

18.2. Linux "ethtool" Command Parameters

The Linux "ethtool -C" command provides a wide array of different types of parametersthat can be configured in various ways to set values related to interrupt coalescing.

Please note that even though "ethtool" provides support for these parameters, the NICdriver itself may not. Use the "man ethtool" page along with the NIC documentation toresearch the exact parameters available in more detail.

The tables below are organized by type of parameter: RX Parameters, TX Parameters, andOther Parameters.

RX Parameters

Parameter	Definition
`rx-usecs`	Maximum number of microseconds to delay an RX interrupt after receiving a packet. If0, only rx-max-frames is used. Do not set both rx-usecs and rx-max-frames to 0 as thiswould cause no RX interrupts to be generated.
`rx-usecs-low`	Same as `rx-usecs`, but used in concert with `pkt-rate-low` (see below).
`rx-usecs-high`	Same as `rx-usecs`, but used in concert with `pkt-rate-high` (see below).
`rx-usecs-irq`	Maximum number of microseconds to delay an RX interrupt after receiving a packet whilean IRQ is also being serviced by the host. Some NIC drivers may not support thisfeature.
`rx-max-frames`	Maximum number of packets to delay an RX interrupt after receiving a packet. If 0,only `rx-usecs` is used. Do not set both `rx-usecs` and `rx-max-frames` to 0 as thiswould cause no RX interrupts to be generated.
`rx-max-frames-low`	Same as `rx-max-frames`, but used in concert with `pkt-rate-low` (see below).
`rx-max-frames-high`	Same as `rx-max-frames`, but used in concert with `pkt-rate-high` (see below).
`rx-max-frames-irq`	Maximum number of packets to delay an RX interrupt after receiving a packet while anIRQ is also being serviced by the host. Some NIC drivers may not support thisfeature.

TX Parameters

Parameter	Definition
`tx-usecs`	Maximum number of microseconds to delay a TX interrupt after sending a packet. If 0,only tx-max-frames is used. Do not set both tx-usecs and tx-max-frames to 0 as this wouldcause no TX interrupts to be generated.
`tx-usecs-low`	Same as `tx-usecs`, but used in concert with `pkt-rate-low` (see below).
`tx-usecs-high`	Same as `tx-usecs`, but used in concert with `pkt-rate-high` (see below).
`tx-usecs-irq`	Maximum number of microseconds to delay a TX interrupt after sending a packet while anIRQ is also being serviced by the host. Some NICs may not support this feature.
`tx-max-frames`	Maximum number of packets to delay a TX interrupt after sending a packet. If 0, onlytx-usecs is used. Do not set both tx-usecs and tx-max-frames to 0 as this would cause noTX interrupts to be generated.
`tx-max-frames-low`	Same as `tx-max-frames`, but used in concert with `pkt-rate-low` (see below).
`tx-max-frames-high`	Same as `tx-max-frames`, but used in concert with `pkt-rate-high` (see below).
`tx-max-frames-irq`	Maximum number of packets to delay a TX interrupt after sending a packet while an IRQis also being serviced by the host. Some NICs may not support this feature.

Other Parameters

Parameter	Definition
`adaptive-rx`	An algorithm to improve rx latency under low packet rates and improve throughput underhigh packet rates. Some NIC drivers do not support this feature.
`adaptive-tx`	An algorithm to improve tx latency under low packet rates and improve throughput underhigh packet rates. Some NIC drivers do not support this feature.
`pkt-rate-low`	Rate of packets per second below which a different set of `-usecs` and `-max-frames` parameters areused: `rx-usecs-low` `rx-max-frames-low` `tx-usecs-low` `tx-max-frames-low` Above this rate, the normal `-usecs` and `-max-frames` parameters are used.
`pkt-rate-high`	Rate of packets per second above which a different set of `-usecs` and `-max-frames` parameters areused: `rx-usecs-high` `rx-max-frames-high` `tx-usecs-high` `tx-max-frames-high` Below this rate, the normal `-usecs` and `-max-frames` parameters are used.
`sample-interval`	Number of seconds to use as packet sampling rate for adaptive coalescing. Must benon-zero.

18.3. Windows Interrupt Coalescing

We haven't had as much experience configuring interrupt coalescing on Windows as wehave had on Linux. For a given type of NIC, we'd expect the relevant parameters name tobe similar to those given for Linux above.

18.4. Windows NIC Loss Avoidance andDetection

We know of a few ways to avoid and detect loss at the NIC level, depending on thebrand of NIC on your Windows machine. Below, we cover Windows and Intel NICs and Windows and Broadcom NICs.

18.4.1.Windows and Intel NICs

If you have Intel NICs, you might have success doing what one customer did: avoid UDPloss by increasing the Receive Descriptors. It defaultedto 256 and this customer's NIC model allowed a maximum of 2048. (We recommend that youuse the maximum Receive Descriptors value for your NICmodel whatever it is.) See Figure 3.

Figure 3. Windows Intel NIC Receive Descriptors Parameter Setting

Increasing the Receive Descriptors parameter doesn'tchange interrupt coalescing settings. It simply increases the size of the ring buffer the NICuses for receiving. This allows for more interrupt servicing latency before loss.

Note: As shown in the screen shot, each ReceiveDescriptor requires 2 KB of memory. Our customer's increase to 2048 means thateach Intel NIC would allocate 2 MB of physical memory for its receive ring buffer.

18.4.2. Windows and Broadcom NICs

With Broadcom NICs, we recommend that you download and install the BACS tool to detectloss, and possibly to adjust tuning options on the NIC to help address it.

First, download BACS (Broadcom Advanced Control Suites) from the install CD that camewith your NIC, or your hardware vendor. You might find this Broadcom NICs FAQ page useful.

After downloading and installing it, start BACS. Then run a workload on your machinethat will drive the network traffic you are interested in diagnosing. Then clickStart-->Control Panel, and look in the list for Broadcom Control Suite. Double clickon it to open the Broadcom Advanced Control Suite dialog.

In this dialog, first check the list of Network Interfaces and make sure the BroadcomNIC is highlighted. Then, over on the upper right, click the Statistics tab, and scrolldown the list of statistics to a statistic called "Out of Recv. Buffer". This value isthe number of packets dropped due to a shortage of buffer space for the NIC. If the valueis nonzero, you may wish to increase the size of your buffers.

19. Latency Measurement Overview

Measuing latency in messaging systems likeInformaticaUMS requires using microseconds (1/1,000,000 second) becausesub-millisecond (1/1,000 second) results are common. In fact, today, even the nanosecond threshold is(1/1,000,000,000 second) often tested.Therefore, this raises measurement issues not often found with leadingmessaging systems. Following sections discuss some of these issues and provide resultsfrom some of our latency measurements. See Section 17 forbackground on the latency sources we've noted during testing in our Ultra Messaging lab.

One of the largest sources of latency we saw in our tests was network loss repair.Time is required for the transport layer to first detect the loss, requestretransmission, and finally for the repair to arrive. See Section 17.3 for a general discussion ofretransmissions and latency. See Section 2 for a discussion ofsuch latency in TCP. For reliable multicast transport protocols such as Informatica's LBT-RM,minimum loss repair latency comes at a cost of efficiency. Systems using reliablemulticast protocols for their scalability often have to trade off efficiency to get thelowest possible loss repair latency.

No matter what transport protocol is in use, it's important to check for loss at thefirst hint of latency. See the end of Section2.2 for some advice on diagnosing TCP loss. See Section 15.3 for advice on how to avoid losswhen using multicast.

20. Measuring CPU Scheduling Latency

Once latency due to repair loss has been removed from a messaging system, the nextlargest source is often CPU scheduling latency. This happens when a process is ready torun, but is unable to get a CPU on which to run. There is usually both a fixed andvariable component in CPU scheduling latency. The remainder of this section describestests we've run to better understand latency in messaging systems.

We ran a series of tests in our Ultra Messaging labusing Solaris 2.8 on a pair of SPARC machines. One machine ran 5 UMS sources, each producing a message stream of 2,000-bytemessages at the rate of 10 messages per second. The other machine ran 10 UMS receivers, each consuming all the messages from each of the 5sources. Hence the aggregate payload rate across all sources was 100,000 BPS and theaggregate payload rate across all receivers was 1,000,000 BPS. Our LBT-RM reliablemulticast protocol was used so that the network bandwidth between machines was only100,000 BPS plus protocol overhead.

We chose a relatively low message rate with a modest size payload for these tests toemphasize latency over throughput. Note that the 2,000-byte message payload together withthe 1500-byte MTU of Ethernet force UMS to break every messageinto two Ethernet packets. This magnifies the effect of network loss on latency becausethe loss of either of the two packets would delay delivery.

Our first test run was meant to establish a baseline for further tests. We wereparticularly interested in the maximum latency measured over a long-running (20 minute)test. We measured a maximum latency of about 86 ms even though the average was only 1.8ms. The plot below shows the latency measured for each of the 600,000 messages receivedfor this test.

Figure 4. Baseline Latency Measurement

Only 1 in 1,000 messages had latency over 5.6 ms., however the above plot makes itapparent that many messages had much higher latencies.

Working on the assumption that CPU scheduling latency might be a significant factor inthe large latency variance observed in the baseline, we tried increasing the CPUscheduling priority of the UMS processes. We used the commandprefix nice --19 to elevate the priority of all UMS source and receiver processes to the maximum allowable underthe Solaris "time sharing" scheduling class. Surprisingly, this had little effect onlatency. The maximum measured latency remained unchanged. The plot below shows this.

Figure 5. Latency with Maximum Time Sharing Priority

The lack of a significant change lead us to conclude that if CPU scheduling latencywas the cause of the large maximum latency, then it wasn't due to CPU contention withother user time sharing processes on the machine.

This led us to try a CPU scheduling class offered by Solaris called "real time." Weused the command prefix priocntl -c RT -e to request real timescheduling for all UMS processes. This had a dramatic effecton the maximum latency, reducing it over 16 fold to just 5.2 ms. The plot below showsthis.

Figure 6. Latency with Real Time Priority

The seemingly random incidence of high-latency messages completely disappeared when UMS ran with real time priority. We conclude that CPU schedulinglatency was the primary cause of the large maximum latency we saw in earlier test.However, it seems that the source of CPU contention was something within the Solariskernel since maximum user CPU priority produced no reduction in latency.

21. Measuring CPU Contention Latency

A special form of CPU scheduling latency called CPU contention latency happens whenthere are more runnable processes than CPUs. In effect, this separates the fixed portionof CPU scheduling latency from the variable portion. In our final round of tests, wewanted to investigate the effect of CPU contention on latency. In the preceding tests, 10receiver processes had to contend for time on 4 physical CPUs. Note that a singlemulticast packet wakes up all 10 receiver process at the same time. Assuming an otherwiseidle machine, 4 processes get scheduled onto CPUs immediately while 6 wait.

For this test, we ran only 3 receiver processes on the same 4-CPU machine. The resultsare shown in the lower line on the plot below. The upper line shows the usual 10 receivercase from above. But note that the vertical axis covers only about 1/7 the latency rangeof the preceding plots. The range of the vertical scale was reduced to show moredetail.

Figure 7. Effect of CPU Contention on Scheduling Latency

The lower band is much thinner than the upper one since there were fewer receiversthan CPUs and hence no CPU contention. The difference in thickness is a good indicationof how long the additional receivers in the earlier tests had to wait for a CPU to runthem (the CPU contention latency). From inspection of the plot above, we'd estimate theCPU contention latency for the 10 receiver case to be about 1.5 ms.

With the magnified vertical scale of the above plot, the clock drift over the durationof the test is also apparent. This makes it easy to visualize the operation of ntpd in trying to keep the system clocks synchronized. Note that theapproximately 7 ms. of clock drift during the test run significantly exceeds the measuredlatency for almost all of the test. Also note that the clock skew was negative for theentire 20-minute duration of the test.

你可能感兴趣的:(网络资源)

人应该追求多少钱？还是追求自由，陪伴，互相依存？阿尚青子自由写作人
人应该追求多少钱？还是追求自由，陪伴，互相依存？（原问题）回答这样的问题应该有难度，因为此问题问的几个方面好像不属于同一个价值平台，而同一个价值平台的和钱几乎等同的概念又是什么呢？好像又没有什么标准答案，认同不同，问题不同，权当一个不妥帖的解释罢了。首先回答，人应该追求多少钱？看你到底对自己生活的要求和精神要求有多高了，精神追求也是需要定量金钱为支撑的，比如即使看电影，你也得花钱，就网络资源来讲你
流量牵引技术与传统防火墙的区别 666IDCaaa ddos
在网络安全领域，流量牵引技术和传统防火墙都起着重要的作用，但它们在很多方面存在着明显的区别。一、工作原理不同传统防火墙主要是通过设置访问控制规则来过滤网络流量。它基于预先设定的策略，对进入和离开网络的数据包进行检查，根据源地址、目的地址、端口号等信息决定是否允许数据包通过。例如，企业可以设置防火墙规则，只允许特定IP地址的设备访问内部网络资源，或者禁止某些端口的流量进入，以防止潜在的攻击。而流量牵
KVM虚拟化平台大西瓜不爱告诉你姓名虚拟化 KVM虚拟化
前言1、云计算的定义云计算的定义用户可以在任何时间、地点通过网络获取所需要的计算资源、网络资源、存储资源并且按量计费、弹性伸缩云计算就是一个大的租赁渠首云计算这个大的资源池中的各种资源(以租赁的形式)云计算所汇聚的这部分资源(通过云平台的方式汇聚这些资源)而云平台比如:阿里云、华为云这些云平台使用到的底层平台技术为OpenStack而OpenStack利用了什么技术将资源可以划分给不同的用户使用呢
最新 Kubernetes 常见面试题汇总闻道☞ k8s kubernetes 容器云原生
1.什么是Kubernetes？Kubernetes是一个用于自动部署、扩展和管理容器化应用程序的开源平台。它提供了一个可扩展的、高可用的集群，并包含了自动化部署、负载平衡、存储管理、自我修复、自动扩容等功能。2.Kubernetes中的Pod是什么？Pod是Kubernetes中最小的部署单元，它包含一个或多个紧密耦合的容器和共享的存储/网络资源。Pod提供了一种抽象层，使得容器可以在同一个主机
WebKit 简介及工作流程深度解析不知名靓仔 webkit chrome safari edge
WebKit是一个开源的浏览器引擎，最初由苹果公司为Safari浏览器开发，后来被多个浏览器和应用采纳，如早期的Chrome和众多移动平台上的浏览器。WebKit不仅仅是一个渲染引擎，它集成了HTML、CSS、JavaScript的解析与渲染能力，以及网络资源加载、图形处理、排版布局等功能，为用户提供丰富的网页浏览体验。本文将深入探讨WebKit的核心组件以及其复杂的工作流程。WebKit核心架构
防火墙部署基本方法坚持可信信息安全 php 开发语言
防火墙部署基本方法防火墙部署是确保网络安全的重要步骤，通过合理的部署和配置防火墙，可以有效地保护网络资源，防止未经授权的访问和网络攻击。以下是防火墙部署的基本方法，包括部署步骤、配置建议和常见的部署拓扑。一、部署步骤1.需求分析步骤：评估网络结构：了解当前网络拓扑结构、流量模式和主要的网络设备。明确安全需求：确定需要保护的资源、潜在威胁以及安全目标。选择防火墙类型：根据需求选择合适的防火墙类型（硬
Flask-Caching Botiway FlaskWeb python flask 后端 linux web3
Flask-Caching是Flask的一个扩展，它为Flask应用提供了缓存支持。缓存是一种优化技术，可以存储那些费时且不经常改变的运算结果或页面内容，从而加快应用的响应速度，减少对数据库或网络资源的重复访问。Flask-Caching的主要特点多种缓存后端支持：Flask-Caching支持多种缓存后端，包括内存缓存（如SimpleCache）、文件系统缓存、Memcached缓存、Redis
浅聊kubernetes RBAC 企鹅侠客云原生 kubernetes 容器云原生 RBAC
RBAC基于角色（Role）的访问控制（RBAC）是一种基于组织中用户的角色来调节控制对计算机或网络资源的访问的方法。RBAC鉴权机制使用rbac.authorization.k8s.ioAPI组来驱动鉴权决定，允许你通过KubernetesAPI动态配置策略要启用RBAC，在启动API服务器时将--authorization-mode参数设置为一个逗号分隔的列表并确保其中包含RBAC。Role和
tcp 流量控制秋夫人计算机网络 tcp/ip 网络服务器计算机网络
TCP流量控制是TCP/IP协议中用于控制发送方和接收方之间数据传输速率的一种机制，以防止网络拥塞和确保网络资源的有效利用。流量控制主要通过调整TCP窗口大小来实现，确保发送方不会发送超出接收方处理能力的数据量。以下是TCP流量控制的关键概念和工作原理：关键概念TCP窗口：TCP窗口是指接收方能够接收但尚未确认的数据量，它是接收方根据自身缓冲区的空闲容量动态通告给发送方的。接收窗口（rwnd）：接
无法访问请与这台计算机,计算机无法访问,您可能没有权限使用网络资源.请与这台服务器的管理员联系以查... 陆盖盖无法访问请与这台计算机
计算机无法访问,您可能没有权限使用网络资源.请与这台服务器的管理员联系以查明您(1)安装NWlinkIPX/SPX/NetBIOSCompatibleTransportProtocol协议。(2)开启guest账号：右击我的电脑\管理\用户有个guest，双击之去掉“账户已停用”前面的勾。(3)右击我的电脑\属性\计算机名，查看该选项卡中出现的局域网工作组名称(4)使用winxp防火墙的例外：wi
zhuan 漫谈C语言及如何学习C语言 weixin_34112900
转载云风最近写了一篇博客《C语言的前世今生》。作为长期使用C语言开发网络游戏服务器的程序员，云风是有理由写这样一篇文字，不过还是感觉谈的不够深入，C语言在业界使用的现状没有怎么描写，有些意犹未尽。在这里想比较系统的谈谈个人对C语言学习方式方法的理解。分别按照书籍、实验环境搭建、网络资源来分别介绍，希望能写的比较完整全面一些，给想学习C语言的朋友一个有价值的参考。为什么要学习C语言？为什么要学习、使
深入解析HarmonyOS Image组件的使用与优化神码兄弟 HarmonyOS学习开发语言 harmonyos
在现代移动应用开发中，图像处理是一个至关重要的部分。HarmonyOS提供了功能强大的图像组件，允许开发者从多种来源显示图像，如本地资源、网络资源、资源文件、媒体库和Base64图像编码。本篇博客将深入探讨如何接地使用图像组件，并高效涵盖一些实用的进阶技巧，如图像的缩放、插值、重复、渲染模式、解码大小以及滤镜效果。一、图像组件的基本使用1.从本地资源加载图像ImagemyImage=newImag
江苏BGP大带宽服务器所适用的业务有哪些？ wanhengwangluo 服务器运维机房
随着网络业务的快速发展，企业对于服务器的性能与网络质量有着很高的要求，而江苏BGP大带宽服务器则有着优质的网络资源和高性能的服务器硬件配置，是大部分企业的理想选择，本文就来介绍一下江苏BGP大带宽服务器都适用于哪些业务。江苏BGP大带宽服务器能够适用于在线教育平台当中，可以为平台提供稳定可靠的数据传输和处理大量的音频与视频，支持多个用户同时在线学习，确保师生之间的互动顺畅无阻，并且可以保证教育课程
什么是网络交换，常用的网络交换设备有哪些？罗兰Yolanda 网络交换智能路由器
网络交换是一种通过特定设备（如交换机）将数据从源节点传输到目标节点的技术，它通过数据包的转发和交换实现了数据在网络中的传输。为了进一步了解网络交换的具体类型、工作原理及其应用场景，可以从以下几个方面进行详细分析：按照层次划分的网络交换物理层交换：物理层交换主要应用于电话网，通过电路交换提供专用网络资源，适用于需要保证服务质量的通信场景。链路层交换：链路层交换（二层交换），主要对MAC地址进行变更，
ICANN成立20周年年会，全球互联网的变革与未来！ 62474c0b60b9
10月20-25日，ICANN年会在巴塞罗那召开，恰逢ICANN成立20周年，使得本次大会更具有内涵。随着斯蒂芬·克罗克（SteveCrocker）董事会主席去年到期，创造互联网的第一代先驱基本退出ICANN的舞台。ICANN年会2016年ICANN正式脱离美国政府的直接控制，如今，这个掌控着人类网络时代最关键网络资源的机构，已经进入了一个全新的时代。究竟是完成了成人礼，还是更多是危机和坎坷，目前
vos3000系统，如何防护ddos数据攻击？ wx_jiuyun383838 tel18554512818 vos3000 web安全安全架构系统安全
大家可能都有vos3000服务器被ddos攻击的烦恼，不仅浪费时间还会影响业务。那么什么是ddos攻击呢？分布式拒绝服务（‌‌DDoS）‌攻击是一种通过大量请求拥塞目标服务器的网络资源，‌从而阻止合法用户访问的攻击方式。‌为了有效防御DDoS攻击，‌可以采取以下策略：‌服务器高防ip，这个成本太高不考虑。尽早发现攻击：‌及时觉察是防御的第一步。‌例如，‌当网站流量突然激增或收到大量垃圾邮件时，‌可
python怎么获取免费代理IP 傻啦嘿哟关于python那些事儿关于代理IP那些事儿服务器网络运维
什么是免费代理IP免费代理IP是指可以免费使用的代理服务器的IP地址。代理服务器充当客户端和目标服务器之间的中间人，通过转发请求和响应来实现网络数据的传输。使用代理IP可以隐藏真实的客户端IP地址，实现匿名访问网络资源。免费代理IP通常由个人或组织提供，供用户免费使用。这些免费代理IP可能是公开的、匿名的或透明的，提供不同级别的匿名性和安全性。通过使用免费代理IP，用户可以绕过一些网络限制、访问被
设备运维故障排查与修复技巧一去二三里... 配置运维运维
运维中最常见的40个故障问题及其解决方法：1.网络不通问题：无法访问网络资源。解决方法：检查物理线路、交换机端口、网卡驱动和配置，使用ping、traceroute等工具定位问题。2.网络速度慢问题：访问网络资源速度慢。解决方法：分析带宽使用情况，检查是否存在广播风暴，调整端口速率协商设置。3.设备掉线频繁问题：网络设备频繁掉线。解决方法：检查电源稳定性、散热情况、链路质量及设备固件版本，确保设备
linux之DNS配置运维不屈的一生 linux 运维服务器
一、DNS概述1.1DNS简介DNS（DomainNameSystem）是互联网中用于将域名转换为对应IP地址的分布式命名系统。IP地址不便于记忆，DNS使用户可以通过易记的域名快速访问各种网络资源。域名结构:域名采用层次结构表示，从右向左依次表示更高的级别。以www.example.com为例.com是一级域名example是二级域名www是主机名主机名用于标识特定的设备或计算机，而子域名用于对
云计算虚拟化的主要类型有哪些？有哪些优势？ weixin_54503231 云计算
云计算虚拟化是一种资源管理技术，其核心在于将计算机的各种实体资源（如CPU、内存、磁盘空间、网络适配器等）进行抽象、转换，并呈现出来供用户分割、组合成一个或多个逻辑上的资源。主要类型：服务器虚拟化：将物理服务器虚拟化成多个独立的虚拟服务器，每个虚拟服务器都可以运行自己的操作系统和应用程序。存储虚拟化：将存储资源抽象成独立的虚拟存储单元，以便更高效地管理和利用存储资源。网络虚拟化：将网络资源（如交换
为什么网站域名大多以www开头？电子邮件地址中的@是怎么来的？爱沙沙
大家可能注意到大多数网站域名都是以www作为开头，这是为什么呢？图片发自App其实它是万维网。worldwideweb的缩写。十年前，英国科学家博纳斯.李发明了万维网的方式，在互联网中引入了直观的图形界面，取代了抽象难懂的命令格式，从而使“上网”不再是专业人员的特权。互联网因此得以迅速普及，数以亿计的人能够方便的使用浩瀚的网络资源。网络普及后，万维网吸引了很多网站的加入，所以现在大部分网站的域名都
QML | 在QML中导入JavaScript资源、导入JavaScript资源、包含一个JavaScript 资源 Qt历险记 Qt 高级开发工程师 javascript qt qml android-studio
01在QML中导入JavaScript资源JavaScript资源可以被QML文档和其他JavaScript通过相对或者绝对路径进行导入。如果使用相对路径，位置解析需要相对于包含import语句的QML文档或JavaScript资源的位置。如果JavaScript需要从网络资源中进行获取，组件的status属性会被设置为Loading，直到该脚本被下载完成。JavaScript资源也可以导入QML
阿里云游戏访问与下载加速解决方案九河云阿里云游戏云计算
随着全球化的发展，越来越多的企业开始将业务扩展到海外市场。然而，海外市场的挑战也是显而易见的。例如在游戏企业进军海外市场的过程中，网络延迟是一个常见的问题。由于地理位置的限制，海外用户登录访问游戏和游玩时，往往会面临较长的响应时间和加载速度缓慢的问题。此外还需要保证服务的连续性。为了解决这些问题，九河云基于多年从云经验为游戏企业提供了阿里云出海解决方案。方案概述：该方案利用阿里云覆盖全球的网络资源
什么是IP白名单？为什么要设置IP白名单？小熊HTTP 网络 tcp/ip 网络协议
在互联网的世界里，IP地址是每个设备与网络进行通信的关键标识。然而，并不是所有的IP地址都可以无限制地访问所有网络资源。为了保障网络安全和资源管理，很多网站和服务会设置IP白名单。本文将详细介绍IP白名单的定义、作用以及为什么要设置IP白名单。1、IP白名单的定义IP白名单是指将一些特定的IP地址列入允许访问的名单中，这些IP地址被视为可信的来源。当来自这些IP地址的请求访问某个网站或服务时，系统
HTML案例/form标签/frameset标签-（html补充）小山居
HTML案例/form标签/frameset标签一。简历模板image.png简历作业简历作业基本情况姓名性别民族出生年月政治面貌健康籍贯学历电子信箱联系电话专业毕业院校求职意向教育情况时间院校专业学历培训经历时间培训机构课程证书二。内嵌标签学习iframe1.src:要显示的网页资源路径可以是本地(相对路径)也可以是网络资源(URL)注意:默认当前页面打开及加载src指向的资源2.width:设
网站的保姆级攻略 athena1999270 服务器
网站的保姆级攻略什么是域名？域名是互联网上用于识别和定位计算机和网络服务的字符串。它提供了一个便于人们记忆和使用的名称，用来代替复杂的IP地址，可用于从客户端浏览器访问网站。简单来说，域名是用户在浏览器窗口中键入以访问特定网站的字符串。例如，Google的域名是“google.com”什么是网址？网址是指用于标识和访问特定网络资源的字符序列，通常以"http://"或"https://"开头，后跟
服务器相关概念1 爱健身的小刘同学浏览器相关服务器
客户端网络节点中负责消耗资源的电脑服务器负责对外提供网络资源的电脑服务器和普通电脑区别服务器有web服务器软件，IIS，Apache，phpstudyIP地址每台计算机唯一地址；格式（a.b.c.d）点分时进制，为（0-255）十进制整数，如（192.168.1.1）查看：终端运行pingwww.baidu.com查看百度服务器开发可以用本机服务器ip（127.0.0.1）测试域名和域名服务器字符
浅谈web性能测试 .咖啡加剁椒软件测试前端功能测试软件测试自动化测试程序人生职场和发展
什么是性能测试？web性能应该注意些什么？性能测试，简而言之就是模仿用户对一个系统进行大批量的操作，得出系统各项性能指标和性能瓶颈，并从中发现存在的问题，通过多方协助调优的过程。而web端的性能测试应该注意的指标有：用户操作的响应时间、系统的吞吐量（TPS）、系统的硬件资源情况（CPU、硬盘、磁盘）、网络资源占用情况等web性能测试之HTTP请求关于性能测试中，HTTP请求类的性能指标都需要我们去
IP地址受限制怎么办？网络IP遭到限制常见的解决方案 ip动态代理网络爬虫服务器配置
网络IP受限制的问题对于许多网络用户来说可能是一个常见的问题。当你在使用网络时遇到IP地址受限的问题时，不要担心，本文将为你提供一些常见的解决方案。首先，让我们了解一下什么是IP地址受限。IP地址是网络中计算机的唯一标识符，它用于与其他计算机通信。当你的IP地址被限制时，这意味着你无法访问某些网络资源，或者你的网络连接被限制。这通常发生在以下情况下：公共场所网络：在一些公共场所，如咖啡馆、图书馆等
微信圈朋友圈代理怎么做 weixinguanggao 微信信息流微信
在腾讯社交将来优良发展趋势的整体背景之下，尽管微信朋友圈代理商早已健全，但是需求与销售市场。现阶段，广告主对微信朋友圈广告的喜好和信任取决于微信朋友圈广告代理商依然存在销售市场。如果您需要，欢迎您搜索公纵号：全网广告投放推广刘老师，期待与您合作。微信朋友圈代理招聘公司正在全国范围招骋微信发朋友圈媒体代理。代理商网络资源目标一般是本地中小型企业，例如本地的家装公司.4S店.餐馆.教育培训机构.酒店餐
js动画html标签（持续更新中） 843977358 html js 动画 media opacity
1.jQuery 效果 - animate() 方法改变 "div" 元素的高度： $(".btn1").click(function(){ $("#box").animate({height:"300px
springMVC学习笔记 caoyong springMVC
1、搭建开发环境 a>、添加jar文件，在ioc所需jar包的基础上添加spring-web.jar,spring-webmvc.jar b>、在web.xml中配置前端控制器 <servlet> &nbs
POI中设置Excel单元格格式 107x poi style 列宽合并单元格自动换行
引用：http://apps.hi.baidu.com/share/detail/17249059 POI中可能会用到一些需要设置EXCEL单元格格式的操作小结：先获取工作薄对象: HSSFWorkbook wb = new HSSFWorkbook(); HSSFSheet sheet = wb.createSheet(); HSSFCellStyle setBorder = wb.
jquery 获取A href 触发js方法的this参数无效的情况一炮送你回车库 jquery
html如下： <td class=\"bord-r-n bord-l-n c-333\"> <a class=\"table-icon edit\" onclick=\"editTrValues(this);\">修改</a> </td>" j
md5 3213213333332132 MD5
import java.security.MessageDigest; import java.security.NoSuchAlgorithmException; public class MDFive { public static void main(String[] args) { String md5Str = "cq
完全卸载干净Oracle11g sophia天雪 orale数据库卸载干净清理注册表
完全卸载干净Oracle11g A、存在OUI卸载工具的情况下：第一步：停用所有Oracle相关的已启动的服务；第二步：找到OUI卸载工具：在“开始”菜单中找到“oracle_OraDb11g_home”文件夹中 &
apache 的access.log 日志文件太大如何解决 darkranger apache
CustomLog logs/access.log common 此写法导致日志数据一致自增变大。直接注释上面的语法 #CustomLog logs/access.log common 增加： CustomLog "|bin/rotatelogs.exe -l logs/access-%Y-%m-d.log
Hadoop单机模式环境搭建关键步骤 aijuans 分布式
Hadoop环境需要sshd服务一直开启，故，在服务器上需要按照ssh服务，以Ubuntu Linux为例，按照ssh服务如下： sudo apt-get install ssh sudo apt-get install rsync 编辑HADOOP_HOME/conf/hadoop-env.sh文件，将JAVA_HOME设置为Java
PL/SQL DEVELOPER 使用的一些技巧 atongyeye java sql
1 记住密码这是个有争议的功能，因为记住密码会给带来数据安全的问题。但假如是开发用的库，密码甚至可以和用户名相同，每次输入密码实在没什么意义，可以考虑让PLSQL Developer记住密码。位置：Tools菜单－－Preferences－－Oracle－－Logon HIstory－－Store with password 2 特殊Copy 在SQL Window
PHP：在对象上动态添加一个新的方法 bardo 方法动态添加闭包
有关在一个对象上动态添加方法，如果你来自Ruby语言或您熟悉这门语言，你已经知道它是什么...... Ruby提供给你一种方式来获得一个instancied对象，并给这个对象添加一个额外的方法。好！不说Ruby了，让我们来谈谈PHP PHP未提供一个“标准的方式”做这样的事情，这也是没有核心的一部分... 但无论如何，它并没有说我们不能做这样
ThreadLocal与线程安全 bijian1013 java java多线程 threadLocal
首先来看一下线程安全问题产生的两个前提条件： 1.数据共享，多个线程访问同样的数据。 2.共享数据是可变的，多个线程对访问的共享数据作出了修改。实例：定义一个共享数据： public static int a = 0;
Tomcat 架包冲突解决征客丶 tomcat Web
环境： Tomcat 7.0.6 win7 x64 错误表象：【我的冲突的架包是：catalina.jar 与 tomcat-catalina-7.0.61.jar 冲突，不知道其他架包冲突时是不是也报这个错误】严重: End event threw exception java.lang.NoSuchMethodException: org.apache.catalina.dep
【Scala三】分析Spark源代码总结的Scala语法一 bit1129 scala
Scala语法 1. classOf运算符 Scala中的classOf[T]是一个class对象，等价于Java的T.class,比如classOf[TextInputFormat]等价于TextInputFormat.class 2. 方法默认值 defaultMinPartitions就是一个默认值，类似C++的方法默认值
java 线程池管理机制 BlueSkator java线程池管理机制
编辑 Add Tools jdk线程池一、引言第一：降低资源消耗。通过重复利用已创建的线程降低线程创建和销毁造成的消耗。第二：提高响应速度。当任务到达时，任务可以不需要等到线程创建就能立即执行。第三：提高线程的可管理性。线程是稀缺资源，如果无限制的创建，不仅会消耗系统资源，还会降低系统的稳定性，使用线程池可以进行统一的分配，调优和监控。
关于hql中使用本地sql函数的问题（问-答） BreakingBad HQL 存储函数
转自于：http://www.iteye.com/problems/23775 问：我在开发过程中，使用hql进行查询（mysql5）使用到了mysql自带的函数find_in_set()这个函数作为匹配字符串的来讲效率非常好，但是我直接把它写在hql语句里面（from ForumMemberInfo fm,ForumArea fa where find_in_set(fm.userId,f
读《研磨设计模式》-代码笔记-迭代器模式-Iterator bylijinnan java 设计模式
声明：本文只为方便我个人查阅和理解，详细的分析以及源代码请移步原作者的博客http://chjavach.iteye.com/ import java.util.Arrays; import java.util.List; /** * Iterator模式提供一种方法顺序访问一个聚合对象中各个元素，而又不暴露该对象内部表示 * * 个人觉得，为了不暴露该
常用SQL chenjunt3 oracle sql C++c C#
--NC建库 CREATE TABLESPACE NNC_DATA01 DATAFILE 'E:\oracle\product\10.2.0\oradata\orcl\nnc_data01.dbf' SIZE 500M AUTOEXTEND ON NEXT 50M EXTENT MANAGEMENT LOCAL UNIFORM SIZE 256K ; CREATE TABLESPA
数学是科学技术的语言 comsci 工作活动领域模型
从小学到大学都在学习数学，从小学开始了解数字的概念和背诵九九表到大学学习复变函数和离散数学，看起来好像掌握了这些数学知识，但是在工作中却很少真正用到这些知识，为什么？最近在研究一种开源软件-CARROT2的源代码的时候，又一次感觉到数学在计算机技术中的不可动摇的基础作用，CARROT2是一种用于自动语言分类（聚类）的工具性软件，用JAVA语言编写，它
Linux系统手动安装rzsz 软件包 daizj linux sz rz
1、下载软件 rzsz-3.34.tar.gz。登录linux，用命令 wget http://freeware.sgi.com/source/rzsz/rzsz-3.48.tar.gz下载。 2、解压 tar zxvf rzsz-3.34.tar.gz 3、安装 cd rzsz-3.34 ; make posix 。注意：这个软件安装与常规的GNU软件不
读源码之:ArrayBlockingQueue dieslrae java
ArrayBlockingQueue是concurrent包提供的一个线程安全的队列,由一个数组来保存队列元素.通过 takeIndex和 putIndex来分别记录出队列和入队列的下标,以保证在出队列时不进行元素移动. //在出队列或者入队列的时候对takeIndex或者putIndex进行累加,如果已经到了数组末尾就又从0开始,保证数
C语言学习九枚举的定义和应用 dcj3sjt126com c
枚举的定义 # include <stdio.h> enum WeekDay { MonDay, TuesDay, WednesDay, ThursDay, FriDay, SaturDay, SunDay }; int main(void) { //int day; //day定义成int类型不合适 enum WeekDay day = Wedne
Vagrant 三种网络配置详解 dcj3sjt126com vagrant
Forwarded port Private network Public network Vagrant 中一共有三种网络配置，下面我们将会详解三种网络配置各自优缺点。端口映射(Forwarded port)，顾名思义是指把宿主计算机的端口映射到虚拟机的某一个端口上，访问宿主计算机端口时，请求实际是被转发到虚拟机上指定端口的。Vagrantfile中设定语法为： c
16.性能优化-完结 frank1234 性能优化
性能调优是一个宏大的工程，需要从宏观架构(比如拆分，冗余，读写分离，集群，缓存等)，软件设计（比如多线程并行化，选择合适的数据结构），数据库设计层面（合理的表设计，汇总表，索引，分区，拆分，冗余等）以及微观（软件的配置，SQL语句的编写，操作系统配置等）根据软件的应用场景做综合的考虑和权衡，并经验实际测试验证才能达到最优。性能水很深，笔者经验尚浅，赶脚也就了解了点皮毛而已，我觉得
Word Search hcx2013 search
Given a 2D board and a word, find if the word exists in the grid. The word can be constructed from letters of sequentially adjacent cell, where "adjacent" cells are those horizontally or ve
Spring4新特性——Web开发的增强 jinnianshilongnian spring spring mvc spring4
Spring4新特性——泛型限定式依赖注入 Spring4新特性——核心容器的其他改进 Spring4新特性——Web开发的增强 Spring4新特性——集成Bean Validation 1.1(JSR-349)到SpringMVC Spring4新特性——Groovy Bean定义DSL Spring4新特性——更好的Java泛型操作API Spring4新
CentOS安装配置tengine并设置开机启动 liuxingguome centos
yum install gcc-c++ yum install pcre pcre-devel yum install zlib zlib-devel yum install openssl openssl-devel Ubuntu上可以这样安装 sudo aptitude install libdmalloc-dev libcurl4-opens
第14章工具函数（上） onestopweb 函数
index.html <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/
Xelsius 2008 and SAP BW at a glance blueoxygen BO Xelsius
Xelsius提供了丰富多样的数据连接方式，其中为SAP BW专属提供的是BICS。那么Xelsius的各种连接的优缺点比较以及Xelsius是如何直接连接到BEx Query的呢？以下Wiki文章应该提供了全面的概览。 http://wiki.sdn.sap.com/wiki/display/BOBJ/Xcelsius+2008+and+SAP+NetWeaver+BW+Co
oracle表空间相关 tongsh6 oracle
在oracle数据库中，一个用户对应一个表空间，当表空间不足时，可以采用增加表空间的数据文件容量，也可以增加数据文件，方法有如下几种： 1.给表空间增加数据文件 ALTER TABLESPACE "表空间的名字" ADD DATAFILE '表空间的数据文件路径' SIZE 50M; &nb
.Net framework4.0安装失败 yangjuanjava .net windows
上午的.net framework 4.0，各种失败，查了好多答案，各种不靠谱，最后终于找到答案了和Windows Update有关系，给目录名重命名一下再次安装，即安装成功了！下载地址：http://www.microsoft.com/en-us/download/details.aspx?id=17113 方法： 1.运行cmd，输入net stop WuAuServ 2.点击开

Topics in High-Performance Messaging

Topics in High-Performance Messaging

Robert A. Van Valzah Todd L. Montgomery Eric Bowden

1. Introduction

2. TCP Latency

2.1. TCPLatency Behavior

2.2. TCP Receiver-Side Latency

2.3.TCP Sender-Side Latency

2.4. TCP Latency Recommendations

3. Group RateControl

3.1. The CrybabyReceiver Problem

3.2. Group Rate Control Policies

3.2.1. Policy Extreme1

3.2.2. Policy Extreme2

3.2.3. MiddleGround

3.3. Group Rate Control Consequences

3.4. PolicySelection

3.5. Group Rate ControlRecommendations

3.6. Group Rate Control and TransportProtocols

4.Ethernet Flow Control

5. Packet LossMyths

6. Monitoring Messaging Systems

7.UDP Buffering Background

7.1. Functionof UDP Buffering

7.2.Importance of UDP Buffering

7.3. Kernel and UserMode Roles in UDP Buffering

7.4. Unix Kernel UDP Buffer Limit Rationale

7.5. UDPBuffering and CPU Scheduling Latency

7.6.TCP Receive Buffering vs. UDP Receive Buffering

8. UDP BufferSizing

8.1.Optimal UDP Buffer Sizing

8.2. UDP Buffer Space Too Small:Consequences

8.3. UDPLoss: Consequences

8.4. UDP Buffer Space Too Large:Consequences

8.5. UDP Buffer Size Equations: Latency

8.6.UDP Buffer Size Equations: Buffer Size

8.7. UDP Buffer Kernel Defaults

8.8. Setting Kernel UDP Buffer Limits

8.8.1. Making Changes Survive Reboot

8.9. DetectingUDP Loss

8.9.1. Detecting Solaris UDP Loss

8.9.2. Detecting Linux UDP Loss

8.9.3. Detecting Windows UDP Loss

8.9.4.Detecting AIX UDP Loss

8.9.5. Detecting FreeBSD and Darwin UDP Loss

9. MulticastLoopback

10. Sending Multicast on MultipleInterfaces

11. TTL=0 to Keep Multicast Local

12. TTL=1 and Cisco CPU Usage

12.1.Solution: Increase TTL

12.2. Solution: IP MulticastTTL-Threshold

12.3. Solution: IP Multicast Boundary

12.4. Solution: Hardware Rate Limit

13. Intermittent Network MulticastLoss

14. Multicast Address Assignment

15. Multicast Retransmissions

15.1. Retransmission Cost/Benefit

15.2. Multicast Retransmission ControlTechniques

15.3. Avoiding Multicast Receiver Loss

16.Messaging Latency Budget

17. Sources ofLatency

17.1. Intermediaries

17.2. Garbage Collection

17.3. Retransmissions

17.4. Reordering

17.5. Batching

17.6. CPU Scheduling

17.7. Socket Buffers

17.8. Network Queuing

17.9. Network Access Control

17.10. Serialization

17.11. Speed of Light

17.12. AddressResolution Protocol (ARP)

18. Latency from Interrupt Coalescing

18.1. Linux Interrupt Coalescing

18.2. Linux "ethtool" Command Parameters

18.3. Windows Interrupt Coalescing

18.4. Windows NIC Loss Avoidance andDetection

18.4.1.Windows and Intel NICs

18.4.2. Windows and Broadcom NICs

Robert A. Van Valzah
Todd L. Montgomery
Eric Bowden