Why does Red Hat Enterprise Linux system does not respond to SYN requests intermittently ?

 

Why does Red Hat Enterprise Linux system does not respond to SYN requests intermittently ?

Article ID: 40006 - Created on: Aug 17, 2009 7:48 AM - Last Modified: Sep 13, 2010 6:13 AM

Issue

• This issue has been reported in a environment using NAT (Network Address Translation) like the following:
  

   

  Resolution
  Disable TCP time stamps using the following command:

  echo 0 > /proc/sys/net/ipv4/tcp_timestamps

  or by adding the following line to /etc/systcl.conf and rebooting the system:

  net.ipv4.tcp_timestamps = 0

  Disabling TCP time stamps will disable PAWS (Protect Against Wrapped Sequences), checking as defined in RFC 1323.
  Root Cause
  In a TCP dump of the affected packets, the TSVal of SYN packet is a lower value than the preceding packet.

  Packet #30 from x.x.x.x to y.y.y.y has a TSval of 773695118

  Packet #33 from x.x.x.x to y.y.y.y has a TSval of 470318331

  In above TCP trace, packet #33 has a TSval of 470318331 which is lower than its preceding one. As PAWS checking is enabled in the TCP stack, it doesn't send an acknowledge (SYN-ACK) back to the peer.
  PAWS assumes that every received TCP segment (including data and ACK segments) contains a timestamp (SEG.TSval) whose values are non-decreasing in time. The basic idea is that a segment can be discarded as an old duplicate if it is received with a timestamp (SEG.TSval) less than some timestamp recently received on this connection.
  Timestamp issues often indicate a problem with firewalls or NAT routers.
  Diagnostic Steps
  Stage 1
  Since the server is not heavily loaded, you may at first suspect this issue to be an issue with socket exhaustion. However, checking the output of "netstat -paunt" will reveal that there are only a couple of active connections.
  Stage 2
  Increasing the size of syn backlog queue (net.ipv4.tcp_max_syn_backlog) doesn't seem to help.
  Stage 3
  Collecting a TCP dump from the server using a command like the following:

  tcpdump -s0 -w /tmp/tcpdump.pcap -i any host <client ip> and port 80

  and generating traffic (HTTP or LDAP) to the server captures evidence of the server not responding to the TCP SYN packets. The output file can be analyzed with a command like

  tcpdump -r /<path>/<to>/tcpdump.pcap

  or Wireshark can be used to analyze the TCP dump file graphically. Wireshark is a standard package in most Red Hat Enterprise Linux and Fedora distributions. Analysis of the TCP dump should show cases where the client will send several TCP SYN packets to the host, but the host will not respond. Further analysis should show that the TSVal of the SYN packets is less than a preceding TCP packet.