imuxsock begins to drop messages

It turns out that many modern Linux distributions come with 'rsyslog', which is a replacement for 'syslogd' or 'sysklogd', but starting with version 5.7.1 of rsyslog, a feature known as rate-limiting was added to the utility, and if a given process ID (PID) were to send more than 200 messages to /var/log/messages in a 5 second interval (the default setting in rsyslog), it will start to drop messages and place the following warning inside of /var/log/messages:

 

Feb  5 13:07:52 plugh rsyslogd-2177: imuxsock begins to drop messages

from pid 12105 due to rate-limiting

 

In the case of daemons or processes logging to /var/log/messages (or any other directory/file which rsyslog happens to be handling  logging for), a great deal of important and/or critical logging  data could be lost to security or system administrators.

 

While rate-limiting on routers/firewalls/web servers is a useful method of containing certain types of network based attacks, in the case of system and/or application logging, this may create a logistical nightmare for SIEM's or applications which collect and analyze large amounts of system and/or application logs for event information/messages/warnings.

 

In doing some research on rsyslog, I found two solutions which can be used to solve this condition on systems where rsyslog is the default system logging method.

 

Note - Back up any file(s) listed below before proceeding!

 

The first solution is to simply increase the messages allowed and the time interval before rate-limiting occurs in rsyslog.  To do this, locate the rsyslog.conf and/or rsyslog.early.conf (usually in /etc) and add the following lines:

 

$SystemLogRateLimitInterval 10

$SystemLogRateLimitBurst 500

 

after any ModLoad commands in rsyslog.conf and/or rsyslog.early.conf

 

This will tell rsyslog to start rate-limiting (discarding messages) when more than 500 messages from a single PID are received within a 10 second interval (these numbers are not absolutes, they can be tailored to any given system, btw).

 

The second solution is to simply turn off rate-limiting for rsyslog, and to do this, add the following line to rsyslog.early.conf and/or rsyslog.conf using your favorite editor (Iím a vi/vim/gvim hound):

 

$SystemLogRateLimitInterval 0

 

after any ModLoad commands in rsyslog.conf and/or rsyslog.early.conf

 

This will disable any rate-limiting in effect for the rsyslog process running on this system.  Note that by doing this, an out of control process ID on your system can fill up /var/log/messages with a lot of useless messages (which is why rate-limiting is enabled by default in rsyslog).

 

Remember to stop/start or restart the rsyslog daemon in order to make the changes to rsyslog.conf and/or rsyslog.early.conf take effect.
Reboot rsyslog:

/etc/init.d/rsyslog resart

 

The following Linux systems use 'rsyslog' as the default system logger (these are distributions which I am actively using, btw):

 

CentOS 6.x

Debian 5.0 or greater

Fedora 13 or greater

OpenSuSE 11.x/12.x

Ubuntu 10.0 or greater

 

BSD based systems (FreeBSD 8.x/9.0, OpenBSD 5.x, and NetBSD 5.x/6.0) use traditional syslogd as the default system logging utility.