Detecting Loadable Kernel Modules (LKM)

 

在学linux内核rootkit 时看过的一篇文章，介绍检测linux内核rootkit的一些方法。

 Detecting   Loadable Kernel Modules (LKM)

Purpose

      The purpose of this paper is cover LKM basics, detecting "trojaned" LKM's and figuring out which LKM is installed on your machine.

 

LKM

 

      What is a LKM? Loadable Kernel Modules (LKM) are files that contain dynamically loadable kernel components. LKM's are normally used to load device drivers and other hardware drivers. LKM's can be found on Linux, Solaris and BSD (Open, Free and Net). This paper will focus on Linux.

      Linux comes with various tools to assist the system administrator in loading, listing and unloading of the kernel modules. Before we cover the tools, lets look at some important files and directories associated with Kernel Modules. The first directory we want to look at is /lib/modules/"kernel_version". Lets look and see what files we can find under /lib/modules."Kernel_version":

 

block build cdrom fs ipv4 misc modules.dep modules.isapnpmap modules.pcimap modules.usbmap net pcmcia scsi usb video

         Table 1. /lib/modules listing

 Table 1 shows us both directories and files related to LKM's. The only ones listed that are not directories are modules.dep, modules.isapnpmap, modules.pcimap and mosules.usbmap. Those are actual files that list modules within the system (combined). Lets take a quick look at the net directory (This is not a complete listing, do to space):

3c59x.o 3c90x.o 82596.o 8390.o ac3200.o acenic.o arlan-proc.o

                        Table 2. net directory 

There are a couple of important bits of information we can gather from this table. The first bit is, modules are listed as .o. Why? Because they are object files that contain the actual module itself. The second bit is, the listing is not complete. I did not have room to put all of the modules.

      The second important directory/file we want to look at it /etc/conf.modules. Conf.modules is the configuration file that allows the system administrator to specify a variety of parameters that control the loading of the modules. The conf.modules file typically looks like this:

alias parport_lowlevel parport_pc

 

 

 

 

                                Table 3. conf.modules file

               

The conf.modules file allows the administrator to assign alias to commonly used modules. Note: This file is not required. A system administrator can create his/her own configuration file. A system administrator can do that by using the modprobe –C command.

      Since I mentioned modprobe earlier, lets briefly take a look at modprobe and the other tools available in Linux that load and unload modules. The first tool is modprobe. Modprobe can load single and multiple modules. It uses the modules.dep file to look up dependencies. Depmod creates a “Makefile” like dependency file. This file is called modules.dep.  The last three commands I will cover are lsmod, insmod and rmmod. lsmod provides the administrator a listing of all modules currently loaded in the kernel. This list can also be found at /proc/modules. This is helpful if you want to figure out what modules are currently running. You will see a little later that lsmod is not always a good tool to use for the detection of rootkit LKM’s. insmod loads modules(seems simple). Some of the “trojaned” LKM code I have seen uses insmod -f to load the “trojaned” module. rmmod is normally used to remove modules from the kernel. Normally rmmod will NOT remove any “trojaned” LKM. We will discuss details of that a little bit later. Hopefully, this section provided everybody the basics of Loadable Kernel Modules. If this was not detailed enough see http://www.kernel.org/LDP/.

 

 

LKM rootkits and kstat      

      Recently, there has been a lot of press about adore. Well, the worm adore not the LKM adore. So I decided to look into the LKM adore(everyone else has looked at the worm). BTW, if want to download adore go to http://packetstorm.securify.com/filedesc/adore-0.34.html.

      Adore is a Linux LKM rootkit. It is easy to install and only requires a few minor adjustments when configuring. Adore can be installed with a default configuration or the user can make changes to some of the code. The readme file recommends that the user change the settings for ELITE_CMD and for HIDDEN_PORT. When you run ./configure, adore asks you for a password. This is for the backdoor port (hence, change the HIDDEN_PORT) If you want more information on the install you will have to download it. I ran the default installation, which consisted of running./configure and make. After running make, you will see two files (ava, startadore). In order for adore too execute you have to run startadore. Once you run startadore the user can then run ava. Table 4 is the output from ./ava.

 

Usage: ./ava {h,u,r,R,i,v,U} [file, PID or dummy (for U)]          h hide file        u unhide file        r execute as root        R remove PID forever        U uninstall adore        i make PID invisible        v make PID visible

Table 4. Ava output

 

Table 4 shows us the options that adore provides to us. I will not cover each switch and what it does only because it does not help us detect this rootkit. Now that we have covered the basics of adore (LKM) lets look at how we detect adore and other rootkits.

      Many rootkits hide processes, directories, files and even connections. But many of them do so by modifying the source code of binaries such as ps, df, netstat, top and lsof. There are a couple of ways to detect these types of rootkits (i.e. t0rn):

1) md5 checksums

2) Compiling these binaries from a known good source (i.e. cd-rom, disk).      

3) Some rootkits have a default port that an administrator can focus in on.

4) Running a program such as chkrootkit

Many of the techniques used to detect rootkits like t0rn are not effective against LKM rootkits (chkrootkit can detect some of them). Since LKM rootkits access the kernel, they can hide processes, connections, directories and files without modifying the binaries. MD5 checksums become useless because there are no files being modified. This means the checksums will not change. Checking ports MIGHT be an option BUT that only tells you that you have been “rooted”.

      Well, how can I detect these monsters? Easy. There is a program I highly recommend to everyone concerned with LKM rootkits. This program is called KSTAT. You can find it at

http://s0ftpj.org/en/site.html. KSTAT works by checking the memory (/dev/kmem) for information about the host(including LKMs). Want to see what kstat looks like? Good. Here is the output for kstat:

Usage: kstat [-i iff] [-P] [-p pid] [-M] [-m addr] [-s]

-i    iff may be specified as 'all' or as name (e.g. eth0)

      displays info about the queried interface

 

-P    displays all processes

 

-p    pid is the process id of the queried task

 

-M    displays the kernel's LKMs' linked list

 

-m    addr is the hex address of the queried module

      displays info about the module to be found at addr

 

-s    displays info about the system calls' table

 

As you can see, kstat provides a person with many options for detecting these rootkits. Lets go through some of the options and from these options we will learn how to detect the following LKM rootkits:

 

1) knark

 

2) adore

 

3) rkit

 

BTW, if there’s any rootkits I have omitted please let me know and I will update the paper. : ) Kstat –s seems to be the best way to detect LKM rootkits. The other options are really great, but –s works all the time. Here is an output from kstat –s:

SysCall                         Address sys_exit                        0xc0117ce4 sys_fork                        0xc0108ebc sys_read                        0xc012604c sys_write                       0xc0126110 sys_open                        0xc0125c10 sys_close                       0xc0125d60 sys_waitpid                     0xc0117ff8 sys_creat                       0xc0125ca4 sys_link                        0xc012de60 sys_unlink                      0xc012dc90 sys_execve                      0xc0108f18 sys_chdir                       0xc01254a0 sys_time                        0xc01184b4 sys_mknod                       0xc012d77c sys_chmod                       0xc01256e4

           Table 5. kstat –s

 

Table 5 is not a complete output from kstat –s, but it does give us an idea as too what the output looks like. Remember –s provides us with a picture of the sys_call_table. Keep this information fresh as we will revisit this again later.

      Kstat –P is another switch that is very effective. Kstat -P shows us all of the processes running at that time. This includes the processes hidden by an LKM rootkit.

Table 6 is an example of kstat –P:

 

PID  PPID  UID   GID   COMMAND  1     0     0   0     init  2     1     0   0     kflushd  3     1     0   0     kupdate  4     1     0   0     kpiod  5     1     0   0     kswapd  6     1     0   0     mdrecoveryd 241   1      1   0     portmap   256   1      0   0     lockd 257  256     0   0     rpciod 266   1      0   0     rpc.statd 280   1      0   0     apmd 331   1      0   0     syslogd

Table 6. kstat –P

 

When I first ran this program I wasn’t sure kstat could do what it said but…it proved my doubts WRONG. Here is what I did to verify this switch. I used ava -i 241(portmap). I then ran kstat –P. As you can see from table 6,  portmap(bold) shows up. I then ran ps-ef(Table 7)and could not find it. Lsof also did not show it. Table 7 is the output from ps –ef:

 

UID        PID  PPID  C STIME TTY          TIME CMD

root         1     0  0 Mar30 ?        00:00:06 init [3]

root         2     1  0 Mar30 ?        00:00:00 [kflushd]

root         3     1  0 Mar30 ?        00:00:00 [kupdate]

root         4     1  0 Mar30 ?        00:00:00 [kpiod]

root         5     1  0 Mar30 ?        00:00:00 [kswapd]

root         6     1  0 Mar30 ?        00:00:00 [mdrecoveryd]

root       256     1  0 Mar30 ?        00:00:00 [lockd]

root       257   256  0 Mar30 ?        00:00:00 [rpciod]

root       266     1  0 Mar30 ?        00:00:00 rpc.statd

root       280     1  0 Mar30 ?        00:00:00 /usr/sbin/apmd -p 10 -w 5 -W -s

root       331     1  0 Mar30 ?        00:00:00 syslogd -m 0

root       340     1  0 Mar30 ?        00:00:00 klogd

nobody     354     1  0 Mar30 ?        00:00:00 identd -e -o

nobody     357   354  0 Mar30 ?        00:00:00 identd -e -o

nobody     359   357  0 Mar30 ?        00:00:00 identd -e -o

nobody     360   357  0 Mar30 ?        00:00:00 identd -e -o

nobody     361   357  0 Mar30 ?        00:00:00 identd -e -o

daemon     372     1  0 Mar30 ?        00:00:00 /usr/sbin/atd

root       386     1  0 Mar30 ?        00:00:00 crond

root       404     1  0 Mar30 ?        00:00:00 inetd

root       418     1  0 Mar30 ?        00:00:00 lpd

root       462     1  0 Mar30 ?        00:00:00 sendmail: accepting connections

root       477     1  0 Mar30 ?        00:00:00 gpm -t ps/2

root       491     1  0 Mar30 ?        00:00:01 httpd

xfs        531     1  0 Mar30 ?        00:00:00 xfs -droppriv -daemon -port -1

root       571     1  0 Mar30 tty1     00:00:00 login -- root   

root       572     1  0 Mar30 tty2     00:00:00 /sbin/mingetty tty2

root       573     1  0 Mar30 tty3     00:00:00 /sbin/mingetty tty3

root       574     1  0 Mar30 tty4     00:00:00 /sbin/mingetty tty4

root       575     1  0 Mar30 tty5     00:00:00 /sbin/mingetty tty5

root       576     1  0 Mar30 tty6     00:00:00 /sbin/mingetty tty6

nobody    4290   491  0 Apr01 ?        00:00:00 httpd

nobody    4291   491  0 Apr01 ?        00:00:00 httpd

nobody    4292   491  0 Apr01 ?        00:00:00 httpd

nobody    4293   491  0 Apr01 ?        00:00:00 httpd

nobody    4294   491  0 Apr01 ?        00:00:00 httpd

nobody    4295   491  0 Apr01 ?        00:00:00 httpd

nobody    4298   491  0 Apr01 ?        00:00:00 httpd

nobody    4299   491  0 Apr01 ?        00:00:00 httpd

root      8073   571  0 Apr02 tty1     00:00:00 -bash

root     10659  8073  0 11:24 tty1     00:00:00 ps -ef

            Table 7. ps –ef output

There are two other switches we will go over, they are kstat –p and kstat –M. First, lets look at kstat –p. kstat –p gives us more information about a process. In order to run kstat –p you will need to provide a process id. For example: kstat –p 241. This checks process 241 (portmap) and provide an output. Sometimes it can provide us with more information about a LKM rootkit. TABLE 8 is the output from kstat –p 241:

 

Name:     portmap State:    S (sleeping) Pid:      241 Ppid:     1 (init) Uid:      1     1     1     1 Gid:      0     0     0     0 Flags:    PF_FORKNOEXEC PF_SUPERPRIV Crucial Capabilities Check Open Files     0     CHAR        /dev/null     1     CHAR        /dev/null     2     CHAR        /dev/null     3     0.0.0.0:111 0.0.0.0:0     4     0.0.0.0:111 0.0.0.0:0     7     FIFO        ///     8     FIFO        ///     21    CHAR        /dev/null

Table 8. kstat –p output

 

Finally, lets look at kstat –M. Under normal conditions an administrator can perform lsmod or more /proc/modules and find out what modules he/she are running. When a machine has been “rooted” you can’t trust lsmod for accurate information. Kstat –M will catch many of the basic LKM rootkits that I have tested. Kstat –M list all of the modules loaded. I have seen it where kstat –M did list anything for knark, but…nothings perfect. The output from kstat –M is similar to lsmod.

 

Detecting knark, adore and other LKM rootkits

 

So far we have covered a lot of material about LKM’s, rootkits and kstat. Now we are going to put all of that together and learn how to detect some of the LKM rootkits available today. 

The first LKM rootkit we want to look at is knark. Knark is probably the best-known LKM rootkit and one of the best written as well. Detecting it with kstat is fairly straight up as well. Remember Table 5? Well in order to detect a knark LKM rootkit you will need to run kstat –s. Once ran you need to look for the following:

 

 

sys_fork       0xc284652c WARNING! Should be at 0xc0108c88 sys_read       0xc2846868 WARNING! Should be at 0xc012699c sys_execve     0xc2846bb8 WARNING! Should be at 0xc0108ce4 sys_kill       0xc28465d4 WARNING! Should be at 0xc01106b4 sys_ioctl      0xc2846640 WARNING! Should be at 0xc012ff78 sys_settimeofday 0xc2846a8c WARNING! Should be at 0xc0118364 sys_clone      0xc2846580 WARNING! Should be at 0xc0108ca4

 Table 9. knark detection

 

Lets take a closer look at this table and see what all of this mess means to us. First, we see that there are seven (7) sys_call_table entries that have warnings. Lets look at sys_settimeofday, here we see that currently it is making it’s home in memory at 0xc2846a8c. kstat –s tells us that this is wrong and it should be at 0xc0118364. When knark  is installed it changes the sys_call_table and the memory locations where you can find: sys_fork, sys_read, sys_execve, sys_kill, sys_ioctl, sys_settimeofday and sys_clone. This is how you could detect knark. Knowing which sys_call_table entries have been changed could help you identify the actual rootkit itself. Knark changes seven(7) total. After running kstat –s, the next step would be to run ps-ef along with kstat –P and compare the two. If the are any differences in the two you can then take the correct action. Keep in mind that memory locations can change from box to box. As I said earlier, detecting LKM rootkits with kstat is quite simple.

      Lets look at adore. Table 10 will show us what adore changes:

 

sys_fork       0xc4051428 WARNING! Should be at 0xc0108c88 sys_write      0xc4051590 WARNING! Should be at 0xc01269b8 sys_close      0xc405163c WARNING! Should be at 0xc01264a4 sys_kill       0xc40514d0 WARNING! Should be at 0xc011060c sys_mkdir      0xc405172c WARNING! Should be at 0xc012e540 sys_clone      0xc405147c WARNING! Should be at 0xc0108ca4 sys_getdents   0xc40512a4 WARNING! Should be at 0xc013022c

Table 10. Adore detection

Adore changes seven (7) entries as well (just like knark). Knark and adore only change three (3) of the same sys_call_table entries. They are sys_fork, sys_kill, sys_clone. This is a good thing because it allows us the ability to detect each one separately. Again, once this has been done, the administrator can then run kstat –P and ps-ef and compare the two to figure out what processes are running and hiding.

The last rootkit we will look at is rkit. This rootkit does not hide itself as well as the other two do. As a matter of fact it only changes sys_setuid and can be found using kstat –M.

Conclusion

                LKM rootkits can make a system administrator’s life a nightmare. They are hard to detect, but using tools like kstat and understanding what the rootkit changes can make our life easier. Since tools like kstat are available, it would help systems administrators if they took a “picture” of the sys_call_table after a fresh install and any upgrades. This will help in identifying the good from the bad.  Although I have not had the time, one could write a shell | perl script to automate this process and check weekly, daily, every 10 minutes or whatever.

Resources

Kstat

http://s0ftpj.org/en/site.html

Knark

http://members.prestige.net/tmiller12/papers/KNARK.htm

 

LKM rootkits

 

http://packetstorm.securify.com/groups/thc/LKM_HACKING.html

Author

Toby Miller is a GIAC Certified Analyst and MCP. He is currently working towards his CISSP and RHCE. Toby has contributed to 2 books, written papers for SANS, Securityfocus, performs Risk Assessments and runs a lab for a living. For entertainment Toby likes to analyze exploit signatures on his home network. Toby can be reached at [email protected].