Linux排查服务器是否已经被入侵

入侵者可能会删除机器的日志信息

可以查看日志信息是否还存在或者是否被清空，相关命令示例：

[root@centos8 ~]# ls -h /var/log/*
/var/log/boot.log                  /var/log/dnf.librepo.log-20210502  /var/log/hawkey.log-20210502
/var/log/boot.log-20200828         /var/log/dnf.librepo.log-20210509  /var/log/hawkey.log-20210509
/var/log/boot.log-20210318         /var/log/dnf.log                   /var/log/lastlog
/var/log/btmp                      /var/log/dnf.log.1                 /var/log/maillog
/var/log/btmp-20210501             /var/log/dnf.log.2                 /var/log/messages
/var/log/cloud-init.log            /var/log/dnf.log-20200828          /var/log/qcloud_action.log
/var/log/cloud-init-output.log     /var/log/dnf.rpm.log               /var/log/secure
/var/log/cron                      /var/log/dnf.rpm.log-20200828      /var/log/secure-202103281616874601.gz
/var/log/dnf.librepo.log           /var/log/hawkey.log                /var/log/spooler
/var/log/dnf.librepo.log-20210418  /var/log/hawkey.log-20210418       /var/log/wtmp
/var/log/dnf.librepo.log-20210425  /var/log/hawkey.log-20210425

/var/log/anaconda:
anaconda.log  dnf.librepo.log  ifcfg.log    ks-script-64obidnb.log  ks-script-xc9zm2f2.log  program.log  syslog
dbus.log      hawkey.log       journal.log  ks-script-drwrp_wh.log  packaging.log           storage.log

/var/log/audit:
audit.log  audit.log.1  audit.log.2  audit.log.3

/var/log/chrony:

/var/log/insights-client:

/var/log/journal:
33790f3e0323419f9a055840e9d10b13

/var/log/nginx:
access.log  access.log-20210319.gz  error.log  error.log-20210319.gz

/var/log/private:

/var/log/qemu-ga:

/var/log/samba:
old

/var/log/sssd:
sssd_implicit_files.log              sssd_kcm.log           sssd.log-20210502.gz  sssd_nss.log-20200828.gz
sssd_implicit_files.log-20200828.gz  sssd_kcm.log-20210318  sssd.log-20210509     sssd_nss.log-20210318
sssd_implicit_files.log-20210318     sssd.log               sssd_nss.log

/var/log/tuned:
tuned.log

[root@centos8 ~]# du -sh /var/log/*
56K	/var/log/anaconda
29M	/var/log/audit
0	/var/log/boot.log
4.0K	/var/log/boot.log-20200828
4.0K	/var/log/boot.log-20210318
34M	/var/log/btmp
184M	/var/log/btmp-20210501
4.0K	/var/log/chrony
212K	/var/log/cloud-init.log
16K	/var/log/cloud-init-output.log
14M	/var/log/cron
52K	/var/log/dnf.librepo.log
136K	/var/log/dnf.librepo.log-20210418
136K	/var/log/dnf.librepo.log-20210425
136K	/var/log/dnf.librepo.log-20210502
132K	/var/log/dnf.librepo.log-20210509
112K	/var/log/dnf.log
1.1M	/var/log/dnf.log.1
1.1M	/var/log/dnf.log.2
4.0K	/var/log/dnf.log-20200828
192K	/var/log/dnf.rpm.log
4.0K	/var/log/dnf.rpm.log-20200828
4.0K	/var/log/hawkey.log
4.0K	/var/log/hawkey.log-20210418
4.0K	/var/log/hawkey.log-20210425
4.0K	/var/log/hawkey.log-20210502
4.0K	/var/log/hawkey.log-20210509
4.0K	/var/log/insights-client
1001M	/var/log/journal
8.0K	/var/log/lastlog
4.0K	/var/log/maillog
2.8M	/var/log/messages
12K	/var/log/nginx
4.0K	/var/log/private
4.0K	/var/log/qcloud_action.log
4.0K	/var/log/qemu-ga
8.0K	/var/log/samba
0	/var/log/secure
3.8M	/var/log/secure-202103281616874601.gz
4.0K	/var/log/spooler
36K	/var/log/sssd
8.0K	/var/log/tuned
4.0K	/var/log/wtmp

入侵者可能创建一个新的存放用户名及密码文件

[root@centos8 ~]# ll /etc/passwd*
-rw-r--r--  1 root root 1578 Mar 18 16:41 /etc/passwd
-rw-r--r--. 1 root root 1516 Mar 18 16:18 /etc/passwd-
[root@centos8 ~]# ll /etc/shadow*
----------  1 root root 799 Mar 18 16:41 /etc/shadow
----------. 1 root root 778 Mar 18 16:18 /etc/shadow-

入侵者可能修改用户名及密码文件

[root@centos8 ~]# more /etc/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
operator:x:11:0:operator:/root:/sbin/nologin
games:x:12:100:games:/usr/games:/sbin/nologin
ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin
nobody:x:65534:65534:Kernel Overflow User:/:/sbin/nologin
dbus:x:81:81:System message bus:/:/sbin/nologin
systemd-coredump:x:999:997:systemd Core Dumper:/:/sbin/nologin
systemd-resolve:x:193:193:systemd Resolver:/:/sbin/nologin
tss:x:59:59:Account used by the trousers package to sandbox the tcsd daemon:/dev/null:/sbin/nologin
polkitd:x:998:996:User for polkitd:/:/sbin/nologin
unbound:x:997:994:Unbound DNS resolver:/etc/unbound:/sbin/nologin
libstoragemgmt:x:996:993:daemon account for libstoragemgmt:/var/run/lsm:/sbin/nologin
cockpit-ws:x:995:991:User for cockpit-ws:/:/sbin/nologin
setroubleshoot:x:994:990::/var/lib/setroubleshoot:/sbin/nologin
sssd:x:993:989:User for sssd:/:/sbin/nologin
insights:x:992:988:Red Hat Insights:/var/lib/insights:/sbin/nologin
sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin
chrony:x:991:987::/var/lib/chrony:/sbin/nologin
tcpdump:x:72:72::/:/sbin/nologin
syslog:x:990:986::/home/syslog:/bin/false
cockpit-wsinstance:x:989:985:User for cockpit-ws instances:/nonexisting:/sbin/nologin
nginx:x:988:984:Nginx web server:/var/lib/nginx:/sbin/nologin

[root@centos8 ~]# more /etc/shadow
root:$1$MiAxHcSj$H9Peb.P53VkD4YxSkr6g9.:18704:0:99999:7:::
bin:*:18027:0:99999:7:::
daemon:*:18027:0:99999:7:::
adm:*:18027:0:99999:7:::
lp:*:18027:0:99999:7:::
sync:*:18027:0:99999:7:::
shutdown:*:18027:0:99999:7:::
halt:*:18027:0:99999:7:::
mail:*:18027:0:99999:7:::
operator:*:18027:0:99999:7:::
games:*:18027:0:99999:7:::
ftp:*:18027:0:99999:7:::
nobody:*:18027:0:99999:7:::
dbus:!!:18226::::::
systemd-coredump:!!:18226::::::
systemd-resolve:!!:18226::::::
tss:!!:18226::::::
polkitd:!!:18226::::::
unbound:!!:18226::::::
libstoragemgmt:!!:18226::::::
cockpit-ws:!!:18226::::::
setroubleshoot:!!:18226::::::
sssd:!!:18226::::::
insights:!!:18226::::::
sshd:!!:18226::::::
chrony:!!:18226::::::
tcpdump:!!:18226::::::
syslog:!!:18240::::::
cockpit-wsinstance:!!:18704::::::
nginx:!!:18704::::::

查看机器最近成功登陆的事件和最后一次不成功的登陆事件

对应日志“/var/log/lastlog”，相关命令示例：

[root@centos8 ~]# lastlog 
Username         Port     From             Latest
root             pts/0    221.217.94.106   Tue May 11 13:31:28 +0800 2021
bin                                        **Never logged in**
daemon                                     **Never logged in**
adm                                        **Never logged in**
lp                                         **Never logged in**
sync                                       **Never logged in**
shutdown                                   **Never logged in**
halt                                       **Never logged in**
mail                                       **Never logged in**
operator                                   **Never logged in**
games                                      **Never logged in**
ftp                                        **Never logged in**
nobody                                     **Never logged in**
dbus                                       **Never logged in**
systemd-coredump                           **Never logged in**
systemd-resolve                            **Never logged in**
tss                                        **Never logged in**
polkitd                                    **Never logged in**
unbound                                    **Never logged in**
libstoragemgmt                             **Never logged in**
cockpit-ws                                 **Never logged in**
setroubleshoot                             **Never logged in**
sssd                                       **Never logged in**
insights                                   **Never logged in**
sshd                                       **Never logged in**
chrony                                     **Never logged in**
tcpdump                                    **Never logged in**
syslog                                     **Never logged in**
cockpit-wsinstance                         **Never logged in**
nginx 

查看全部用户的登录事件

Linux查看/var/log/wtmp文件查看可疑IP登陆

[weiyan@VM-6-168-centos ~]$ last -f /var/log/wtmp
weiyan   pts/2        218.94.128.194   Thu Feb  2 13:51 - 13:51  (00:00)    
weiyan   pts/1        218.94.128.194   Thu Feb  2 13:36   still logged in   
weiyan   pts/0        218.94.128.194   Thu Feb  2 13:36   still logged in   
weiyan   pts/4        218.94.128.194   Thu Feb  2 11:30 - 11:33  (00:02)    
weiyan   pts/3        218.94.128.194   Thu Feb  2 11:30 - 11:33  (00:02)    
weiyan   pts/6        218.94.128.194   Thu Feb  2 11:12 - 11:12  (00:00)    
weiyan   pts/3        218.94.128.194   Thu Feb  2 11:12 - 11:12  (00:00)    
weiyan   pts/2        218.94.128.194   Thu Feb  2 10:59 - 12:22  (01:23)  

该日志文件永久记录每个用户登录、注销及系统的启动、停机的事件。因此随着系统正常运行时间的增加，该文件的大小也会越来越大，

增加的速度取决于系统用户登录的次数。该日志文件可以用来查看用户的登录记录，

last命令就通过访问这个文件获得这些信息，并以反序从后向前显示用户的登录记录，last也能根据用户、终端tty或时间显示相应的记录。

查看/var/log/secure文件寻找可疑IP(218.94.128.194)登陆次数

[weiyan@VM-6-168-centos ~]$ sudo cat /var/log/secure|grep 218.94.128.194
Feb  2 10:06:52 VM-6-168-centos sshd[4131]: Accepted password for weiyan from 218.94.128.194 port 6380 ssh2
Feb  2 10:38:57 VM-6-168-centos sshd[8976]: Accepted password for weiyan from 218.94.128.194 port 3308 ssh2
Feb  2 10:38:58 VM-6-168-centos sshd[9026]: Accepted password for weiyan from 218.94.128.194 port 3474 ssh2
Feb  2 10:39:37 VM-6-168-centos sshd[9839]: Accepted password for weiyan from 218.94.128.194 port 4872 ssh2
Feb  2 10:39:42 VM-6-168-centos sshd[10259]: Received disconnect from 218.94.128.194 port 4872:11: disconnected by user
Feb  2 10:39:42 VM-6-168-centos sshd[10259]: Disconnected from 218.94.128.194 port 4872

查看机器当前登录的全部用户

对应日志文件“/var/run/utmp”，相关命令示例：

[root@centos8 ~]# who 
root pts/0 2021-05-11 13:31 (221.217.94.106) 
root pts/1 2021-05-11 13:45 (221.217.94.106)

查看机器所有用户的连接时间（小时）

对应日志文件“/var/log/wtmp”，相关命令示例：

[root@centos8 ~]#  yum install psacct -y
[root@centos8 ~]# ac -dp
	root                                 1.47
Mar 18	total        1.47
	root                                 1.07
Today	total        1.07

如果发现机器产生了异常流量

可以使用命令“tcpdump”抓取网络包查看流量情况或者使用工具”iperf”查看流量情况

可以查看/var/log/secure日志文件
尝试发现入侵者的信息，相关命令示例：

[root@centos8 ~]# cat /var/log/secure
[root@centos8 ~]# cat /var/log/secure |grep -i "accepted password"

查询异常进程所对应的执行脚本文件

top命令查看异常进程对应的PID

在虚拟文件系统目录查找该进程的可执行文件

[root@centos8 ~]# ll /proc/68405/ |grep -i exe
lrwxrwxrwx  1 root root 0 Mar 18 16:52 exe -> /usr/lib/jvm/java-11-openjdk-11.0.9.11-
3.el8_3.x86_64/bin/java
[root@centos8 ~]#
[root@centos8 ~]# ll /usr/lib/jvm/java-11-openjdk-11.0.9.11-3.el8_3.x86_64/bin/java
-rwxr-xr-x 1 root root 16048 Jan  5 01:07 /usr/lib/jvm/java-11-openjdk-11.0.9.11-3.el8_3.x86_64/bin/java
[root@centos8 ~]#

如果确认机器已被入侵，重要文件已被删除，可以尝试找回被删除的文件Note:

1、当进程打开了某个文件时，只要该进程保持打开该文件，即使将其删除，它依然存在于磁盘中。这意味着，进程并不知道文件已经被删除，它仍然可以向打开该文件时提供给它的文件描述符进行读取和写入。除了该进程之外，这个文件是不可见的，因为已经删除了其相应的目录索引节点。

2、在/proc 目录下，其中包含了反映内核和进程树的各种文件。/proc目录挂载的是在内存中所映射的一块区域，所以这些文件和目录并不存在于磁盘中，因此当我们对这些文件进行读取和写入时，实际上是在从内存中获取相关信息。大多数与 lsof 相关的信息都存储于以进程的 PID 命名的目录中，即 /proc/1234 中包含的是 PID 为 1234 的进程的信息。每个进程目录中存在着各种文件，它们可以使得应用程序简单地了解进程的内存空间、文件描述符列表、指向磁盘上的文件的符号链接和其他系统信息。lsof 程序使用该信息和其他关于内核内部状态的信息来产生其输出。所以lsof 可以显示进程的文件描述符和相关的文件名等信息。也就是我们通过访问进程的文件描述符可以找到该文件的相关信息。

3、当系统中的某个文件被意外地删除了，只要这个时候系统中还有进程正在访问该文件，那么我们就可以通过lsof从/proc目录下恢复该文件的内容。

假设入侵者将/var/log/secure文件删除掉了，尝试将/var/log/secure文件恢复的方法可以参考如下：
a.查看/var/log/secure文件，发现已经没有该文件

[root@centos8 ~]# ll /var/log/secure
-rw------- 1 root root 0 Mar 28 03:50 /var/log/secure
### 假设没有该文件

b.使用lsof命令查看当前是否有进程打开/var/log/secure，

[root@centos8 ~]# lsof |grep /var/log/secure
rsyslogd    29014                             root    7w      REG              253,1 268745466     399444 /var/log/secure-202103281616874601 (deleted)
rsyslogd    29014   29016 in:imjour           root    7w      REG              253,1 268745466     399444 /var/log/secure-202103281616874601 (deleted)
rsyslogd    29014   29017 rs:main             root    7w      REG              253,1 268745466     399444 /var/log/secure-202103281616874601 (deleted)

c.从上面的信息可以看到 PID 29014（rsyslogd）打开文件的文件描述符为7。同时还可以看到/var/log/ secure已经标记为被删除了。因此我们可以在/proc/29014/fd/7（fd下的每个以数字命名的文件表示进程对应的文件描述符）中查看相应的信息，如下：

[root@centos8 ~]# tail  /proc/29014/fd/7
May 11 14:19:22 centos8 sshd[3022959]: Received disconnect from 68.183.84.221 port 54968:11: Bye Bye [preauth]
May 11 14:19:22 centos8 sshd[3022959]: Disconnected from invalid user alldigitalGE_ 68.183.84.221 port 54968 [preauth]
May 11 14:20:12 centos8 sshd[3023088]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=60.191.119.124  user=root
May 11 14:20:14 centos8 sshd[3023088]: Failed password for root from 60.191.119.124 port 64295 ssh2
May 11 14:20:15 centos8 sshd[3023088]: Received disconnect from 60.191.119.124 port 64295:11: Bye Bye [preauth]
May 11 14:20:15 centos8 sshd[3023088]: Disconnected from authenticating user root 60.191.119.124 port 64295 [preauth]
May 11 14:20:24 centos8 sshd[3023106]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=159.203.85.196  user=root
May 11 14:20:26 centos8 sshd[3023106]: Failed password for root from 159.203.85.196 port 58781 ssh2
May 11 14:20:27 centos8 sshd[3023106]: Received disconnect from 159.203.85.196 port 58781:11: Bye Bye [preauth]
May 11 14:20:27 centos8 sshd[3023106]: Disconnected from authenticating user root 159.203.85.196 port 58781 [preauth]

d.从上面的信息可以看出，查看/proc/29014/fd/7就可以得到所要恢复的数据。如果可以通过文件描述符查看相应的数据，那么就可以使用I/O重定向将其重定向到文件中，如:

[root@centos8 ~]# cat /proc/29014/fd/7 > /var/log/secure

e.再次查看/var/log/secure，发现该文件已经存在。对于许多应用程序，尤其是日志文件和数据库，这种恢复删除文件的方法非常有用。

[root@centos8 ~]# ll /var/log/secure
-rw------- 1 root root 0 Mar 28 03:50 /var/log/secure