首先数据文件删除文件系统空间不释放的问题不只出现在Linux平台,所有平台都可能有这样的问题。这里只是在Linux平台做一些测试,其他平台类似;其次只有将数据文件存放在文件系统中才会有此类问题,如果数据库存放在ASM中也不会有类似的问题,这篇文章的目的是对相关问题进行总结,熟悉一些工具和方法,这是比较重要的。
写这篇文章是因为客户有一套数据库,AIX 6.1,数据文件存放在文件系统中,在使用DROP TABLESPACE UNDO INCLUDING CONTENTS AND DATAFILES删除了UNDO表空间之后,文件系统空间没有得到释放。如何解决问题就成了写这篇文章的初衷。
空间没有释放我们可能是通过df命令看确认的,当我们用du去扫描目录的大小可能会发现df和du两个命令看到的空间使用情况是不同的,可能差别很大,下面这篇MOS文章说明了原因:
'du' and 'df' tools report different space utilization (文档 ID 457444.1) |
|
In this Document
Symptoms
Cause
Solution
Applies to:
Linux OS - Version Oracle Linux 4.4 and later
Linux x86-64
Linux x86
Linux Kernel - Version: 4.4 to 5.3
Symptoms
The 'du' (/usr/bin/du) and 'df' (/bin/df) command output displays conflicting space utilisation values, for example:
# df -k /
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/sda6 9288792 8672768 144120 99% /
# du -xsh /
2.1G /
In the example above, 'df' reports 8.6 Gb to have been used on the root (/) filesystem, whereas 'du' reports only 2.1 Gb to have been used.
Cause
The 'df' command reports how many disk blocks are used, whilst 'du' traverses the filesystem and reports the actual number of blocks used (directory by directory), including any relating to files in use by processes.
In most cases, space utilisation values returned from 'df' and 'du' will be consistent. However, the potential exists for a running process to delete a large file, say. In this instance, according to 'du', the large file no longer exists, so the blocks associated with the deleted file are not reported. With the process still running, and with an open file descriptor still held against the deleted file, 'df' continues to track and report all disk blocks used, including those associated with the deleted (phantom) file. In this situation, the disk space associated with the deleted file will only be relinquished back to the system when the process completely releases the deleted file's descriptor or the process terminates (either gracefully or killed).
Solution
The solution is to identify and stop (or kill) the process that continues to hold a file descriptor open for the deleted file. To do so, run the lsof command (/usr/sbin/lsof | grep deleted) as root to identify the holding process, for example:
# lsof | grep deleted
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
cannaserv 3825 canna 0u CHR 136,0 2 /dev/pts/0 (deleted)
vmware 4295 root 6u REG 253,0 6770 13074503 /tmp/vmware-root/ui-4295.log (deleted)
vmware-re 4316 root 6u REG 253,0 6770 13074503 /tmp/vmware-root/ui-4295.log (deleted)
vmnet-nat 4448 root 0u CHR 136,0 2 /dev/pts/0 (deleted)
vmware-se 4454 root 0u CHR 136,0 2 /dev/pts/0 (deleted)
gdm-binar 4506 root 0u CHR 136,0 2 /dev/pts/0 (deleted)
gconfd-2 5392 root 12wW REG 253,0 609 13090818 /tmp/gconfd-root/lock/0t1188207163ut519551u0p5392r346479926k3219784492 (deleted)
vmware-vm 5822 root 57u REG 253,0 6520832 13074477 /tmp/vmware-root/ram0 (deleted)
vmware-vm 16487 root 57u REG 253,0 11153408 13074520 /tmp/vmware-root/ram0 (deleted)
kdeinit 17991 root 17u REG 253,0 26712 13074524 /tmp/kde-root/khtmlcacheM7jXYb.tmp (deleted)
kdeinit 17991 root 18u REG 253,0 5631 13074501 /tmp/kde-root/khtmlcacheZlJmda.tmp (deleted)
kdeinit 17991 root 21u REG 253,0 44718 13074514 /tmp/kde-root/khtmlcacheH5m4lc.tmp (deleted)
In the example above, the 7th column in the output denotes the size of deleted files (in bytes). The 9th column denotes which file remains held open. The 1st and second columns denotes the process and pid that still holds the open file descriptor.
之所以df和du命令看到的空间使用会有差别,原因在于du不统计已经删除的文件,df会统计已经删除的文件,但该文件依然被进程持有,只有等进程释放了该文件,df才不进行统计。通过lsof | grep deleted命令可以找出被删除的文件依然被进程持有的情况。
通过上面这篇文章我们得知,在文件系统中删除某个大文件空间没有释放是因为依然有进程在持有它,如果找到相关进程就需要用到lsof命令和fuser命令,熟悉这两个命令非常的重要。
>lsof(list open files)是一个列出当前系统打开文件的工具。
>fuser用于标识访问文件或socket的进程信息。
fuser也是我们平时常用的找进程的工具,但它不能列出详细的进程信息,更多的只是一个进程号,无法很好的过滤,在这个场景中lsof工具更加的合适。
另外,大家都知道,在Linux平台文件系统中,即使某个文件被删除,但是它如果任然有进程持有它,这个文件是可以被恢复的,这就是为什么数据库所有数据文件被删除,在实例没有停机的情况下可以完整恢复数据文件的原因,下面这篇MOS文章就讨论了在此情况下如何对数据文件进行恢复:
How To Recover Deleted Files on ext3/ext4 Filesystem (文档 ID 2056343.1) |
|
|
In this Document
Goal
Solution
References
Applies to:
Linux OS - Version Oracle Linux 5.0 and later
Linux x86-64
Linux x86
Goal
How to recover deleted file on ext3/ext4 filesystem, Which still has file descriptor opened.
Solution
A file in Linux is a pointer to an inode which contains the file data (permissions, owner and where its actual content resides on the disk).
Deleting the file removes the link, but not the inode itself.
If any other process has this file open then inode is not released for writing until the process releases it.
So if a process still has the file open, the data are there somewhere, even though the directory listing shows no files.
# ll
total 4
-rw-r--r-- 1 root root 27 Sep 16 05:19 test
# rm test
# lsof /opt/test/test
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
less 21353 root 4r REG 252,2 27 260360 /opt/test/test (deleted) <<<<<<<<<<<<<
Understanding output of "lsof" command:
COMMAND: Command using the file.
PID: PID of the file
USER: Owner of the file
FD: File descriptor. Different flags of File descriptor are as below:
#: The number in front of flag(s) is the file descriptor number of used by the process to associated with the file
u: File open with Read and Write permission
r: File open with Read permission
w: File open with Write permission
W: File open with Write permission and with Write Lock on entire file
mem: Memory mapped file, usually for share library
TYPE: File type. Different flags of File type are as below:
REG - Regular file
DIR - Directory
DEVICE: major, minor number of the device where file resides.
SIZE/OFF: File size
NODE: inode number
NAME: File name
Now we know that process 21353 still has the file open, and the file descriptor is 4.
Now we can look into /proc and there will be a reference to the inode, from which the deleted file can be copied.
Following steps will help to recover the deleted files:
# ls -l /proc/21353/fd/4
lr-x------ 1 root root 64 Sep 16 05:28 /proc/21353/fd/4 -> /opt/test/test (deleted)
# cp /proc/21353/fd/4 /opt/test/test.bkp
Now verify the content of the restored file.
Note: Don't use the -a flag with cp, as this will copy the (broken) symbolic link, rather than the actual file contents.
另外,找到是某个进程持有的文件,通过下面的方法可以看到这个进程相关的环境信息:
Checking the environment variables of ASM pmon process: It shows ORACLE_HOME is set to /oracle_grid/product/11.2.0.3/grid/ ( with 'slash' at the end )
# ps -ef | grep pmon
oracle
27232
1 0 May30 ? 00:07:05 asm_pmon_+ASM1
#
cat /proc/27232/environ
__CLSAGFW_TYPE_NAME=ora.asm.type
ORA_CRS_HOME=/oracle_grid/product/11.2.0.3/grid/
HOSTNAME=aude3od015naboi.basdev.aurdev.national.com.auTERM=xtermSHELL=/bin/bash__CR......
总结:对于此类问题,我们首先要明白为什么df和du在空间计算上有所差别,其次要熟悉lsof和fuser两个命令,找出继续持有文件的进程号,通过该进程号可以在/proc目录下恢复文件,查看进程的环境信息,甚至杀掉进程来释放空间。
最后通过一个简单的例子来结束这篇文章:
1.首先确保lsof工具已经安装到操作系统。
[root@rac01 Server]# rpm -ivh lsof-4.78-6.x86_64.rpm
Preparing... ########################################### [100%]
1:lsof ########################################### [100%]
[root@rac01 Server]# which lsof
/usr/sbin/lsof
2.在其中一个会话通过tail -f install2.log命令使tail进程持有该文件,在另一个会话通过rm -rf install2.log命令删除该文件。
3.使用lsof执行如下的操作:
[root@rac01 ~]# lsof | grep deleted
tail 6006 root 3r REG 8,3 29544 4587629 /root/install2.log (deleted)
[root@rac01 ~]# cd /proc/6006/
[root@rac01 6006]# ls
attr cmdline cwd fdinfo loginuid mounts numa_maps pagemap schedstat stat task
auxv comm environ io maps mountstats oom_adj personality sessionid statm wchan
cgroup coredump_filter exe latency mem net oom_score root smaps status
clear_refs cpuset fd limits mountinfo ns oom_score_adj sched stack syscall
[root@rac01 6006]# cd fd
[root@rac01 fd]# ll
total 0
lrwx------ 1 root root 64 Dec 3 19:07 0 -> /dev/pts/0
lrwx------ 1 root root 64 Dec 3 19:07 1 -> /dev/pts/0
lrwx------ 1 root root 64 Dec 3 19:07 2 -> /dev/pts/0
lr-x------ 1 root root 64 Dec 3 19:07 3 -> /root/install2.log (deleted)
[root@rac01 fd]# cd ..
[root@rac01 6006]# cat environ
HOSTNAME=rac01TERM=vt100SHELL=/bin/bashHISTSIZE=1000SSH_CLIENT=172.168.4.123 56823 22OLDPWD=/mnt/ServerSSH_TTY=/dev/pts/0USER=rootLS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:MAIL=/var/spool/mail/rootPATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/binINPUTRC=/etc/inputrcPWD=/rootLANG=en_US.UTF-8SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpassSHLVL=1HOME=/rootLOGNAME=rootSSH_CONNECTION=172.168.4.123 56823 172.168.4.200 22LESSOPEN=|/usr/bin/lesspipe.sh %sG_BROKEN_FILENAMES=1_=/usr/bin/tail
[root@rac01 6006]# lsof /root/ <<<< 找出持有/root目录下文件的进程
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash 4934 root cwd DIR 8,3 4096 4587521 /root/
tail 6006 root cwd DIR 8,3 4096 4587521 /root/
[root@rac01 6006]# lsof -c tail <<<< 找出tail进程持有的文件
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tail 6006 root cwd DIR 8,3 4096 4587521 /root
tail 6006 root rtd DIR 8,3 4096 2 /
tail 6006 root txt REG 8,3 37704 1448826 /usr/bin/tail
tail 6006 root mem REG 8,3 56479136 1446088 /usr/lib/locale/locale-archive
tail 6006 root mem REG 8,3 1720736 5242891 /lib64/libc-2.5.so
tail 6006 root mem REG 8,3 142488 5242884 /lib64/ld-2.5.so
tail 6006 root 0u CHR 136,0 0t0 3 /dev/pts/0
tail 6006 root 1u CHR 136,0 0t0 3 /dev/pts/0
tail 6006 root 2u CHR 136,0 0t0 3 /dev/pts/0
tail 6006 root 3r REG 8,3 29544 4587629 /root/install2.log (deleted)
[root@rac01 6006]# lsof +d /root/ <<<< 显示访问/root目录下文件的进程。
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash 4934 root cwd DIR 8,3 4096 4587521 /root/
tail 6006 root cwd DIR 8,3 4096 4587521 /root/
[root@rac01 6006]# lsof +D /root/ <<<< 显示访问/root目录及子目录下文件的进程。
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash 4934 root cwd DIR 8,3 4096 4587521 /root/
tail 6006 root cwd DIR 8,3 4096 4587521 /root/
[root@rac01 6006]# lsof -d 3 | grep -v grep | grep deleted <<<< 显示持有文件FD为3的进程文件
tail 6006 root 3r REG 8,3 29544 4587629 /root/install2.log (deleted)
[root@rac01 6006]# lsof -p 6006 <<<< 显示6006进程持有的文件
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tail 6006 root cwd DIR 8,3 4096 4587521 /root
tail 6006 root rtd DIR 8,3 4096 2 /
tail 6006 root txt REG 8,3 37704 1448826 /usr/bin/tail
tail 6006 root mem REG 8,3 56479136 1446088 /usr/lib/locale/locale-archive
tail 6006 root mem REG 8,3 1720736 5242891 /lib64/libc-2.5.so
tail 6006 root mem REG 8,3 142488 5242884 /lib64/ld-2.5.so
tail 6006 root 0u CHR 136,0 0t0 3 /dev/pts/0
tail 6006 root 1u CHR 136,0 0t0 3 /dev/pts/0
tail 6006 root 2u CHR 136,0 0t0 3 /dev/pts/0
tail 6006 root 3r REG 8,3 29544 4587629 /root/install2.log (deleted)
[root@rac01 6006]# lsof -u root | grep deleted <<<< 显示以root用户持有的进程文件
tail 6006 root 3r REG 8,3 29544 4587629 /root/install2.log (deleted)
[root@rac01 6006]# cd /proc/6006/fd/
[root@rac01 fd]# cp 3 /root/install2.log <<<< 恢复删除的install2.log文件
[root@rac01 fd]# cd /proc/6006/fd
[root@rac01 fd]# ll
total 0
lrwx------ 1 root root 64 Dec 3 19:07 0 -> /dev/pts/0
lrwx------ 1 root root 64 Dec 3 19:07 1 -> /dev/pts/0
lrwx------ 1 root root 64 Dec 3 19:07 2 -> /dev/pts/0
lr-x------ 1 root root 64 Dec 3 19:07 3 -> /root/install2.log (deleted) <<<< 文件恢复后deleted状态未发生变化。
[root@rac01 fd]# cd /root/
[root@rac01 ~]# ls
anaconda-ks.cfg core.11067 core.526 core.6027 core.7965 Desktop install2.log install.log install.log.syslog
--end--
来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/23135684/viewspace-1852841/,如需转载,请注明出处,否则将追究法律责任。