问题描述:
一个文件正在被进程写 我想查看哪个进程在写这个文件。
解决思路:
Linux 下每个文件都会在某个块设备上存放,当然也都有相应的 inode ,那么透过 vfs.write 我们就可以知道谁在不停的写入特定的设备上的 inode。
那用什么工具呢? 这个神器就是systemtap ,具体的命令就是stap。
参考:http://cccgw.info/2012/07/systemtap%E5%85%A5%E9%97%A8/
安装很简单:yum -y install systemtap
但是这个安装完毕,还需要内核的支持,不然会报错。具体的说就是需要安装kernel-debuginfo包。
安装过程:
1 查看内核版本
uname -r
2.6.32-279.el6.x86_64
2 下载相应的包:
wget http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-2.6.32-279.el6.x86_64.rpm
wget http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-common-2.6.32-279.el6.x86_64.rpm
3 安装rpm包:
rpm -ivh kernel-debuginfo-2.6.32-279.el6.x86_64.rpm kernel-debuginfo-common-2.6.32-279.el6.x86_64.rpm
4 确保安装包正常:
rpm -qa|grep kernel-debug
kernel-debuginfo-common-x86_64-2.6.32-279.el6.x86_64
kernel-debuginfo-2.6.32-279.el6.x86_64
好了,测试下stap的可用性吧:
执行命令:stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'
显示下面结果:
Pass 1: parsed user script and 83 library script(s) using 194448virt/23176res/3056shr kb, in 130usr/10sys/142real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 3 embed(s), 0 global(s) using 427088virt/122744res/8316shr kb, in 1400usr/90sys/1493real ms.
Pass 3: using cached /root/.systemtap/cache/4a/stap_4a1eb85edba807357c24d4e1a07bc9d7_1471.c
Pass 4: using cached /root/.systemtap/cache/4a/stap_4a1eb85edba807357c24d4e1a07bc9d7_1471.ko
Pass 5: starting run.
read performed
Pass 5: run completed in 0usr/30sys/346real ms.
说明stap可以用了。
用法:
stap stap预制脚本.stp major minor 文件inode
(systemstap,major,minor ,这些不明白的去google吧)
比如我想知道我的/root/debug.log 是谁写的,例子:
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root
34G 11G 21G 35% /
ls -l /dev/mapper/VolGroup-lv_root
lrwxrwxrwx. 1 root root 7 Aug 31 2012 /dev/mapper/VolGroup-lv_root -> ../dm-0
继续找:
ls -l /dev/dm-0
brw-rw----. 1 root disk 253, 0 Aug 31 2012 /dev/dm-0
看到了么? 253是major number ,0 是minor number
再找文件的inode:
stat -c '%i' /root/debug.log
523308
组合命令来吧:
stap /usr/share/doc/systemtap-client-1.7/examples/io/inodewatch.stp 253 0 523302
astroc(740) vfs_write 0x800011/25337884
astroc(740) vfs_write 0x800011/25337884
astroc(740) vfs_write 0x800011/25337884
astroc(740) vfs_write 0x800011/25337884
------------------------------------------------------------------------------
查看进程写哪个文件
[root]# ls -l /proc/29390/fd
[root]# crash /usr/lib/debug/lib/modules/2.6.32-431.el6.x86_64/vmlinux -S /boot/System.map-2.6.32-431.el6.x86_64
crash: cannot open /boot/System.map
crash 6.1.0-5.el6
Copyright (C) 2002-2012 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.
GNU gdb (GDB) 7.3.1
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"...
SYSTEM MAP: /boot/System.map-2.6.32-431.el6.x86_64
DEBUG KERNEL: /usr/lib/debug/lib/modules/2.6.32-431.el6.x86_64/vmlinux (2.6.32-431.el6.x86_64)
DUMPFILE: /dev/crash
CPUS: 32
DATE: Sun Jun 14 23:30:39 2015
UPTIME: 1 days, 19:00:27
LOAD AVERAGE: 10.78, 9.96, 9.42
TASKS: 734
NODENAME: localhost
RELEASE: 2.6.32-431.el6.x86_64
VERSION: #1 SMP Fri Nov 22 03:15:09 UTC 2013
MACHINE: x86_64 (2600 Mhz)
MEMORY: 32 GB
PID: 44892
COMMAND: "crash"
TASK: ffff88083724d500 [THREAD_INFO: ffff880838cd0000]
CPU: 1
STATE: TASK_RUNNING (ACTIVE)
crash> set 29399
PID: 29399
COMMAND: "postgres"
TASK: ffff880435ed4080 [THREAD_INFO: ffff88043747e000]
CPU: 2
STATE: TASK_UNINTERRUPTIBLE
crash> files
PID: 29399 TASK: ffff880435ed4080 CPU: 0 COMMAND: "postgres"
ROOT: / CWD: /p3/pgsql/data1
FD FILE DENTRY INODE TYPE PATH
0 ffff880831e78a80 ffff880439dede40 ffff880839860d48 CHR /dev/null
1 ffff88082ec4b080 ffff88043475c140 ffff880439f3e4e8 CHR /dev/pts/4
2 ffff88082ec4b080 ffff88043475c140 ffff880439f3e4e8 CHR /dev/pts/4
3 ffff8808389d0540 ffff880836647c00 ffff88083a28f7f8 FIFO
4 ffff88082fadbcc0 ffff880836647c00 ffff88083a28f7f8 FIFO
5 ffff880011b60600 ffff8802139aa440 ffff88041d2ba8b0 REG /p3/pgsql/data1/pg_xlog/000000010000038A000000C7
6 ffff88042309b440 ffff8803f479ccc0 ffff880426451bd8 FIFO
7 ffff88008772b180 ffff8800bc83c600 ffff8800879628b0 REG /p3/pgsql/data1/base/16384/16408
8 ffff8800113d4440 ffff8803f479c180 ffff8804345c94c8 SOCK
9 ffff880437604600 ffff880410698b40 ffff880076b4f0c0 REG /p3/pgsql/data1/base/16384/16408.1