原创：扣钉日记（微信公众号ID：codelogs），欢迎分享，转载请保留出处。
简介
Linux上有大量的问题诊断工具，如perf、bcc等，但这些诊断工具，虽然功能强大，但却需要很高的权限才可以使用。
而0x.tools这个工具提供了一个很好的思路，通过采样/proc目录来诊断问题，对被测量程序几乎无性能影响，且只要与目标进程拥有同等级的权限，即可正常使用。
不要小看这个权限区别，在互联网大厂，开发同学一般只能获取到一个受限于容器内的shell环境，想要获取机器的root权限几乎是不可能的。
安装
# 下载源码
$ git clone https://github.com/tanelpoder/0xtools.git
# 安装编译器
$ yum install -y make gcc
# 编译并安装程序
$ make && make install实际上0x.tools里的工具大多数是脚本，如psn工具是python脚本，因此直接将代码clone下来，然后执行bin/psn也是可以的。
psn工具
psn工具用来观测系统中当前活跃的线程正在做什么，如线程在做什么系统调用、写什么文件、阻塞在哪个内核函数下？
查看活跃线程
[tanel@linux01 ~]$ psn
Linux Process Snapper v0.18 by Tanel Poder [https://0x.tools]
Sampling /proc/stat for 5 seconds... finished.
=== Active Threads ================================================
samples | avg_threads | comm | state
-------------------------------------------------------------------
10628 | 3542.67 | (kworker/*:*) | Disk (Uninterruptible)
37 | 12.33 | (oracle_*_l) | Running (ON CPU)
17 | 5.67 | (oracle_*_l) | Disk (Uninterruptible)
2 | 0.67 | (xcapture) | Running (ON CPU)
1 | 0.33 | (ora_lg*_xe) | Disk (Uninterruptible)
1 | 0.33 | (ora_lgwr_lin*) | Disk (Uninterruptible)
1 | 0.33 | (ora_lgwr_lin*c) | Disk (Uninterruptible)
samples: 3 (expected: 100)
total processes: 10470, threads: 11530
runtime: 6.13, measure time: 6.03如上，默认情况下，psn采样/proc目录下每个线程的/proc/$pid/stat文件，采样5秒钟，将R(正在运行)或D(不可中断休眠)状态的线程的数据记录下来，并做汇总。
由于R或D状态的线程都是活跃线程，被采样到的次数越多，则越说明这些线程运行得更慢或更频繁。
查看线程读写文件
[tanel@linux01 ~]$ sudo psn -G syscall,filenamesum
Linux Process Snapper v0.18 by Tanel Poder [https://0x.tools]
Sampling /proc/syscall, stat for 5 seconds... finished.
=== Active Threads =======================================================================================================
samples | avg_threads | comm | state | syscall | filenamesum
--------------------------------------------------------------------------------------------------------------------------
2027 | 506.75 | (kworker/*:*) | Disk (Uninterruptible) | [kernel_thread] |
1963 | 490.75 | (oracle_*_l) | Disk (Uninterruptible) | pread64 | /data/oracle/LIN*C/soe_bigfile.dbf
87 | 21.75 | (oracle_*_l) | Running (ON CPU) | [running] |
13 | 3.25 | (kworker/*:*) | Running (ON CPU) | [running] |
4 | 1.00 | (oracle_*_l) | Running (ON CPU) | read | socket:[*]
2 | 0.50 | (collectl) | Running (ON CPU) | [running] |
1 | 0.25 | (java) | Running (ON CPU) | futex |
1 | 0.25 | (ora_ckpt_xe) | Disk (Uninterruptible) | pread64 | /data/oracle/XE/control*.ctl
1 | 0.25 | (ora_m*_linprd) | Running (ON CPU) | [running] |
1 | 0.25 | (ora_m*_lintes) | Running (ON CPU) | [running] |
通过-G可以指定需要查看的列，syscall表示线程正在执行的系统调用，filenamesum表示正在读写的文件，一般来说，线程处于D状态时在做文件io操作，如果D状态线程频繁出现，那么我们肯定想知道线程正在读写哪个文件。
查看线程的内核栈
[tanel@linux01 ~]$ sudo psn -p -G syscall,wchan,kstack
Linux Process Snapper v0.18 by Tanel Poder [https://0x.tools]
Sampling /proc/wchan, stack, syscall, stat for 5 seconds... finished.
=== Active Threads =======================================================================================================================================================================================================================================================================================================================================================================================
samples | avg_threads | comm | state | syscall | wchan | kstack
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
281 | 140.50 | (kworker/*:*) | Disk (Uninterruptible) | [kernel_thread] | blkdev_issue_flush | ret_from_fork_nospec_begin()->kthread()->worker_thread()->process_one_work()->dio_aio_complete_work()->dio_complete()->generic_write_sync()->xfs_file_fsync()->xfs_blkdev_issue_flush()->blkdev_issue_flush()
211 | 105.50 | (kworker/*:*) | Disk (Uninterruptible) | [kernel_thread] | call_rwsem_down_read_failed | ret_from_fork_nospec_begin()->kthread()->worker_thread()->process_one_work()->dio_aio_complete_work()->dio_complete()->generic_write_sync()->xfs_file_fsync()->xfs_ilock()->call_rwsem_down_read_failed()
169 | 84.50 | (oracle_*_li) | Disk (Uninterruptible) | pread64 | call_rwsem_down_write_failed | system_call_fastpath()->SyS_pread64()->vfs_read()->do_sync_read()->xfs_file_aio_read()->xfs_file_dio_aio_read()->touch_atime()->update_time()->xfs_vn_update_time()->xfs_ilock()->call_rwsem_down_write_failed()
64 | 32.00 | (kworker/*:*) | Disk (Uninterruptible) | [kernel_thread] | xfs_log_force_lsn | ret_from_fork_nospec_begin()->kthread()->worker_thread()->process_one_work()->dio_aio_complete_work()->dio_complete()->generic_write_sync()->xfs_file_fsync()->xfs_log_force_lsn()
24 | 12.00 | (oracle_*_li) | Disk (Uninterruptible) | pread64 | call_rwsem_down_read_failed | system_call_fastpath()->SyS_pread64()->vfs_read()->do_sync_read()->xfs_file_aio_read()->xfs_file_dio_aio_read()->__blockdev_direct_IO()->do_blockdev_direct_IO()->xfs_get_blocks_direct()->__xfs_get_blocks()->xfs_ilock_data_map_shared()->xfs_ilock()->call_rwsem_down_read_failed()
5 | 2.50 | (oracle_*_li) | Disk (Uninterruptible) | pread64 | do_blockdev_direct_IO | system_call_fastpath()->SyS_pread64()->vfs_read()->do_sync_read()->xfs_file_aio_read()->xfs_file_dio_aio_read()->__blockdev_direct_IO()->do_blockdev_direct_IO()
3 | 1.50 | (oracle_*_li) | Running (ON CPU) | [running] | 0 | system_call_fastpath()->SyS_pread64()->vfs_read()->do_sync_read()->xfs_file_aio_read()->xfs_file_dio_aio_read()->__blockdev_direct_IO()->do_blockdev_direct_IO()
2 | 1.00 | (kworker/*:*) | Disk (Uninterruptible) | [kernel_thread] | call_rwsem_down_write_failed | ret_from_fork_nospec_begin()->kthread()->worker_thread()->process_one_work()->dio_aio_complete_work()->dio_complete()->xfs_end_io_direct_write()->xfs_iomap_write_unwritten()->xfs_ilock()->call_rwsem_down_write_failed()
2 | 1.00 | (kworker/*:*) | Running (ON CPU) | [running] | 0 | ret_from_fork_nospec_begin()->kthread()->worker_thread()->process_one_work()->dio_aio_complete_work()->dio_complete()->generic_write_sync()->xfs_file_fsync()->xfs_blkdev_issue_flush()->blkdev_issue_flush()
2 | 1.00 | (oracle_*_li) | Disk (Uninterruptible) | io_submit | call_rwsem_down_write_failed | system_call_fastpath()->SyS_io_submit()->do_io_submit()->xfs_file_aio_read()->xfs_file_dio_aio_read()->touch_atime()->update_time()->xfs_vn_update_time()->xfs_ilock()->call_rwsem_down_write_failed()
1 | 0.50 | (java) | Running (ON CPU) | futex | futex_wait_queue_me | system_call_fastpath()->SyS_futex()->do_futex()->futex_wait()->futex_wait_queue_me()
1 | 0.50 | (ksoftirqd/*) | Running (ON CPU) | [running] | 0 | ret_from_fork_nospec_begin()->kthread()->smpboot_thread_fn()
1 | 0.50 | (kworker/*:*) | Disk (Uninterruptible) | [kernel_thread] | worker_thread | ret_from_fork_nospec_begin()->kthread()->worker_thread()
1 | 0.50 | (kworker/*:*) | Disk (Uninterruptible) | [kernel_thread] | worker_thread | ret_from_fork_nospec_begin()->kthread()->worker_thread()->process_one_work()->dio_aio_complete_work()->dio_complete()->generic_write_sync()->xfs_file_fsync()->xfs_blkdev_issue_flush()->blkdev_issue_flush()
1 | 0.50 | (ora_lg*_xe) | Disk (Uninterruptible) | io_submit | inode_dio_wait | system_call_fastpath()->SyS_io_submit()->do_io_submit()->xfs_file_aio_write()->xfs_file_dio_aio_write()->inode_dio_wait()
1 | 0.50 | (oracle_*_li) | Disk (Uninterruptible) | [running] | 0 | - 同理，通过wchan字段可以查看线程阻塞在什么内核方法上，而kstack字段则可以查看线程阻塞时的内核调用栈是什么。
psn的原理
其实psn和ps命令一样，是通过遍历/proc目录来获取线程信息的，如下：
state：取自/proc/$pid/stat文件。
syscall：取自/proc/$pid/syscall文件。
wchan：取自/proc/$pid/wchan文件。
kstack：取自/proc/$pid/stack文件。
与perf、bcc等工具的区别是，读取这些文件只需要与进程同等级的权限即可，不需要使用root账号。
其它工具
除了psn外，0x.tools里面还有一些其它工具，如xcapture、schedlat等，这里就不一一介绍了，感兴趣可以访问 https://0x.tools/ 查看。
另外，由于psn是通过遍历/proc目录实现的，因此我们也可自己编写脚本来实现同样的功能，如下：
active_thread_kstack(){
# 打印当前系统活跃java线程的内核栈
ps h -Lo pid,tid,s,pcpu,comm,wchan:32,min_flt,maj_flt -C java|grep '[RD] '| awk '
BEGIN{
syscall_files["/usr/include/asm/unistd_64.h"]=1;
syscall_files["/usr/include/x86_64-linux-gnu/asm/unistd_64.h"]=1;
syscall_files["/usr/include/asm-x86_64/unistd.h"]=1;
for(tfile in syscall_files){
cmd="test -f "tfile
if(system(cmd)==0){
hfile=tfile;
break;
}
}
if(hfile){
while (getline 这样，我们不用安装0x.tools，就也能得到类似于psn命令的功能了！
往期内容
Linux命令拾遗-入门篇
Linux命令拾遗-文本处理篇
Linux命令拾遗-软件资源观测
mysql的timestamp会存在时区问题？
真正理解可重复读事务隔离级别
字符编码解惑