linux芯片设计工作站,最近linux工作站闹“机瘟”，故障处理过程
一台服务器重启后smtp报警，dmesg查看有以下信息，
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
EXT3-fs error (device sda6) in start_transaction: Journal has aborted
明显是sda6分区文件系统损坏了，退出所有调用/dev/sda6分区的进程。可以通过以下命令查看：
fuser -vm /dev/sda6
也可通过 fuser -kvm /dev/sda6直接强制关闭调用此分区的进程
再umount /dev/sda6把分区卸载下来，要不强制修复会有问题。
开始修复：
[root@localhost ~]#fsck -C -y /dev/sda6
fsck 1.35 (28-Feb-2004)
e2fsck 1.35 (28-Feb-2004)
/mail: recovering journal
/mail contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inodes that were part of a corrupted orphan linked list found. Fix? yes
Inode 11567146 was part of the orphaned inode list. FIXED.
Deleted inode 88002036 has zero dtime. Fix? yes
Inode 142868618, i_blocks is 336, should be 280. Fix? yes
Duplicate blocks found... invoking duplicate block passes.
Pass 1B: Rescan for duplicate/bad blocks
Duplicate/bad block(s) in inode 140690903: 300506198 300531891 301339371 301339372 301339373 301339374 301339375 301339376 301339377 301339378 301339388 301339389 301339955 301339981 301340019 301340020 301340021 301340083 301340084 301340085 301340086 301340087 301340088 301340089 301340090 301340091 301340092 301340093 301340094 301340095 301340096 301340097 301340098 301340099 301340100
Duplicate/bad block(s) in inode 142868618: 300506198 300531891 301339371 301339372 301339373 301339374 301339375 301339376 301339377 301339378 301339388 301339389 301339955 301339981 301340019 301340020 301340021 301340083 301340084 301340085 301340086 301340087 301340088 301340089 301340090 301340091 301340092 301340093 301340094 301340095 301340096 301340097 301340098 301340099 301340100
Pass 1C: Scan directories for inodes with dup blocks.
Pass 1D: Reconciling duplicate blocks
(There are 2 inodes containing duplicate/bad blocks.)
File /l/libotic.com.cn/l/lwg/cur/1341374309.M886694P10938V0000000000000806I000000000862C5D7_0.mail.mil6.c4e.com,S=135350:2,S (inode #140690903, mod time Wed Jul 4 11:58:30 2012)
has 35 duplicate block(s), shared with 1 file(s):
/l/lnb-ch.com/j/jian_pan/.wbaillist_inbox (inode #142868618, mod time Wed Jul 4 11:25:19 2012)
Clone duplicate/bad blocks? yes
File /l/ln-chi.com/j/jian_pan/.weailit_inbox (inode #142868618, mod time Wed Jul 4 11:25:19 2012)
has 35 duplicate block(s), shared with 1 file(s):
/l/liastic.co.c/l/lwg/cur/1341374309.M886694P10938V0000000000000806I000000000862C5D7_0.ml.ml63.c4e.com,S=135350:2,S (inode #140690903, mod time Wed Jul 4 11:58:30 2012)
Duplicated blocks already reassigned or cloned.
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Unattached inode 153240106
Connect to /lost+found? yes
Inode 153240106 ref count is 2, should be 1. Fix? yes
Inode 153240109 ref count is 1, should be 2. Fix? yes
Unattached inode 153240112
Connect to /lost+found? yes
Inode 153240112 ref count is 2, should be 1. Fix? yes
Inode 153240116 ref count is 1, should be 2. Fix? yes
Inode 153240117 ref count is 1, should be 2. Fix? yes
Unattached inode 153240118
Connect to /lost+found? yes
Inode 153240118 ref count is 2, should be 1. Fix? yes
Inode 153240121 ref count is 1, should be 2. Fix? yes
Inode 153240123 ref count is 1, should be 2. Fix? yes
Pass 5: Checking group summary information
Block bitmap differences: -(23153596--23153599) -(51560568--51560573) +(51560580--51560581) +(51560584--51560588) -(51560589--51560591) -(51560593--51560601) -(51560604--51560606) +(51560608--51560619) +(51560648--51560659) -(51560660--51560667) -51560670 -51560724 -(51560726--51560739) +(51560740--51560747) +(51560752--51560757) -176108310 -286972966 -(301339956--301339980) -301340028 -(301340030--301340032) -(301340034--301340037) -(301340070--301340072) -390707346 -390707348 +(390707349--390707350)
Fix? yes
Free blocks count wrong for group #3 (13089, counted=13054).
Fix? yes
Free blocks count wrong for group #706 (9481, counted=9485).
Fix? yes
Free blocks count wrong for group #5374 (14575, counted=14576).
Fix? yes
Free blocks count wrong for group #8720 (0, counted=1).
Fix? yes
Free blocks count wrong for group #8757 (1, counted=2).
Fix? yes
Free blocks count wrong for group #8760 (0, counted=1).
Fix? yes
Free blocks count wrong for group #8774 (0, counted=2).
Fix? yes
Free blocks count wrong for group #9134 (0, counted=1).
Fix? yes
Free blocks count wrong for group #9196 (17050, counted=17086).
Fix? yes
Free blocks count wrong (174274933, counted=174274945).
Fix? yes
Inode bitmap differences: -11567146 -88002036
Fix? yes
Free inodes count wrong for group #706 (16314, counted=16315).
Fix? yes
Free inodes count wrong for group #5371 (11390, counted=11391).
Fix? yes
Free inodes count wrong (226927033, counted=226927035).
Fix? yes
/mail: ***** FILE SYSTEM WAS MODIFIED *****
/mail: 3268165/230195200 files (1.5% non-contiguous), 286097744/460372689 blocks
1.5T的数据，用了4小时10分钟修复完毕，修复完成后挂载上去恢复正常使用。