rm: cannot remove `d': Read-only file system解决过程

rhel5.7出现cannot remove `db0tktpvdvu_4020_1.bak': Read-only file system，权限是没有问题的，怀疑是硬盘有损坏了

[root@qht108 qht108]# ls
c-1194563790-20190216-00  c-1194563790-20190216-01  db0tktpvdvu_4020_1.bak
[root@qht108 qht108]# rm *.bak
rm: remove regular file `db0tktpvdvu_4020_1.bak'? y
rm: cannot remove `db0tktpvdvu_4020_1.bak': Read-only file system

/etc/mtab和/proc/mounts显示的结果不同，/proc/mounts是实际mount的情况

[root@qht108 qht108]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1               20G  5.6G   14G  30% /
/dev/md3              1.8T  847G  872G  50% /u01
/dev/md0               92M   18M   69M  21% /boot
tmpfs                 2.0G  1.6G  439M  78% /dev/shm
/dev/sdc1             1.8T  1.3T  444G  75% /u02
[root@qht108 qht108]# cat /etc/mtab
/dev/md1 / ext3 rw 0 0
proc /proc proc rw 0 0
sysfs /sys sysfs rw 0 0
devpts /dev/pts devpts rw,gid=5,mode=620 0 0
/dev/md3 /u01 ext3 rw 0 0
/dev/md0 /boot ext3 rw 0 0
tmpfs /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
/dev/sdc1 /u02 ext3 rw 0 0
[root@qht108 qht108]# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw,data=ordered 0 0
/dev /dev tmpfs rw 0 0
/proc /proc proc rw 0 0
/sys /sys sysfs rw 0 0
/proc/bus/usb /proc/bus/usb usbfs rw 0 0
devpts /dev/pts devpts rw 0 0
/dev/md3 /u01 ext3 rw,data=ordered 0 0
/dev/md0 /boot ext3 rw,data=ordered 0 0
tmpfs /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
/etc/auto.misc /misc autofs rw,fd=7,pgrp=2585,timeout=300,minproto=5,maxproto=5,indirect 0 0
-hosts /net autofs rw,fd=13,pgrp=2585,timeout=300,minproto=5,maxproto=5,indirect 0 0
/dev/sdc1 /u02 ext3 ro,data=ordered 0 0

这里可以看出sdc1系统在启动之后由rw权限变成了ro权限，所以造成了文件不能删除的问题。

也就是说很有可能磁盘文件被破坏了或者磁盘有损坏。然后使用命令dmesg查看具体情况：

__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
__journal_remove_journal_head: freeing b_committed_data
ext3_abort called.
EXT3-fs error (device sdc1): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only

尝试用fsck -y自动修复

[root@qht108 qht108]# fsck -y /dev/sdc1
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
/dev/sdc1 is mounted.

WARNING!!!  Running e2fsck on a mounted filesystem may cause
SEVERE filesystem damage.

Do you really want to continue (y/n)? yes


/dev/sdc1: recovering journal
/dev/sdc1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes

结束后需要将此坏硬件重新mount一下，如果是系统盘的话需要重启一下。

成功删除了文件！

[root@qht108 ~]# umount /u02
[root@qht108 ~]# mount /dev/sdc1 /u02
[root@qht108 ~]# cd /u02/backup/rman/qht108
[root@qht108 qht108]# rm *.bak