su user oracle does not exist,【案例】Linux文件系统无法mount 由superblock损坏导致

【案例】Linux文件系统无法mount 由superblock损坏导致

时间:2016-12-19 10:33   来源:Oracle研究中心   作者:网络   点击:

次

天萃荷净

Oracle研究中心案例分析：运维DBA反映Oracle数据库遇到linux方面，文件系统mount不了的问题，说是superblock损坏，要进行修复，我这里用vm来进行模拟下，以及进行手工修复。

[root@killdb ~]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda2             3.9G  3.3G  384M  90% /

/dev/sda1              46M   11M   33M  25% /boot

none                  506M     0  506M   0% /dev/shm

/dev/sda3              25G   23G  1.4G  95% /home

[root@killdb ~]# tune2fs -l /dev/sda3

tune2fs 1.35 (28-Feb-2004)

Filesystem volume name:   /home

Last mounted on:

Filesystem UUID:          d3d8a85b-a06f-4765-84c7-62c5d1257151

Filesystem magic number:  0xEF53

Filesystem revision #:    1 (dynamic)

Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file

Default mount options:    (none)

Filesystem state:         clean

Errors behavior:          Continue

Filesystem OS type:       Linux

Inode count:              3286752

Block count:              6568576

Reserved block count:     328428

Free blocks:              2454102

Free inodes:              3184193

First block:              0

Block size:               4096

Fragment size:            4096

Reserved GDT blocks:      1024

Blocks per group:         32768

Fragments per group:      32768

Inodes per group:         16352

Inode blocks per group:   511

Filesystem created:       Sat Sep 24 11:48:16 2011

Last mount time:          Sat Oct  6 00:02:31 2012

Last write time:          Sat Oct  6 00:02:31 2012

Mount count:              86

Maximum mount count:      -1

Last checked:             Sat Sep 24 11:48:16 2011

Check interval:           0 ()

Reserved blocks uid:      0 (user root)

Reserved blocks gid:      0 (group root)

First inode:              11

Inode size:               128

Journal inode:            8

First orphan inode:       1439427

Default directory hash:   tea

Directory Hash Seed:      63cb625d-3db5-45ff-ac6f-fa35dc01e088

Journal backup:           inode blocks

[root@killdb ~]# dumpe2fs /dev/sda3

dumpe2fs 1.35 (28-Feb-2004)

Filesystem volume name:   /home

Last mounted on:

Filesystem UUID:          d3d8a85b-a06f-4765-84c7-62c5d1257151

Filesystem magic number:  0xEF53

Filesystem revision #:    1 (dynamic)

Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file

Default mount options:    (none)

Filesystem state:         clean

Errors behavior:          Continue

Filesystem OS type:       Linux

Inode count:              3286752

Block count:              6568576

Reserved block count:     328428

Free blocks:              2454102

Free inodes:              3184193

First block:              0

Block size:               4096

Fragment size:            4096

Reserved GDT blocks:      1024

Blocks per group:         32768

Fragments per group:      32768

Inodes per group:         16352

Inode blocks per group:   511

Filesystem created:       Sat Sep 24 11:48:16 2011

Last mount time:          Sat Oct  6 00:02:31 2012

Last write time:          Sat Oct  6 00:02:31 2012

Mount count:              86

Maximum mount count:      -1

Last checked:             Sat Sep 24 11:48:16 2011

Check interval:           0 ()

Reserved blocks uid:      0 (user root)

Reserved blocks gid:      0 (group root)

First inode:              11

Inode size:               128

Journal inode:            8

First orphan inode:       1439427

Default directory hash:   tea

Directory Hash Seed:      63cb625d-3db5-45ff-ac6f-fa35dc01e088

Journal backup:           inode blocks

Group 0: (Blocks 0-32767)

Primary superblock at 0, Group descriptors at 1-2

Block bitmap at 1027 (+1027), Inode bitmap at 1028 (+1028)

Inode table at 1029-1539 (+1029)

0 free blocks, 14770 free inodes, 237 directories

Free blocks:

Free inodes: 727, 944-951, 1592-16352

Group 1: (Blocks 32768-65535)

Backup superblock at 32768, Group descriptors at 32769-32770

Block bitmap at 33795 (+1027), Inode bitmap at 33796 (+1028)

Inode table at 33797-34307 (+1029)

0 free blocks, 11741 free inodes, 270 directories

Free blocks:

Free inodes: 17277-17287, 17828-17836, 20828-20831, 20946, 20948, 20965, 20991-32704

Group 2: (Blocks 65536-98303)

Block bitmap at 65536 (+0), Inode bitmap at 65537 (+1)

Inode table at 65538-66048 (+2)

0 free blocks, 14821 free inodes, 226 directories

Free blocks:

Free inodes: 34236-49056

Group 3: (Blocks 98304-131071)

Backup superblock at 98304, Group descriptors at 98305-98306

Block bitmap at 99331 (+1027), Inode bitmap at 99332 (+1028)

Inode table at 99333-99843 (+1029)

0 free blocks, 16352 free inodes, 0 directories

Free blocks:

Free inodes: 49057-65408

Group 4: (Blocks 131072-163839)

Block bitmap at 131072 (+0), Inode bitmap at 131073 (+1)

Inode table at 131074-131584 (+2)

0 free blocks, 16341 free inodes, 1 directories

Free blocks:

Free inodes: 65416-65417, 65422-81760

Group 5: (Blocks 163840-196607)

Backup superblock at 163840, Group descriptors at 163841-163842

Block bitmap at 164867 (+1027), Inode bitmap at 164868 (+1028)

Inode table at 164869-165379 (+1029)

0 free blocks, 11588 free inodes, 289 directories

Free blocks:

Free inodes: 86525-98112

Group 6: (Blocks 196608-229375)

Block bitmap at 196608 (+0), Inode bitmap at 196609 (+1)

Inode table at 196610-197120 (+2)

3978 free blocks, 16351 free inodes, 0 directories

Free blocks: 210922-212834, 212836-212842, 213344-215401

Free inodes: 98113-98251, 98253-114464

Group 7: (Blocks 229376-262143)

Backup superblock at 229376, Group descriptors at 229377-229378

Block bitmap at 230403 (+1027), Inode bitmap at 230404 (+1028)

Inode table at 230405-230915 (+1029)

22495 free blocks, 16352 free inodes, 0 directories

Free blocks: 230916-230917, 230920-230925, 237568-238068, 240137-240192, 240200-242816, 242824-244800, Oracleoracleplus.net244808-262143

Free inodes: 114465-130816

。。。。。。

Free inodes: 3239782-3254048

Group 199: (Blocks 6520832-6553599)

Block bitmap at 6520832 (+0), Inode bitmap at 6520833 (+1)

Inode table at 6520834-6521344 (+2)

0 free blocks, 11925 free inodes, 213 directories

Free blocks:

Free inodes: 3258210-3258212, 3258215-3258220, 3258222-3258230, 3258494-3270400

Group 200: (Blocks 6553600-6568575)

Block bitmap at 6553600 (+0), Inode bitmap at 6553601 (+1)

Inode table at 6553602-6554112 (+2)

2 free blocks, 16348 free inodes, 0 directories

Free blocks: 6554326-6554327

Free inodes: 3270405-3286752

[root@killdb ~]# dumpe2fs /dev/sda3|grep superblock

dumpe2fs 1.35 (28-Feb-2004)

Primary superblock at 0, Group descriptors at 1-2

Backup superblock at 32768, Group descriptors at 32769-32770

Backup superblock at 98304, Group descriptors at 98305-98306

Backup superblock at 163840, Group descriptors at 163841-163842

Backup superblock at 229376, Group descriptors at 229377-229378

Backup superblock at 294912, Group descriptors at 294913-294914

Backup superblock at 819200, Group descriptors at 819201-819202

Backup superblock at 884736, Group descriptors at 884737-884738

Backup superblock at 1605632, Group descriptors at 1605633-1605634

Backup superblock at 2654208, Group descriptors at 2654209-2654210

Backup superblock at 4096000, Group descriptors at 4096001-4096002

从上面的信息可以得出如下结论：

1. /dev/sda3 blocksize是4k

2. 第一个主超级块在第2个block上;

3. /dev/sda3文件系统除了一个主超级块之外，还有10个备份的superblock,

下面我们来模拟超级坏块损坏的情况：

[root@killdb /]# umount /home

umount: /home: device is busy

umount: /home: device is busy

[root@killdb /]# ps -ef|grep pmon

ora10g     851     1  0 16:52          00:00:01 ora_pmon_roger

root     10194  9421  0 21:43 pts/4    00:00:00 grep pmon

[root@killdb /]# su - oracle

su: user oracle does not exist

[root@killdb /]# su - ora10g

[ora10g@killdb ~]$ sqlplus "/as sysdba"

SQL*Plus: Release 10.2.0.5.0 - Production on Thu Nov 1 21:43:54 2012

Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.

Connected to:

Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - Production

With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> shutdown immediate

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> exit

Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - Production

With the Partitioning, OLAP, Data Mining and Real Application Testing options

[ora10g@killdb ~]$ exit

logout

[root@killdb /]# umount /home

[root@killdb /]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda2             3.9G  3.3G  384M  90% /

/dev/sda1              46M   11M   33M  25% /boot

none                  506M     0  506M   0% /dev/shm

[root@killdb /]# dd if=/dev/zero  of=/dev/sda3 bs=4096 skip=1 count=1

1+0 records in

1+0 records out

[root@killdb /]#  mount -t ext3 /dev/sda3 /home

mount: wrong fs type, bad option, bad superblock on /dev/sda3,

or too many mounted file systems

提示超级坏块损坏，我们利用后面备份的超级坏块来进行修复：

[root@killdb /]# fsck.ext3 -b 32768 /dev/sda3

e2fsck 1.35 (28-Feb-2004)

Clearing orphaned inode 1439497 (uid=501, gid=501, mode=040755, size=0)

Illegal inode 1350886196 in orphaned inode list.

/home was not cleanly unmounted, check forced.

Pass 1: Checking inodes, blocks, and sizes

Pass 2: Checking directory structure

Pass 3: Checking directory connectivity

Pass 4: Checking reference counts

Pass 5: Checking group summary information

Free blocks count wrong for group #6 (3983, counted=3978).

Fix  yes

Free blocks count wrong for group #7 (22516, counted=22495).

Fix  yes

Free blocks count wrong for group #11 (31807, counted=31806).

Fix  yes

Free blocks count wrong for group #12 (32241, counted=31954).

Fix  yes

Free blocks count wrong for group #13 (32241, counted=31759).

Fix  yes

Free blocks count wrong for group #14 (32248, counted=0).

Fix  yes

Free blocks count wrong for group #15 (32247, counted=0).

Fix  yes

Free blocks count wrong for group #16 (32241, counted=0).

Fix  yes

Free blocks count wrong for group #17 (26822, counted=0).

Fix  yes

Free blocks count wrong for group #18 (22528, counted=0).

Fix  yes

Free blocks count wrong for group #19 (20424, counted=0).

Fix  yes

Free blocks count wrong for group #20 (32119, counted=0).

Fix  yes

Free blocks count wrong for group #21 (32253, counted=0).

Fix  yes

Free blocks count wrong for group #22 (22983, counted=0).

Fix  yes

Free blocks count wrong for group #23 (30372, counted=0).

Fix  yes

Free blocks count wrong for group #24 (32254, counted=0).

Fix  yes

Free blocks count wrong for group #25 (31228, counted=0).

Fix  yes

Free blocks count wrong for group #26 (32255, counted=0).

Fix  yes

Free blocks count wrong for group #27 (31225, counted=0).

Fix  yes

Free blocks count wrong for group #28 (32251, counted=0).

Fix  yes

Free blocks count wrong for group #29 (32244, counted=0).

Fix  yes

Free blocks count wrong for group #30 (32251, counted=0).

Fix  yes

Free blocks count wrong for group #31 (19972, counted=0).

Fix  yes

Free blocks count wrong for group #32 (16403, counted=0).

Fix  yes

Free blocks count wrong for group #33 (253, counted=0).

Fix  yes

Free blocks count wrong for group #34 (19706, counted=0).

Fix  yes

Free blocks count wrong for group #35 (13924, counted=0).

Fix  yes

Free blocks count wrong for group #36 (23864, counted=0).

Fix  yes

Free blocks count wrong for group #37 (21452, counted=0).

Fix  yes

Free blocks count wrong for group #38 (22105, counted=0).

Fix  yes

Free blocks count wrong for group #40 (11888, counted=0).

Fix  yes

Free blocks count wrong for group #41 (19443, counted=0).

Fix  yes

Free blocks count wrong for group #42 (23458, counted=0).

Fix  yes

Free blocks count wrong for group #43 (13582, counted=0).

Fix  yes

Free blocks count wrong for group #50 (9, counted=0).

Fix  yes

Free blocks count wrong for group #52 (7315, counted=0).

Fix  yes

Free blocks count wrong for group #53 (28773, counted=0).

Fix  yes

Free blocks count wrong for group #54 (6, counted=0).

Fix  yes

Free blocks count wrong for group #57 (7776, counted=0).

Fix  yes

Free blocks count wrong for group #58 (32247, counted=0).

Fix  yes

Free blocks count wrong for group #59 (32109, counted=0).

Fix  yes

Free blocks count wrong for group #60 (31922, counted=0).

Fix  yes

Free blocks count wrong for group #61 (31851, counted=0).

Fix  yes

Free blocks count wrong for group #62 (24979, counted=0).

Fix  yes

Free blocks count wrong for group #63 (32249, counted=0).

Fix  yes

Free blocks count wrong for group #64 (1201, counted=0).

Fix  yes

Free blocks count wrong for group #66 (18430, counted=12).

Fix  yes

Free blocks count wrong for group #67 (7196, counted=0).

Fix  yes

Free blocks count wrong for group #68 (24941, counted=1663).

Fix  yes

Free blocks count wrong for group #71 (32241, counted=32175).

Fix  yes

Free blocks count wrong for group #73 (23876, counted=23873).

Fix  yes

Free blocks count wrong for group #83 (23843, counted=23840).

Fix  yes

Free blocks count wrong for group #88 (923, counted=5710).

Fix  yes

Free blocks count wrong for group #89 (26, counted=5688).

Fix  yes

Free blocks count wrong for group #92 (0, counted=21).

Fix  yes

Free blocks count wrong for group #93 (7076, counted=24401).

Fix  yes

Free blocks count wrong for group #94 (4187, counted=4179).

Fix  yes

Free blocks count wrong for group #95 (5511, counted=5495).

Fix  yes

Free blocks count wrong for group #97 (94, counted=85).

Fix  yes

Free blocks count wrong for group #98 (204, counted=173).

Fix  yes

Free blocks count wrong for group #99 (6199, counted=6198).

Fix  yes

Free blocks count wrong for group #104 (18306, counted=18143).

Fix  yes

Free blocks count wrong for group #105 (29955, counted=29904).

Fix  yes

Free blocks count wrong for group #106 (7376, counted=7362).

Fix  yes

Free blocks count wrong for group #109 (9, counted=0).

Fix  yes

Free blocks count wrong for group #163 (4, counted=0).

Fix  yes

Free blocks count wrong for group #164 (7333, counted=0).

Fix  yes

Free blocks count wrong for group #165 (32114, counted=0).

Fix  yes

Free blocks count wrong for group #166 (32183, counted=0).

Fix  yes

Free blocks count wrong for group #167 (30440, counted=0).

Fix  yes

Free blocks count wrong for group #168 (32240, counted=0).

Fix  yes

Free blocks count wrong for group #169 (27021, counted=0).

Fix  yes

Free blocks count wrong for group #170 (32233, counted=16).

Fix  yes

Free blocks count wrong for group #171 (32233, counted=0).

Fix  yes

Free blocks count wrong for group #172 (32241, counted=0).

Fix  yes

Free blocks count wrong for group #173 (32219, counted=0).

Fix  yes

Free blocks count wrong for group #174 (32248, counted=0).

Fix  yes

Free blocks count wrong for group #175 (32228, counted=0).

Fix  yes

Free blocks count wrong for group #176 (31762, counted=0).

Fix  yes

Free blocks count wrong for group #177 (31965, counted=0).

Fix  yes

Free blocks count wrong for group #178 (29609, counted=0).

Fix  yes

Free blocks count wrong for group #179 (31979, counted=0).

Fix  yes

Free blocks count wrong for group #180 (31702, counted=0).

Fix  yes

Free blocks count wrong for group #181 (24876, counted=0).

Fix  yes

Free blocks count wrong for group #182 (31551, counted=0).

Fix  yes

Free blocks count wrong for group #183 (32255, counted=0).

Fix  yes

Free blocks count wrong for group #184 (32243, counted=0).

Fix  yes

Free blocks count wrong for group #185 (31842, counted=0).

Fix  yes

Free blocks count wrong for group #186 (32170, counted=0).

Fix  yes

Free blocks count wrong for group #187 (32252, counted=1).

Fix  yes

Free blocks count wrong for group #188 (32249, counted=0).

Fix  yes

Free blocks count wrong for group #189 (32253, counted=14).

Fix  yes

Free blocks count wrong for group #190 (25549, counted=2960).

Fix  yes

Free blocks count wrong for group #192 (32252, counted=32009).

Fix  yes

Free blocks count wrong (2454102, counted=671157).

Fix  yes

Free inodes count wrong for group #14 (16352, counted=16351).

Fix  yes

Free inodes count wrong for group #54 (16327, counted=16336).

Fix  yes

Free inodes count wrong for group #59 (16203, counted=16158).

Fix  yes

Free inodes count wrong for group #60 (16026, counted=16017).

Fix  yes

Free inodes count wrong for group #64 (16146, counted=16132).

Fix  yes

Free inodes count wrong for group #66 (16352, counted=16343).

Fix  yes

Free inodes count wrong for group #71 (16352, counted=16351).

Fix  yes

Free inodes count wrong for group #73 (16344, counted=16338).

Fix  yes

Free inodes count wrong for group #88 (12512, counted=12501).

Fix  yes

Directories count wrong for group #88 (309, counted=312).

Fix  yes

Free inodes count wrong for group #90 (16341, counted=16343).

Fix  yes

Free inodes count wrong for group #109 (14179, counted=14137).

Fix  yes

Directories count wrong for group #109 (345, counted=346).

Fix  yes

Free inodes count wrong for group #110 (16348, counted=16339).

Fix  yes

Free inodes count wrong for group #170 (16351, counted=15971).

Fix  yes

Directories count wrong for group #170 (0, counted=7).

Fix  yes

Free inodes count wrong (3184192, counted=3183676).

Fix  yes

/home: ***** FILE SYSTEM WAS MODIFIED *****

/home: 103076/3286752 files (1.3% non-contiguous), 5897419/6568576 blocks

修复完成之后，我们来挂载文件系统，是否能够正常mount：

[root@killdb /]#  mount -t ext3 /dev/sda3 /home

[root@killdb /]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda2             3.9G  3.3G  384M  90% /

/dev/sda1              46M   11M   33M  25% /boot

none                  506M     0  506M   0% /dev/shm

/dev/sda3              25G   23G  1.4G  95% /home

最后来挂载oracle数据库看看是否正常。

[root@killdb /]# su - ora10g

s[ora10g@killdb ~]$ sqlplus "/as sysdba"

SQL*Plus: Release 10.2.0.5.0 - Production on Thu Nov 1 21:57:46 2012

Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.

Connected to an idle instance.

SQL> !pwd

/home/ora10g

SQL> startup

ORACLE instance started.

Total System Global Area  167772160 bytes

Fixed Size                  1272600 bytes

Variable Size             146801896 bytes

Database Buffers           16777216 bytes

Redo Buffers                2920448 bytes

Database mounted.

Database opened.

SQL> select open_mode from V$database;

OPEN_MODE

----------

READ WRITE

一切正常。

补充：

这里有篇文章关于superblock的结构进行了详细描述：

http://homepage.smc.edu/morgan_david/cs40/analyze-ext2.htm

--------------------------------------ORACLE-DBA----------------------------------------

最权威、专业的Oracle案例资源汇总之【案例】Linux文件系统无法mount 由superblock损坏导致