在数据库正常运行时,运维人员在无意中将部分数据文件删除了,此时数据库管理员并不知道,且数据库运行正常,并没有立即抛出错误和告警;但是开发人员在对某张表进行更新的时候,正好这张表在被删除的数据文件中,报出ORA-01110和ORA-27041错误。随即数据库管理人员来看现象,发现有一个文件已经从系统层面删除了,并且数据库数据库没有进行重启操作,所以在没有进行更新的时候业务并没有造成影响。
非常幸运的是,在数据文件删除的情况下,数据库没有重启,使得这次的恢复变的较为简单,基于这次的恢复,我在我得实验设备上进行了重演,记录了完整的数据文件删除和恢复的完整步骤,如下:
SQL> create tablespace woo datafile
2 '/u01/app/oracle/oradata/PROD1/woo01.dbf'
3 size 20m autoextend on;
Tablespace created.
SQL> select name from v$dbfile;
NAME
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/PROD1/users01.dbf
/u01/app/oracle/oradata/PROD1/undotbs01.dbf
/u01/app/oracle/oradata/PROD1/sysaux01.dbf
/u01/app/oracle/oradata/PROD1/system01.dbf
/u01/app/oracle/oradata/PROD1/example01.dbf
/u01/app/oracle/oradata/PROD1/tools.dbf
/u01/app/oracle/oradata/PROD1/test.dbf
/u01/app/oracle/oradata/PROD1/woo01.dbf
8 rows selected.
SQL> !rm -rf /u01/app/oracle/oradata/PROD1/woo01.dbf
SQL> !tail -100f /u01/app/oracle/diag/rdbms/prod1/PROD1/trace/alert*
Wed Aug 26 14:31:01 2015
create tablespace woo datafile
'/u01/app/oracle/oradata/PROD1/woo01.dbf'
size 20m autoextend on
Completed: create tablespace woo datafile
'/u01/app/oracle/oradata/PROD1/woo01.dbf'
size 20m autoextend on
#在这里我们可以看到数据文件在系统层面被删除之后数据库并没产生告警。
SQL> create table test tablespace woo as select * from dba_users;
create table test tablespace woo as select * from dba_users
*
ERROR at line 1:
ORA-01116: error in opening database file 8
ORA-01110: data file 8: '/u01/app/oracle/oradata/PROD1/woo01.dbf'
ORA-27041: unable to open file
Linux Error: 2: No such file or directory
Additional information: 3
#而使在我们需要往这个删除的文件中写入数据的时候才发现数据文件被删除了。
SQL> !tail -50 /u01/app/oracle/diag/rdbms/prod1/PROD1/trace/alert*
Wed Aug 26 14:34:33 2015
Errors in file /u01/app/oracle/diag/rdbms/prod1/PROD1/trace/PROD1_smon_9564.trc:
ORA-01116: error in opening database file 8
ORA-01110: data file 8: '/u01/app/oracle/oradata/PROD1/woo01.dbf'
ORA-27041: unable to open file
Linux Error: 2: No such file or directory
Additional information: 3
Wed Aug 26 14:34:33 2015
Checker run found 1 new persistent data failures
Wed Aug 26 14:39:44 2015
Errors in file /u01/app/oracle/diag/rdbms/prod1/PROD1/trace/PROD1_m000_14015.trc:
ORA-01116: error in opening database file 8
ORA-01110: data file 8: '/u01/app/oracle/oradata/PROD1/woo01.dbf'
ORA-27041: unable to open file
Linux Error: 2: No such file or directory
Additional information: 3
SQL> select instance_name,status from v$instance;
INSTANCE_NAME STATUS
---------------- ------------
PROD1 OPEN
#即便数据库已经知道了数据库文件丢失,因为不是系统表空间的数据文件,所以数据库的运行并没有收到影响。
因为数据库并没有停止运行,这个时候我们可以通过dbwr写数据文件进程来找到进程句柄号,进入该句柄号就可以找到该进程锁定的相关数据文件了。
SQL> !ps -ef|grep dbw|grep -v grep
oracle 9554 1 0 07:16 ? 00:00:01 ora_dbw0_PROD1
#我们可以看到进程的ID为9554,通过进程ID查找到进程下锁定的所有文件
SQL> !ls -rtl /proc/9554/fd
[oracle@edbjr2p1 trace]$ ls -rtl /proc//9554/fd
total 0
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 0 -> /dev/null
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 9 -> /dev/null
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 8 -> /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 7 -> /dev/null
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 6 -> /u01/app/oracle/product/11.2.0/dbhome_1/hpatch/orapatchPROD1.cfg
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 5 -> /u01/app/oracle/product/11.2.0/dbhome_1/dbs/hc_PROD1.dat
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 4 -> /dev/null
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 3 -> /dev/null
l-wx------ 1 oracle oinstall 64 Aug 26 14:37 2 -> /dev/null
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 19 -> /u01/app/oracle/product/11.2.0/dbhome_1/hpatch/orapatchPROD1.cfg
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 18 -> /proc/9554/fd
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 17 -> /u01/app/oracle/product/11.2.0/dbhome_1/rdbms/mesg/oraus.msb
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 16 -> /u01/app/oracle/product/11.2.0/dbhome_1/dbs/hc_PROD1.dat
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 15 -> /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 14 -> /dev/zero
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 13 -> /dev/zero
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 12 -> /u01/app/oracle/product/11.2.0/dbhome_1/hpatch/orapatchPROD1.cfg
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 11 -> /dev/null
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 10 -> /dev/null
l-wx------ 1 oracle oinstall 64 Aug 26 14:37 1 -> /dev/null
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 268 -> /u01/app/oracle/oradata/PROD1/woo01.dbf (deleted)//我们找到了这个文件,处于deleted
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 267 -> /u01/app/oracle/oradata/PROD1/temp2.dbf
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 266 -> /u01/app/oracle/oradata/PROD1/temp1.dbf
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 265 -> /u01/app/oracle/oradata/PROD1/temp01.dbf
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 264 -> /u01/app/oracle/oradata/PROD1/test.dbf
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 263 -> /u01/app/oracle/oradata/PROD1/tools.dbf
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 262 -> /u01/app/oracle/oradata/PROD1/example01.dbf
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 261 -> /u01/app/oracle/oradata/PROD1/users01.dbf
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 260 -> /u01/app/oracle/oradata/PROD1/undotbs01.dbf
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 259 -> /u01/app/oracle/oradata/PROD1/sysaux01.dbf
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 258 -> /u01/app/oracle/oradata/PROD1/system01.dbf
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 257 -> /u01/app/oracle/oradata/PROD1/control02.ctl
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 256 -> /u01/app/oracle/oradata/PROD1/control01.ctl
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 24 -> /u01/app/oracle/product/11.2.0/dbhome_1/rdbms/mesg/oraus.msb
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 23 -> /u01/app/oracle/product/11.2.0/dbhome_1/dbs/lkPROD1
lrwx------ 1 oracle oinstall 64 Aug 26 14:37 22 -> /u01/app/oracle/product/11.2.0/dbhome_1/dbs/hc_PROD1.dat
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 21 -> /u01/app/oracle/product/11.2.0/dbhome_1/bin/oracle
lr-x------ 1 oracle oinstall 64 Aug 26 14:37 20 -> /dev/zero
lrwx------ 1 oracle oinstall 64 Aug 26 15:09 25 -> socket:[173283]
SQL> !cp /proc//9554/fd/268 /u01/app/oracle/oradata/PROD1/woo01.dbf
SQL> col name format a50
SQL> select file#,status,name from v$datafile;
FILE# STATUS NAME
---------- ------- --------------------------------------------------
1 SYSTEM /u01/app/oracle/oradata/PROD1/system01.dbf
2 ONLINE /u01/app/oracle/oradata/PROD1/sysaux01.dbf
3 ONLINE /u01/app/oracle/oradata/PROD1/undotbs01.dbf
4 ONLINE /u01/app/oracle/oradata/PROD1/users01.dbf
5 ONLINE /u01/app/oracle/oradata/PROD1/example01.dbf
6 ONLINE /u01/app/oracle/oradata/PROD1/tools.dbf
7 ONLINE /u01/app/oracle/oradata/PROD1/test.dbf
8 ONLINE /u01/app/oracle/oradata/PROD1/woo01.dbf #我们可以看到这个时候该数据文件是ONLINE状态
8 rows selected.
#由于这是一套在线库,且有其它业务,不可随意停机,所以这个时候将需要恢复的数据文件offline,就可以直接在线执行恢复了。
SQL> alter database datafile 8 offline;
Database altered.
SQL> select file#,status,name from v$datafile;
FILE# STATUS NAME
---------- ------- --------------------------------------------------
1 SYSTEM /u01/app/oracle/oradata/PROD1/system01.dbf
2 ONLINE /u01/app/oracle/oradata/PROD1/sysaux01.dbf
3 ONLINE /u01/app/oracle/oradata/PROD1/undotbs01.dbf
4 ONLINE /u01/app/oracle/oradata/PROD1/users01.dbf
5 ONLINE /u01/app/oracle/oradata/PROD1/example01.dbf
6 ONLINE /u01/app/oracle/oradata/PROD1/tools.dbf
7 ONLINE /u01/app/oracle/oradata/PROD1/test.dbf
8 RECOVER /u01/app/oracle/oradata/PROD1/woo01.dbf #在对数据文件进行操作,触发了该文件,发现文件头部的scn不一致,提示需要进行恢复。’
8 rows selected.
#执行在线恢复并且online该数据文件。
SQL> recover datafile 8;
Media recovery complete.
SQL> alter database datafile 8 online;
Database altered.
SQL> select file#,status,name from v$datafile;
FILE# STATUS NAME
---------- ------- --------------------------------------------------
1 SYSTEM /u01/app/oracle/oradata/PROD1/system01.dbf
2 ONLINE /u01/app/oracle/oradata/PROD1/sysaux01.dbf
3 ONLINE /u01/app/oracle/oradata/PROD1/undotbs01.dbf
4 ONLINE /u01/app/oracle/oradata/PROD1/users01.dbf
5 ONLINE /u01/app/oracle/oradata/PROD1/example01.dbf
6 ONLINE /u01/app/oracle/oradata/PROD1/tools.dbf
7 ONLINE /u01/app/oracle/oradata/PROD1/test.dbf
8 ONLINE /u01/app/oracle/oradata/PROD1/woo01.dbf
8 rows selected.
SQL> create table test tablespace woo as select * from dba_users;
Table created.