VC Appliance停机是用待机方式将Oracle数据库停了。再启动时,Oracle不能启动。用SQLPLUS中的Starup命令。出现下面的提示:
由于ORA-03113错误是一个通用的错误提示,能给的信息量很少。
于是一步步的来Debug,由于startup相当于3个命令的集合,就一步步地执行
先执行
SQL>startup nomount
并没有发现什么明显的失败信息,说明启动数据库实例是没有问题的,再执行第二步
SQL>alter database mount;
出现故障
看来故障出现在挂载数据库文件时,这一步骤主要是“允许特定的维护操作, 例如, 重命名数据文件, 添加, 删除或重命名重做日志文件, 启用和禁用重做归档选项, 执行完全数据库恢复。它不允许对数据库进行一般的访问。”
找到$ORACLE_HOME/rdbms/log目录,用ls -ltr找到最近的日志文件。在本例中是sales_ora_27010.trc
看来是olr.loc和ocr.loc两个文件不能打开。但是我的机器上根本没有这个文件。在网上搜索,发现这两个文件和Oracle的集群有关,我这里是单节点的,按说是不需要Oracle Local Registry的。
在网上搜索,也没有什么特别有用的提示,但是发现了另一个问题,就是我找的日志文件可能不对,找了原来安装时导出的备份虚拟机,用命令
SQL>show parameter background
找到正确的日志路径/usr/local/oracle/diag/rdbms/oracle/sales/trace
查看启动时的日志,在最后发现
Starting ORACLE instance (normal)LICENSE_MAX_SESSION = 0LICENSE_SESSIONS_WARNING = 0Picked latch-free SCN scheme 3Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DESTAutotune of undo retention is turned on.…Wed Aug 07 15:37:46 2013SMON started with pid=13, OS id=30687Wed Aug 07 15:37:46 2013RECO started with pid=14, OS id=30689Wed Aug 07 15:37:46 2013MMON started with pid=15, OS id=30691starting up 1 dispatcher(s) for network address ‘(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))’…Wed Aug 07 15:37:46 2013MMNL started with pid=16, OS id=30693starting up 1 shared server(s) …ORACLE_BASE from environment = /usr/local/oracleWed Aug 07 15:37:46 2013ALTER DATABASE MOUNTUSER (ospid: 30698): terminating the instanceInstance terminated by USER, pid = 30698
重试了多次,结果大多类似。在日志相同目录里,找到最近的trc文件,发现如下内容:
[root@localhost trace]# vi sales_ora_20498.trc
Release: 2.6.32-279.el6.x86_64
Version: #1 SMP Fri Jun 22 12:19:21 UTC 2012
Machine: x86_64
Instance name: sales
Redo thread mounted by this instance: 0
Oracle process number: 19
Unix process pid: 20498, image: [email protected] (TNS V1-V3)
*** 2013-08-08 09:23:49.818
*** SESSION ID:(125.37) 2013-08-08 09:23:49.818
*** CLIENT ID:() 2013-08-08 09:23:49.818
*** SERVICE NAME:() 2013-08-08 09:23:49.818
*** MODULE NAME:([email protected] (TNS V1-V3)) 2013-08-08 09:23:49.818
*** ACTION NAME:() 2013-08-08 09:23:49.818
Error: kccpb_sanity_check_2
Control file sequence number mismatch!
fhcsq: 38637 bhcsq: 38638 cfn 0
*** 2013-08-08 09:23:49.819
USER (ospid: 20498): terminating the instance
Create controlfile reuse set database “oracle”MAXINSTANCES 8MAXLOGHISTORY 1MAXLOGFILES 16MAXLOGMEMBERS 3MAXDATAFILES 100Datafile‘/usr/local/oradata/ora11g/ORA11G/oracle/system01.dbf’,‘/usr/local/oradata/ora11g/ORA11G/oracle/sysaux01.dbf’,‘/usr/local/oradata/ora11g/ORA11G/oracle/undotbs01.dbf’,‘/usr/local/oradata/ora11g/ORA11G/oracle/users01.dbf’LOGFILE GROUP 1 (‘/usr/local/oradata/ora11g/ORA11G/oracle/redo01.log’) SIZE 51200K,GROUP 2 (‘/usr/local/oradata/ora11g/ORA11G/oracle/redo02.log’) SIZE 51200K,GROUP 3 (‘/usr/local/oradata/ora11g/ORA11G/oracle/redo03.log’) SIZE 51200K RESETLOGSWARNING: Default Temporary Tablespace not specified in CREATE DATABASE commandDefault Temporary Tablespace will be necessary for a locally managed database in future releaseSuccessful mount of redo thread 1, with mount id 1691519286Completed: Create controlfile reuse set database “oracle”
STARTUP NOMOUNTCREATE CONTROLFILE REUSE DATABASE “PRIMARY” NORESETLOGS ARCHIVELOG– SET STANDBY TO MAXIMIZE PERFORMANCEMAXLOGFILES 5MAXLOGMEMBERS 3MAXDATAFILES 100MAXINSTANCES 1MAXLOGHISTORY 226LOGFILEGROUP 1 ‘/opt/oracle/oradata/primary/redo01.log’ SIZE 10M,GROUP 2 ‘/opt/oracle/oradata/primary/redo02.log’ SIZE 10M,GROUP 3 ‘/opt/oracle/oradata/primary/redo03.log’ SIZE 10M– STANDBY LOGFILEDATAFILE‘/opt/oracle/oradata/primary/system01.dbf’,‘/opt/oracle/oradata/primary/undotbs01.dbf’,‘/opt/oracle/oradata/primary/users01.dbf’CHARACTER SET ZHS16GBK;RECOVER DATABASEALTER SYSTEM ARCHIVE LOG ALL;ALTER DATABASE OPEN;ALTER TABLESPACE TEMP ADD TEMPFILE ‘/opt/oracle/oradata/primary/temp01.dbf’SIZE 41943040 REUSE AUTOEXTEND ON NEXT 655360 MAXSIZE 32767M;
CREATE CONTROLFILE REUSE DATABASE “oracle” NORESETLOGS ARCHIVELOG– SET STANDBY TO MAXIMIZE PERFORMANCEMAXLOGFILES 16MAXLOGMEMBERS 3MAXDATAFILES 100MAXINSTANCES 8MAXLOGHISTORY 1LOGFILEGROUP 1 ‘/usr/local/oradata/ora11g/ORA11G/oracle/redo01.log’ SIZE 51200K,GROUP 2 ‘/usr/local/oradata/ora11g/ORA11G/oracle/redo02.log’ SIZE 51200K,GROUP 3 ‘/usr/local/oradata/ora11g/ORA11G/oracle/redo03.log’ SIZE 51200K– STANDBY LOGFILEDATAFILE‘/usr/local/oradata/ora11g/ORA11G/oracle/vpx01.dbf’,–’/usr/local/oradata/ora11g/ORA11G/oracle/temp01.dbf’,‘/usr/local/oradata/ora11g/ORA11G/oracle/system01.dbf’,‘/usr/local/oradata/ora11g/ORA11G/oracle/sysaux01.dbf’,‘/usr/local/oradata/ora11g/ORA11G/oracle/undotbs01.dbf’,‘/usr/local/oradata/ora11g/ORA11G/oracle/users01.dbf’
ALTER TABLESPACE TEMP ADD TEMPFILE ‘/usr/local/oradata/ora11g/ORA11G/oracle/temp01.dbf’ SIZE 20480K REUSE AUTOEXTEND ON NEXT 640K MAXSIZE UNLIMITED
[oracle@localhost ~]$ dbv FILE=’/usr/local/oradata/ora11g/ORA11G/oracle/vpx01.dbf’ BLOCKSIZE=8192
Select tablespace_name,segment_type,owner,segment_name From dba_extents Where file_id=5 and 45953 between block_id and block_id+blocks-1;
select table_name, column_name from dba_lobs where segment_name=’SYS_LOB0000075100C00016$$’ and owner =’VPXADMIN’;
select count(*) from VPXADMIN.VPX_RESOURCE_POOL;
ROWID ID—————— ———-AAASVcAAFAAACfjAAA 8AAASVcAAFAAACfkAAB 523AAASVcAAFAAACfkAAA 816AAASVcAAFAAACfkAAC 826AAASVcAAFAAACfkAAD 833AAASVcAAFAAACfkAAE 839AAASVcAAFAAACflAAB 1212AAASVcAAFAAACfjAAC 1601
select * from VPXADMIN.VPX_RESOURCE_POOL where id>1212;
delete from VPXADMIN.VPX_RESOURCE_POOL where id=1212;
recover datafile 5alter database datafile 5 online;alter tablespace VPX online;
…DBV-00201: Block, DBA 21044390, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044391, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044393, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044394, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044395, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044397, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044398, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044399, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044401, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044402, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044403, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044406, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044407, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044410, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044411, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044412, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044414, marked corrupt for invalid redo applicationDBV-00201: Block, DBA 21044415, marked corrupt for invalid redo application
select dbms_utility.data_block_address_file(21044415) from dual;select dbms_utility.data_block_address_block(21044415) from dual;
Select tablespace_name,segment_type,owner,segment_name From dba_extents Where file_id=5 and 72895 between block_id and block_id+blocks-1;
select table_name, column_name from dba_lobs where segment_name=’SYS_LOB0000075068C00059$$’ and owner =’VPXADMIN’
create table corrupted_data (corrupted_rowid rowid);
set concat offdeclareerror_1578 exception;pragma exception_init(error_1578,-1578);n number;beginfor cursor_lob in (select rowid r, &&lob_column from &table_owner.&table_with_lob) loopbeginn:=dbms_lob.instr(cursor_lob.&&lob_column,hextoraw(’889911′));exceptionwhen error_1578 theninsert into corrupted_data values (cursor_lob.r);commit;end;end loop;end;
set concat offupdate &table_owner.&table_with_lob set &lob_column=empty_blob() where rowid in (select corrupted_rowid from corrupted_data);
ORA-00932: inconsistent datatypes: expected NCLOB got BLOB
update &table_owner.&table_with_lob set &lob_column=empty_clob() where rowid in (select corrupted_rowid from corrupted_data);