数据库故障处理的案例-归档目录空间满(一)

某单位机房突然掉电,导致数据库异常,是否发现存储也有异常以及数据库管理的不善,导致一序列的问题接踵而来,还好本次的故障虽然问题好几次,但幸运的是数据库还是救了回来。
    数据库环境:Oracle 10.2.0.5 Rac For Linux X86_64,使用OCFS+ASM方式, 下面请看日志:
ORA-06512: at line 1
Tue May 13 10:59:26 2014
Thread 2 advanced to log sequence 11976 (LGWR switch)
  Current log# 4 seq# 11976 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Tue May 13 13:09:24 2014
Thread 2 advanced to log sequence 11977 (LGWR switch)
  Current log# 3 seq# 11977 mem# 0: +DG/dqb/onlinelog/group_3.268.803845083
Tue May 13 16:30:39 2014
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Interface type 1 eth1 192.168.1.0 configured from OCR for use as a cluster interconnect
Interface type 1 eth0 10.177.1.0 configured from OCR for use as  a public interface
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on. 
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
  processes                = 300
  sessions                 = 335
  __shared_pool_size       = 637534208
  __large_pool_size        = 16777216
  __java_pool_size         = 16777216
  __streams_pool_size      = 0
  spfile                   = +DG/dqb/spfiledqb.ora
  sga_target               = 1610612736
  control_files            = +DG/dqb/controlfile/current.272.803844991
  control_file_record_keep_time= 8
  db_block_size            = 8192
  __db_cache_size          = 922746880
  compatible               = 10.2.0.3.0
  log_archive_dest_1       = LOCATION=+dg/dqb/arch/arch1
  db_file_multiblock_read_count= 16
  cluster_database         = TRUE
  cluster_database_instances= 2
  db_create_file_dest      = +DG
  thread                   = 2
  instance_number          = 2
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS2
  remote_login_passwordfile= EXCLUSIVE
  db_domain                = 
  dispatchers              = (PROTOCOL=TCP) (SERVICE=dqbXDB)
  remote_listener          = LISTENERS_DQB
  job_queue_processes      = 10
  background_dump_dest     = /oracle/product/admin/dqb/bdump
  user_dump_dest           = /oracle/product/admin/dqb/udump
  core_dump_dest           = /oracle/product/admin/dqb/cdump
  audit_file_dest          = /oracle/product/admin/dqb/adump
  db_name                  = dqb
  open_cursors             = 300
  pga_aggregate_target     = 3363831808
Cluster communication is configured to use the following interface(s) for this instance
  192.168.1.2
Tue May 13 16:30:39 2014
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=4, OS id=13983
DIAG started with pid=8, OS id=13985
PSP0 started with pid=12, OS id=13987
LMON started with pid=16, OS id=13989
LMD0 started with pid=2, OS id=13991
LMS0 started with pid=6, OS id=13993
LMS1 started with pid=3, OS id=13997
LMS2 started with pid=20, OS id=14001
LMS3 started with pid=5, OS id=14005
LMS4 started with pid=10, OS id=14009
LMS5 started with pid=7, OS id=14014
LMS6 started with pid=24, OS id=14022
LMS7 started with pid=9, OS id=14026
MMAN started with pid=28, OS id=14030
DBW0 started with pid=14, OS id=14032
DBW1 started with pid=11, OS id=14034
DBW2 started with pid=32, OS id=14036
DBW3 started with pid=13, OS id=14038
LGWR started with pid=15, OS id=14040
CKPT started with pid=36, OS id=14042
SMON started with pid=40, OS id=14044
RECO started with pid=44, OS id=14046
CJQ0 started with pid=18, OS id=14048
MMON started with pid=48, OS id=14050
Tue May 13 16:30:40 2014
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=52, OS id=14052
Tue May 13 16:30:40 2014
starting up 1 shared server(s) ...
Tue May 13 16:30:40 2014
lmon registered with NM - instance id 2 (internal mem no 1)
Tue May 13 16:30:40 2014
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
 1
 Global Resource Directory frozen
* allocate domain 0, invalid = TRUE 
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Tue May 13 16:30:40 2014
 LMS 1: 0 GCS shadows cancelled, 0 closed
Tue May 13 16:30:40 2014
 LMS 2: 0 GCS shadows cancelled, 0 closed
Tue May 13 16:30:40 2014
 LMS 6: 0 GCS shadows cancelled, 0 closed
Tue May 13 16:30:40 2014
 LMS 3: 0 GCS shadows cancelled, 0 closed
Tue May 13 16:30:40 2014
 LMS 0: 0 GCS shadows cancelled, 0 closed
Tue May 13 16:30:40 2014
 LMS 4: 0 GCS shadows cancelled, 0 closed
Tue May 13 16:30:40 2014
 LMS 5: 0 GCS shadows cancelled, 0 closed
Tue May 13 16:30:40 2014
 LMS 7: 0 GCS shadows cancelled, 0 closed
 Set master node info 
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Post SMON to start 1st pass IR
Tue May 13 16:30:40 2014
 LMS 2: 0 GCS shadows traversed, 0 replayed
Tue May 13 16:30:40 2014
 LMS 3: 0 GCS shadows traversed, 0 replayed
Tue May 13 16:30:40 2014
 LMS 0: 0 GCS shadows traversed, 0 replayed
Tue May 13 16:30:40 2014
 LMS 7: 0 GCS shadows traversed, 0 replayed
Tue May 13 16:30:40 2014
 LMS 1: 0 GCS shadows traversed, 0 replayed
Tue May 13 16:30:40 2014
 LMS 5: 0 GCS shadows traversed, 0 replayed
Tue May 13 16:30:40 2014
 LMS 4: 0 GCS shadows traversed, 0 replayed
Tue May 13 16:30:40 2014
 LMS 6: 0 GCS shadows traversed, 0 replayed
Tue May 13 16:30:40 2014
 Submitted all GCS remote-cache requests
 Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=30, OS id=14059
Tue May 13 16:30:41 2014
ALTER DATABASE   MOUNT
Tue May 13 16:30:41 2014
This instance was first to mount
Tue May 13 16:30:41 2014
Starting background process ASMB
ASMB started with pid=56, OS id=14070
Starting background process RBAL
RBAL started with pid=60, OS id=14074
Tue May 13 16:30:44 2014
SUCCESS: diskgroup DG was mounted
Tue May 13 16:30:49 2014
Setting recovery target incarnation to 2
Tue May 13 16:30:49 2014
Successful mount of redo thread 2, with mount id 4126418993
Tue May 13 16:30:49 2014
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE   MOUNT
Tue May 13 16:30:49 2014
ALTER DATABASE OPEN
This instance was first to open
Tue May 13 16:30:53 2014
Beginning crash recovery of 2 threads
 parallel recovery started with 16 processes
Tue May 13 16:30:55 2014
Started redo scan
Tue May 13 16:30:55 2014
Completed redo scan
 0 redo blocks read, 0 data blocks need recovery
Tue May 13 16:30:55 2014
Started redo application at
 Thread 1: logseq 15107, block 142214
 Thread 2: logseq 11977, block 43783
Tue May 13 16:30:55 2014
Recovery of Online Redo Log: Thread 1 Group 1 Seq 15107 Reading mem 0
  Mem# 0: +DG/dqb/onlinelog/group_1.271.803844993
Tue May 13 16:30:55 2014
Recovery of Online Redo Log: Thread 2 Group 3 Seq 11977 Reading mem 0
  Mem# 0: +DG/dqb/onlinelog/group_3.268.803845083
Tue May 13 16:30:55 2014
Completed redo application
Tue May 13 16:30:55 2014
Completed crash recovery at
 Thread 1: logseq 15107, block 142214, scn 13881050933462
 Thread 2: logseq 11977, block 43783, scn 13881050931946
 0 data blocks read, 0 data blocks written, 0 redo blocks read
Tue May 13 16:30:55 2014
Thread 1 advanced to log sequence 15108 (thread recovery)
Picked broadcast on commit scheme to generate SCNs
Tue May 13 16:30:55 2014
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=58, OS id=14186
Tue May 13 16:30:55 2014
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=39, OS id=14188
Tue May 13 16:30:55 2014
Thread 2 advanced to log sequence 11978 (thread open)
Thread 2 opened at log sequence 11978
  Current log# 4 seq# 11978 mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Successful open of redo thread 2
Tue May 13 16:30:55 2014
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Tue May 13 16:30:55 2014
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Tue May 13 16:30:55 2014
ARC0: Becoming the heartbeat ARCH
Tue May 13 16:30:55 2014
SMON: enabling cache recovery
Tue May 13 16:30:56 2014
ARC1: Log corruption near block 43776 change 13881059254746 time ?
Tue May 13 16:30:56 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_14188.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 43776 change 13881059254746 time 05/13/2014 13:09:24
ORA-00312: online log 3 thread 2: '+DG/dqb/onlinelog/group_3.268.803845083'
ARC1: All Archive destinations made inactive due to error 354
Tue May 13 16:30:56 2014
ARC1: Closing local archive destination LOG_ARCHIVE_DEST_1: '+DG/dqb/arch/arch1/2_11977_803844992.dbf' (error 354)
 (dqb2)
Committing creation of archivelog '+DG/dqb/arch/arch1/2_11977_803844992.dbf' (error 354)
ARCH: Archival stopped, error occurred. Will continue retrying
Tue May 13 16:30:56 2014
ORACLE Instance dqb2 - Archival Error
Tue May 13 16:30:56 2014
ORA-16038: log 3 sequence# 11977 cannot be archived
ORA-00354: corrupt redo log block header
ORA-00312: online log 3 thread 2: '+DG/dqb/onlinelog/group_3.268.803845083'
Tue May 13 16:30:56 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_arc1_14188.trc:
ORA-16038: log 3 sequence# 11977 cannot be archived
ORA-00354: corrupt redo log block header
ORA-00312: online log 3 thread 2: '+DG/dqb/onlinelog/group_3.268.803845083'
Tue May 13 16:30:56 2014
Successfully onlined Undo Tablespace 5.
Tue May 13 16:30:56 2014
SMON: enabling tx recovery
Tue May 13 16:30:56 2014
Database Characterset is ZHS16GBK
Opening with internal Resource Manager plan
where NUMA PG = 4, CPUs = 20
Tue May 13 16:30:57 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35
ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'
Tue May 13 16:30:57 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:
ORA-00600: internal error code, arguments: [4194], [65], [47], [], [], [], [], []
Tue May 13 16:30:59 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35
ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'
ORA-00600: internal error code, arguments: [4194], [65], [47], [], [], [], [], []
Tue May 13 16:31:00 2014
Trace dumping is performing id=[cdmp_20140513163100]
Tue May 13 16:31:00 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35
ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'
ORA-00600: internal error code, arguments: [4194], [65], [47], [], [], [], [], []
Tue May 13 16:31:00 2014
Doing block recovery for file 6 block 4405
Block recovery from logseq 11978, block 89 to scn 13881066792885
Tue May 13 16:31:00 2014
Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0
  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Block recovery stopped at EOT rba 11978.91.16
Block recovery completed at rba 11978.91.16, scn 3231.4027457777
Doing block recovery for file 6 block 121
Block recovery from logseq 11978, block 89 to scn 13881066791152
Tue May 13 16:31:00 2014
Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0
  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Block recovery completed at rba 11978.91.16, scn 3231.4027457777
Tue May 13 16:31:00 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35
ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'
Tue May 13 16:31:00 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:
ORA-00600: internal error code, arguments: [4193], [25330], [25385], [], [], [], [], []
Tue May 13 16:31:00 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_smon_14044.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35
ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'
Tue May 13 16:31:00 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_smon_14044.trc:
ORA-00600: internal error code, arguments: [4194], [43], [26], [], [], [], [], []
Tue May 13 16:31:01 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_smon_14044.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35
ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'
ORA-00600: internal error code, arguments: [4194], [43], [26], [], [], [], [], []
Tue May 13 16:31:01 2014
Doing block recovery for file 6 block 4718
Block recovery from logseq 11978, block 100 to scn 13881066792905
Tue May 13 16:31:01 2014
Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0
  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Block recovery stopped at EOT rba 11978.101.16
Block recovery completed at rba 11978.101.16, scn 3231.4027459528
Doing block recovery for file 6 block 105
Block recovery from logseq 11978, block 100 to scn 13881066792903
Tue May 13 16:31:01 2014
Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0
  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Block recovery completed at rba 11978.101.16, scn 3231.4027459528
Tue May 13 16:31:01 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_smon_14044.trc:
ORA-01595: error freeing extent (4) of rollback segment (17))
ORA-00600: internal error code, arguments: [4194], [43], [26], [], [], [], [], []
Tue May 13 16:31:03 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35
ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'
ORA-00600: internal error code, arguments: [4193], [25330], [25385], [], [], [], [], []
Tue May 13 16:31:03 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35
ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'
ORA-00600: internal error code, arguments: [4193], [25330], [25385], [], [], [], [], []
Tue May 13 16:31:03 2014
Doing block recovery for file 6 block 301
Block recovery from logseq 11978, block 91 to scn 13881066792908
Tue May 13 16:31:03 2014
Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0
  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Block recovery stopped at EOT rba 11978.101.16
Block recovery completed at rba 11978.101.16, scn 3231.4027459528
Doing block recovery for file 6 block 153
Block recovery from logseq 11978, block 91 to scn 13881066792887
Tue May 13 16:31:03 2014
Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0
  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Block recovery completed at rba 11978.93.16, scn 3231.4027459513
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=66, OS id=14288
Tue May 13 16:31:04 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35
ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'
Tue May 13 16:31:04 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:
ORA-00600: internal error code, arguments: [4193], [25330], [25385], [], [], [], [], []
Tue May 13 16:31:07 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35
ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'
ORA-00600: internal error code, arguments: [4193], [25330], [25385], [], [], [], [], []
Tue May 13 16:31:07 2014
Doing block recovery for file 6 block 301
Block recovery from logseq 11978, block 91 to scn 13881066792908
Tue May 13 16:31:07 2014
Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0
  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Tue May 13 16:31:07 2014
Trace dumping is performing id=[cdmp_20140513163107]
Tue May 13 16:31:07 2014
Block recovery completed at rba 11978.101.16, scn 3231.4027459534
Doing block recovery for file 6 block 153
Block recovery from logseq 11978, block 91 to scn 13881066792913
Tue May 13 16:31:07 2014
Recovery of Online Redo Log: Thread 2 Group 4 Seq 11978 Reading mem 0
  Mem# 0: +DG/dqb/onlinelog/group_4.266.803845083
Block recovery completed at rba 11978.104.16, scn 3231.4027459538
Error 0 in kwqmnpartition(), aborting txn 
Tue May 13 16:31:07 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_14120.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 43776 change 13881059258085 time 05/13/2014 13:58:35
ORA-00334: archived log: '+DG/dqb/onlinelog/group_3.268.803845083'
Tue May 13 16:31:07 2014
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
  processes                = 300
  sessions                 = 335
  __shared_pool_size       = 637534208
  __large_pool_size        = 16777216
  __java_pool_size         = 16777216
  __streams_pool_size      = 0
  spfile                   = +DG/dqb/spfiledqb.ora
  sga_target               = 1610612736
  control_files            = +DG/dqb/controlfile/current.272.803844991
  control_file_record_keep_time= 8
  db_block_size            = 8192
  __db_cache_size          = 922746880
  compatible               = 10.2.0.3.0
  log_archive_dest_1       = LOCATION=+dg/dqb/arch/arch1
  db_file_multiblock_read_count= 16
  cluster_database         = TRUE
  cluster_database_instances= 2
  db_create_file_dest      = +DG
  db_recovery_file_dest_size= 10737418240
  thread                   = 2
  instance_number          = 2
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS2
  remote_login_passwordfile= EXCLUSIVE
  db_domain                = 
  dispatchers              = (PROTOCOL=TCP) (SERVICE=dqbXDB)
  remote_listener          = LISTENERS_DQB
  job_queue_processes      = 10
  background_dump_dest     = /oracle/product/admin/dqb/bdump
  user_dump_dest           = /oracle/product/admin/dqb/udump
  core_dump_dest           = /oracle/product/admin/dqb/cdump
  audit_file_dest          = /oracle/product/admin/dqb/adump
  db_name                  = dqb
  open_cursors             = 300
  pga_aggregate_target     = 3363831808
Cluster communication is configured to use the following interface(s) for this instance
  192.168.1.2
Wed May 14 11:31:48 2014
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
PMON started with pid=2, OS id=32760
DIAG started with pid=6, OS id=32762
PSP0 started with pid=10, OS id=32764
LMON started with pid=14, OS id=32766
LMD0 started with pid=18, OS id=300
LMS0 started with pid=22, OS id=302
LMS1 started with pid=3, OS id=306
LMS2 started with pid=4, OS id=310
LMS3 started with pid=5, OS id=314
LMS4 started with pid=26, OS id=318
LMS5 started with pid=7, OS id=322
LMS6 started with pid=8, OS id=326
LMS7 started with pid=9, OS id=330
MMAN started with pid=30, OS id=334
DBW0 started with pid=34, OS id=336
DBW1 started with pid=11, OS id=338
DBW2 started with pid=12, OS id=340
DBW3 started with pid=13, OS id=342
LGWR started with pid=15, OS id=344
CKPT started with pid=38, OS id=346
SMON started with pid=42, OS id=348
RECO started with pid=46, OS id=350
CJQ0 started with pid=50, OS id=395
MMON started with pid=54, OS id=397
Wed May 14 11:31:49 2014
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=58, OS id=399
Wed May 14 11:31:49 2014
starting up 1 shared server(s) ...
Wed May 14 11:31:49 2014
lmon registered with NM - instance id 2 (internal mem no 1)
Wed May 14 11:31:49 2014
Reconfiguration started (old inc 0, new inc 2)
List of nodes:
 1
 Global Resource Directory frozen
* allocate domain 0, invalid = TRUE 
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Wed May 14 11:31:49 2014
 LMS 0: 0 GCS shadows cancelled, 0 closed
Wed May 14 11:31:49 2014
 LMS 1: 0 GCS shadows cancelled, 0 closed
Wed May 14 11:31:49 2014
 LMS 3: 0 GCS shadows cancelled, 0 closed
Wed May 14 11:31:49 2014
 LMS 2: 0 GCS shadows cancelled, 0 closed
Wed May 14 11:31:49 2014
 LMS 4: 0 GCS shadows cancelled, 0 closed
Wed May 14 11:31:49 2014
 LMS 5: 0 GCS shadows cancelled, 0 closed
Wed May 14 11:31:49 2014
 LMS 6: 0 GCS shadows cancelled, 0 closed
Wed May 14 11:31:49 2014
 LMS 7: 0 GCS shadows cancelled, 0 closed
 Set master node info 
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Post SMON to start 1st pass IR
Wed May 14 11:31:49 2014
 LMS 1: 0 GCS shadows traversed, 0 replayed
Wed May 14 11:31:49 2014
 LMS 0: 0 GCS shadows traversed, 0 replayed
Wed May 14 11:31:49 2014
 LMS 6: 0 GCS shadows traversed, 0 replayed
Wed May 14 11:31:49 2014
 LMS 2: 0 GCS shadows traversed, 0 replayed
Wed May 14 11:31:49 2014
 LMS 7: 0 GCS shadows traversed, 0 replayed
Wed May 14 11:31:49 2014
 LMS 3: 0 GCS shadows traversed, 0 replayed
Wed May 14 11:31:49 2014
 LMS 5: 0 GCS shadows traversed, 0 replayed
Wed May 14 11:31:49 2014
 LMS 4: 0 GCS shadows traversed, 0 replayed
Wed May 14 11:31:49 2014
 Submitted all GCS remote-cache requests
 Fix write in gcs resources
Reconfiguration complete
LCK0 started with pid=70, OS id=405
Wed May 14 11:31:50 2014
ALTER DATABASE   MOUNT
Wed May 14 11:31:50 2014
This instance was first to mount
Wed May 14 11:31:50 2014
Starting background process ASMB
ASMB started with pid=78, OS id=416
Starting background process RBAL
RBAL started with pid=82, OS id=420
Wed May 14 11:31:53 2014
SUCCESS: diskgroup DG was mounted
Wed May 14 11:31:57 2014
Setting recovery target incarnation to 2
Wed May 14 11:31:57 2014
Successful mount of redo thread 2, with mount id 4126510502
Wed May 14 11:31:57 2014
Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)
Completed: ALTER DATABASE   MOUNT
Wed May 14 11:31:57 2014
ALTER DATABASE OPEN
This instance was first to open
Picked broadcast on commit scheme to generate SCNs
Wed May 14 11:31:58 2014
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=90, OS id=475
Wed May 14 11:31:58 2014
ARC0: Archival started
ARC1 started with pid=23, OS id=477
Wed May 14 11:31:58 2014
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
Wed May 14 11:31:58 2014
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Wed May 14 11:31:58 2014
ARC0: Becoming the heartbeat ARCH
Wed May 14 11:31:58 2014
ARCH: Log corruption near block 43776 change 13881059254746 time ?
Wed May 14 11:31:58 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_466.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 43776 change 13881059254746 time 05/13/2014 13:09:24
ORA-00312: online log 3 thread 2: '+DG/dqb/onlinelog/group_3.268.803845083'
ARCH: All Archive destinations made inactive due to error 354
Wed May 14 11:32:01 2014
ARCH: Closing local archive destination LOG_ARCHIVE_DEST_1: '+DG/dqb/arch/arch1/2_11977_803844992.dbf' (error 354)
 (dqb2)
Committing creation of archivelog '+DG/dqb/arch/arch1/2_11977_803844992.dbf' (error 354)
Wed May 14 11:32:01 2014
Errors in file /oracle/product/admin/dqb/udump/dqb2_ora_466.trc:
ORA-16038: log 3 sequence# 11977 cannot be archived
ORA-00354: corrupt redo log block header
ORA-00312: online log 3 thread 2: '+DG/dqb/onlinelog/group_3.268.803845083'
USER: terminating instance due to error 16038
Wed May 14 11:32:01 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms1_306.trc:
ORA-16038: log  sequence#  cannot be archived
Wed May 14 11:32:01 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms2_310.trc:
ORA-16038: log  sequence#  cannot be archived
Wed May 14 11:32:01 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms3_314.trc:
ORA-16038: log  sequence#  cannot be archived
Wed May 14 11:32:01 2014
Errors in file /oracle/product/admin/dqb/bdump/dqb2_lms6_326.trc:
ORA-16038: log  sequence#  cannot be archived
Wed May 14 11:32:01 2014
System state dump is made for local instance
System State dumped to trace file /oracle/product/admin/dqb/bdump/dqb2_diag_32762.trc
Wed May 14 11:32:01 2014
Trace dumping is performing id=[cdmp_20140514113201]
Wed May 14 11:32:04 2014
Instance terminated by USER, pid = 466
Wed May 14 11:32:53 2014

出现上面的解决办法当然是删除无用的过期归档日志,腾出归档空间,以便数据库启动后执行归档进程恢复动作。
1、登陆到数据库服务器,并切换到oracle安装用户
#su - ora10g
$export ORACLE_SID=+ASM1   输出环境变量进入asm实例命令行
$asmcmd
asmcmd>pwd
+DG
asmcmd>cd +DG/.....
.......

你可能感兴趣的:(异常,数据库服务器)