RAC 实例不能启动 ORA-1589 signalled during ALTER DATABASE OPEN

 

今天启动RAC 做实验，发现RAC 实例怎么启动不了。

 

[oracle@rac2 bin]$ crs_stat -t

Name           Type           Target    State     Host

------------------------------------------------------------

ora.orcl.db    application    ONLINE    OFFLINE

ora....oltp.cs application    ONLINE    OFFLINE

ora....cl1.srv application    ONLINE    OFFLINE

ora....cl2.srv application    ONLINE    OFFLINE

ora....l1.inst application    ONLINE    OFFLINE

ora....l2.inst application    ONLINE    OFFLINE

ora....SM1.asm application    ONLINE    ONLINE    rac1

ora....C1.lsnr application    ONLINE    ONLINE    rac1

ora.rac1.gsd   application    ONLINE    ONLINE    rac1

ora.rac1.ons   application    ONLINE    ONLINE    rac1

ora.rac1.vip   application    ONLINE    ONLINE    rac1

ora....SM2.asm application    ONLINE    ONLINE    rac2

ora....C2.lsnr application    ONLINE    ONLINE    rac2

ora.rac2.gsd   application    ONLINE    ONLINE    rac2

ora.rac2.ons   application    ONLINE    ONLINE    rac2

ora.rac2.vip   application    ONLINE    ONLINE    rac2

 

 

看了一下ASM 的磁盘组，都挂载成功。

 

[oracle@rac2 bin]$ export ORACLE_SID=+ASM2

[oracle@rac2 bin]$ sqlplus / as sysdba;

SQL*Plus: Release 10.2.0.1.0 - Production on Tue Sep 28 04:33:24 2010

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

Connected to:

Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production

With the Partitioning, Real Application Clusters, OLAP and Data Mining options

 

SQL> select name,state from v$asm_diskgroup;

 

NAME                           STATE

------------------------------ -----------

DATA                           MOUNTED

FLASH_RECOVERY_AREA            MOUNTED

 

SQL>

 

 

之前遇到过这种情况情况，所以第一时间就查看了这个ASM 磁盘组的信息。

 

ASM 没有问题，只能查看数据库的alert log了。

 

 

[oracle@rac1 bdump]$ pwd

/u01/app/oracle/admin/orcl/bdump

[oracle@rac1 bdump]$ ls

alert_orcl1.log       orcl1_diag_22153.trc  orcl1_lms0_22169.trc

cdmp_20100917173112   orcl1_lgwr_22186.trc  orcl1_smon_22190.trc

orcl1_arc0_24180.trc  orcl1_lmd0_22167.trc

orcl1_arc1_24182.trc  orcl1_lmon_22165.trc

[oracle@rac1 bdump]$ tail -100 alert_orcl1.log

 

Tue Sep 28 04:05:20 2010

starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...

MMON started with pid=15, OS id=5650

Tue Sep 28 04:05:20 2010

starting up 1 shared server(s) ...

Tue Sep 28 04:05:20 2010

lmon registered with NM - instance id 1 (internal mem no 0)

Tue Sep 28 04:05:21 2010

Reconfiguration started (old inc 0, new inc 2)

List of nodes:

 0

 Global Resource Directory frozen

* allocate domain 0, invalid = TRUE

 Communication channels reestablished

 Master broadcasted resource hash value bitmaps

 Non-local Process blocks cleaned out

Tue Sep 28 04:05:21 2010

 LMS 0: 0 GCS shadows cancelled, 0 closed

 Set master node info

 Submitted all remote-enqueue requests

 Dwn-cvts replayed, VALBLKs dubious

 All grantable enqueues granted

 Post SMON to start 1st pass IR

Tue Sep 28 04:05:21 2010

 LMS 0: 0 GCS shadows traversed, 0 replayed

Tue Sep 28 04:05:21 2010

 Submitted all GCS remote-cache requests

 Post SMON to start 1st pass IR

 Fix write in gcs resources

Reconfiguration complete

LCK0 started with pid=19, OS id=5683

Tue Sep 28 04:05:22 2010

ALTER DATABASE   MOUNT

Tue Sep 28 04:05:22 2010

This instance was first to mount

Tue Sep 28 04:05:22 2010

Starting background process ASMB

ASMB started with pid=21, OS id=5706

Starting background process RBAL

RBAL started with pid=22, OS id=5710

Loaded ASM Library - Generic Linux, version 2.0.4 (KABI_V2) library for asmlib interface

Tue Sep 28 04:05:26 2010

SUCCESS: diskgroup DATA was mounted

SUCCESS: diskgroup FLASH_RECOVERY_AREA was mounted

Tue Sep 28 04:05:34 2010

Setting recovery target incarnation to 3

Tue Sep 28 04:05:35 2010

Successful mount of redo thread 1, with mount id 1258660482

Tue Sep 28 04:05:35 2010

Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)

Completed: ALTER DATABASE   MOUNT

Tue Sep 28 04:05:35 2010

ALTER DATABASE OPEN

This instance was first to open

ORA-1589 signalled during: ALTER DATABASE OPEN...

Tue Sep 28 04:05:39 2010

Shutting down instance (abort)

License high water mark = 1

Instance terminated by USER, pid = 6080

 

 

从日志里我们发现了一个很重要的信息：

 

ALTER DATABASE OPEN

This instance was first to open

ORA-1589 signalled during: ALTER DATABASE OPEN...

Tue Sep 28 04:05:39 2010

Shutting down instance (abort)

 

 

查看了一下错误代码：

ORA-01589:	must use RESETLOGS or NORESETLOGS option for database open
Cause:	Either incomplete or backup control file recovery has been performed. After these types of recovery you must specify either the RESETLOGS option or the NORESETLOGS option to open your database.
Action:	Specify the appropriate option.

 

 

提示我们必须使用resetlogs 模式打开数据。 开到这才明白，昨天做RMAN 恢复后直接把数据库关了，没有用resetlogs打开。 杯具中..

 

 

手工启动数据库至mount 状态，在用resetlogs 打开：

 

[oracle@rac1 bdump]$ export ORACLE_SID=orcl1

[oracle@rac1 bdump]$ sqlplus /nolog

 

SQL*Plus: Release 10.2.0.1.0 - Production on Tue Sep 28 04:17:47 2010

 

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

 

SQL> conn / as sysdba;

Connected to an idle instance.

SQL> startup mount;

ORACLE instance started.

 

Total System Global Area  180355072 bytes

Fixed Size                  1218388 bytes

Variable Size             109054124 bytes

Database Buffers           67108864 bytes

Redo Buffers                2973696 bytes

Database mounted.

SQL> alter database open resetlogs;

Database altered.

SQL>

 

 

在另一个节点，直接正常启动就可以了。

 

[oracle@rac2 bin]$ export ORACLE_SID=orcl2

[oracle@rac2 bin]$ sqlplus /nolog

 

SQL*Plus: Release 10.2.0.1.0 - Production on Tue Sep 28 04:22:50 2010

 

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

 

SQL> conn / as sysdba

Connected to an idle instance.

SQL> startup mount;

ORACLE instance started.

 

Total System Global Area  180355072 bytes

Fixed Size                  1218388 bytes

Variable Size              88082604 bytes

Database Buffers           88080384 bytes

Redo Buffers                2973696 bytes

Database mounted.

SQL> alter database open;

 

Database altered.

 

SQL>

 

 

查看一下RAC 进程的状态：

 

[oracle@rac2 bin]$ crs_stat -t

Name           Type           Target    State     Host

------------------------------------------------------------

ora.orcl.db    application    ONLINE    ONLINE    rac2

ora....oltp.cs application    ONLINE    ONLINE    rac2

ora....cl1.srv application    ONLINE    ONLINE    rac1

ora....cl2.srv application    ONLINE    ONLINE    rac2

ora....l1.inst application    ONLINE    ONLINE    rac1

ora....l2.inst application    ONLINE    ONLINE    rac2

ora....SM1.asm application    ONLINE    ONLINE    rac1

ora....C1.lsnr application    ONLINE    ONLINE    rac1

ora.rac1.gsd   application    ONLINE    ONLINE    rac1

ora.rac1.ons   application    ONLINE    ONLINE    rac1

ora.rac1.vip   application    ONLINE    ONLINE    rac1

ora....SM2.asm application    ONLINE    ONLINE    rac2

ora....C2.lsnr application    ONLINE    ONLINE    rac2

ora.rac2.gsd   application    ONLINE    ONLINE    rac2

ora.rac2.ons   application    ONLINE    ONLINE    rac2

ora.rac2.vip   application    ONLINE    ONLINE    rac2

 

 

 

一切正常，从这个小错误中明白，遇到错误，要养成看alert log 的习惯。 不管是单实例还是RAC 实例，看log 永远是第一条。 还有就是要心细，昨天偷了懒，为了补这个懒，花的时间就比昨天多了很多。 杯具啊.

 

 

 

 

 

------------------------------------------------------------------------------

Blog： http://blog.csdn.net/tianlesoftware

网上资源： http://tianlesoftware.download.csdn.net

相关视频：http://blog.csdn.net/tianlesoftware/archive/2009/11/27/4886500.aspx

DBA1 群：62697716(满); DBA2 群：62697977(满)

DBA3 群：63306533;     聊天 群：40132017

--加群需要在备注说明SGA的组成部分，否则拒绝申请