Can not startup the second node on RAC.
Got the following error from alert log.
Wed Apr 8 03:41:38 2009
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=23, OS id=26761
Wed Apr 8 03:41:38 2009
starting up 1 shared server(s) ...
Wed Apr 8 03:41:38 2009
lmon registered with NM - instance id 2 (internal mem no 1)
Warning: cluster_database_instances (1) of node 0 is <= my node id (1)
cluster_database_instances parameter mismatch!
Instance 0 = 1, mine = 2
This instance has not mounted yet. Exiting!
Wed Apr 8 03:41:39 2009
USER: terminating instance due to error 29707
Instance terminated by USER, pid = 26601
I took the following action from first node.
$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.3.0 - Production on Wed Apr 8 03:42:05 2009
Copyright (c) 1982, 2006, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Oracle Label Security, OLAP
and Data Mining options
SQL> show parameter instance
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
active_instance_count integer
cluster_database_instances integer 1
instance_groups string
instance_name string RTST1
instance_number integer 1
instance_type string RDBMS
open_links_per_instance integer 4
parallel_instance_group string
parallel_server_instances integer 1
SQL> show parameter cluster
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cluster_database boolean FALSE
cluster_database_instances integer 1
cluster_interconnects string
SQL> create pfile='/export/home/oracle/henry/pfile.txt' from spfile;
File created.
SQL> alter system set cluster_database=true scope=spfile;
System altered.
SQL> alter system set cluster_database_instances=2 scope=spfile;
System altered.
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Oracle Label Security, OLAP
and Data Mining options
$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....ST1.srv application OFFLINE OFFLINE
ora....TST1.cs application OFFLINE OFFLINE
ora....T1.inst application OFFLINE OFFLINE
ora....ST2.srv application OFFLINE OFFLINE
ora....TST2.cs application OFFLINE OFFLINE
ora....T2.inst application ONLINE OFFLINE
ora.RTST.db application OFFLINE OFFLINE
ora....QX.lsnr application ONLINE ONLINE tst-...-eqx
ora....eqx.ons application ONLINE ONLINE tst-...-eqx
ora....eqx.vip application ONLINE ONLINE tst-...-eqx
ora....QX.lsnr application ONLINE ONLINE tst-...-eqx
ora....eqx.ons application ONLINE ONLINE tst-...-eqx
ora....eqx.vip application ONLINE ONLINE tst-...-eqx
$ crs_start
Usage: crs_start resource_name [...] [-c cluster_member] [-f] [-q] ["attrib=value ..."]
crs_start -all [-q]
$ crs_start -all
Attempting to start `ora.RTST.RTST1.inst` on member `tst-rac01-ud-us-eqx`
Attempting to start `ora.RTST.RTST2.inst` on member `tst-rac02-ud-us-eqx`
Start of `ora.RTST.RTST2.inst` on member `tst-rac02-ud-us-eqx` succeeded.
Start of `ora.RTST.RTST1.inst` on member `tst-rac01-ud-us-eqx` succeeded.
Attempting to start `ora.RTST.RTST1.cs` on member `tst-rac01-ud-us-eqx`
Attempting to start `ora.RTST.RTST1.RTST1.srv` on member `tst-rac01-ud-us-eqx`
Attempting to start `ora.RTST.db` on member `tst-rac01-ud-us-eqx`
Attempting to start `ora.RTST.RTST2.RTST2.srv` on member `tst-rac02-ud-us-eqx`
Attempting to start `ora.RTST.RTST2.cs` on member `tst-rac02-ud-us-eqx`
Start of `ora.RTST.RTST1.cs` on member `tst-rac01-ud-us-eqx` succeeded.
Start of `ora.RTST.RTST2.cs` on member `tst-rac02-ud-us-eqx` succeeded.
Start of `ora.RTST.RTST1.RTST1.srv` on member `tst-rac01-ud-us-eqx` succeeded.
Start of `ora.RTST.RTST2.RTST2.srv` on member `tst-rac02-ud-us-eqx` succeeded.
Start of `ora.RTST.db` on member `tst-rac01-ud-us-eqx` succeeded.
$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....ST1.srv application ONLINE ONLINE tst-...-eqx
ora....TST1.cs application ONLINE ONLINE tst-...-eqx
ora....T1.inst application ONLINE ONLINE tst-...-eqx
ora....ST2.srv application ONLINE ONLINE tst-...-eqx
ora....TST2.cs application ONLINE ONLINE tst-...-eqx
ora....T2.inst application ONLINE ONLINE tst-...-eqx
ora.RTST.db application ONLINE ONLINE tst-...-eqx
ora....QX.lsnr application ONLINE ONLINE tst-...-eqx
ora....eqx.ons application ONLINE ONLINE tst-...-eqx
ora....eqx.vip application ONLINE ONLINE tst-...-eqx
ora....QX.lsnr application ONLINE ONLINE tst-...-eqx
ora....eqx.ons application ONLINE ONLINE tst-...-eqx
ora....eqx.vip application ONLINE ONLINE tst-...-eqx
$ srvctl status database -d RTST
sInstance RTST1 is running on node tst-rac01-ud-us-eqx
Instance RTST2 is running on node tst-rac02-ud-us-eqx