Troubleshooting 11.2 Grid Infrastructure root.sh Issues (Doc ID 1053970.1)

In this Document

Purpose

Troubleshooting Steps

Advanced Root.sh Troubleshooting

Community Discussions

References

APPLIES TO:

Oracle Database - Enterprise Edition - Version 11.2.0.1 to 11.2.0.3 [Release 11.2]
Information in this document applies to any platform.

PURPOSE

This document is to provide a reference for troubleshooting root.sh issues after installing an 11.2 Grid Infrastructure home for a cluster.  For versions prior to 11.2, see Note: 240001.1

TROUBLESHOOTING STEPS

At the end of a grid infrastructure installation, the user is prompted to run the "root.sh" script.  This script configures and starts the Oracle Clusterware stack.  A root.sh script can error out and/or fail under one of the following conditions:

• Problem with the network configuration.
• Problem with the storage location for the OCR and/or voting files.  
• Permission problem with/var/tmp (specifically /var/tmp/.oracle).
• Problem with the vendor clusterware (if used).
• Some other configuration issue.
• An Oracle bug.

Most configuration issues should be detectable by running the Cluster Verification Utility with the following syntax (input the nodelist):

cd <GRID_HOME>/bin
./cluvfy stage -pre crsinst -n <nodelist> -r 11gR2 -verbose

Additional options can be used for a more thorough check:

USAGE:
cluvfy stage -pre crsinst -n <node_list> [-r {10gR1|10gR2|11gR1|11gR2}]
[-c <ocr_location_list>] [-q <voting_disk_list>]
[-osdba <osdba_group>]
[-orainv <orainventory_group>]
[-asm -asmgrp <asmadmin_group>]
[-asm -asmdev <asm_device_list>]
[-fixup [-fixupdir <fixup_dir>]] [-verbose]

If the Cluster Verification Utility is unable to find a configuration problem and your root.sh still fails, you may need the assistance of Oracle Support to troubleshoot further and/or see the "Advanced Root.sh Troubleshooting" section:

Advanced Root.sh Troubleshooting

The root.sh is simply a parent script that calls the following scripts:

<GRID_HOME>/install/utl/rootmacro.sh   # small - validates home and user
<GRID_HOME>/install/utl/rootinstall.sh    # small - creates some local files
<GRID_HOME>/network/install/sqlnet/setowner.sh   # small - opens up /tmp permissions
<GRID_HOME>/rdbms/install/rootadd_rdbms.sh  # small - misc file/permission checks
<GRID_HOME>/rdbms/install/rootadd_filemap.sh  # small - misc file/permission checks
<GRID_HOME>/crs/install/rootcrs.pl  # MAIN CLUSTERWARE CONFIG SCRIPT

If your root.sh is failing on one of the first 5 scripts, it should be an easy fix since those fix are small and easy to troubleshoot.  However, most problems are likely going to happen in the rootcrs.pl script which is the main clusterware config script.  This script will log useful trace data to <GRID_HOME>/cfgtoollogs/crsconfig/rootcrs_<nodename>.log.  However, you should check the clusterware alert log under <GRID_HOME>/log/<nodename> first for any obvious problems or errors.

In the following section I will show the log output of a new installation on a 2 node cluster (racbde1 and racbde2) where the OCR and Voting files will be stored in ASM on a diskgroup called +SYSTEMDG.  This log information is posted for reference.  It might be useful to compare the clusterware alert log from a working root.sh (mine) to a failing one (yours) to see where it went wrong.  I will bold the major landmarks in the clusterware alert log so that you can see how far you got:

Node 1 (racbde1) Clusterware Alert Log During root.sh:

Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
2009-12-23 19:24:33.844
[client(17368)]CRS-2106:The OLR location /u01/app/grid/cdata/racbde1.olr is inaccessible. Details in /u01/app/grid/log/racbde1/client/ocrconfig_17368.log.
2009-12-23 19:24:33.956
[client(17368)]CRS-2101:The OLR was formatted using version 3.
2009-12-23 19:25:02.495
[ohasd(17767)]CRS-2112:The OLR service started on node racbde1.
2009-12-23 19:25:02.833
[ohasd(17767)]CRS-2772:Server 'racbde1' has been assigned to pool 'Free'.
2009-12-23 19:25:34.801
[cssd(18791)]CRS-1713:CSSD daemon is started in exclusive mode
2009-12-23 19:25:37.126
[cssd(18791)]CRS-1709:Lease acquisition failed for node racbde1 because no voting file has been configured; Details at (:CSSNM00031:) in /u01/app/grid/log/racbde1/cssd/ocssd.log
2009-12-23 19:25:54.705
[cssd(18791)]CRS-1601:CSSD Reconfiguration complete. Active nodes are racbde1 .
2009-12-23 19:25:55.431
[ctssd(18848)]CRS-2403:The Cluster Time Synchronization Service on host racbde1 is in observer mode.
2009-12-23 19:25:55.575
[ctssd(18848)]CRS-2407:The new Cluster Time Synchronization Service reference node is host racbde1.
2009-12-23 19:25:56.312
[ctssd(18848)]CRS-2401:The Cluster Time Synchronization Service started on host racbde1.
[client(19034)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(19038)]CRS-10001:ACFS-9322: done.
2009-12-23 19:30:26.790
[client(19423)]CRS-1006:The OCR location +SYSTEMDG is inaccessible. Details in /u01/app/grid/log/racbde1/client/ocrconfig_19423.log.
2009-12-23 19:30:27.883
[client(19423)]CRS-1001:The OCR was formatted using version 3.
2009-12-23 19:30:40.473
[crsd(19480)]CRS-1012:The OCR service started on node racbde1.
2009-12-23 19:31:53.331
[cssd(18791)]CRS-1605:CSSD voting file is online: /dev/sdb1; details in /u01/app/grid/log/racbde1/cssd/ocssd.log.
2009-12-23 19:31:53.373
[cssd(18791)]CRS-1605:CSSD voting file is online: /dev/sdb2; details in /u01/app/grid/log/racbde1/cssd/ocssd.log.
2009-12-23 19:31:53.417
[cssd(18791)]CRS-1605:CSSD voting file is online: /dev/sdb3; details in /u01/app/grid/log/racbde1/cssd/ocssd.log.
2009-12-23 19:31:54.413
[cssd(18791)]CRS-1626:A Configuration change request completed successfully
2009-12-23 19:31:54.424
[cssd(18791)]CRS-1601:CSSD Reconfiguration complete. Active nodes are racbde1 .
2009-12-23 19:32:10.831
[ctssd(18848)]CRS-2405:The Cluster Time Synchronization Service on host racbde1 is shutdown by user
2009-12-23 19:32:26.536
[cssd(18791)]CRS-1603:CSSD on node racbde1 shutdown by user.
2009-12-23 19:32:26.856
[cssd(18791)]CRS-1625:Node racbde1, number 1, was manually shut down
2009-12-23 19:32:44.826
[cssd(20125)]CRS-1713:CSSD daemon is started in clustered mode
2009-12-23 19:34:07.568
[cssd(20125)]CRS-1707:Lease acquisition for node racbde1 number 1 completed
2009-12-23 19:34:07.690
[cssd(20125)]CRS-1605:CSSD voting file is online: /dev/sdb3; details in /u01/app/grid/log/racbde1/cssd/ocssd.log.
2009-12-23 19:34:07.731
[cssd(20125)]CRS-1605:CSSD voting file is online: /dev/sdb2; details in /u01/app/grid/log/racbde1/cssd/ocssd.log.
2009-12-23 19:34:07.774
[cssd(20125)]CRS-1605:CSSD voting file is online: /dev/sdb1; details in /u01/app/grid/log/racbde1/cssd/ocssd.log.
2009-12-23 19:34:25.380
[cssd(20125)]CRS-1601:CSSD Reconfiguration complete. Active nodes are racbde1 .
2009-12-23 19:34:26.324
[ctssd(20269)]CRS-2403:The Cluster Time Synchronization Service on host racbde1 is in observer mode.
2009-12-23 19:34:26.448
[ctssd(20269)]CRS-2407:The new Cluster Time Synchronization Service reference node is host racbde1.
2009-12-23 19:34:27.278
[ctssd(20269)]CRS-2401:The Cluster Time Synchronization Service started on host racbde1.
2009-12-23 19:34:41.941
[crsd(20392)]CRS-1012:The OCR service started on node racbde1.
2009-12-23 19:34:44.734
[crsd(20392)]CRS-1201:CRSD started on node racbde1.

Node 2 (racbde2) Clusterware Alert Log During root.sh:

Oracle Database 11g Clusterware Release 11.2.0.1.0 - Production Copyright 1996, 2009 Oracle. All rights reserved.
2009-12-23 19:33:43.687
[client(12019)]CRS-2106:The OLR location /u01/app/grid/cdata/racbde2.olr is inaccessible. Details in /u01/app/grid/log/racbde2/client/ocrconfig_12019.log.
2009-12-23 19:33:43.700
[client(12019)]CRS-2101:The OLR was formatted using version 3.
2009-12-23 19:33:50.660
[ohasd(12058)]CRS-2112:The OLR service started on node racbde2.
2009-12-23 19:33:50.946
[ohasd(12058)]CRS-2772:Server 'racbde2' has been assigned to pool 'Free'.
2009-12-23 19:34:15.140
[ohasd(12058)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2009-12-23 19:34:17.910
[cssd(13108)]CRS-1713:CSSD daemon is started in exclusive mode
2009-12-23 19:35:39.019
[cssd(13108)]CRS-1707:Lease acquisition for node racbde2 number 2 completed
[cssd(13108)]CRS-1636:The CSS daemon was started in exclusive mode but found an active CSS daemon on node racbde1 and is terminating; details at (:CSSNM00006:) in /u01/app/grid/log/racbde2/cssd/ocssd.log
2009-12-23 19:35:39.043
[cssd(13108)]CRS-1603:CSSD on node racbde2 shutdown by user.
2009-12-23 19:35:39.152
[ohasd(12058)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'racbde2'.
2009-12-23 19:36:00.206
[cssd(13376)]CRS-1713:CSSD daemon is started in clustered mode
2009-12-23 19:36:19.648
[cssd(13376)]CRS-1707:Lease acquisition for node racbde2 number 2 completed
2009-12-23 19:36:19.762
[cssd(13376)]CRS-1605:CSSD voting file is online: /dev/sdb1; details in /u01/app/grid/log/racbde2/cssd/ocssd.log.
2009-12-23 19:36:19.810
[cssd(13376)]CRS-1605:CSSD voting file is online: /dev/sdb3; details in /u01/app/grid/log/racbde2/cssd/ocssd.log.
2009-12-23 19:36:19.857
[cssd(13376)]CRS-1605:CSSD voting file is online: /dev/sdb2; details in /u01/app/grid/log/racbde2/cssd/ocssd.log.
2009-12-23 19:36:31.342
[cssd(13376)]CRS-1601:CSSD Reconfiguration complete. Active nodes are racbde1 racbde2 .
2009-12-23 19:36:32.707
[ctssd(13443)]CRS-2403:The Cluster Time Synchronization Service on host racbde2 is in observer mode.
2009-12-23 19:36:32.860
[ctssd(13443)]CRS-2407:The new Cluster Time Synchronization Service reference node is host racbde1.
2009-12-23 19:36:33.600
[ctssd(13443)]CRS-2401:The Cluster Time Synchronization Service started on host racbde2.
[client(13473)]CRS-10001:ACFS-9327: Verifying ADVM/ACFS devices.
[client(13477)]CRS-10001:ACFS-9322: done.
2009-12-23 19:39:27.166
[crsd(13606)]CRS-1012:The OCR service started on node racbde2.
2009-12-23 19:39:30.419
[crsd(13606)]CRS-1201:CRSD started on node racbde2.

If further analysis is needed, it might be useful to compare a working rootcrs output (mine) to one that is failing (yours) to see what went wrong.  Again the rootcrs log is in <GRID_HOME>/cfgtoollogs/crsconfig/rootcrs_<nodename>.log.  I will divide the log into the following rootcrs sections:

• First Node Initial Setup (racbde1)
• First Node Setup OLR for storing Oracle local registry data
• First Node Setup GPnP wallet and profile
• First Node Setup and copy files for OHASD daemon
• First Node Start OHASD Daemon
• First Node Copy required CRS resources for OHASD to start
• First Node Start in Exclusive Mode and Configure Diskgroup
• First Node Push GPnP Profile to Remote Node(s)
• First Node Start Full Clusterware Stack
• First Node Adding Clusterware Resources
• Secondary Note Initial Setup (racbde2)
• Secondary Note Get GPnP Profile
• Secondary Node Setup OLR for storing Oracle local registry data
• Secondary Node Setup and copy files for OHASD daemon
• Secondary Node Start OHASD Daemon
• Secondary Node Copy required CRS resources for OHASD to start
• Secondary Node Start Full Clusterware Stack
• Secondary Node Adding Clusterware Resources

First Node Initial Setup (racbde1):

2009-12-23 19:24:30: The configuration parameter file /u01/app/grid/crs/install/crsconfig_params is valid
2009-12-23 19:24:30: Checking for super user privileges
2009-12-23 19:24:30: User has super user privileges
2009-12-23 19:24:30: ### Printing the configuration values from files:
2009-12-23 19:24:30:    /u01/app/grid/crs/install/crsconfig_params
2009-12-23 19:24:30:    /u01/app/grid/crs/install/s_crsconfig_defs
2009-12-23 19:24:30: ASM_DISCOVERY_STRING=/dev/sdb*
2009-12-23 19:24:30: ASM_DISKS=/dev/sdb1,/dev/sdb2,/dev/sdb3
2009-12-23 19:24:30: ASM_DISK_GROUP=SYSTEMDG
2009-12-23 19:24:30: ASM_REDUNDANCY=NORMAL
2009-12-23 19:24:30: ASM_SPFILE=
2009-12-23 19:24:30: ASM_UPGRADE=false
2009-12-23 19:24:30: CLSCFG_MISSCOUNT=
2009-12-23 19:24:30: CLUSTER_GUID=
2009-12-23 19:24:30: CLUSTER_NAME=racbde-scan
2009-12-23 19:24:30: CRS_NODEVIPS='racbde1-vip/255.255.254.0/eth1,racbde2-vip/255.255.254.0/eth1'
2009-12-23 19:24:30: CRS_STORAGE_OPTION=1
2009-12-23 19:24:30: CSS_LEASEDURATION=400
2009-12-23 19:24:30: DIRPREFIX=
2009-12-23 19:24:30: DISABLE_OPROCD=0
2009-12-23 19:24:30: EMBASEJAR_NAME=oemlt.jar
2009-12-23 19:24:30: EWTJAR_NAME=ewt3.jar
2009-12-23 19:24:30: EXTERNAL_ORACLE_BIN=/opt/oracle/bin
2009-12-23 19:24:30: GNS_ADDR_LIST=
2009-12-23 19:24:30: GNS_ALLOW_NET_LIST=
2009-12-23 19:24:30: GNS_CONF=false
2009-12-23 19:24:30: GNS_DENY_ITF_LIST=
2009-12-23 19:24:30: GNS_DENY_NET_LIST=
2009-12-23 19:24:30: GNS_DOMAIN_LIST=
2009-12-23 19:24:30: GPNPCONFIGDIR=/u01/app/grid
2009-12-23 19:24:30: GPNPGCONFIGDIR=/u01/app/grid
2009-12-23 19:24:30: GPNP_PA=
2009-12-23 19:24:30: HELPJAR_NAME=help4.jar
2009-12-23 19:24:30: HOST_NAME_LIST=racbde1,racbde2
2009-12-23 19:24:30: ID=/etc/init.d
2009-12-23 19:24:30: INIT=/sbin/init
2009-12-23 19:24:30: IT=/etc/inittab
2009-12-23 19:24:30: JEWTJAR_NAME=jewt4.jar
2009-12-23 19:24:30: JLIBDIR=/u01/app/grid/jlib
2009-12-23 19:24:30: JREDIR=/u01/app/grid/jdk/jre/
2009-12-23 19:24:30: LANGUAGE_ID=AMERICAN_AMERICA.AL32UTF8
2009-12-23 19:24:30: MSGFILE=/var/adm/messages
2009-12-23 19:24:30: NETCFGJAR_NAME=netcfg.jar
2009-12-23 19:24:30: NETWORKS="eth1"/10.141.154.0:public,"eth0"/192.168.0.0:cluster_interconnect
2009-12-23 19:24:30: NEW_HOST_NAME_LIST=
2009-12-23 19:24:30: NEW_NODEVIPS='racbde1-vip/255.255.254.0/eth1,racbde2-vip/255.255.254.0/eth1'
2009-12-23 19:24:30: NEW_NODE_NAME_LIST=
2009-12-23 19:24:30: NEW_PRIVATE_NAME_LIST=
2009-12-23 19:24:30: NODELIST=racbde1,racbde2
2009-12-23 19:24:30: NODE_NAME_LIST=racbde1,racbde2
2009-12-23 19:24:30: OCFS_CONFIG=
2009-12-23 19:24:30: OCRCONFIG=/etc/oracle/ocr.loc
2009-12-23 19:24:30: OCRCONFIGDIR=/etc/oracle
2009-12-23 19:24:30: OCRID=
2009-12-23 19:24:30: OCRLOC=ocr.loc
2009-12-23 19:24:30: OCR_LOCATIONS=NO_VAL
2009-12-23 19:24:30: OLASTGASPDIR=/etc/oracle/lastgasp
2009-12-23 19:24:30: OLRCONFIG=/etc/oracle/olr.loc
2009-12-23 19:24:30: OLRCONFIGDIR=/etc/oracle
2009-12-23 19:24:30: OLRLOC=olr.loc
2009-12-23 19:24:30: OPROCDCHECKDIR=/etc/oracle/oprocd/check
2009-12-23 19:24:30: OPROCDDIR=/etc/oracle/oprocd
2009-12-23 19:24:30: OPROCDFATALDIR=/etc/oracle/oprocd/fatal
2009-12-23 19:24:30: OPROCDSTOPDIR=/etc/oracle/oprocd/stop
2009-12-23 19:24:30: ORACLE_BASE=/u01/app/oracle
2009-12-23 19:24:30: ORACLE_HOME=/u01/app/grid
2009-12-23 19:24:30: ORACLE_OWNER=oracle
2009-12-23 19:24:30: ORA_ASM_GROUP=oinstall
2009-12-23 19:24:30: ORA_DBA_GROUP=oinstall
2009-12-23 19:24:30: PRIVATE_NAME_LIST=
2009-12-23 19:24:30: RCALLDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc3.d /etc/rc.d/rc4.d /etc/rc.d/rc5.d /etc/rc.d/rc6.d
2009-12-23 19:24:30: RCKDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc4.d /etc/rc.d/rc6.d
2009-12-23 19:24:30: RCSDIR=/etc/rc.d/rc3.d /etc/rc.d/rc5.d
2009-12-23 19:24:30: RC_KILL=K19
2009-12-23 19:24:30: RC_KILL_OLD=K96
2009-12-23 19:24:30: RC_START=S96
2009-12-23 19:24:30: SCAN_NAME=racbde-scan.us.oracle.com
2009-12-23 19:24:30: SCAN_PORT=1521
2009-12-23 19:24:30: SCRBASE=/etc/oracle/scls_scr
2009-12-23 19:24:30: SHAREJAR_NAME=share.jar
2009-12-23 19:24:30: SILENT=false
2009-12-23 19:24:30: SO_EXT=so
2009-12-23 19:24:30: SRVCFGLOC=srvConfig.loc
2009-12-23 19:24:30: SRVCONFIG=/var/opt/oracle/srvConfig.loc
2009-12-23 19:24:30: SRVCONFIGDIR=/var/opt/oracle
2009-12-23 19:24:30: VNDR_CLUSTER=false
2009-12-23 19:24:30: VOTING_DISKS=NO_VAL
2009-12-23 19:24:30: ### Printing other configuration values ###
2009-12-23 19:24:30: CLSCFG_EXTRA_PARMS=
2009-12-23 19:24:30: CRSDelete=0
2009-12-23 19:24:30: CRSPatch=0
2009-12-23 19:24:30: DEBUG=
2009-12-23 19:24:30: DOWNGRADE=
2009-12-23 19:24:30: HAS_GROUP=oinstall
2009-12-23 19:24:30: HAS_USER=root
2009-12-23 19:24:30: HOST=racbde1
2009-12-23 19:24:30: IS_SIHA=0
2009-12-23 19:24:30: OLR_DIRECTORY=/u01/app/grid/cdata
2009-12-23 19:24:30: OLR_LOCATION=/u01/app/grid/cdata/racbde1.olr
2009-12-23 19:24:30: ORA_CRS_HOME=/u01/app/grid
2009-12-23 19:24:30: SUPERUSER=root
2009-12-23 19:24:30: UPGRADE=
2009-12-23 19:24:30: VF_DISCOVERY_STRING=
2009-12-23 19:24:30: addfile=/u01/app/grid/crs/install/crsconfig_addparams
2009-12-23 19:24:30: crscfg_trace=1
2009-12-23 19:24:30: crscfg_trace_file=/u01/app/grid/cfgtoollogs/crsconfig/rootcrs_racbde1.log
2009-12-23 19:24:30: hosts=
2009-12-23 19:24:30: oldcrshome=
2009-12-23 19:24:30: oldcrsver=
2009-12-23 19:24:30: osdfile=/u01/app/grid/crs/install/s_crsconfig_defs
2009-12-23 19:24:30: parameters_valid=1
2009-12-23 19:24:30: paramfile=/u01/app/grid/crs/install/crsconfig_params
2009-12-23 19:24:30: platform_family=unix
2009-12-23 19:24:30: srvctl_trc_suff=0
2009-12-23 19:24:30: unlock_crshome=
2009-12-23 19:24:30: user_is_superuser=1
2009-12-23 19:24:30: ### Printing of configuration values complete ###
2009-12-23 19:24:30: Oracle CRS stack is not configured yet
2009-12-23 19:24:30: CRS is not yet configured. Hence, will proceed to configure CRS
2009-12-23 19:24:30: Cluster-wide one-time actions... Done!
2009-12-23 19:24:32: Oracle CRS home = /u01/app/grid
2009-12-23 19:24:32: Host name = racbde1
2009-12-23 19:24:32: CRS user = oracle
2009-12-23 19:24:32: Oracle CRS home = /u01/app/grid
2009-12-23 19:24:32: GPnP host = racbde1
2009-12-23 19:24:32: Oracle GPnP home = /u01/app/grid/gpnp
2009-12-23 19:24:32: Oracle GPnP local home = /u01/app/grid/gpnp/racbde1
2009-12-23 19:24:32: GPnP directories verified. 
2009-12-23 19:24:32: Checking to see if Oracle CRS stack is already configured
2009-12-23 19:24:32: Oracle CRS stack is not configured yet
2009-12-23 19:24:32: ---Checking local gpnp setup...
2009-12-23 19:24:32: The setup file "/u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml" does not exist
2009-12-23 19:24:32: The setup file "/u01/app/grid/gpnp/racbde1/wallets/peer/cwallet.sso" does not exist
2009-12-23 19:24:32: The setup file "/u01/app/grid/gpnp/racbde1/wallets/prdr/cwallet.sso" does not exist
2009-12-23 19:24:32: chk gpnphome /u01/app/grid/gpnp/racbde1: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
2009-12-23 19:24:32: chk gpnphome /u01/app/grid/gpnp/racbde1: INVALID (bad profile/wallet)
2009-12-23 19:24:32: ---Checking cluster-wide gpnp setup...
2009-12-23 19:24:32: The setup file "/u01/app/grid/gpnp/profiles/peer/profile.xml" does not exist
2009-12-23 19:24:32: The setup file "/u01/app/grid/gpnp/wallets/peer/cwallet.sso" does not exist
2009-12-23 19:24:32: The setup file "/u01/app/grid/gpnp/wallets/prdr/cwallet.sso" does not exist
2009-12-23 19:24:32: chk gpnphome /u01/app/grid/gpnp: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
2009-12-23 19:24:32: chk gpnphome /u01/app/grid/gpnp: INVALID (bad profile/wallet)
2009-12-23 19:24:32: gpnp setup checked: local valid? 0 cluster-wide valid? 0
2009-12-23 19:24:32: gpnp setup: NONE
2009-12-23 19:24:32: GPNP configuration required
2009-12-23 19:24:32: Validating for SI-CSS configuration

First Node Setup OLR for storing Oracle local registry data:

2009-12-23 19:24:32: Retrieving OCR main disk location
2009-12-23 19:24:32: Opening file OCRCONFIG
2009-12-23 19:24:32: Value () is set for key=ocrconfig_loc
2009-12-23 19:24:32: Unable to retrieve ocr disk info
2009-12-23 19:24:32: Checking to see if any 9i GSD is up
2009-12-23 19:24:32: libskgxnBase_lib = /etc/ORCLcluster/oracm/lib/libskgxn2.so
2009-12-23 19:24:32: libskgxn_lib = /opt/ORCLcluster/lib/libskgxn2.so
2009-12-23 19:24:32: SKGXN library file does not exists
2009-12-23 19:24:32: OLR location = /u01/app/grid/cdata/racbde1.olr
2009-12-23 19:24:32: Oracle CRS Home = /u01/app/grid
2009-12-23 19:24:32: Validating /etc/oracle/olr.loc file for OLR location /u01/app/grid/cdata/racbde1.olr
2009-12-23 19:24:32: /etc/oracle/olr.loc already exists. Backing up /etc/oracle/olr.loc to /etc/oracle/olr.loc.orig
2009-12-23 19:24:32: Oracle CRS home = /u01/app/grid
2009-12-23 19:24:32: Oracle cluster name = racbde-scan
2009-12-23 19:24:32: OCR locations = +SYSTEMDG
2009-12-23 19:24:32: Validating OCR
2009-12-23 19:24:32: Retrieving OCR location used by previous installations
2009-12-23 19:24:32: Opening file OCRCONFIG
2009-12-23 19:24:32: Value () is set for key=ocrconfig_loc
2009-12-23 19:24:32: Opening file OCRCONFIG
2009-12-23 19:24:32: Value () is set for key=ocrmirrorconfig_loc
2009-12-23 19:24:32: Opening file OCRCONFIG
2009-12-23 19:24:32: Value () is set for key=ocrconfig_loc3
2009-12-23 19:24:32: Opening file OCRCONFIG
2009-12-23 19:24:32: Value () is set for key=ocrconfig_loc4
2009-12-23 19:24:32: Opening file OCRCONFIG
2009-12-23 19:24:32: Value () is set for key=ocrconfig_loc5
2009-12-23 19:24:32: Checking if OCR sync file exists
2009-12-23 19:24:32: No need to sync OCR file
2009-12-23 19:24:32: OCR_LOCATION=+SYSTEMDG
2009-12-23 19:24:32: OCR_MIRROR_LOCATION=
2009-12-23 19:24:32: OCR_MIRROR_LOC3=
2009-12-23 19:24:32: OCR_MIRROR_LOC4=
2009-12-23 19:24:32: OCR_MIRROR_LOC5=
2009-12-23 19:24:32: Current OCR location=
2009-12-23 19:24:32: Current OCR mirror location=
2009-12-23 19:24:32: Current OCR mirror loc3=
2009-12-23 19:24:32: Current OCR mirror loc4=
2009-12-23 19:24:32: Current OCR mirror loc5=
2009-12-23 19:24:32: Verifying current OCR settings with user entered values
2009-12-23 19:24:32: Setting OCR locations in /etc/oracle/ocr.loc
2009-12-23 19:24:32: Validating OCR locations in /etc/oracle/ocr.loc
2009-12-23 19:24:32: Checking for existence of /etc/oracle/ocr.loc
2009-12-23 19:24:32: Backing up /etc/oracle/ocr.loc to /etc/oracle/ocr.loc.orig
2009-12-23 19:24:32: Setting ocr location +SYSTEMDG
2009-12-23 19:24:32: Creating or upgrading Oracle Local Registry (OLR)
2009-12-23 19:24:34: OLR successfully created or upgraded
2009-12-23 19:24:34: /u01/app/grid/bin/clscfg -localadd
2009-12-23 19:24:34: Keys created in the OLR successfully

First Node Setup GPnP wallet and profile:

2009-12-23 19:24:34: GPnP setup state: none
2009-12-23 19:24:34: Creating local GPnP setup for clustered node...
2009-12-23 19:24:34: Oracle CRS home = /u01/app/grid
2009-12-23 19:24:34: Oracle GPnP wallets home = /u01/app/grid/gpnp/racbde1/wallets
2009-12-23 19:24:34: Checking if GPnP setup exists
2009-12-23 19:24:34: /u01/app/grid/gpnp/racbde1/wallets/peer/cwallet.sso wallet must be created
2009-12-23 19:24:34: Removing old wallets/certificates, if any
2009-12-23 19:24:34: Creating GPnP Root Wallet...
2009-12-23 19:24:34: /u01/app/grid/bin/orapki wallet create -wallet "/u01/app/grid/gpnp/racbde1/wallets/root" -pwd gpnp_wallet1 -nologo
2009-12-23 19:24:35: Creating GPnP Root Certificate...
2009-12-23 19:24:35: /u01/app/grid/bin/orapki wallet add -wallet "/u01/app/grid/gpnp/racbde1/wallets/root" -pwd gpnp_wallet1 -self_signed -dn "CN=GPnP_root" -keysize 1024 -validity 9999 -nologo
2009-12-23 19:24:37: Exporting GPnP Root Certificate...
2009-12-23 19:24:37: /u01/app/grid/bin/orapki wallet export -wallet "/u01/app/grid/gpnp/racbde1/wallets/root" -pwd gpnp_wallet1 -dn "CN=GPnP_root" -cert "/u01/app/grid/gpnp/racbde1/wallets/root/b64certificate.txt" -nologo
2009-12-23 19:24:38: Creating GPnP Peer Wallet...
2009-12-23 19:24:38: /u01/app/grid/bin/orapki wallet create -wallet "/u01/app/grid/gpnp/racbde1/wallets/peer" -pwd gpnp_wallet1 -auto_login -nologo
2009-12-23 19:24:38: Creating GPnP Profile Reader Wallet...
2009-12-23 19:24:38: /u01/app/grid/bin/orapki wallet create -wallet "/u01/app/grid/gpnp/racbde1/wallets/prdr" -pwd gpnp_wallet1 -auto_login -nologo
2009-12-23 19:24:39: Creating GPnP PA Wallet...
2009-12-23 19:24:39: /u01/app/grid/bin/orapki wallet create -wallet "/u01/app/grid/gpnp/racbde1/wallets/pa" -pwd gpnp_wallet1 -auto_login -nologo
2009-12-23 19:24:40: Adding private key to GPnP Peer Wallet...
2009-12-23 19:24:40: /u01/app/grid/bin/orapki wallet add -wallet "/u01/app/grid/gpnp/racbde1/wallets/peer" -pwd gpnp_wallet1 -dn "CN=GPnP_peer" -keysize 1024 -nologo
2009-12-23 19:24:41: Adding private key to GPnP PA Wallet...
2009-12-23 19:24:41: /u01/app/grid/bin/orapki wallet add -wallet "/u01/app/grid/gpnp/racbde1/wallets/pa" -pwd gpnp_wallet1 -dn "CN=GPnP_pa" -keysize 1024 -nologo
2009-12-23 19:24:42: Creating certificate request for GPnP Peer Wallet...
2009-12-23 19:24:42: /u01/app/grid/bin/orapki wallet export -wallet "/u01/app/grid/gpnp/racbde1/wallets/peer" -pwd gpnp_wallet1 -dn "CN=GPnP_peer" -request "/u01/app/grid/gpnp/racbde1/wallets/peer/certreq.txt" -nologo
2009-12-23 19:24:43: Creating certificate request for GPnP PA Wallet...
2009-12-23 19:24:43: /u01/app/grid/bin/orapki wallet export -wallet "/u01/app/grid/gpnp/racbde1/wallets/pa" -pwd gpnp_wallet1 -dn "CN=GPnP_pa" -request "/u01/app/grid/gpnp/racbde1/wallets/pa/certreq.txt" -nologo
2009-12-23 19:24:44: Creating certificate for GPnP Peer Wallet...
2009-12-23 19:24:44: /u01/app/grid/bin/orapki cert create -wallet "/u01/app/grid/gpnp/racbde1/wallets/root" -pwd gpnp_wallet1 -request "/u01/app/grid/gpnp/racbde1/wallets/peer/certreq.txt" -cert "/u01/app/grid/gpnp/racbde1/wallets/peer/cert.txt" -validity 9999 -nologo
2009-12-23 19:24:45: Creating certificate for GPnP PA Wallet...
2009-12-23 19:24:45: /u01/app/grid/bin/orapki cert create -wallet "/u01/app/grid/gpnp/racbde1/wallets/root" -pwd gpnp_wallet1 -request "/u01/app/grid/gpnp/racbde1/wallets/pa/certreq.txt" -cert "/u01/app/grid/gpnp/racbde1/wallets/pa/cert.txt" -validity 9999 -nologo
2009-12-23 19:24:46: Adding Root Certificate TP to GPnP Peer Wallet...
2009-12-23 19:24:46: /u01/app/grid/bin/orapki wallet add -wallet "/u01/app/grid/gpnp/racbde1/wallets/peer" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/grid/gpnp/racbde1/wallets/root/b64certificate.txt" -nologo
2009-12-23 19:24:47: Adding Root Certificate TP to GPnP Profile Reader Wallet...
2009-12-23 19:24:47: /u01/app/grid/bin/orapki wallet add -wallet "/u01/app/grid/gpnp/racbde1/wallets/prdr" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/grid/gpnp/racbde1/wallets/root/b64certificate.txt" -nologo
2009-12-23 19:24:48: Adding Root Certificate TP to GPnP PA Wallet...
2009-12-23 19:24:48: /u01/app/grid/bin/orapki wallet add -wallet "/u01/app/grid/gpnp/racbde1/wallets/pa" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/grid/gpnp/racbde1/wallets/root/b64certificate.txt" -nologo
2009-12-23 19:24:49: Adding PA Certificate as a TP into a GPnP Peer Wallet...
2009-12-23 19:24:49: /u01/app/grid/bin/orapki wallet add -wallet "/u01/app/grid/gpnp/racbde1/wallets/peer" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/grid/gpnp/racbde1/wallets/pa/cert.txt" -nologo
2009-12-23 19:24:50: Adding peer Certificate as a TP into a GPnP PA Wallet...
2009-12-23 19:24:50: /u01/app/grid/bin/orapki wallet add -wallet "/u01/app/grid/gpnp/racbde1/wallets/pa" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/grid/gpnp/racbde1/wallets/peer/cert.txt" -nologo
2009-12-23 19:24:51: Adding PA Certificate as a TP into a GPnP Profile Reader Wallet...
2009-12-23 19:24:51: /u01/app/grid/bin/orapki wallet add -wallet "/u01/app/grid/gpnp/racbde1/wallets/prdr" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/grid/gpnp/racbde1/wallets/pa/cert.txt" -nologo
2009-12-23 19:24:52: Adding peer Certificate as a TP into a GPnP Profile Reader Wallet...
2009-12-23 19:24:52: /u01/app/grid/bin/orapki wallet add -wallet "/u01/app/grid/gpnp/racbde1/wallets/prdr" -pwd gpnp_wallet1 -trusted_cert -cert "/u01/app/grid/gpnp/racbde1/wallets/peer/cert.txt" -nologo
2009-12-23 19:24:53: Adding PA Certificate as a TP into a GPnP Peer Wallet...
2009-12-23 19:24:53: /u01/app/grid/bin/orapki wallet add -wallet "/u01/app/grid/gpnp/racbde1/wallets/peer" -pwd gpnp_wallet1 -user_cert -cert "/u01/app/grid/gpnp/racbde1/wallets/peer/cert.txt" -nologo
2009-12-23 19:24:54: Adding peer Certificate as a TP into a GPnP PA Wallet...
2009-12-23 19:24:54: /u01/app/grid/bin/orapki wallet add -wallet "/u01/app/grid/gpnp/racbde1/wallets/pa" -pwd gpnp_wallet1 -user_cert -cert "/u01/app/grid/gpnp/racbde1/wallets/pa/cert.txt" -nologo
2009-12-23 19:24:55: GPnP Wallets ownership/permissions successfully set.
2009-12-23 19:24:55: GPnP Wallets successfully created.
2009-12-23 19:24:55: <--- GPnP wallets successfully created
2009-12-23 19:24:55: Creating GPnP peer profile --->
2009-12-23 19:24:55: Oracle CRS home = /u01/app/grid
2009-12-23 19:24:55: Oracle GPnP profiles home = /u01/app/grid/gpnp/racbde1/profiles
2009-12-23 19:24:55: Oracle GPnP profiles parameters: 
2009-12-23 19:24:55:    paloc==
2009-12-23 19:24:55:    cname=racbde-scan=
2009-12-23 19:24:55:    cssdisco==
2009-12-23 19:24:55:    cssld=400=
2009-12-23 19:24:55:    asmdisco=/dev/sdb*=
2009-12-23 19:24:55:    asmspf==
2009-12-23 19:24:55:    netlst="eth1"/10.141.154.0:public,"eth0"/192.168.0.0:cluster_interconnect=
2009-12-23 19:24:55:    ocrid==
2009-12-23 19:24:55:    clusterguid==
2009-12-23 19:24:55: Checking if GPnP setup exists
2009-12-23 19:24:55: /u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml profile must be created
2009-12-23 19:24:55: OCRID is not available, hence not set in GPnP Profile
2009-12-23 19:24:55: ClusterGUID is not available, hence not set in GPnP Profile
2009-12-23 19:24:55: gpnptool: run /u01/app/grid/bin/gpnptool create -o="/u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml" -ovr -prf -prf_sq=1 -prf_cn=racbde-scan -prf_pa="" -hnet=gen -gen:hnet_nm="*" -gen:net=net1 -net1:net_ip="10.141.154.0" -net1:net_ada="eth1" -net1:net_use="public" -gen:net=net2 -net2:net_ip="192.168.0.0" -net2:net_ada="eth0" -net2:net_use="cluster_interconnect" -css=css -css:css_dis="" -css:css_ld=400 -asm=asm -asm:asm_dis="/dev/sdb*" -asm:asm_spf=""
2009-12-23 19:24:55: Running as user oracle: /u01/app/grid/bin/gpnptool create -o="/u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml" -ovr -prf -prf_sq=1 -prf_cn=racbde-scan -prf_pa="" -hnet=gen -gen:hnet_nm="*" -gen:net=net1 -net1:net_ip="10.141.154.0" -net1:net_ada="eth1" -net1:net_use="public" -gen:net=net2 -net2:net_ip="192.168.0.0" -net2:net_ada="eth0" -net2:net_use="cluster_interconnect" -css=css -css:css_dis="" -css:css_ld=400 -asm=asm -asm:asm_dis="/dev/sdb*" -asm:asm_spf=""
2009-12-23 19:24:55: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/gpnptool create -o="/u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml" -ovr -prf -prf_sq=1 -prf_cn=racbde-scan -prf_pa="" -hnet=gen -gen:hnet_nm="*" -gen:net=net1 -net1:net_ip="10.141.154.0" -net1:net_ada="eth1" -net1:net_use="public" -gen:net=net2 -net2:net_ip="192.168.0.0" -net2:net_ada="eth0" -net2:net_use="cluster_interconnect" -css=css -css:css_dis="" -css:css_ld=400 -asm=asm -asm:asm_dis="/dev/sdb*" -asm:asm_spf="" '
2009-12-23 19:24:55: Removing file /tmp/fileDple0c
2009-12-23 19:24:55: Successfully removed file: /tmp/fileDple0c
2009-12-23 19:24:55: /bin/su successfully executed

2009-12-23 19:24:55: gpnptool: rc=0
2009-12-23 19:24:55: gpnptool output:
Resulting profile written to "/u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml".
Success.

2009-12-23 19:24:55: gpnptool: run /u01/app/grid/bin/gpnptool sign -p="/u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml" -o="/u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml" -ovr -w="file:/u01/app/grid/gpnp/racbde1/wallets/peer" -rmws
2009-12-23 19:24:55: Running as user oracle: /u01/app/grid/bin/gpnptool sign -p="/u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml" -o="/u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml" -ovr -w="file:/u01/app/grid/gpnp/racbde1/wallets/peer" -rmws
2009-12-23 19:24:55: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/gpnptool sign -p="/u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml" -o="/u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml" -ovr -w="file:/u01/app/grid/gpnp/racbde1/wallets/peer" -rmws '
2009-12-23 19:24:55: Removing file /tmp/fileWLaWwU
2009-12-23 19:24:55: Successfully removed file: /tmp/fileWLaWwU
2009-12-23 19:24:55: /bin/su successfully executed

2009-12-23 19:24:55: gpnptool: rc=0
2009-12-23 19:24:55: gpnptool output:
Resulting profile written to "/u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml".
Success.

2009-12-23 19:24:55: GPnP peer profile create successfully completed.
2009-12-23 19:24:55: <--- GPnP peer profile successfully created
2009-12-23 19:24:55: GPnP local setup successfully created

First Node Setup and copy files for OHASD daemon:

2009-12-23 19:24:55: Registering ohasd
2009-12-23 19:24:55: init file = /u01/app/grid/crs/init/init.ohasd
2009-12-23 19:24:55: Copying file /u01/app/grid/crs/init/init.ohasd to /etc/init.d directory
2009-12-23 19:24:55: Setting init.ohasd permission in /etc/init.d directory
2009-12-23 19:24:55: init file = /u01/app/grid/crs/init/ohasd
2009-12-23 19:24:55: Copying file /u01/app/grid/crs/init/ohasd to /etc/init.d directory
2009-12-23 19:24:55: Setting ohasd permission in /etc/init.d directory
2009-12-23 19:24:55: Removing "/etc/rc.d/rc3.d/S96ohasd"
2009-12-23 19:24:55: Removing file /etc/rc.d/rc3.d/S96ohasd
2009-12-23 19:24:55: Failure with return code 1 from command rm /etc/rc.d/rc3.d/S96ohasd
2009-12-23 19:24:55: Failed to remove file: 
2009-12-23 19:24:55: Creating a link "/etc/rc.d/rc3.d/S96ohasd" pointing to /etc/init.d/ohasd
2009-12-23 19:24:55: Removing "/etc/rc.d/rc5.d/S96ohasd"
2009-12-23 19:24:55: Removing file /etc/rc.d/rc5.d/S96ohasd
2009-12-23 19:24:55: Failure with return code 1 from command rm /etc/rc.d/rc5.d/S96ohasd
2009-12-23 19:24:55: Failed to remove file: 
2009-12-23 19:24:55: Creating a link "/etc/rc.d/rc5.d/S96ohasd" pointing to /etc/init.d/ohasd
2009-12-23 19:24:55: Removing "/etc/rc.d/rc0.d/K19ohasd"
2009-12-23 19:24:55: Removing file /etc/rc.d/rc0.d/K19ohasd
2009-12-23 19:24:55: Failure with return code 1 from command rm /etc/rc.d/rc0.d/K19ohasd
2009-12-23 19:24:55: Failed to remove file: 
2009-12-23 19:24:55: Creating a link "/etc/rc.d/rc0.d/K19ohasd" pointing to /etc/init.d/ohasd
2009-12-23 19:24:55: Removing "/etc/rc.d/rc1.d/K19ohasd"
2009-12-23 19:24:55: Removing file /etc/rc.d/rc1.d/K19ohasd
2009-12-23 19:24:55: Failure with return code 1 from command rm /etc/rc.d/rc1.d/K19ohasd
2009-12-23 19:24:55: Failed to remove file: 
2009-12-23 19:24:55: Creating a link "/etc/rc.d/rc1.d/K19ohasd" pointing to /etc/init.d/ohasd
2009-12-23 19:24:55: Removing "/etc/rc.d/rc2.d/K19ohasd"
2009-12-23 19:24:55: Removing file /etc/rc.d/rc2.d/K19ohasd
2009-12-23 19:24:55: Failure with return code 1 from command rm /etc/rc.d/rc2.d/K19ohasd
2009-12-23 19:24:55: Failed to remove file: 
2009-12-23 19:24:55: Creating a link "/etc/rc.d/rc2.d/K19ohasd" pointing to /etc/init.d/ohasd
2009-12-23 19:24:55: Removing "/etc/rc.d/rc4.d/K19ohasd"
2009-12-23 19:24:55: Removing file /etc/rc.d/rc4.d/K19ohasd
2009-12-23 19:24:56: Failure with return code 1 from command rm /etc/rc.d/rc4.d/K19ohasd
2009-12-23 19:24:56: Failed to remove file: 
2009-12-23 19:24:56: Creating a link "/etc/rc.d/rc4.d/K19ohasd" pointing to /etc/init.d/ohasd
2009-12-23 19:24:56: Removing "/etc/rc.d/rc6.d/K19ohasd"
2009-12-23 19:24:56: Removing file /etc/rc.d/rc6.d/K19ohasd
2009-12-23 19:24:56: Failure with return code 1 from command rm /etc/rc.d/rc6.d/K19ohasd
2009-12-23 19:24:56: Failed to remove file: 
2009-12-23 19:24:56: Creating a link "/etc/rc.d/rc6.d/K19ohasd" pointing to /etc/init.d/ohasd
2009-12-23 19:24:56: The file ohasd has been successfully linked to the RC directories

First Node Start OHASD Daemon:

2009-12-23 19:24:56: Starting ohasd
2009-12-23 19:24:56: itab entries=
2009-12-23 19:25:01: Created backup /etc/inittab.no_crs
2009-12-23 19:25:01: Appending to /etc/inittab.tmp:
2009-12-23 19:25:01: h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

2009-12-23 19:25:01: Done updating /etc/inittab.tmp
2009-12-23 19:25:01: Saved /etc/inittab.crs
2009-12-23 19:25:01: Installed new /etc/inittab
2009-12-23 19:25:06: ohasd is starting
2009-12-23 19:25:06: Checking ohasd
2009-12-23 19:25:06: ohasd started successfully

First Node Copy required CRS resources for OHASD to start:

2009-12-23 19:25:06: Creating CRS resources and dependencies
2009-12-23 19:25:06: Configuring HASD
2009-12-23 19:25:06: Registering type ora.daemon.type
2009-12-23 19:25:07: Registering type ora.mdns.type
2009-12-23 19:25:07: Registering type ora.gpnp.type
2009-12-23 19:25:07: Registering type ora.gipc.type
2009-12-23 19:25:07: Registering type ora.cssd.type
2009-12-23 19:25:08: Registering type ora.cssdmonitor.type
2009-12-23 19:25:08: Registering type ora.crs.type
2009-12-23 19:25:08: Registering type ora.evm.type
2009-12-23 19:25:08: Registering type ora.ctss.type
2009-12-23 19:25:08: Registering type ora.asm.type
2009-12-23 19:25:09: Registering type ora.drivers.acfs.type
2009-12-23 19:25:09: Registering type ora.diskmon.type
2009-12-23 19:25:28: ADVM/ACFS is configured

2009-12-23 19:25:28: Successfully created CRS resources for cluster daemon and ASM
2009-12-23 19:25:28: Checking if initial configuration has been performed

First Node Start in Exclusive Mode and Configure Diskgroup:

2009-12-23 19:25:28: Starting CSS in exclusive mode
2009-12-23 19:25:55: CRS-2672: Attempting to start 'ora.gipcd' on 'racbde1'
2009-12-23 19:25:55: CRS-2672: Attempting to start 'ora.mdnsd' on 'racbde1'
2009-12-23 19:25:55: CRS-2676: Start of 'ora.gipcd' on 'racbde1' succeeded
2009-12-23 19:25:55: CRS-2676: Start of 'ora.mdnsd' on 'racbde1' succeeded
2009-12-23 19:25:55: CRS-2672: Attempting to start 'ora.gpnpd' on 'racbde1'
2009-12-23 19:25:55: CRS-2676: Start of 'ora.gpnpd' on 'racbde1' succeeded
2009-12-23 19:25:55: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racbde1'
2009-12-23 19:25:55: CRS-2676: Start of 'ora.cssdmonitor' on 'racbde1' succeeded
2009-12-23 19:25:55: CRS-2672: Attempting to start 'ora.cssd' on 'racbde1'
2009-12-23 19:25:55: CRS-2672: Attempting to start 'ora.diskmon' on 'racbde1'
2009-12-23 19:25:55: CRS-2676: Start of 'ora.diskmon' on 'racbde1' succeeded
2009-12-23 19:25:55: CRS-2676: Start of 'ora.cssd' on 'racbde1' succeeded
2009-12-23 19:25:55: Querying for existing CSS voting disks
2009-12-23 19:25:55: Performing initial configuration for cluster
2009-12-23 19:25:56: Start of resource "ora.ctssd -init" Succeeded
2009-12-23 19:25:56: Configuring ASM via ASMCA
2009-12-23 19:25:56: Executing as oracle: /u01/app/grid/bin/asmca -silent -diskGroupName SYSTEMDG -diskList /dev/sdb1,/dev/sdb2,/dev/sdb3 -redundancy NORMAL -diskString '/dev/sdb*' -configureLocalASM
2009-12-23 19:25:56: Running as user oracle: /u01/app/grid/bin/asmca -silent -diskGroupName SYSTEMDG -diskList /dev/sdb1,/dev/sdb2,/dev/sdb3 -redundancy NORMAL -diskString '/dev/sdb*' -configureLocalASM
2009-12-23 19:25:56:   Invoking "/u01/app/grid/bin/asmca -silent -diskGroupName SYSTEMDG -diskList /dev/sdb1,/dev/sdb2,/dev/sdb3 -redundancy NORMAL -diskString '/dev/sdb*' -configureLocalASM" as user "oracle"
2009-12-23 19:30:21: Creating or upgrading OCR keys
2009-12-23 19:30:31: OCR keys are successfully populated
2009-12-23 19:30:31: Invalidating repository location for Oracle 9i deployments
2009-12-23 19:30:31: Executing clscfg
2009-12-23 19:30:38: Oracle Cluster Registry initialization completed
2009-12-23 19:30:40: Start of resource "ora.crsd -init" Succeeded
2009-12-23 19:30:40: Creating voting files
2009-12-23 19:30:40: Creating voting files in ASM diskgroup SYSTEMDG
2009-12-23 19:30:40: Executing crsctl replace votedisk +SYSTEMDG

First Node Push GPnP Profile to Remote Node(s):

2009-12-23 19:31:55: Promoting local gpnp setup to cluster-wide. Nodes {racbde1,racbde2}
2009-12-23 19:31:55: Pushing local gpnpsetup to cluster nodes: {racbde1,racbde2}
2009-12-23 19:31:55:   /tmp/fileg6883M =>  /u01/app/grid/gpnp/manifest.txt
2009-12-23 19:31:55:      rmtcpy: /u01/app/grid/bin/cluutil -sourcefile /tmp/fileg6883M -destfile /u01/app/grid/gpnp/manifest.txt -nodelist racbde1,racbde2
2009-12-23 19:31:55: Running as user oracle: /u01/app/grid/bin/cluutil -sourcefile /tmp/fileg6883M -destfile /u01/app/grid/gpnp/manifest.txt -nodelist racbde1,racbde2
2009-12-23 19:31:55: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/cluutil -sourcefile /tmp/fileg6883M -destfile /u01/app/grid/gpnp/manifest.txt -nodelist racbde1,racbde2 '
2009-12-23 19:31:56: Removing file /tmp/fileMlmyGF
2009-12-23 19:31:56: Successfully removed file: /tmp/fileMlmyGF
2009-12-23 19:31:56: /bin/su successfully executed

2009-12-23 19:31:56:   /u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml =>  /u01/app/grid/gpnp/profiles/peer/profile.xml
2009-12-23 19:31:56:      rmtcpy: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml -destfile /u01/app/grid/gpnp/profiles/peer/profile.xml -nodelist racbde1,racbde2
2009-12-23 19:31:56: Running as user oracle: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml -destfile /u01/app/grid/gpnp/profiles/peer/profile.xml -nodelist racbde1,racbde2
2009-12-23 19:31:56: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/profiles/peer/profile.xml -destfile /u01/app/grid/gpnp/profiles/peer/profile.xml -nodelist racbde1,racbde2 '
2009-12-23 19:31:56: Removing file /tmp/filePm9QYA
2009-12-23 19:31:56: Successfully removed file: /tmp/filePm9QYA
2009-12-23 19:31:56: /bin/su successfully executed

2009-12-23 19:31:56:   /u01/app/grid/gpnp/racbde1/wallets/peer/cwallet.sso =>  /u01/app/grid/gpnp/wallets/peer/cwallet.sso
2009-12-23 19:31:56:      rmtcpy: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/peer/cwallet.sso -destfile /u01/app/grid/gpnp/wallets/peer/cwallet.sso -nodelist racbde1,racbde2
2009-12-23 19:31:56: Running as user oracle: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/peer/cwallet.sso -destfile /u01/app/grid/gpnp/wallets/peer/cwallet.sso -nodelist racbde1,racbde2
2009-12-23 19:31:56: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/peer/cwallet.sso -destfile /u01/app/grid/gpnp/wallets/peer/cwallet.sso -nodelist racbde1,racbde2 '
2009-12-23 19:31:57: Removing file /tmp/file2trrwx
2009-12-23 19:31:57: Successfully removed file: /tmp/file2trrwx
2009-12-23 19:31:57: /bin/su successfully executed

2009-12-23 19:31:57:   /u01/app/grid/gpnp/racbde1/profiles/peer/profile_orig.xml =>  /u01/app/grid/gpnp/profiles/peer/profile_orig.xml
2009-12-23 19:31:57:      rmtcpy: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/profiles/peer/profile_orig.xml -destfile /u01/app/grid/gpnp/profiles/peer/profile_orig.xml -nodelist racbde1,racbde2
2009-12-23 19:31:57: Running as user oracle: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/profiles/peer/profile_orig.xml -destfile /u01/app/grid/gpnp/profiles/peer/profile_orig.xml -nodelist racbde1,racbde2
2009-12-23 19:31:57: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/profiles/peer/profile_orig.xml -destfile /u01/app/grid/gpnp/profiles/peer/profile_orig.xml -nodelist racbde1,racbde2 '
2009-12-23 19:31:57: Removing file /tmp/fileZqRXjv
2009-12-23 19:31:57: Successfully removed file: /tmp/fileZqRXjv
2009-12-23 19:31:57: /bin/su successfully executed

2009-12-23 19:31:57:   /u01/app/grid/gpnp/racbde1/wallets/root/ewallet.p12 =>  /u01/app/grid/gpnp/wallets/root/ewallet.p12
2009-12-23 19:31:57:      rmtcpy: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/root/ewallet.p12 -destfile /u01/app/grid/gpnp/wallets/root/ewallet.p12 -nodelist racbde1,racbde2
2009-12-23 19:31:57: Running as user oracle: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/root/ewallet.p12 -destfile /u01/app/grid/gpnp/wallets/root/ewallet.p12 -nodelist racbde1,racbde2
2009-12-23 19:31:57: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/root/ewallet.p12 -destfile /u01/app/grid/gpnp/wallets/root/ewallet.p12 -nodelist racbde1,racbde2 '
2009-12-23 19:31:57: Removing file /tmp/filecX5Reu
2009-12-23 19:31:57: Successfully removed file: /tmp/filecX5Reu
2009-12-23 19:31:57: /bin/su successfully executed

2009-12-23 19:31:57:   /u01/app/grid/gpnp/racbde1/wallets/prdr/cwallet.sso =>  /u01/app/grid/gpnp/wallets/prdr/cwallet.sso
2009-12-23 19:31:57:      rmtcpy: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/prdr/cwallet.sso -destfile /u01/app/grid/gpnp/wallets/prdr/cwallet.sso -nodelist racbde1,racbde2
2009-12-23 19:31:57: Running as user oracle: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/prdr/cwallet.sso -destfile /u01/app/grid/gpnp/wallets/prdr/cwallet.sso -nodelist racbde1,racbde2
2009-12-23 19:31:57: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/prdr/cwallet.sso -destfile /u01/app/grid/gpnp/wallets/prdr/cwallet.sso -nodelist racbde1,racbde2 '
2009-12-23 19:31:58: Removing file /tmp/filejENXcu
2009-12-23 19:31:58: Successfully removed file: /tmp/filejENXcu
2009-12-23 19:31:58: /bin/su successfully executed

2009-12-23 19:31:58:   /u01/app/grid/gpnp/racbde1/wallets/pa/cwallet.sso =>  /u01/app/grid/gpnp/wallets/pa/cwallet.sso
2009-12-23 19:31:58:      rmtcpy: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/pa/cwallet.sso -destfile /u01/app/grid/gpnp/wallets/pa/cwallet.sso -nodelist racbde1,racbde2
2009-12-23 19:31:58: Running as user oracle: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/pa/cwallet.sso -destfile /u01/app/grid/gpnp/wallets/pa/cwallet.sso -nodelist racbde1,racbde2
2009-12-23 19:31:58: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/pa/cwallet.sso -destfile /u01/app/grid/gpnp/wallets/pa/cwallet.sso -nodelist racbde1,racbde2 '
2009-12-23 19:31:58: Removing file /tmp/filed8XZpv
2009-12-23 19:31:58: Successfully removed file: /tmp/filed8XZpv
2009-12-23 19:31:58: /bin/su successfully executed

2009-12-23 19:31:58:   /u01/app/grid/gpnp/racbde1/wallets/root/b64certificate.txt =>  /u01/app/grid/gpnp/wallets/root/b64certificate.txt
2009-12-23 19:31:58:      rmtcpy: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/root/b64certificate.txt -destfile /u01/app/grid/gpnp/wallets/root/b64certificate.txt -nodelist racbde1,racbde2
2009-12-23 19:31:58: Running as user oracle: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/root/b64certificate.txt -destfile /u01/app/grid/gpnp/wallets/root/b64certificate.txt -nodelist racbde1,racbde2
2009-12-23 19:31:58: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/root/b64certificate.txt -destfile /u01/app/grid/gpnp/wallets/root/b64certificate.txt -nodelist racbde1,racbde2 '
2009-12-23 19:31:59: Removing file /tmp/file2UPPUx
2009-12-23 19:31:59: Successfully removed file: /tmp/file2UPPUx
2009-12-23 19:31:59: /bin/su successfully executed

2009-12-23 19:31:59:   /u01/app/grid/gpnp/racbde1/wallets/peer/cert.txt =>  /u01/app/grid/gpnp/wallets/peer/cert.txt
2009-12-23 19:31:59:      rmtcpy: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/peer/cert.txt -destfile /u01/app/grid/gpnp/wallets/peer/cert.txt -nodelist racbde1,racbde2
2009-12-23 19:31:59: Running as user oracle: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/peer/cert.txt -destfile /u01/app/grid/gpnp/wallets/peer/cert.txt -nodelist racbde1,racbde2
2009-12-23 19:31:59: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/peer/cert.txt -destfile /u01/app/grid/gpnp/wallets/peer/cert.txt -nodelist racbde1,racbde2 '
2009-12-23 19:31:59: Removing file /tmp/fileRL2LFB
2009-12-23 19:31:59: Successfully removed file: /tmp/fileRL2LFB
2009-12-23 19:31:59: /bin/su successfully executed

2009-12-23 19:31:59:   /u01/app/grid/gpnp/racbde1/wallets/pa/cert.txt =>  /u01/app/grid/gpnp/wallets/pa/cert.txt
2009-12-23 19:31:59:      rmtcpy: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/pa/cert.txt -destfile /u01/app/grid/gpnp/wallets/pa/cert.txt -nodelist racbde1,racbde2
2009-12-23 19:31:59: Running as user oracle: /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/pa/cert.txt -destfile /u01/app/grid/gpnp/wallets/pa/cert.txt -nodelist racbde1,racbde2
2009-12-23 19:31:59: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/cluutil -sourcefile /u01/app/grid/gpnp/racbde1/wallets/pa/cert.txt -destfile /u01/app/grid/gpnp/wallets/pa/cert.txt -nodelist racbde1,racbde2 '
2009-12-23 19:32:00: Removing file /tmp/fileWwdzFG
2009-12-23 19:32:00: Successfully removed file: /tmp/fileWwdzFG
2009-12-23 19:32:00: /bin/su successfully executed

2009-12-23 19:32:00: Exiting exclusive mode

First Node Start Full Clusterware Stack:

2009-12-23 19:32:39: Starting 
2009-12-23 19:32:40: Start of resource "ora.mdnsd -init" Succeeded
2009-12-23 19:32:41: Start of resource "ora.gipcd -init" Succeeded
2009-12-23 19:32:43: Start of resource "ora.gpnpd -init" Succeeded
2009-12-23 19:32:43: Successfully started requested Oracle stack daemons
2009-12-23 19:32:43: Starting CSS in clustered mode
2009-12-23 19:34:26: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racbde1'
2009-12-23 19:34:26: CRS-2676: Start of 'ora.cssdmonitor' on 'racbde1' succeeded
2009-12-23 19:34:26: CRS-2672: Attempting to start 'ora.cssd' on 'racbde1'
2009-12-23 19:34:26: CRS-2672: Attempting to start 'ora.diskmon' on 'racbde1'
2009-12-23 19:34:26: CRS-2676: Start of 'ora.diskmon' on 'racbde1' succeeded
2009-12-23 19:34:26: CRS-2676: Start of 'ora.cssd' on 'racbde1' succeeded
2009-12-23 19:34:27: Start of resource "ora.ctssd -init -env USR_ORA_ENV=CTSS_REBOOT=TRUE" Succeeded
2009-12-23 19:34:40: Start of resource "ora.asm -init" Succeeded
2009-12-23 19:34:41: Start of resource "ora.crsd -init" Succeeded
2009-12-23 19:34:42: Start of resource "ora.evmd -init" Succeeded
2009-12-23 19:34:42: Successfully started Oracle clusterware stack
2009-12-23 19:34:42: Waiting for Oracle CRSD and EVMD to start
2009-12-23 19:34:48: Oracle CRS stack installed and running

First Node Adding Clusterware Resources:

2009-12-23 19:34:48: Configuring node
2009-12-23 19:34:48: adding nodeapps...
2009-12-23 19:34:48: upgrade_opt=
2009-12-23 19:34:48: nodevip=racbde1-vip/255.255.254.0/eth1
2009-12-23 19:34:48: DHCP_flag=0
2009-12-23 19:34:48: nodes_to_add=racbde1
2009-12-23 19:34:48: add nodeapps for static IP
2009-12-23 19:34:48: Running srvctl config nodeapps to detect if VIP exists
2009-12-23 19:34:49: output=PRKO-2312 : GSD does not exist.
 PRKO-2331 : ONS daemon does not exist.
 PRKO-2339 : eONS daemon does not exist.

2009-12-23 19:34:49: Invoking "/u01/app/grid/bin/srvctl add nodeapps -n racbde1 -A "racbde1-vip/255.255.254.0/eth1" "
2009-12-23 19:34:49: trace file=/u01/app/grid/cfgtoollogs/crsconfig/srvmcfg0.log
2009-12-23 19:35:16: add nodeapps -n racbde1 -A racbde1-vip/255.255.254.0/eth1 on node=racbde1 ... passed
2009-12-23 19:35:16: nodes_to_start=racbde1
2009-12-23 19:35:16: Configuring first node
2009-12-23 19:35:16: DHCP_flag=0
2009-12-23 19:35:16: nodes_to_start=racbde1
2009-12-23 19:35:17: check ASM exists done and ASM does not exist
2009-12-23 19:35:17: Prior version ASM does not exist , Invoking add asm
2009-12-23 19:35:17: Invoking "/u01/app/grid/bin/srvctl add asm"
2009-12-23 19:35:17: trace file=/u01/app/grid/cfgtoollogs/crsconfig/srvmcfg1.log
2009-12-23 19:35:17: Running as user oracle: /u01/app/grid/bin/srvctl add asm
2009-12-23 19:35:17:   Invoking "/u01/app/grid/bin/srvctl add asm" as user "oracle"
2009-12-23 19:35:24: add asm ... success
2009-12-23 19:35:24: Adding ASM diskgroup resource
2009-12-23 19:35:24: Running as user oracle: /u01/app/grid/bin/crsctl create diskgroup SYSTEMDG
2009-12-23 19:35:24:   Invoking "/u01/app/grid/bin/crsctl create diskgroup SYSTEMDG" as user "oracle"
2009-12-23 19:35:26: create diskgroup SYSTEMDG ... success
2009-12-23 19:35:26: Successfully created disk group resource
2009-12-23 19:35:27: Invoking: /u01/app/grid/bin/crsctl add type ora.registry.acfs.type -basetype ora.local_resource.type -file /u01/app/grid/crs/template/registry.acfs.type
2009-12-23 19:35:29: add ora.registry.acfs.type ... success
2009-12-23 19:35:29: Invoking: /u01/app/grid/bin/crsctl add resource ora.registry.acfs -attr ACL='owner:root:rwx,pgrp:oinstall:r-x,other::r--' -type ora.registry.acfs.type -f
2009-12-23 19:35:30: add resource ora.registry.acfs ... success
2009-12-23 19:35:30: GNS is not to be configured - skipping
2009-12-23 19:35:30: Invoking "/u01/app/grid/bin/srvctl add scan -n racbde-scan.us.oracle.com"
2009-12-23 19:35:30: trace file=/u01/app/grid/cfgtoollogs/crsconfig/srvmcfg2.log
2009-12-23 19:35:35: add scan=racbde-scan.us.oracle.com ... success
2009-12-23 19:35:35: Invoking "/u01/app/grid/bin/srvctl add scan_listener -p 1521"
2009-12-23 19:35:35: trace file=/u01/app/grid/cfgtoollogs/crsconfig/srvmcfg3.log
2009-12-23 19:35:35: Running as user oracle: /u01/app/grid/bin/srvctl add scan_listener -p 1521
2009-12-23 19:35:35:   Invoking "/u01/app/grid/bin/srvctl add scan_listener -p 1521" as user "oracle"
2009-12-23 19:35:41: add scan listener ... success
2009-12-23 19:35:41: Running as user oracle: /u01/app/grid/bin/srvctl add oc4j
2009-12-23 19:35:41:   Invoking "/u01/app/grid/bin/srvctl add oc4j" as user "oracle"
2009-12-23 19:35:44: J2EE (OC4J) Container Resource Add ... passed ...
2009-12-23 19:35:44: starting nodeapps...
2009-12-23 19:35:44: DHCP_flag=0
2009-12-23 19:35:44: nodes_to_start=racbde1
2009-12-23 19:35:54: exit value of start nodeapps/vip is 0
2009-12-23 19:35:54: GNS is not to be configured - skipping
2009-12-23 19:35:54: Invoking "/u01/app/grid/bin/srvctl start scan"
2009-12-23 19:35:54: trace file=/u01/app/grid/cfgtoollogs/crsconfig/srvmcfg5.log
2009-12-23 19:36:04: start scan ... success
2009-12-23 19:36:04: Invoking "/u01/app/grid/bin/srvctl start scan_listener"
2009-12-23 19:36:04: trace file=/u01/app/grid/cfgtoollogs/crsconfig/srvmcfg6.log
2009-12-23 19:36:04: Running as user oracle: /u01/app/grid/bin/srvctl start scan_listener
2009-12-23 19:36:04:   Invoking "/u01/app/grid/bin/srvctl start scan_listener" as user "oracle"
2009-12-23 19:36:07: start scan listener ... success
2009-12-23 19:36:07: Running as user oracle: /u01/app/grid/bin/srvctl disable oc4j
2009-12-23 19:36:07:   Invoking "/u01/app/grid/bin/srvctl disable oc4j" as user "oracle"
2009-12-23 19:36:08: J2EE (OC4J) Container Resource Disable ... passed ...
2009-12-23 19:36:08: starting acfs_registry...
2009-12-23 19:36:08: nodes_to_start=racbde1
2009-12-23 19:36:09: /u01/app/grid/bin/crsctl start res ora.registry.acfs -n racbde1 ... success
2009-12-23 19:36:09: SRVCONFIG=/var/opt/oracle/srvConfig.loc
2009-12-23 19:36:09: Oracle 92 configuration and SKGXN library does not exists
2009-12-23 19:36:09: /u01/app/grid/bin/ocrconfig -local -manualbackup ... passed
2009-12-23 19:36:09: Install cvuqdisk rpm on Linux...
2009-12-23 19:36:09: New package to install is /u01/app/grid/cv/rpm/cvuqdisk-1.0.7-1.rpm
2009-12-23 19:36:09: Invoking "/bin/rpm -q cvuqdisk" command
2009-12-23 19:36:10: no existing cvuqdisk found
2009-12-23 19:36:10: removing old rpm
2009-12-23 19:36:10: installing/upgrading new rpm
2009-12-23 19:36:11: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

Secondary Note Initial Setup (racbde2):

2009-12-23 19:33:41: The configuration parameter file /u01/app/grid/crs/install/crsconfig_params is valid
2009-12-23 19:33:41: Checking for super user privileges
2009-12-23 19:33:41: User has super user privileges
2009-12-23 19:33:41: ### Printing the configuration values from files:
2009-12-23 19:33:41:    /u01/app/grid/crs/install/crsconfig_params
2009-12-23 19:33:41:    /u01/app/grid/crs/install/s_crsconfig_defs
2009-12-23 19:33:41: ASM_DISCOVERY_STRING=/dev/sdb*
2009-12-23 19:33:41: ASM_DISKS=/dev/sdb1,/dev/sdb2,/dev/sdb3
2009-12-23 19:33:41: ASM_DISK_GROUP=SYSTEMDG
2009-12-23 19:33:41: ASM_REDUNDANCY=NORMAL
2009-12-23 19:33:41: ASM_SPFILE=
2009-12-23 19:33:41: ASM_UPGRADE=false
2009-12-23 19:33:41: CLSCFG_MISSCOUNT=
2009-12-23 19:33:41: CLUSTER_GUID=
2009-12-23 19:33:41: CLUSTER_NAME=racbde-scan
2009-12-23 19:33:41: CRS_NODEVIPS='racbde1-vip/255.255.254.0/eth1,racbde2-vip/255.255.254.0/eth1'
2009-12-23 19:33:41: CRS_STORAGE_OPTION=1
2009-12-23 19:33:41: CSS_LEASEDURATION=400
2009-12-23 19:33:41: DIRPREFIX=
2009-12-23 19:33:41: DISABLE_OPROCD=0
2009-12-23 19:33:41: EMBASEJAR_NAME=oemlt.jar
2009-12-23 19:33:41: EWTJAR_NAME=ewt3.jar
2009-12-23 19:33:41: EXTERNAL_ORACLE_BIN=/opt/oracle/bin
2009-12-23 19:33:41: GNS_ADDR_LIST=
2009-12-23 19:33:41: GNS_ALLOW_NET_LIST=
2009-12-23 19:33:41: GNS_CONF=false
2009-12-23 19:33:41: GNS_DENY_ITF_LIST=
2009-12-23 19:33:41: GNS_DENY_NET_LIST=
2009-12-23 19:33:41: GNS_DOMAIN_LIST=
2009-12-23 19:33:41: GPNPCONFIGDIR=/u01/app/grid
2009-12-23 19:33:41: GPNPGCONFIGDIR=/u01/app/grid
2009-12-23 19:33:41: GPNP_PA=
2009-12-23 19:33:41: HELPJAR_NAME=help4.jar
2009-12-23 19:33:41: HOST_NAME_LIST=racbde1,racbde2
2009-12-23 19:33:41: ID=/etc/init.d
2009-12-23 19:33:41: INIT=/sbin/init
2009-12-23 19:33:41: IT=/etc/inittab
2009-12-23 19:33:41: JEWTJAR_NAME=jewt4.jar
2009-12-23 19:33:41: JLIBDIR=/u01/app/grid/jlib
2009-12-23 19:33:41: JREDIR=/u01/app/grid/jdk/jre/
2009-12-23 19:33:41: LANGUAGE_ID=AMERICAN_AMERICA.AL32UTF8
2009-12-23 19:33:41: MSGFILE=/var/adm/messages
2009-12-23 19:33:41: NETCFGJAR_NAME=netcfg.jar
2009-12-23 19:33:41: NETWORKS="eth1"/10.141.154.0:public,"eth0"/192.168.0.0:cluster_interconnect
2009-12-23 19:33:41: NEW_HOST_NAME_LIST=
2009-12-23 19:33:41: NEW_NODEVIPS='racbde1-vip/255.255.254.0/eth1,racbde2-vip/255.255.254.0/eth1'
2009-12-23 19:33:41: NEW_NODE_NAME_LIST=
2009-12-23 19:33:41: NEW_PRIVATE_NAME_LIST=
2009-12-23 19:33:41: NODELIST=racbde1,racbde2
2009-12-23 19:33:41: NODE_NAME_LIST=racbde1,racbde2
2009-12-23 19:33:41: OCFS_CONFIG=
2009-12-23 19:33:41: OCRCONFIG=/etc/oracle/ocr.loc
2009-12-23 19:33:41: OCRCONFIGDIR=/etc/oracle
2009-12-23 19:33:41: OCRID=
2009-12-23 19:33:41: OCRLOC=ocr.loc
2009-12-23 19:33:41: OCR_LOCATIONS=NO_VAL
2009-12-23 19:33:41: OLASTGASPDIR=/etc/oracle/lastgasp
2009-12-23 19:33:41: OLRCONFIG=/etc/oracle/olr.loc
2009-12-23 19:33:41: OLRCONFIGDIR=/etc/oracle
2009-12-23 19:33:41: OLRLOC=olr.loc
2009-12-23 19:33:41: OPROCDCHECKDIR=/etc/oracle/oprocd/check
2009-12-23 19:33:41: OPROCDDIR=/etc/oracle/oprocd
2009-12-23 19:33:41: OPROCDFATALDIR=/etc/oracle/oprocd/fatal
2009-12-23 19:33:41: OPROCDSTOPDIR=/etc/oracle/oprocd/stop
2009-12-23 19:33:41: ORACLE_BASE=/u01/app/oracle
2009-12-23 19:33:41: ORACLE_HOME=/u01/app/grid
2009-12-23 19:33:41: ORACLE_OWNER=oracle
2009-12-23 19:33:41: ORA_ASM_GROUP=oinstall
2009-12-23 19:33:41: ORA_DBA_GROUP=oinstall
2009-12-23 19:33:41: PRIVATE_NAME_LIST=
2009-12-23 19:33:41: RCALLDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc3.d /etc/rc.d/rc4.d /etc/rc.d/rc5.d /etc/rc.d/rc6.d
2009-12-23 19:33:41: RCKDIR=/etc/rc.d/rc0.d /etc/rc.d/rc1.d /etc/rc.d/rc2.d /etc/rc.d/rc4.d /etc/rc.d/rc6.d
2009-12-23 19:33:41: RCSDIR=/etc/rc.d/rc3.d /etc/rc.d/rc5.d
2009-12-23 19:33:41: RC_KILL=K19
2009-12-23 19:33:41: RC_KILL_OLD=K96
2009-12-23 19:33:41: RC_START=S96
2009-12-23 19:33:41: SCAN_NAME=racbde-scan.us.oracle.com
2009-12-23 19:33:41: SCAN_PORT=1521
2009-12-23 19:33:41: SCRBASE=/etc/oracle/scls_scr
2009-12-23 19:33:41: SHAREJAR_NAME=share.jar
2009-12-23 19:33:41: SILENT=false
2009-12-23 19:33:41: SO_EXT=so
2009-12-23 19:33:41: SRVCFGLOC=srvConfig.loc
2009-12-23 19:33:41: SRVCONFIG=/var/opt/oracle/srvConfig.loc
2009-12-23 19:33:41: SRVCONFIGDIR=/var/opt/oracle
2009-12-23 19:33:41: VNDR_CLUSTER=false
2009-12-23 19:33:41: VOTING_DISKS=NO_VAL
2009-12-23 19:33:41: ### Printing other configuration values ###
2009-12-23 19:33:41: CLSCFG_EXTRA_PARMS=
2009-12-23 19:33:41: CRSDelete=0
2009-12-23 19:33:41: CRSPatch=0
2009-12-23 19:33:41: DEBUG=
2009-12-23 19:33:41: DOWNGRADE=
2009-12-23 19:33:41: HAS_GROUP=oinstall
2009-12-23 19:33:41: HAS_USER=root
2009-12-23 19:33:41: HOST=racbde2
2009-12-23 19:33:41: IS_SIHA=0
2009-12-23 19:33:41: OLR_DIRECTORY=/u01/app/grid/cdata
2009-12-23 19:33:41: OLR_LOCATION=/u01/app/grid/cdata/racbde2.olr
2009-12-23 19:33:41: ORA_CRS_HOME=/u01/app/grid
2009-12-23 19:33:41: SUPERUSER=root
2009-12-23 19:33:41: UPGRADE=
2009-12-23 19:33:41: VF_DISCOVERY_STRING=
2009-12-23 19:33:41: addfile=/u01/app/grid/crs/install/crsconfig_addparams
2009-12-23 19:33:41: crscfg_trace=1
2009-12-23 19:33:41: crscfg_trace_file=/u01/app/grid/cfgtoollogs/crsconfig/rootcrs_racbde2.log
2009-12-23 19:33:41: hosts=
2009-12-23 19:33:41: oldcrshome=
2009-12-23 19:33:41: oldcrsver=
2009-12-23 19:33:41: osdfile=/u01/app/grid/crs/install/s_crsconfig_defs
2009-12-23 19:33:41: parameters_valid=1
2009-12-23 19:33:41: paramfile=/u01/app/grid/crs/install/crsconfig_params
2009-12-23 19:33:41: platform_family=unix
2009-12-23 19:33:41: srvctl_trc_suff=0
2009-12-23 19:33:41: unlock_crshome=
2009-12-23 19:33:41: user_is_superuser=1
2009-12-23 19:33:41: ### Printing of configuration values complete ###
2009-12-23 19:33:41: Oracle CRS stack is not configured yet
2009-12-23 19:33:41: CRS is not yet configured. Hence, will proceed to configure CRS
2009-12-23 19:33:41: Cluster-wide one-time actions... Done!
2009-12-23 19:33:41: Oracle CRS home = /u01/app/grid
2009-12-23 19:33:41: Host name = racbde2
2009-12-23 19:33:41: CRS user = oracle
2009-12-23 19:33:41: Oracle CRS home = /u01/app/grid
2009-12-23 19:33:41: GPnP host = racbde2
2009-12-23 19:33:41: Oracle GPnP home = /u01/app/grid/gpnp
2009-12-23 19:33:41: Oracle GPnP local home = /u01/app/grid/gpnp/racbde2
2009-12-23 19:33:41: GPnP directories verified. 
2009-12-23 19:33:41: Checking to see if Oracle CRS stack is already configured
2009-12-23 19:33:41: Oracle CRS stack is not configured yet
2009-12-23 19:33:41: ---Checking local gpnp setup...
2009-12-23 19:33:41: The setup file "/u01/app/grid/gpnp/racbde2/profiles/peer/profile.xml" does not exist
2009-12-23 19:33:41: The setup file "/u01/app/grid/gpnp/racbde2/wallets/peer/cwallet.sso" does not exist
2009-12-23 19:33:41: The setup file "/u01/app/grid/gpnp/racbde2/wallets/prdr/cwallet.sso" does not exist
2009-12-23 19:33:41: chk gpnphome /u01/app/grid/gpnp/racbde2: profile_ok 0 wallet_ok 0 r/o_wallet_ok 0
2009-12-23 19:33:41: chk gpnphome /u01/app/grid/gpnp/racbde2: INVALID (bad profile/wallet)

Secondary Note Get GPnP Profile:

2009-12-23 19:33:41: ---Checking cluster-wide gpnp setup...
2009-12-23 19:33:41: chk gpnphome /u01/app/grid/gpnp: profile_ok 1 wallet_ok 1 r/o_wallet_ok 1
2009-12-23 19:33:41: gpnptool: run /u01/app/grid/bin/gpnptool verify -p="/u01/app/grid/gpnp/profiles/peer/profile.xml" -w="file:/u01/app/grid/gpnp/wallets/peer" -wu=peer
2009-12-23 19:33:41: Running as user oracle: /u01/app/grid/bin/gpnptool verify -p="/u01/app/grid/gpnp/profiles/peer/profile.xml" -w="file:/u01/app/grid/gpnp/wallets/peer" -wu=peer
2009-12-23 19:33:41: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/gpnptool verify -p="/u01/app/grid/gpnp/profiles/peer/profile.xml" -w="file:/u01/app/grid/gpnp/wallets/peer" -wu=peer '
2009-12-23 19:33:43: Removing file /tmp/file1cIHO0
2009-12-23 19:33:43: Successfully removed file: /tmp/file1cIHO0
2009-12-23 19:33:43: /bin/su successfully executed

2009-12-23 19:33:43: gpnptool: rc=0
2009-12-23 19:33:43: gpnptool output:
Profile signature is valid.

2009-12-23 19:33:43: Profile "/u01/app/grid/gpnp/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/app/grid/gpnp/wallets/peer"
2009-12-23 19:33:43: gpnptool: run /u01/app/grid/bin/gpnptool verify -p="/u01/app/grid/gpnp/profiles/peer/profile.xml" -w="file:/u01/app/grid/gpnp/wallets/prdr" -wu=peer
2009-12-23 19:33:43: Running as user oracle: /u01/app/grid/bin/gpnptool verify -p="/u01/app/grid/gpnp/profiles/peer/profile.xml" -w="file:/u01/app/grid/gpnp/wallets/prdr" -wu=peer
2009-12-23 19:33:43: s_run_as_user2: Running /bin/su oracle -c ' /u01/app/grid/bin/gpnptool verify -p="/u01/app/grid/gpnp/profiles/peer/profile.xml" -w="file:/u01/app/grid/gpnp/wallets/prdr" -wu=peer '
2009-12-23 19:33:43: Removing file /tmp/filev2P2aX
2009-12-23 19:33:43: Successfully removed file: /tmp/filev2P2aX
2009-12-23 19:33:43: /bin/su successfully executed

2009-12-23 19:33:43: gpnptool: rc=0
2009-12-23 19:33:43: gpnptool output:
Profile signature is valid.

2009-12-23 19:33:43: Profile "/u01/app/grid/gpnp/profiles/peer/profile.xml" signature is VALID for wallet "file:/u01/app/grid/gpnp/wallets/prdr"
2009-12-23 19:33:43: chk gpnphome /u01/app/grid/gpnp: OK
2009-12-23 19:33:43: GPnP Wallets ownership/permissions successfully set.
2009-12-23 19:33:43: gpnp setup checked: local valid? 0 cluster-wide valid? 1
2009-12-23 19:33:43: Taking cluster-wide setup as local
2009-12-23 19:33:43:   copy "/u01/app/grid/gpnp/profiles/peer/profile.xml" => "/u01/app/grid/gpnp/racbde2/profiles/peer/profile.xml"
2009-12-23 19:33:43:   set ownership on "/u01/app/grid/gpnp/racbde2/profiles/peer/profile.xml" => (oracle,oinstall)
2009-12-23 19:33:43:   copy "/u01/app/grid/gpnp/wallets/peer/cwallet.sso" => "/u01/app/grid/gpnp/racbde2/wallets/peer/cwallet.sso"
2009-12-23 19:33:43:   set ownership on "/u01/app/grid/gpnp/racbde2/wallets/peer/cwallet.sso" => (oracle,oinstall)
2009-12-23 19:33:43:   copy "/u01/app/grid/gpnp/wallets/prdr/cwallet.sso" => "/u01/app/grid/gpnp/racbde2/wallets/prdr/cwallet.sso"
2009-12-23 19:33:43:   set ownership on "/u01/app/grid/gpnp/racbde2/wallets/prdr/cwallet.sso" => (oracle,oinstall)
2009-12-23 19:33:43:   copy "/u01/app/grid/gpnp/profiles/peer/profile_orig.xml" => "/u01/app/grid/gpnp/racbde2/profiles/peer/profile_orig.xml"
2009-12-23 19:33:43:   set ownership on "/u01/app/grid/gpnp/racbde2/profiles/peer/profile_orig.xml" => (oracle,oinstall)
2009-12-23 19:33:43:   copy "/u01/app/grid/gpnp/wallets/root/ewallet.p12" => "/u01/app/grid/gpnp/racbde2/wallets/root/ewallet.p12"
2009-12-23 19:33:43:   set ownership on "/u01/app/grid/gpnp/racbde2/wallets/root/ewallet.p12" => (oracle,oinstall)
2009-12-23 19:33:43:   copy "/u01/app/grid/gpnp/wallets/pa/cwallet.sso" => "/u01/app/grid/gpnp/racbde2/wallets/pa/cwallet.sso"
2009-12-23 19:33:43:   set ownership on "/u01/app/grid/gpnp/racbde2/wallets/pa/cwallet.sso" => (oracle,oinstall)
2009-12-23 19:33:43:   copy "/u01/app/grid/gpnp/wallets/root/b64certificate.txt" => "/u01/app/grid/gpnp/racbde2/wallets/root/b64certificate.txt"
2009-12-23 19:33:43:   set ownership on "/u01/app/grid/gpnp/racbde2/wallets/root/b64certificate.txt" => (oracle,oinstall)
2009-12-23 19:33:43:   copy "/u01/app/grid/gpnp/wallets/peer/cert.txt" => "/u01/app/grid/gpnp/racbde2/wallets/peer/cert.txt"
2009-12-23 19:33:43:   set ownership on "/u01/app/grid/gpnp/racbde2/wallets/peer/cert.txt" => (oracle,oinstall)
2009-12-23 19:33:43:   copy "/u01/app/grid/gpnp/wallets/pa/cert.txt" => "/u01/app/grid/gpnp/racbde2/wallets/pa/cert.txt"
2009-12-23 19:33:43:   set ownership on "/u01/app/grid/gpnp/racbde2/wallets/pa/cert.txt" => (oracle,oinstall)
2009-12-23 19:33:43: GPnP Wallets ownership/permissions successfully set.
2009-12-23 19:33:43: gpnp setup: GOTCLUSTERWIDE
2009-12-23 19:33:43: Validating for SI-CSS configuration

Secondary Node Setup OLR for storing Oracle local registry data:

2009-12-23 19:33:43: Retrieving OCR main disk location
2009-12-23 19:33:43: Opening file OCRCONFIG
2009-12-23 19:33:43: Value () is set for key=ocrconfig_loc
2009-12-23 19:33:43: Unable to retrieve ocr disk info
2009-12-23 19:33:43: Checking to see if any 9i GSD is up
2009-12-23 19:33:43: libskgxnBase_lib = /etc/ORCLcluster/oracm/lib/libskgxn2.so
2009-12-23 19:33:43: libskgxn_lib = /opt/ORCLcluster/lib/libskgxn2.so
2009-12-23 19:33:43: SKGXN library file does not exists
2009-12-23 19:33:43: OLR location = /u01/app/grid/cdata/racbde2.olr
2009-12-23 19:33:43: Oracle CRS Home = /u01/app/grid
2009-12-23 19:33:43: Validating /etc/oracle/olr.loc file for OLR location /u01/app/grid/cdata/racbde2.olr
2009-12-23 19:33:43: /etc/oracle/olr.loc already exists. Backing up /etc/oracle/olr.loc to /etc/oracle/olr.loc.orig
2009-12-23 19:33:43: Oracle CRS home = /u01/app/grid
2009-12-23 19:33:43: Oracle cluster name = racbde-scan
2009-12-23 19:33:43: OCR locations = +SYSTEMDG
2009-12-23 19:33:43: Validating OCR
2009-12-23 19:33:43: Retrieving OCR location used by previous installations
2009-12-23 19:33:43: Opening file OCRCONFIG
2009-12-23 19:33:43: Value () is set for key=ocrconfig_loc
2009-12-23 19:33:43: Opening file OCRCONFIG
2009-12-23 19:33:43: Value () is set for key=ocrmirrorconfig_loc
2009-12-23 19:33:43: Opening file OCRCONFIG
2009-12-23 19:33:43: Value () is set for key=ocrconfig_loc3
2009-12-23 19:33:43: Opening file OCRCONFIG
2009-12-23 19:33:43: Value () is set for key=ocrconfig_loc4
2009-12-23 19:33:43: Opening file OCRCONFIG
2009-12-23 19:33:43: Value () is set for key=ocrconfig_loc5
2009-12-23 19:33:43: Checking if OCR sync file exists
2009-12-23 19:33:43: No need to sync OCR file
2009-12-23 19:33:43: OCR_LOCATION=+SYSTEMDG
2009-12-23 19:33:43: OCR_MIRROR_LOCATION=
2009-12-23 19:33:43: OCR_MIRROR_LOC3=
2009-12-23 19:33:43: OCR_MIRROR_LOC4=
2009-12-23 19:33:43: OCR_MIRROR_LOC5=
2009-12-23 19:33:43: Current OCR location= 
2009-12-23 19:33:43: Current OCR mirror location= 
2009-12-23 19:33:43: Current OCR mirror loc3=
2009-12-23 19:33:43: Current OCR mirror loc4=
2009-12-23 19:33:43: Current OCR mirror loc5=
2009-12-23 19:33:43: Verifying current OCR settings with user entered values
2009-12-23 19:33:43: Setting OCR locations in /etc/oracle/ocr.loc
2009-12-23 19:33:43: Validating OCR locations in /etc/oracle/ocr.loc
2009-12-23 19:33:43: Checking for existence of /etc/oracle/ocr.loc
2009-12-23 19:33:43: Backing up /etc/oracle/ocr.loc to /etc/oracle/ocr.loc.orig
2009-12-23 19:33:43: Setting ocr location +SYSTEMDG
2009-12-23 19:33:43: Creating or upgrading Oracle Local Registry (OLR)
2009-12-23 19:33:43: OLR successfully created or upgraded
2009-12-23 19:33:43: /u01/app/grid/bin/clscfg -localadd
2009-12-23 19:33:43: Keys created in the OLR successfully
2009-12-23 19:33:44: GPnP setup state: new-cluster-wide
2009-12-23 19:33:44: GPnP cluster configuration already performed

Secondary Node Setup and copy files for OHASD daemon:

2009-12-23 19:33:44: Registering ohasd
2009-12-23 19:33:44: init file = /u01/app/grid/crs/init/init.ohasd
2009-12-23 19:33:44: Copying file /u01/app/grid/crs/init/init.ohasd to /etc/init.d directory
2009-12-23 19:33:44: Setting init.ohasd permission in /etc/init.d directory
2009-12-23 19:33:44: init file = /u01/app/grid/crs/init/ohasd
2009-12-23 19:33:44: Copying file /u01/app/grid/crs/init/ohasd to /etc/init.d directory
2009-12-23 19:33:44: Setting ohasd permission in /etc/init.d directory
2009-12-23 19:33:44: Removing "/etc/rc.d/rc3.d/S96ohasd"
2009-12-23 19:33:44: Removing file /etc/rc.d/rc3.d/S96ohasd
2009-12-23 19:33:44: Failure with return code 1 from command rm /etc/rc.d/rc3.d/S96ohasd
2009-12-23 19:33:44: Failed to remove file: 
2009-12-23 19:33:44: Creating a link "/etc/rc.d/rc3.d/S96ohasd" pointing to /etc/init.d/ohasd
2009-12-23 19:33:44: Removing "/etc/rc.d/rc5.d/S96ohasd"
2009-12-23 19:33:44: Removing file /etc/rc.d/rc5.d/S96ohasd
2009-12-23 19:33:44: Failure with return code 1 from command rm /etc/rc.d/rc5.d/S96ohasd
2009-12-23 19:33:44: Failed to remove file: 
2009-12-23 19:33:44: Creating a link "/etc/rc.d/rc5.d/S96ohasd" pointing to /etc/init.d/ohasd
2009-12-23 19:33:44: Removing "/etc/rc.d/rc0.d/K19ohasd"
2009-12-23 19:33:44: Removing file /etc/rc.d/rc0.d/K19ohasd
2009-12-23 19:33:44: Failure with return code 1 from command rm /etc/rc.d/rc0.d/K19ohasd
2009-12-23 19:33:44: Failed to remove file: 
2009-12-23 19:33:44: Creating a link "/etc/rc.d/rc0.d/K19ohasd" pointing to /etc/init.d/ohasd
2009-12-23 19:33:44: Removing "/etc/rc.d/rc1.d/K19ohasd"
2009-12-23 19:33:44: Removing file /etc/rc.d/rc1.d/K19ohasd
2009-12-23 19:33:44: Failure with return code 1 from command rm /etc/rc.d/rc1.d/K19ohasd
2009-12-23 19:33:44: Failed to remove file: 
2009-12-23 19:33:44: Creating a link "/etc/rc.d/rc1.d/K19ohasd" pointing to /etc/init.d/ohasd
2009-12-23 19:33:44: Removing "/etc/rc.d/rc2.d/K19ohasd"
2009-12-23 19:33:44: Removing file /etc/rc.d/rc2.d/K19ohasd
2009-12-23 19:33:44: Failure with return code 1 from command rm /etc/rc.d/rc2.d/K19ohasd
2009-12-23 19:33:44: Failed to remove file: 
2009-12-23 19:33:44: Creating a link "/etc/rc.d/rc2.d/K19ohasd" pointing to /etc/init.d/ohasd
2009-12-23 19:33:44: Removing "/etc/rc.d/rc4.d/K19ohasd"
2009-12-23 19:33:44: Removing file /etc/rc.d/rc4.d/K19ohasd
2009-12-23 19:33:44: Failure with return code 1 from command rm /etc/rc.d/rc4.d/K19ohasd
2009-12-23 19:33:44: Failed to remove file: 
2009-12-23 19:33:44: Creating a link "/etc/rc.d/rc4.d/K19ohasd" pointing to /etc/init.d/ohasd
2009-12-23 19:33:44: Removing "/etc/rc.d/rc6.d/K19ohasd"
2009-12-23 19:33:44: Removing file /etc/rc.d/rc6.d/K19ohasd
2009-12-23 19:33:44: Failure with return code 1 from command rm /etc/rc.d/rc6.d/K19ohasd
2009-12-23 19:33:44: Failed to remove file: 
2009-12-23 19:33:44: Creating a link "/etc/rc.d/rc6.d/K19ohasd" pointing to /etc/init.d/ohasd
2009-12-23 19:33:44: The file ohasd has been successfully linked to the RC directories
2009-12-23 19:33:44: Starting ohasd
2009-12-23 19:33:44: itab entries=
2009-12-23 19:33:49: Created backup /etc/inittab.no_crs
2009-12-23 19:33:49: Appending to /etc/inittab.tmp:
2009-12-23 19:33:49: h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

2009-12-23 19:33:49: Done updating /etc/inittab.tmp
2009-12-23 19:33:49: Saved /etc/inittab.crs
2009-12-23 19:33:49: Installed new /etc/inittab

Secondary Node Start OHASD Daemon:

2009-12-23 19:33:55: ohasd is starting
2009-12-23 19:33:55: Checking ohasd
2009-12-23 19:33:55: ohasd started successfully

Secondary Node Copy required CRS resources for OHASD to start:

2009-12-23 19:33:55: Creating CRS resources and dependencies
2009-12-23 19:33:55: Configuring HASD
2009-12-23 19:33:55: Registering type ora.daemon.type
2009-12-23 19:33:55: Registering type ora.mdns.type
2009-12-23 19:33:55: Registering type ora.gpnp.type
2009-12-23 19:33:55: Registering type ora.gipc.type
2009-12-23 19:33:56: Registering type ora.cssd.type
2009-12-23 19:33:56: Registering type ora.cssdmonitor.type
2009-12-23 19:33:56: Registering type ora.crs.type
2009-12-23 19:33:56: Registering type ora.evm.type
2009-12-23 19:33:56: Registering type ora.ctss.type
2009-12-23 19:33:57: Registering type ora.asm.type
2009-12-23 19:33:57: Registering type ora.drivers.acfs.type
2009-12-23 19:33:57: Registering type ora.diskmon.type
2009-12-23 19:34:12: ADVM/ACFS is configured

2009-12-23 19:34:12: Successfully created CRS resources for cluster daemon and ASM
2009-12-23 19:34:12: Checking if initial configuration has been performed
2009-12-23 19:34:12: Starting CSS in exclusive mode
2009-12-23 19:35:54: CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node racbde1, number 1, and is terminating
2009-12-23 19:35:54: An active cluster was found during exclusive startup, restarting to join the cluster

Secondary Node Start Full Clusterware Stack:

2009-12-23 19:35:54: Starting 
2009-12-23 19:35:56: Start of resource "ora.mdnsd -init" Succeeded
2009-12-23 19:35:57: Start of resource "ora.gipcd -init" Succeeded
2009-12-23 19:35:58: Start of resource "ora.gpnpd -init" Succeeded
2009-12-23 19:35:58: Successfully started requested Oracle stack daemons
2009-12-23 19:35:58: Starting CSS in clustered mode
2009-12-23 19:36:32: CRS-2672: Attempting to start 'ora.cssdmonitor' on 'racbde2'
2009-12-23 19:36:32: CRS-2676: Start of 'ora.cssdmonitor' on 'racbde2' succeeded
2009-12-23 19:36:32: CRS-2672: Attempting to start 'ora.cssd' on 'racbde2'
2009-12-23 19:36:32: CRS-2672: Attempting to start 'ora.diskmon' on 'racbde2'
2009-12-23 19:36:32: CRS-2676: Start of 'ora.diskmon' on 'racbde2' succeeded
2009-12-23 19:36:32: CRS-2676: Start of 'ora.cssd' on 'racbde2' succeeded
2009-12-23 19:36:33: Start of resource "ora.ctssd -init -env USR_ORA_ENV=CTSS_REBOOT=TRUE" Succeeded
2009-12-23 19:39:25: Start of resource "ora.asm -init" Succeeded
2009-12-23 19:39:26: Start of resource "ora.crsd -init" Succeeded
2009-12-23 19:39:27: Start of resource "ora.evmd -init" Succeeded
2009-12-23 19:39:27: Successfully started Oracle clusterware stack
2009-12-23 19:39:27: Waiting for Oracle CRSD and EVMD to start
2009-12-23 19:39:33: Oracle CRS stack installed and running

Secondary Node Adding Clusterware Resources:

2009-12-23 19:39:33: Configuring node
2009-12-23 19:39:34: adding nodeapps...
2009-12-23 19:39:34: upgrade_opt=
2009-12-23 19:39:34: nodevip=racbde2-vip/255.255.254.0/eth1
2009-12-23 19:39:34: DHCP_flag=0
2009-12-23 19:39:34: nodes_to_add=racbde2
2009-12-23 19:39:34: add nodeapps for static IP
2009-12-23 19:39:34: Running srvctl config nodeapps to detect if VIP exists
2009-12-23 19:39:35: vip exists
2009-12-23 19:39:35: Invoking "/u01/app/grid/bin/srvctl add vip -n racbde2 -k 1 -A "racbde2-vip/255.255.254.0/eth1" "
2009-12-23 19:39:35: trace file=/u01/app/grid/cfgtoollogs/crsconfig/srvmcfg0.log
2009-12-23 19:39:38: add vip -n racbde2 -k 1 -A racbde2-vip/255.255.254.0/eth1 on node=racbde2 ... passed
2009-12-23 19:39:38: nodes_to_start=racbde2
2009-12-23 19:39:38: starting nodeapps...
2009-12-23 19:39:38: DHCP_flag=0
2009-12-23 19:39:38: nodes_to_start=racbde2
2009-12-23 19:39:43: exit value of start nodeapps/vip is 2
2009-12-23 19:39:43: output for start nodeapps is  PRKO-2421 : Network resource is already started on node(s): racbde2 PRKO-2422 : ONS is already started on node(s): racbde2 PRKO-2423 : eONS is already started on node(s): racbde2
2009-12-23 19:39:43: output of startnodeapp after removing already started mesgs is 
2009-12-23 19:39:43: /u01/app/grid/bin/srvctl start nodeapps -n racbde2 ... passed
2009-12-23 19:39:43: /u01/app/grid/bin/ocrconfig -local -manualbackup ... passed
2009-12-23 19:39:43: Install cvuqdisk rpm on Linux...
2009-12-23 19:39:43: New package to install is /u01/app/grid/cv/rpm/cvuqdisk-1.0.7-1.rpm
2009-12-23 19:39:43: Invoking "/bin/rpm -q cvuqdisk" command
2009-12-23 19:39:44: no existing cvuqdisk found
2009-12-23 19:39:44: removing old rpm
2009-12-23 19:39:44: installing/upgrading new rpm
2009-12-23 19:39:45: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

If the root.sh is successful but there are subsequent startup problems, see Note: 1050908.1 for more troubleshooting information