How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure (CRS) [ID 969254.1]

How to Proceed from Failed Upgrade to 11gR2 Grid Infrastructure (CRS) [ID 969254.1]
	Modified 10-OCT-2010     Type HOWTO     Status PUBLISHED

In this Document
  Goal
  Solution
     Failure before rootupgrade.sh is ran:
     Failure during rootupgrade.sh:
     Failure during plug-in:
  References

Applies to:

Oracle Server - Enterprise Edition - Version: 11.2.0.1 to 11.2.0.1 - Release: 11.2 to 11.2
Generic UNIX
Generic Linux

Goal

This goal of this note is to provide steps to proceed from a failed upgrade to 11gR2  (11.2) Grid Infrastructure(CRS), so another attempt can be pursued.

Solution

 

Failure before rootupgrade.sh is ran:

Case applies here is runInstaller failed before rootupgrade.sh is executed.

You will need identify cause of the failure from screen output and installation logs in $ORACLE_BASE/oraInventory/logs (three files .log, .err and .out will be generated each time you run runInstaller) before execute clean up steps.

  A. As grid user, remove the failed installation by running $GRID_HOME/deinstall/deinstall

  B. If step A succeeded, skip to step C, otherwise perform the following:

export ORACLE_HOME=$OLD_CRS_HOME

## To detach failed GRID_HOME:
$OLD_CRS_HOME/oui/bin/runInstaller -detachHome -silent ORACLE_HOME=$GRID_HOME

## To confirm $GRID_HOME is removed from central inventory:
$OLD_CRS_HOME/OPatch/opatch lsinventory -all  

## To remove files in failed GRID_HOME on all nodes manually:
/bin/rm -rf $GRID_HOME

unset ORACLE_HOME

 

  C. Detach and attach pre-11gR2 CRS home to set CRS flag to true on all nodes:

 

export ORACLE_HOME=$OLD_CRS_HOME

$OLD_CRS_HOME/oui/bin/runInstaller -detachHome -silent -local ORACLE_HOME=$OLD_CRS_HOME

$OLD_CRS_HOME/oui/bin/runInstaller -silent -local -ignoreSysPrereqs -attachHome ORACLE_HOME=$OLD_CRS_HOME ORACLE_HOME_NAME=$OLD_CRS_HOME_NAME LOCAL_NODE=local_nodename CLUSTER_NODES=local_nodename,remote_nodename CRS=true

 

  D. Restart runInstaller to upgrade to 11.2 Grid Infrastructure

Failure during rootupgrade.sh:

You will need identify cause of the rootupgrade.sh failure by reviewing logs in $GRID_HOME/cfgtoollogs/crsconfig, $GRID_HOME/log and $ORACLE_BASE/oraInventory/logs; once cause is identified and issue is fixed, please proceed with steps below:

  0. Restartable root script - rootupgrade.sh

For 11.2.0.2 and above, rootupgrade.sh is restartable. Once cause is identified and fixed, rootupgrade.sh can be executed again on the failed node, if it succeeds, you can continue with the installation, otherwise proceed to next step (Step 1) of the note.

  1. Identify "last node"

Mark the node as "last node" where you find pre-upgrade OCR backup in $GRID_HOME/cdata/ocr$OLD_CRS_VERSION (for example, /ocw/grid/cdata/ocr10.2.0.4.0); in most case it's first node where you execute rootupgrade.sh

  2. As root, on all but "last node", execute:

$GRID_HOME/crs/install/rootcrs.pl -downgrade -force

      You can skip the command on node(s) where you never executed rootupgrade.sh
 
  3. As root, on "last node", execute:

 

$GRID_HOME/crs/install/rootcrs.pl -downgrade -force -lastnode -oldcrshome $OLD_CRS_HOME -version $OLD_CRS_VERSION

 

      Please note $OLD_CRS_VERSION need to be in format of n.n.n.n.n, for example, 10.2.0.4.0

      If OCR downgraded successfully, skip to step 5, otherwise go to step 4.

  4. Deconfigure 11gR2
 
      Execute this step only if step 3 failed.

        4A. As root, on all but last node, execute:

$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force

        4B. As root, on last node, execute:

$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode

• If rootcrs.pl fails with following error:

 

Using configuration parameter file: /ocw/grid/crs/install/crsconfig_params
Can't open /etc/oracle/scls_scr/node1/root/ohasdrun for write: No such file or directory at /ocw/grid/crs/install/s_crsconfig_lib.pm line 1021.

        As root, please create a empty ohasdrun and try the same again:

mkdir -p /etc/oracle/scls_scr/$HOSTNAME/root
touch /etc/oracle/scls_scr/$HOSTNAME/root/ohasdrun

 

• If rootcrs.pl hang forever, please disable 11gR2 from auto startup, reboot the node, and re-execute the same command.  To disable CRS from auto startup, as root execute:

    $GRID_HOME/bin/crsctl disable crs

• Once Step 4 finishes, checkpoint file shouldn't exist. To verify:

              ls -l $ORACLE_BASE/Clusterware/ckptGridHA_<nodename>.xml

If it's still there, please remove it manually with "rm" command.

  5. Execute $OLD_CRS_HOME/root.sh

     - If  new node were added or any of the following were changed after original setup of pre-11.2 CRS, please engage Oracle Support for instruction on how to modify pre-11.2 configuration manually.

            OCR or Voting disk location
            Network information for public, VIP or private


     - If OCR or Voting Disk are located on Cluster File System or Network File System, please make sure they exist before execution of  root.sh. If they don't exist, please create dummy OCR/Voting Disk files with /bin/touch.

      As root, execute $OLD_CRS_HOME/root.sh on each node sequentially, it will start old CRS at end of the script.

      If root.sh failed to bring up pre-11.2 CRS, please go to step 6; otherwise skip to step 7.

  6. Deconfigure and reconfigure pre-11.2 CRS

      Execute this step only if step 5 failed.

        6A. As root, on all nodes, execute $OLD_CRS_HOME/install/rootdelete.sh
        6B. As root, on first node, execute $OLD_CRS_HOME/install/rootdeinstall.sh
        6C. As root, execute $OLD_CRS_HOME/root.sh on all nodes sequentially.

  7. Restore OCR

    If you execute either step 4 or step 6, backup of OCR need to be restored.

    If you backed up OCR prior to upgrade, you can use that to restore, otherwise go through $OLD_CRS_HOME/cdata/{CLUSTERNAME} on all nodes to identify a good automatic backup

      7A. As root, on all nodes, stop CRS:

$OLD_CRS_HOME/bin/crsctl stop crs

      7B. As root, on node where identified OCR backup resides:

$OLD_CRS_HOME/bin/ocrconfig -restore name_of_ocr_backup

  8. At this point, pre-11.2 CRS should be running, as CRS user, execute following on all nodes to validate:

 

export ORACLE_HOME=$OLD_CRS_HOME
$OLD_CRS_HOME/bin/crsctl query crs activeversion
$OLD_CRS_HOME/bin/crsctl query crs softwareversion
$OLD_CRS_HOME/bin/crsctl check crs
$OLD_CRS_HOME/bin/crs_stat -t
$OLD_CRS_HOME/bin/crs_stat
$OLD_CRS_HOME/bin/oifcfg iflist -p -n
$OLD_CRS_HOME/bin/oifcfg getif
$OLD_CRS_HOME/bin/ocrcheck
cat /etc/oracle/ocr.loc /var/opt/oracle/ocr.loc
$OLD_CRS_HOME/bin/crsctl get css diagwait
$OLD_CRS_HOME/bin/srvctl config nodeapps -n `hostname | cut -d'.' -f1 | awk '{ print tolower($0); }'` -a
$OLD_CRS_HOME/OPatch/opatch lsinventory oh=$OLD_CRS_HOME
unset ORACLE_HOME
ifconfig              ## location and exact syntax in note 1054902.1

 

  9. Execute rootupgrade.sh

      Unset ORACLE_HOME, ORACLE_BASE, ORACLE_SID, ORA_CRS_HOME, TNS_ADMIN in both root and oracle's profile and current shell environment; ensure firewall is down on all nodes, and execute $GRID_HOME/rootupgrade.sh sequentially on all nodes.

  10. Post rootupgrade.sh steps:

      If any configuration assistant already finished successfully previously and you'd like to skip it, please be sure RERUN is set to false in $GRID_HOME/cfgtoollogs/configToolAllCommands, for example:

 

$ cat $GRID_HOME/cfgtoollogs/configToolAllCommands
..
/ogrid/gbase/oui/bin/runConfig.sh ORACLE_HOME=/ogrid/gbase MODE=perform ACTION=configure RERUN=false $*

 

      As GRID user, execute $GRID_HOME/cfgtoollogs/configToolAllCommands on first node.

Failure during plug-in:

    Please identify the cause of failed plug-in and go to Step 10.

References

NOTE:1050908.1 - How to Troubleshoot Grid Infrastructure Startup Issues
NOTE:1053147.1 - 11gR2 Clusterware and Grid Home - What You Need to Know
NOTE:1054902.1 - How to Validate Network and Name Resolution Setup for the Clusterware and RAC
NOTE:1056322.1 - How to Troubleshoot 11gR2 Installation Issue
NOTE:942166.1 - How to Proceed from Failed 11gR2 Grid Infrastructure (CRS) Installation
NOTE:948456.1 - Pre 11.2 Database Issues in 11gR2 Grid Infrastructure Environment