CRS-215 Srvctl unable to start ASM, Listener, RDBMS Resources OR CRS-0223

Applies to:

Oracle Server Enterprise Edition - Version: 10.2.0.1 to 10.2.0.4
This problem can occur on any platform.
Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 10.2.0.4

Symptoms

After patching CRS, RDBMS & ASM ORACLE_HOME, unable to start/stop ASM, Database & Listener resources via srvctl:

node68:/u01>srvctl start asm -n node68
PRKS-1009 : Failed to start ASM instance "+ASM4" on node "node68", [PRKS-1009
: Failed to start ASM instance "+ASM4" on node "node68", [CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource 'ora.node68.ASM4.asm' has placement error.]]
[PRKS-1009 : Failed to start ASM instance "+ASM4" on node "node68", [CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource 'ora.node68.ASM4.asm' has placement error.]]

node68:/u01>srvctl stop asm -n node68
Attempting to stop `ora.node68.ASM4.asm` on member `node68`
`ora.node68.ASM4.asm` on member `node68` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
CRS-0216: Could not stop resource 'ora.node68.ASM4.asm'.

node68:/u01>crs_stat -t | grep 68

ora....TA4.srv application ONLINE UNKNOWN node68
ora....SM4.asm application ONLINE UNKNOWN node68
ora....68.lsnr application ONLINE UNKNOWN node68
ora....d68.gsd application ONLINE ONLINE node68
ora....d68.ons application ONLINE ONLINE node68
ora....d68.vip application ONLINE ONLINE node68

Starting ASM instance via sqlplus is fine. Starting listener via lsnrctl is fine too.

Changes

CRS, RDBMS & ASM is patched with the latest bundle patch.

Cause

While the resource is in an UNKNOWN state, one can not start resource via srvctl command. Need to force resource OFFLINE with command:
% $ORA_CRS_HOME/bin/crs_stop -f <resource name>

E.g.

$ ./crs_stop -f ora.node68ASM4.asm
Attempting to stop `ora.node68.ASM4.asm` on member `node68`
Stop of `ora.node68.ASM4.asm` on member `node68` succeeded

Please note the preferred method for management of a resource is through srvctl, any issues with the usage of crs_start & crs_stop should be done under the guidance of Oracle Support.

After this, the real error shows up:

node68:/u01>srvctl start asm -n node68
[PRKS-1009 : Failed to start ASM instance "ASM4" on node "node68",
[node68:ora.node68.ASM4.asm:/u01/oracle/product/10.2.0/asm/bin/racgwrap: %ORACLE_HOME%/bin/racgmain:
not found
CRS-0215: Could not start resource 'ora.node68.ASM4.asm'.]]

$ORACLE_HOME/bin/racgwrap does not have ORACLE_HOME variable correctly defined.

The above likely occured because of an issue with either patching or installation by which the contents of racgwrap took an incorrect value of ORACLE_HOME possible at the environment level.

Solution

1). Stop the resource with crs_stop -f <resource name>
Only do this to ensure the state of the resource change from UNKNOWN status to OFFLINE status.

% cd $ORA_CRS_HOME/bin
% ./crs_stop -f ora.node4.ASM4.asm

2). Locate the racgwrap script for the problem resource.

% ./crs_stat -p ora.node68.ASM4.asm

Identify the location of the racgwrap script defined in ACTION_SCRIPT value using the above
command.

3). Modify racgwrap script to ensure the ORACLE_HOME is correct.
For example:
Change from:

ORACLE_HOME=%ORACLE_HOME%
export ORACLE_HOME

ORACLE_BASE=%ORACLE_BASE%
# export ORACLE_BASE if it is set

Change To

ORACLE_HOME=/u01/oracle/product/10.2.0/asm
export ORACLE_HOME

ORACLE_BASE=/u01/oracle/product/10.2.0/asm
# export ORACLE_BASE if it is set

Please replace above value with your real setting.
Once the above is done, start the resource via srvctl is fine.

Keywords

START~INSTANCE ; ASM ; ORACLE_BASE ; SRVCTL ; START~LISTENER ; LSNRCTL ; START~INSTANCE ;