CRS-215 Srvctl unable to start ASM, Listener, RDBMS Resources

CRS-215 Srvctl unable to start ASM, Listener, RDBMS Resources [ID 740319.1]
		修改时间 14-DEC-2009     类型 PROBLEM     状态 PUBLISHED

In this Document
  Symptoms
  Changes
  Cause
  Solution

---

Applies to:

Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 10.2.0.4
Information in this document applies to any platform.
Oracle Server Enterprise Edition - Version: 10.2.0.1 to 10.2.0.4

Symptoms

After patching CRS, RDBMS & ASM ORACLE_HOME, unable to start/stop ASM, Database & Listener resources via srvctl:

node68:/u01>srvctl start asm -n node68
PRKS-1009 : Failed to start ASM instance "+ASM4" on node "node68", [PRKS-1009
: Failed to start ASM instance "+ASM4" on node "node68", [CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource 'ora.node68.ASM4.asm' has placement error.]]
[PRKS-1009 : Failed to start ASM instance "+ASM4" on node "node68", [CRS-1028: Dependency analysis failed because of:
CRS-0223: Resource 'ora.node68.ASM4.asm' has placement error.]]

node68:/u01>srvctl stop asm -n node68
Attempting to stop `ora.node68.ASM4.asm` on member `node68`
`ora.node68.ASM4.asm` on member `node68` has experienced an unrecoverable failure.
Human intervention required to resume its availability.
CRS-0216: Could not stop resource 'ora.node68.ASM4.asm'.

node68:/u01>crs_stat -t | grep 68

ora....TA4.srv  application ONLINE UNKNOWN node68
ora....SM4.asm application ONLINE UNKNOWN node68
ora....68.lsnr     application ONLINE UNKNOWN node68
ora....d68.gsd   application ONLINE     ONLINE     node68
ora....d68.ons   application ONLINE     ONLINE     node68
ora....d68.vip   application ONLINE     ONLINE     node68

Starting ASM instance via sqlplus is fine. Starting listener via lsnrctl is fine too.

Changes

CRS, RDBMS & ASM is patched with the latest bundle patch.

Cause

While the resource is in an UNKNOWN state, one can not start resource via srvctl command. Need to force resource OFFLINE with command:
% $ORA_CRS_HOME/bin/crs_stop -f

E.g.

$ ./crs_stop -f ora.node68ASM4.asm
Attempting to stop `ora.node68.ASM4.asm` on member `node68`
Stop of `ora.node68.ASM4.asm` on member `node68` succeeded

Please note the preferred method for management of a resource is through srvctl, any issues with the usage of crs_start & crs_stop should be done under the guidance of Oracle Support.

After this, the real error shows up:

node68:/u01>srvctl start asm -n node68
[PRKS-1009 : Failed to start ASM instance "ASM4" on node "node68",
[node68:ora.node68.ASM4.asm:/u01/oracle/product/10.2.0/asm/bin/racgwrap: %ORACLE_HOME%/bin/racgmain:
not found
CRS-0215: Could not start resource 'ora.node68.ASM4.asm'.]]

$ORACLE_HOME/bin/racgwrap does not have ORACLE_HOME variable correctly defined.

The cause is actually that scripts like racgwrap weren't modified when the CRS bundle was applied against the ASM/database home.
In most cases this will be a result of not having run the pre & post scripts as per patch README. 

Example of Patch# 7715304

% cd patch location
% custom/server/7715304/custom/scripts/prepatch.sh -dbhome
% opatch napply custom/server/ -local -oh -id 7715304
% custom/server/7715304/custom/scripts/postpatch.sh -dbhome

The last step parses all scripts that have been included in the patch, replacing installer variables like %ORACLE_HOME% with the actual path to the ORACLE_HOME. If steps 1 & 3 are omitted any patched scripts may have incorrect values for such variables and subsequently scripts like 'racgwrap' fail causing the status of the associated resource to become 'UNKNOWN'.

Solution

Given that the root cause to this is actually the result of the patch being applied incorrectly you should re-apply the patch following the instructions but first rolling back the patch.

Steps..

1. shutdown any instances running out of this RDBMS home, you can do this with sqlplus for each instance.  You may also need to place the resource in an OFFLINE state instead of UNKNOWN, can do this with crs_stop -f

E.g.
% cd $ORA_CRS_HOME/bin
% ./crs_stop -f ora.node4.ASM4.asm

This will ensure the state of the resource changes from UNKNOWN status to OFFLINE status.

2. rollback patch 7715304 (as the oracle user) as per README:

E.g.
% opatch rollback -id 7715304 -local -oh

3. Re-apply the patch against the RDBMS home as the oracle user:

% cd
% custom/server/7715304/custom/scripts/prepatch.sh -dbhome
% opatch napply custom/server/ -local -oh -id 7715304
% custom/server/7715304/custom/scripts/postpatch.sh -dbhome

4. when done check the script. 'racgwrap' if it now contains the correct path for ORACLE_HOME by running the following command

grep 'ORACLE_HOME=' $ORACLE_HOME/bin/racgwrap

Above needs to be repeated on each node.

---

<> FOLDER:ST.Server.Scalability.RAC TOPIC:Cluster Ready Services - CRS DOCUMENT-ID:740319.1 ALIAS: SOURCE:AWIZ 7116256.993 DOCUMENT-TYPE:PROBLEM ZCXTECH TITLE:CRS-215 Srvctl unable to start ASM, Listener, RDBMS Resources IMPACT:LOW SKILL-LEVEL:NOVICE STATUS:PUBLISHED DISTRIBUTION:EXTERNAL ZCXPUBLIC ZCXCURRENT AUTHOR:AMASTROM.AU ERROR:CRS-1028 ERROR:CRS-215 ERROR:CRS-216 ERROR:CRS-223 ERROR:PRKS-1009 KEYWORD:ASM KEYWORD:LSNRCTL KEYWORD:ORACLE_BASE KEYWORD:SRVCTL KEYWORD:START~INSTANCE KEYWORD:START~LISTENER PRODID-5 COMPONENT:RAC.RACM MINVER:10.2.0.1 MAXVER:10.2.0.4 PORTID-0 FDRSEG-824 FDRSEG-1221 FDRSEG-773 FDRSEG-465

相关的 产品 Oracle Database Products > Oracle Database > Oracle Database > Oracle Server - Enterprise Edition 关键字 LSNRCTL; START INSTANCE; START LISTENER; ORACLE_BASE; ASM; SRVCTL 错误 CRS-216; CRS-223; CRS-1028; CRS-215; PRKS-1009; 216 ERROR; 215 ERROR; 651028 ERROR; 1028 ERROR; 223 ERROR

来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/7865774/viewspace-625994/，如需转载，请注明出处，否则将追究法律责任。

转载于:http://blog.itpub.net/7865774/viewspace-625994/