Failed To Connect ASM with ORA-12547: TNS:lost contact (Doc ID 2258977.1)

SYMPTOMS

When connecting to ASM instance in Oracle Clusterware failing with ORA-12547: TNS:lost contact

[grid@rac1 bin]$ export ORACLE_SID=+ASM1
[grid@rac1 bin]$ export ORACLE_HOME=/
[grid@rac1 bin]$ ./sqlplus "/ as sysasm"

SQL*Plus: Release 12.1.0.2.0 Production on Thu Apr 20 09:15:47 2017

Copyright (c) 1982, 2014, Oracle. All rights reserved.

ERROR:
ORA-12547: TNS:lost contact 

CAUSE

1.  From ohasd_oraagent_oracle.trc it shows the following 

2017-04-20 09:57:08.796826*:kgfn.c@1281: kgfnFindLocalNode: found no members
2017-04-20 09:57:08.796826*:kgfn.c@1332: kgfnFindLocalNode: not ok
2017-04-20 09:57:08.796826*:kgfn.c@2097: kgfnTgtInit: not found
2017-04-20 09:57:08.796826*:kgfn.c@2207: kgfnTgtDestroy: sid=(null) host=(null) port=0
cstr=(null) asminst=(null) flags=0x0
2017-04-20 09:57:08.796826*:kgfn.c@5281: kgfnGetBeqData: kgfnTgtInit failed, inst=NULL flags=0x2000
2017-04-20 09:57:08.796826*:kgfn.c@4607: kgfnConnect2: kgfnGetBeqData failed
2017-04-20 09:57:08.796826*:kgfn.c@4868: kgfnConnect2: failed to connect
2017-04-20 09:57:08.796826*:kgfn.c@4887: kgfnConnect: conn=(nil)
2017-04-20 09:57:08.796826*:kgfp.c@669: kgfpInitComplete2 hdl=0x7f54f80e6198 conn=0x7f54f80e61b0 ok=0
2017-04-20 09:57:08.796826*:kgfo.c@947: kgfo_kge2slos error stack at kgfoAl06: ORA-15077: could not locate ASM instance serving a required diskgroup

2017-04-20 09:57:08.796826*:kgfo.c@1058: kgfoSaveError: ctx=0x7f54f810d090 hdl=(nil) gph=0x7f54f80aae58 ose=0x7f5511770dc0 at kgfo.c:1006
2017-04-20 09:57:08.796826*:kgfo.c@1115: kgfoSaveError: ignoring existing error: ORA-15077: could not locate ASM instance serving a required diskgroup

2017-04-20 09:57:08.799791 :kgfo.c@817: kgfoFreeHandle ctx=0x7f54f810d090 hdl=0x7f54f80e6178 conn=0x7f54f80e61b0 disconnect=0
2017-04-20 09:57:08.799791*:kgfo.c@846: disconnect hdl 0x7f54f80e6178 (recycling)
2017-04-20 09:57:08.799791*:kgfo.c@2757: Handle Alloc failed - kgfoCheckMount Reconnecting
2017-04-20 09:57:08.799791*:kgfo.c@2846: kgfoCheckMount dg=DATADG ok=0
2017-04-20 09:57:08.799869 : USRTHRD:293054208: {0:0:2} -- trace dump on error exit --

2017-04-20 09:57:08.799886 : USRTHRD:293054208: {0:0:2} Error [kgfoAl06] in [kgfokge] at kgfo.c:2850

2017-04-20 09:57:08.799900 : USRTHRD:293054208: {0:0:2} ORA-15077: could not locate ASM instance serving a required diskgroup

2017-04-20 09:57:08.799911 : USRTHRD:293054208: {0:0:2} Category: 7

2017-04-20 09:57:08.799923 : USRTHRD:293054208: {0:0:2} DepInfo: 15077

2017-04-20 09:57:08.800649 : USRTHRD:293054208: {0:0:2} -- trace dump end --

2017-04-20 09:57:08.800717 :CLSDYNAM:293054208: [ora.storage]{0:0:2} [check] retcode = 7, kgfoCheckMount(DATADG)
2017-04-20 09:57:08.800736 :CLSDYNAM:293054208: [ora.storage]{0:0:2} [check] (null) category: 7, operation: kgfoAl06, loc: kgfokge, OS error: 15077, other: ORA-15077: could not locate ASM instance serving a required diskgroup

2017-04-20 09:57:08.800766 :CLSDYNAM:293054208: [ora.storage]{0:0:2} [check] kgfo returncode 1
2017-04-20 09:57:08.800787 :CLSDYNAM:293054208: [ora.storage]{0:0:2} [check] (:CLSN00140:)StorageAgent::parsekgforretcodes OCR dgName DATADG state 1
2017-04-20 09:57:08.800787*:kgfn.c@3203: kgfnEnvDestroy: env=0x7f54f80e61b0
2017-04-20 09:57:09.656011 : USRTHRD:293054208: {0:9:3} Check: 0-1
2017-04-20 09:57:32.729956 :CLSDYNAM:286750464: [ora.ctssd]{0:9:3} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0
2017-04-20 09:57:32.730012 :CLSDYNAM:286750464: [ora.ctssd]{0:9:3} [check] translateReturnCodes, return = 0, state detail = OBSERVERCheckcb data [0x7f54e4052280]: mode[0xee] offset[0 ms].
2017-04-20 09:57:36.903322 :CLSDYNAM:286750464: [ora.gipcd]{0:0:2} [check] ClsdmClient::sendMessage clsdmc_respget return: status=0, ecode=0

2.  Some files ownership of $GRID_HOME/bin changed to non-grid user. Here GRID_HOME owner is "grid" user, but some files' ownership has been changed to "oracle".

[root@rac1 ]# ls -lstr
8 -rwxr-xr-x 1 oracle oinstall 6263 Apr 24 2016 dbca
4 -rwxr-xr-x 1 oracle oinstall 3898 Apr 24 2016 rdtool
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 oranetmonitor
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 oradnssd
8 -rwxr-x--- 1 oracle oinstall 4265 Apr 24 2016 ologdbg.pl
4 -rwxr-x--- 1 oracle oinstall 2025 Apr 24 2016 oclumon.pl
4 -rwxr-xr-x 1 oracle oinstall 2944 Apr 24 2016 mgmtca
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 mdnsd
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 gpnptool
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 gpnpd
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 gipcd
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 crs_setperm
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 crs_relocate
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 crs_register
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 crs_profile
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 crs_getperm
4 -rwxr-xr-x 1 oracle oinstall 3472 Apr 24 2016 cluutil
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 clssproxy
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 cemutls
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 cemutlo
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 appagent
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 scriptagent
4 -rwxr-xr-x 1 oracle oinstall 1507 Apr 24 2016 racgwrap
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 oraagent
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 olsnodes
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 oifcfg
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 odnsd
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 lsdb
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 evmwatch
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 evmsort
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 evmshow
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 evmpost
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 evmmklib
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 evmmkbin
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 evmlogger
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 evminfo
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 evmd
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 emcrsp
8 -rwxr-xr-x 1 oracle oinstall 8085 Apr 24 2016 crs_unregister 

SOLUTION

1.  Stop Clusterware, if it is still partially up.

As root user:

# /bin/crsctl stop crs -f

2. Change the owner of the above mentioned files from "oracle" to "grid". 

 

 

 

3. Start the clusterware.

As root user:

# /bin/crsctl start crs

4. Check the status of the clusterware.

# /bin/crsctl check crs

CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

 5. Check the connection to ASM1 instance.

[grid@rac1 ~]$ sqlplus "/ as sysasm"

SQL*Plus: Release 12.1.0.2.0 Production on Thu Apr 20 10:02:04 2017

Copyright (c) 1982, 2014, Oracle. All rights reserved.

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL>

 5. Validate the permissions of the binaries under the GI Home "bin" folder.

==========================类似11.2.0.4可能遇到的问题===================

SYMPTOMS

 

1. The crsd and evmd are in INTERMEDIATE/OFFLINE status $crsctl stat res -t -init

--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
      1        ONLINE  ONLINE       node2                Started             
ora.cluster_interconnect.haip
      1        ONLINE  ONLINE       node2
ora.crf
      1        ONLINE  OFFLINE      node2
ora.crsd
      1        ONLINE  INTERMEDIATE       node2
ora.cssd
      1        ONLINE  ONLINE       node2
ora.cssdmonitor
      1        ONLINE  ONLINE       node2
ora.ctssd
      1        ONLINE  ONLINE       node2               OBSERVER            
ora.diskmon
      1        ONLINE  ONLINE       node2

ora.evmd
      1        ONLINE  INTERMEDIATE       node2
ora.gipcd
      1        ONLINE  ONLINE       node2
ora.gpnpd
      1        ONLINE  ONLINE       node2
ora.mdnsd
      1        ONLINE  ONLINE       node2

-----------

 

2. All DISKGROUPs are in UNMOUNTED status.

 

3. /log//alert.log :
------------

2016-05-11 04:52:02.896:
[cssd(39230)]CRS-1713:CSSD daemon is started in clustered mode
2016-05-11 04:52:23.746:
[cssd(39230)]CRS-1707:Lease acquisition for node node2 number 2 completed

.
2016-05-11 04:52:52.838:
[ohasd(38960)]CRS-2765:Resource 'ora.crsd' has failed on server 'node2'.
2016-05-11 04:52:56.125:
[crsd(40040)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /log//crsd/crsd.log.
2016-05-11 04:52:56.131:
[crsd(40040)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
ORA-01017: invalid username/password; logon denied
]. Details at (:CRSD00111:) in /log//crsd/crsd.log.
-------------
 

4. /log//crsd.log of the problematic node :
-------------------------------------

2016-05-11 04:52:50.433: [ CRSMAIN][2876733168] Initializing OCR
[   CLWAL][2876733168]clsw_Initialize: OLR initlevel [70000]
2016-05-11 04:52:51.709: [  OCRASM][2876733168]proprasmo: Error in open/create file in dg [DBFS_DG]
[  OCRASM][2876733168]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=1017, loc=kgfokge

2016-05-11 04:52:51.709: [  OCRASM][2876733168]ASM Error Stack : ORA-01017: invalid username/password; logon denied

2016-05-11 04:52:52.726: [  OCRASM][2876733168]proprasmo: kgfoCheckMount returned [7]
2016-05-11 04:52:52.726: [  OCRASM][2876733168]proprasmo: The ASM instance is down
2016-05-11 04:52:52.726: [  OCRRAW][2876733168]proprioo: Failed to open [+DBFS_DG]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
-------------

 

5. Connecting to ASM instance of node2 using "sqlplus / as sysasm" as the grid owner also failed with 

"ORA-01017: invalid username/password; logon denied".

 

 

CHANGES

 The Grid has been upgraded from 11.2.0.3 to 11.2.0.4 recently.

CAUSE

 The ownership and permission of "/bin/oracle"  was not correct, it was owned by root and set-user/set-group identification permissions was not set.

-rw-r----x 1 root oinstall 291315984 Feb 25 14:10 /bin/oracle

 

SOLUTION

1. Stop CRS on problematic node.

#crsctl stop crs -f

2. Change the permission and ownership of  "/bin/oracle"  as below.

#chmod 6751 /bin/oracle

e.g.

$ ls -lrt /bin/oracle
-rwsr-s--x 1 oracle oinstall 291315984 Feb 25 14:10 /bin/oracle

3. Start CRS

#crsctl start crs