"crsctl check crs" command hangs at EVMD check



Pre-11gR2: "crsctl check crs" command hangs at EVMD check (文档 ID 1578875.1)

APPLIES TO:

Oracle Database - Enterprise Edition - Version 10.2.0.3 to 11.1.0.7 [Release 10.2 to 11.1]
Information in this document applies to any platform.
SYMPTOMS

In a 2 node RAC environment, with 11.1.0.7 CRS, execution of the command "crsctl check crs" hangs at EVMD check only in Node 1

[oracle@srv03401 bin]$ ./crsctl check crs
Cluster Synchronization Services appears healthy
Cluster Ready Services appears healthy

From Node1, below is the output of strace for the command "crsctl check crs"

# strace -f -t -o /tmp/crschk.trc crsctl check crs

Content of the generated output file :/tmp/crschk.trc is as follows:  

28268 11:47:03 execve("./crsctl", ["./crsctl", "check", "crs"], [/* 23 vars */]) = 0
28268 11:47:03 brk(0)                   = 0x193d2000
28268 11:47:03 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b35b9436000
28268 11:47:03 uname({sys="Linux", node="srv03401.metra.com", ...}) = 0
28268 11:47:03 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
28268 11:47:03 open("/etc/ld.so.cache", O_RDONLY) = 3
28268 11:47:03 fstat(3, {st_mode=S_IFREG|0644, st_size=92563, ...}) = 0
28268 11:47:03 mmap(NULL, 92563, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2b35b9437000
28268 11:47:03 close(3)                 = 0
28268 11:47:03 open("/lib64/libtermcap.so.2", O_RDONLY) = 3
28268 11:47:03 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\20\300z2\0\0\0"..., 832) = 832
28268 11:47:03 fstat(3, {st_mode=S_IFREG|0755, st_size=15840, ...}) = 0
28268 11:47:03 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b35b944e000
28268 11:47:03 mmap(0x327ac00000, 2108944, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x327ac00000
28268 11:47:03 mprotect(0x327ac03000, 2093056, PROT_NONE) = 0
28268 11:47:03 mmap(0x327ae02000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x327ae02000
28268 11:47:03 close(3)                 = 0
28268 11:47:03 open("/lib64/libdl.so.2", O_RDONLY) = 3
..
..
28268 11:47:03 close(3)                 = 0
28268 11:47:03 write(1, "Cluster Ready Services appears h"..., 39) = 39
28268 11:47:03 socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = 3
28268 11:47:03 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
28268 11:47:03 bind(3, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0
28268 11:47:03 getsockname(3, {sa_family=AF_INET6, sin6_port=htons(42027), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [140733193388060]) = 0
28268 11:47:03 getpeername(3, 0x7fff5f19e1e0, [140733193388060]) = -1 ENOTCONN (Transport endpoint is not connected)
28268 11:47:03 getsockopt(3, SOL_SOCKET, SO_SNDBUF, [5536382933839118336], [4]) = 0
28268 11:47:03 getsockopt(3, SOL_SOCKET, SO_RCVBUF, [5536382933843050496], [4]) = 0
28268 11:47:03 fcntl(3, F_SETFD, FD_CLOEXEC) = 0
28268 11:47:03 fcntl(3, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
28268 11:47:03 geteuid()                = 700
28268 11:47:03 times({tms_utime=1, tms_stime=2, tms_cutime=0, tms_cstime=0}) = 7422615891
28268 11:47:03 socket(PF_FILE, SOCK_STREAM, 0) = 4
28268 11:47:03 access("/var/tmp/.oracle/sSYSTEM.evm.acceptor.auth", F_OK) = 0
28268 11:47:03 connect(4, {sa_family=AF_FILE, path="/var/tmp/.oracle/sSYSTEM.evm.acceptor.auth"...}, 110
  

CAUSE

Analysing  the strace output, looks like it was trying to write to a socket.

========
28268 11:47:03 socket(PF_FILE, SOCK_STREAM, 0) = 4
28268 11:47:03 access("/var/tmp/.oracle/sSYSTEM.evm.acceptor.auth", F_OK) = 0
28268 11:47:03 connect(4, {sa_family=AF_FILE, path="/var/tmp/.oracle/sSYSTEM.evm.acceptor.auth"...}, 110   <<<<<<<
========
This, indicates a problem with the network socket file.

SOLUTION

Get the PID of evmd.bin process and kill it

$ ps -ef | grep 'd.bin'

oracle   21046 21045  0  2012 ?        00:07:46 /u01/app/ract/crs/bin/evmd.bin         
root     21054 15845  0  2012 ?        11:34:47 /u01/app/ract/crs/bin/crsd.bin reboot
oracle   22072 21453  0  2012 ?        05:44:50 /u01/app/ract/crs/bin/ocssd.bin
root     22135     1  0  2012 ?        00:00:00 /u01/app/ract/crs/bin/oclskd.bin
oracle   22410     1  0  2012 ?        00:00:00 /u01/app/ract/crs/bin/oclskd.bin
oracle   29834 27854  0 13:22 pts/8    00:00:00 egrep d.bin

$ kill -9 21046

After killing evmd.bin process, the command "crsctl check crs" returns the complete output without any hangs.

[oracle@srv03401 bin]$ ./crsctl check crs

CSS appears healthy
CRS appears healthy
EVM appears healthy