shareplex 错误处理一例 Cop: Address already in use tcp server bind addr port - exiting

今天遇到一个问题，shareplex启动之后，sp_ctrl进入不了。查看进程，只有几个，很不完整，查看event_log， 发现如下提示：

Cop: Address already in use tcp server bind addr 1.1.1.1 port 11111 - exiting  [module cop]

看了文档，kill当前sp_开头的进程，重启shareplex，即可。

文档URL，自己备用参考：

https://support.software.dell.com/kb/sol98270

Knowledge Article 98270

• Title

   
  Error starting cop tcp server bind addr 192.168.1.212 port 40481 - exiting

• Description

   

  When trying to start cop with a command like sp_cop -uPORT & an error like the following is thrown "tcp server bind addr 192.168.1.212 port 40481 - exiting"

  See example below
   

  [korcl@oel5u7x64 bin]$ sp_ctrl

  *******************************************************
  * SharePlex for Oracle Command Utility
  * © 2010 Quest Software, Inc.
  * ALL RIGHTS RESERVED.
  * Protected by U.S. Patents: 7,461,103 and 7,065,538
  *******************************************************

  sp_ctrl (oel5u7x64:8606)> shutdown
  [korcl@oel5u7x64 bin]$ ps -ef | grep sp_
  korcl 9280 9219 0 21:21 pts/1 00:00:00 grep sp_
  [korcl@oel5u7x64 bin]$ pwd
  /splex/proddir/bin
  [korcl@oel5u7x64 bin]$ /splex/proddir/bin/sp_cop -u8606 &
  [1] 9299
  [korcl@oel5u7x64 bin]$
  *******************************************************
  * SharePlex for Oracle Startup
  * © 2010 Quest Software, Inc.
  * ALL RIGHTS RESERVED.
  * Protected by U.S. Patents: 7,461,103 and 7,065,538
  * Version: 7.6.2.59-m64-oracle110
  * VarDir : /splex/vardir
  * Port : 8606
  *******************************************************

  [korcl@oel5u7x64 bin]$ /splex/proddir/bin/sp_cop -u8606 &
  [2] 9312
  [korcl@oel5u7x64 bin]$
  *******************************************************
  * SharePlex for Oracle Startup
  * © 2010 Quest Software, Inc.
  * ALL RIGHTS RESERVED.
  * Protected by U.S. Patents: 7,461,103 and 7,065,538
  * Version: 7.6.2.59-m64-oracle110
  * VarDir : /splex/vardir
  * Port : 8606
  *******************************************************
  tcp server bind addr 192.168.1.212 port 40481 - exiting

  [korcl@oel5u7x64 bin]$
  [2]+ Exit 1 /splex/proddir/bin/sp_cop -u8606
  [korcl@oel5u7x64 bin]$

• Cause

   

  This is usually caused by one of two errors, either the cop is already running and you are trying to restart it, or the cop has hung/stuck memory segments and or
  from a previous cop.

• Resolution

   

  Check to see if the cop or any children are running.
  unix> sp_ctrl
  If you can get into sp_ctrl then cop was already running
  unix> ps -ef | grep sp_
  Are any processes left over that need to killed
  example below
  ps -ef | grep sp_
  orcl 20721 1 0 Oct09 ? 00:00:20 /splex/proddir/.app-modules/sp_cop -u8606
  orcl 21552 20721 5 Oct09 ? 02:06:17 /splex/proddir/.app-modules/sp_ocap -u8606
  Kill these processes
  kill -9 20721 
  kill -9 21552
  now try and start cop in the background

  If the issue is stuck/hung memory segments or semaphores the follow this
  go to the $SP_SYS_VARDIR/rim and
  [korcl@oel5u7x64 rim]$ od -x shmaddr.loc
  0000000 800c 0049 2000 a274 7f29 0000 50b7 0017
  0000020 2000 0002 0004 000a
  0000030
  [korcl@oel5u7x64 rim]$ od -x shstinfo.ipc
  0000000 000b 0049 c000 a181 7f29 0000 50b7 0007
  0000020 0000 0080 8003 0009
  0000030
  Take the 8th field to find the key and

  [korcl@oel5u7x64 rim]$ ipcs -a | grep 50b7
  0x000750b7 4784139 korcl 644 8388608 6
  0x001750b7 4816908 korcl 644 139264 6
  0x003750b7 4849677 korcl 644 8388608 2
  0x004750b7 4882446 korcl 644 33554432 2
  0x005750b7 4915215 korcl 644 8388608 3
  0x000750b7 622595 korcl 666 12
  0x001750b7 655364 korcl 666 26
  [korcl@oel5u7x64 rim]$
  You will want to run the ipcs -a without the grep to see which are memory segments and which are semaphores.
   

  [korcl@oel5u7x64 rim]$ ipcs -a

  ------ Shared Memory Segments --------
  key shmid owner perms bytes nattch status
  0x00000000 3637248 root 644 80 2
  0x00000000 3670017 root 644 16384 2
  0x00000000 3702786 root 644 280 2
  0x00000000 3833861 korcl 640 4096 0
  0x00000000 3866630 korcl 640 4096 0
  0x3b1671d0 3899399 korcl 640 4096 0
  0x00000000 3964936 korcl 640 4096 0
  0x00000000 3997705 korcl 640 4096 0
  0xf000e0d4 4030474 korcl 640 4096 0
  0x000750b7 4784139 korcl 644 8388608 6
  0x001750b7 4816908 korcl 644 139264 6
  0x003750b7 4849677 korcl 644 8388608 2
  0x004750b7 4882446 korcl 644 33554432 2
  0x005750b7 4915215 korcl 644 8388608 3

  ------ Semaphore Arrays --------
  key semid owner perms nsems
  0x6f5dd7b8 131073 korcl 640 124
  0xf274e4d0 262146 korcl 640 154
  0x000750b7 622595 korcl 666 12
  0x001750b7 655364 korcl 666 26

  ------ Message Queues --------
  key msqid owner perms used-bytes messages
  now kill them using the ipcrm command with the flag -m for memory segments and -s for semaphores
  
  ipcrm -s 622595 
  ipcrm -s 655364

  ipcrm -m 4784139 
  ipcrm -m 4816908 
  ipcrm -m 4849677 
  ipcrm -m 4882446 
  ipcrm -m 4915215