kernel.sem设置过小导致RAC节点无法启动

同时新搭RAC，搭到cluster，成功了，重启了一下，RAC2可以正常启动，但是RAC1起不来了。其实听到这个消息的时候，我是happy的，练手的机会又来了！

果断日志看起来！

1.操作系统日志---未发现异常

2.grid用户下的alert日志：

[crsd(3499)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/rac1/crsd/crsd.log.
2016-11-03 10:34:50.988
[ohasd(2668)]CRS-2765:Resource 'ora.crsd' has failed on server 'rac1'.

3.找到crsd日志：

2016-11-03 10:34:45.903: [  OCRASM][2386199264]proprasmo: kgfoCheckMount returned [7]
2016-11-03 10:34:45.903: [  OCRASM][2386199264]proprasmo: The ASM instance is down
2016-11-03 10:34:45.903: [  OCRRAW][2386199264]proprioo: Failed to open [+OCR]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2016-11-03 10:34:45.903: [  OCRRAW][2386199264]proprioo: No OCR/OLR devices are usable
2016-11-03 10:34:45.903: [  OCRASM][2386199264]proprasmcl: asmhandle is NULL
2016-11-03 10:34:45.903: [  OCRRAW][2386199264]proprinit: Could not open raw device 
2016-11-03 10:34:45.904: [  OCRASM][2386199264]proprasmcl: asmhandle is NULL
2016-11-03 10:34:45.904: [  OCRAPI][2386199264]a_init:16!: Backend init unsuccessful : [26]
2016-11-03 10:34:45.904: [  CRSOCR][2386199264] OCR context init failure.  Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge
ORA-15077: could not locate ASM instance serving a required diskgroup
] [7]
2016-11-03 10:34:45.904: [    CRSD][2386199264][PANIC] CRSD exiting: Could not init OCR, code: 26
2016-11-03 10:34:45.904: [    CRSD][2386199264] Done.

看起来是OCR盘的问题，但是检查了一下，盘没有问题，盘的权限、空间等等都没有问题。

尝试开启asm实例：

su - grid

sqlplus / as sysasm

SQL> startup
ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpsemsper

这个报错。。MOS一下；

找到了：ORA-27300, ORA-27301 and ORA-27302: failure occurred at: sskgpsemsper when starting ASM instance (文档 ID 1494285.1)

按照文档，检查了kernel.sem的设置，果然较小，修改之后重启操作系统，RAC1恢复正常。

kernel.sem = 256 32768 100 228

kernel.sem
从左至右 依次是  SEMMSL SEMMNS SEMOPM SEMMNI
SEMMAL   信号集容纳最大信号数量
SEMMNS   所有信号的最大数量
SEMOPM   调用单个信号集中最大信号数量
SEMMNI    信号集的最大值