记一次Oracle Clusterware成功安装后的故障处理

 

记一次Oracle Clusterware安装成功后的故障处理

 

1. 环境

[grid@rac1 rac1]$ cat /etc/issue

Red Hat Enterprise Linux Server release 5.8 (Tikanga)

Kernel \r on an \m

 

2. 问题描述在安装RAC的过程中, 成功安装好grid (clusterware) 后关闭了各节点. 在下次开启各节点后, 检查crs资源状态, 出现如下错误:

[grid@rac1 ~]$ crs_stat -t -v

CRS-0184: Cannot communicate with the CRS daemon.

 

3. 分析解决

检查crs状态:

[grid@rac1 ~]$ crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4535: Cannot communicate with Cluster Ready Services  # 无法与CRS通信

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

查看crsd对应日志:

2014-11-21 15:18:13.490: [GIPCXCPT][1002185440] gipcShutdownF: skipping shutdown, count 2, from [ clsinet.c : 1732], ret gipcretSuccess (0)

2014-11-21 15:18:13.492: [GIPCXCPT][1002185440] gipcShutdownF: skipping shutdown, count 1, from [ clsgpnp0.c : 1021], ret gipcretSuccess (0)

2014-11-21 15:18:13.498: [  OCRASM][1002185440]proprasmo: Error in open/create file in dg [DATA] # 打开磁盘组失败

[  OCRASM][1002185440]SLOS : SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge

ORA-15077: could not locate ASM instance serving a required diskgroup  # 没有ASM实例



2014-11-21 15:18:13.498: [  OCRASM][1002185440]proprasmo: kgfoCheckMount returned [7]

2014-11-21 15:18:13.498: [  OCRASM][1002185440]proprasmo: The ASM instance is down # ASM实例处于关闭状态

2014-11-21 15:18:13.499: [  OCRRAW][1002185440]proprioo: Failed to open [+DATA]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.

2014-11-21 15:18:13.499: [  OCRRAW][1002185440]proprioo: No OCR/OLR devices are usable

2014-11-21 15:18:13.499: [  OCRASM][1002185440]proprasmcl: asmhandle is NULL

2014-11-21 15:18:13.499: [  OCRRAW][1002185440]proprinit: Could not open raw device 

2014-11-21 15:18:13.499: [  OCRASM][1002185440]proprasmcl: asmhandle is NULL

2014-11-21 15:18:13.499: [  OCRAPI][1002185440]a_init:16!: Backend init unsuccessful : [26]

2014-11-21 15:18:13.499: [  CRSOCR][1002185440] OCR context init failure.  Error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat=7, opn=kgfoAl06, dep=15077, loc=kgfokge

ORA-15077: could not locate ASM instance serving a required diskgroup

] [7]

2014-11-21 15:18:13.499: [    CRSD][1002185440][PANIC] CRSD exiting: Could not init OCR, code: 26

2014-11-21 15:18:13.499: [    CRSD][1002185440] Done.

 

日志信息表明, ASM实例未能启动, 导致crsd进程无法启动

尝试手动启动ASM实例:

[grid@rac1 ~]$ asmcmd

Connected to an idle instance.

ASMCMD> startup

ORA-27154: post/wait create failed

ORA-27300: OS system dependent operation:semget failed with status: 28

ORA-27301: OS failure message: No space left on device

ORA-27302: failure occurred at: sskgpsemsper

Connected to an idle instance.

上述信息表明, 失败的操作是semget.
semget的任务是获得信号量集(get set of semaphores), 这里的No space left on device并不是指存储空间，而是信号量资源.

检查系统中的信号量使用情况:

[grid@rac1 ~]$ ipcs



------ Shared Memory Segments --------

key        shmid      owner      perms      bytes      nattch     status      

0x00000000 3407873    root      644        80         2                       

0x00000000 3440643    root      644        16384      2                       

0x00000000 3473412    root      644        280        2                       



------ Semaphore Arrays --------

key        semid      owner      perms      nsems     



------ Message Queues --------

key        msqid      owner      perms      used-bytes   messages 

未发现异常. 继续检查内核参数中的semmns:

root@rac1 ~]# sysctl -a|grep sem

kernel.sem = 250    100    32    128

四个参数分别为:
semmsl---每个信号集包含的信号数，该值应比ORACLE进程的最大数大10左右
semmns---系统中的信号数
semopm---每个信号操作呼叫的最大操作数
semmni---信号集标识符数,用于控制可随时创建的信号集数

加大系统中的信号量(/etc/sysctl.conf):

kernel.sem = 256 32768 100 228

 

重新启动ASM实例:

ASMCMD> startup

ORA-03113: end-of-file on communication channel

Connected to an idle instance.

因着急继续做实验, 此时直接对两个节点进行了重新启动, 重启后ASM实例正常启动, crs资源状态正常, 问题得到解决.

后来实验结束后查询ORA-03113, 导致该错误的可能原因有:

1) Unix核心参数设置不当

2) Oracle执行文件权限不正确／环境变量问题

3) 客户端通信不能正确处理

4) 数据库服务器崩溃/操作系统崩溃/进程被kill

5) Oracle 内部错误

6) 特定SQL、PL/SQL引起的错误

7) 空间不够

8) 防火墙的问题

但因错误环境已消失, 未能排查解决, 很是遗憾, 仅留做以后参考.

 

4. 参考
1) [Oracle 11g RAC CRS-4535/ORA-15077] http://blog.csdn.net/l106439814/article/details/8969060
2) [ASM启动报错ORA-27300, ORA-27301 and ORA-27302: failure occurred at: sskgpsemsper] http://www.51itstudy.com/33735.html
3) [DBA手记:共享内存无法正常释放的处理] http://www.eygle.com/archives/2011/03/ipcs_semaphore.html
4) [ORA-03113: end-of-file on communication channel 错误定位过程] http://www.51itstudy.com/6628.html