skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3



oracle 无法 fork 新的进程

今天客户的系统突然遇到一个问题，说不能连接新的进程。

这里做个札记，简单记录下，基本诊断如下：

首先先看下系统层面是否有问题：

#errpt -aj  xxxxxx

LABEL:          PGSP_KILL
IDENTIFIER:     C5C09FFA

Date/Time:       Sun Feb 23 08:05:14 BEIS
Sequence Number: 381582
Machine Id:      00CF786F4C00
Node Id:         xxxxx
Class:           S
Type:            PERM
Resource Name:   SYSVMM         

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SYSTEM RUNNING OUT OF PAGING SPACE  ---------------------------这里报错已经很明确了：系统交换空间不足

Failure Causes
INSUFFICIENT PAGING SPACE DEFINED FOR THE SYSTEM
PROGRAM USING EXCESSIVE AMOUNT OF PAGING SPACE

        Recommended Actions
        DEFINE ADDITIONAL PAGING SPACE
        REDUCE PAGING SPACE REQUIREMENTS OF PROGRAM(S)

Detail Data
PROGRAM
nmon_aix52ml5
USER'S PROCESS ID:
      153542
PROGRAM'S PAGING SPACE USE IN 1KB BLOCKS
         204
$

$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$

接着 ，我们再看下alert.log 如下：

alert.log
Sun Feb 23 09:23:32 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:23:49 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:23:49 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:23:50 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:23:52 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:23:53 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:23:55 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:24:06 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:24:06 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:24:16 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:24:16 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:24:27 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:24:27 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:24:38 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:24:38 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:24:49 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:24:49 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:24:59 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:24:59 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:25:10 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:25:10 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:25:21 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:25:21 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:25:32 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:25:32 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:25:42 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:25:42 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:25:53 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:25:53 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:26:04 2014
skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3
Sun Feb 23 09:26:04 2014
Errors in file /oracle/app/oracle/admin/gis/bdump/gis_smon_152640.trc:
ORA-27142: could not create new process
Sun Feb 23 09:26:14 2014

注意这一点 ：

skgpspawn failed:category = 27142, depinfo = 12, op = fork, loc = skgpspawn3

这里的depinfo =12 的 os 定义为： 

#define ENOMEM  12      /* Not enough space                     */

3. 这里我们再确认下系统配置情况：

系统内存 ： 8g
交换空间 ：
$ lsps -a
Page Space      Physical Volume   Volume Group    Size %Used Active  Auto  Type
hd6             hdisk1            rootvg        8192MB    35     yes   yes    lv

oracle的内存配置为 2.3g

根据mos：OracleDatabase (RDBMS) on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix OperatingSystems Installation and Configuration Requirements Quick Reference (8.0.5 to11.2) (文档 ID 169706.1) 确认，当前的系统参数配置完全恰当

for aix ora9i swap  ：1Gbor match RAM

最后，推断为可能是用户在短期内做了某些，大排序的操作等，导致oracle 内存耗尽，进而使用os的swap 空间，最后系统资源不足，oracle 无法fork新的进程。

后来系统突然又正常了，因此也没做什么操作。

总结：

1，根据os配置内存和实际业务情况，来确定 sga的大小，并且要保证os有足够内存使用。

2， 如果os内存过小，可以适当调大 swap空间的大小，可以在一定层面上缓解上述状况。

3， 实在万不得已重启下服务器吧！！！！！

FYI：http://blog.csdn.net/eygle/article/details/84166 

     aix 环境下不封depinfo = xx 代表的错误号信息   :

http://blog.csdn.net/lixora/article/details/19758549