ORA-27300: OS system dependent operation:fork failed with status

问题现象：
 

Sun Sep 30 21:12:28 2018
Errors in file /DBSoft2/oracle/diag/rdbms/fzdbcgbk/fzdbcgbk1/trace/fzdbcgbk1_psp0_221463.trc:
ORA-27300: ͳϵͳز: fork ʧ ״̬Ϊ: 11
ORA-27301: ͳϢ: Resource temporarily unavailable
ORA-27302: skgpspawn3
Process J000 died, see its trace file
Sun Sep 30 21:12:28 2018
kkjcre1p: unable to spawn jobq slave process 
Sun Sep 30 21:12:28 2018
Errors in file /DBSoft2/oracle/diag/rdbms/fzdbcgbk/fzdbcgbk1/trace/fzdbcgbk1_cjq0_222890.trc:
Sun Sep 30 21:12:29 2018
Process startup failed, error stack:
Sun Sep 30 21:12:29 2018
Errors in file /DBSoft2/oracle/diag/rdbms/fzdbcgbk/fzdbcgbk1/trace/fzdbcgbk1_psp0_221463.trc:
ORA-27300: ͳϵͳز: fork ʧ ״̬Ϊ: 11
ORA-27301: ͳϢ: Resource temporarily unavailable
ORA-27302: skgpspawn3

参考资料：

Mos ID:392006.1

SYMPTOMS：
The following message reported in alert log:

Mon Sep 18 18:10:34 2006
Errors in file /u01/oracle/admin/orcl/bdump/orcl1_psp0_954436.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Mon Sep 18 18:10:35 2006
Process P073 died, see its trace file

 

CAUSE：
The error messages indicating that oracle has problem in forking more process, the maximum number of PROCESSES allowed per user could be too low.

This is configured by the following kernel settings:

  AIX: maxuproc
  HP and Solaris: maxuprc
  Solaris 10/11: project.max-lwps
  Linux: nproc (/etc/security/limits.conf)

 

检查配置：

[root@fzdb1 trace]# su - oracle2
[oracle2@fzdb1 ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1031748
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 3047
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[oracle2@fzdb1 ~]$ ulimit -u 
3047

[root@fzdb1 ~]# grep proc /etc/security/limits.conf
#        - nproc - max number of processes
#        - priority - the priority to run user process with
#@student        hard    nproc           20
#@faculty        soft    nproc           20
#@faculty        hard    nproc           50
#ftp             hard    nproc           0
oracle           soft    nproc           3047
oracle           hard    nproc           16384
grid             soft    nproc           3047
grid             hard    nproc           16384

问题分析：
   从上面检查我们可以发现oracle2的用户进程数量已经达到了1491，几乎接近oracle2用户的user processes数量，所以数据库日志里面频繁出现大量的ORA-27300,ORA-27301,ORA-27302，根据MOS:392006.1建议修改/etc/security/limits.conf文件里的oracle2 nproc值为更大的值，但是实际上我们并没有发现有关于oracle2用户的配置，所以我们手工添加如下配置后重启数据库服务器就可以了:
oracle2         soft    nproc           16384
oracle2         hard    nproc           16384
oracle2         soft    nofile          65536
oracle2         hard    nofile          65536
oracle2         soft    memlock         3145728
oracle2         hard    memlock         3145728

 

开始修改：

[root@fzdb1 ~]# cat  >> /etc/security/limits.conf << EOF
> oracle2         soft    nproc           16384
> oracle2         hard    nproc           16384
> oracle2         soft    nofile          65536
> oracle2         hard    nofile          65536
> oracle2         soft    memlock         3145728
> oracle2         hard    memlock         3145728
> EOF

重启服务器：

[oracle2@fzdb1 ~]$ su - root
[root@fzdb1 ~]$ reboot

检查刚才的参数修改：
 

[oracle2@fzdb1 ~]$ ulimit -u
16384

检查数据库日志：
   修改参数之后通过一天的观察未见ORA-27300,ORA-27301,ORA-27302等报错信息，此故障处理完毕。