centos7.4下ORA-27300 ORA-27301 ORA-27302处理

alert日志：

Errors in file /oracle/diag/rdbms/dbepdb/dbepdb/trace/dbepdb_psp0_1092.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Cannot allocate memory
ORA-27302: failure occurred at: skgpspawn3
2018-08-14T05:07:43.565893+08:00
Process m000 died, see its trace file
2018-08-14T05:07:43.577097+08:00
Process startup failed, error stack:
2018-08-14T05:07:43.577460+08:00
Errors in file /oracle/diag/rdbms/dbepdb/dbepdb/trace/dbepdb_psp0_1092.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Cannot allocate memory
2018-08-14T05:07:42.182453+08:00
Process startup failed, error stack:
2018-08-14T05:07:42.182731+08:00
Errors in file /oracle/diag/rdbms/dbepdb/dbepdb/trace/dbepdb_psp0_1092.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Cannot allocate memory
ORA-27302: failure occurred at: skgpspawn3
2018-08-14T05:07:42.255869+08:00
Process m001 died, see its trace file
2018-08-14T05:07:42.571428+08:00
Process startup failed, error stack:
2018-08-14T05:07:42.571736+08:00
Errors in file /oracle/diag/rdbms/dbepdb/dbepdb/trace/dbepdb_psp0_1092.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Cannot allocate memory
ORA-27302: failure occurred at: skgpspawn3
2018-08-14T05:07:43.565893+08:00
Process m000 died, see its trace file
2018-08-14T05:07:43.577097+08:00
Process startup failed, error stack:
2018-08-14T05:07:43.577460+08:00

trace文件

oracle@dz-dbepdb01[/oracle]$view /oracle/diag/rdbms/dbepdb/dbepdb/trace/dbepdb_psp0_1092.trc

*** 2018-08-14T05:07:40.519013+08:00
*** SESSION ID:(765.22310) 2018-08-14T05:07:40.519113+08:00
*** CLIENT ID:() 2018-08-14T05:07:40.519128+08:00
*** SERVICE NAME:(SYS$BACKGROUND) 2018-08-14T05:07:40.519146+08:00
*** MODULE NAME:() 2018-08-14T05:07:40.519160+08:00
*** ACTION NAME:() 2018-08-14T05:07:40.519172+08:00
*** CLIENT DRIVER:() 2018-08-14T05:07:40.519184+08:00

Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Cannot allocate memory
ORA-27302: failure occurred at: skgpspawn3
OS - DIAGNOSTICS
----------------
loadavg : 19.73 15.36 18.73
Memory (Avail / Total) = 1056.80M / 7822.25M
Swap (Avail / Total) = 0.00M /  5120.00M
Max user processes limits(s / h) =  16384 / 16384
----------------

*** 2018-08-14T05:07:42.182482+08:00
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Cannot allocate memory
ORA-27302: failure occurred at: skgpspawn3
OS - DIAGNOSTICS
----------------
loadavg : 19.73 15.36 18.73
Memory (Avail / Total) = 1056.95M / 7822.25M
Swap (Avail / Total) = 0.00M /  5120.00M
Max user processes limits(s / h) =  16384 / 16384
----------------

*** 2018-08-14T05:07:42.571454+08:00
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Cannot allocate memory
ORA-27302: failure occurred at: skgpspawn3
OS - DIAGNOSTICS
----------------
loadavg : 19.73 15.36 18.73
Memory (Avail / Total) = 1057.00M / 7822.25M
Swap (Avail / Total) = 0.00M /  5120.00M
 

HugePages on Oracle Linux 64-bit (文档 ID 361468.1)

处理办法：

1、增大内存，由原来的8g增大到16g。

2、增大swap，原8g内存，swap给了5g。明显是不合理的，现在16g内存，分配16g。

oracle12.2官方建议：

Between 1 GB and 2 GB: 1.5 times the size of the RAM
Between 2 GB and 16 GB: Equal to the size of the RAM
More than 16 GB: 16 GB

3、cpu目前是8核，但是平均负载都在15-19，已经验证超重运算了，但是这个问题可能是内存导致的。

4、扩大内存参数以匹配现有的物理内存：

kernel.shmmax=16642998272

kernel.shmall=4063232

vm.min_free_kbytes=1048576

5、开启hugepages：

APPLIES TO:

Linux OS - Version Enterprise Linux 4.0 to Oracle Linux 7.2 with Unbreakable Enterprise Kerne [3.8.13] [Release RHEL4 to OL7U2]
Oracle Database - Enterprise Edition
Linux x86-64
Oracle Linux
Red Hat Enterprise Linux (RHEL)
SUSE Linux Enterprise Server (SLES)

PURPOSE

This document aims to provide.

• Basic configuration of HugePages on 64-bit Linux
• Fundemental reasons to use HugePages on Linux
• References to known problems
• References to technical background on HugePages

SCOPE

Information in this document is useful for Linux system administrators and Oracle database administrators working with system administrators.

This document covers information about Linux HugePages for 64-bit architectures. For more generic and uses on 32-bit and for references please see Document 361323.1

The configuration steps provided here is primarily for Oracle Linux. Still the same concepts and configurations should apply to other Linux distributions.

DETAILS

Introduction

HugePages is a feature of the Linux kernel  which allows larger pages to manage memory as the alternative to the small 4KB pagesize. For a detailed introduction, see Document 361323.1

Why Do You Need HugePages?

HugePages is crucial for faster Oracle database performance on Linux if you have a large RAM and SGA. If your combined database SGAs is large (like more than 8GB, can even be important for smaller), you will need HugePages configured. Note that the size of the SGA matters. Advantages of HugePages are:

• Larger Page Size and Less # of Pages: Default page size is 4K whereas the HugeTLB size is 2048K. That means the system would need to handle 512 times less pages.
• Reduced Page Table Walking: Since a HugePage covers greater contiguous virtual address range than a regular sized page, a probability of getting a TLB hit per TLB entry with HugePages are higher than with regular pages. This reduces the number of times page tables are walked to obtain physical address from a virtual address.
• Less Overhead for Memory Operations: On virtual memory systems (any modern OS) each memory operation is actually two abstract memory operations. With HugePages, since there are less number of pages to work on, the possible bottleneck on page table access is clearly avoided.
• Less Memory Usage: From the Oracle Database perspective, with HugePages, the Linux kernel will use less memory to create pagetables to maintain virtual to physical mappings for SGA address range, in comparison to regular size pages. This makes more memory to be available for process-private computations or PGA usage.
• No Swapping: We must avoid swapping to happen on Linux OS at all Document 1295478.1. HugePages are not swappable (whereas regular pages are). Therefore there is no page replacement mechanism overhead. HugePages are universally regarded as pinned.
• No 'kswapd' Operations: kswapd will get very busy if there is a very large area to be paged (i.e. 13 million page table entries for 50GB memory) and will use an incredible amount of CPU resource. When HugePages are used, kswapd is not involved in managing them. See also Document 361670.1

There is a general misconception where the HugePages is a feature specific to 32-bit Linux.  HugePages is a generic feature available to all word-sizes and architectures. Just that there are some specifics with 32-bit platforms. Please see Document 361323.1 for further references.

The HugePages configuration described in this document does not cause the O/S components to use HugePages. HugePages will be used by applications which explicitly make use of HugePages in their code (like majority of Oracle RDBMS SGA given proper configuration). Therefore, you will still see swap usage on the system as the regular O/S components, or non-HugePages-aware applications use swappable pages.

How to Configure

The configuration steps below will guide you to do a persistent system configuration where you would need to do a complete reboot of the system. Please plan your operations accordingly:

Step 1: Have the memlock user limit set in /etc/security/limits.conf file. Set the value (in KB) slightly smaller than installed RAM. e.g. If you have 64GB RAM installed, you may set:

*   soft   memlock    60397977
*   hard   memlock    60397977

There is no harm in setting this value larger than your SGA requirements. 

The parameters will be set by default on:

• Oracle Linux with oracle-validated package (See Document 437743.1) installed.
• Oracle Exadata DB compute nodes

 

These settings can also be done in a *.conf file under /etc/security/limits.d/ directory. Also if there is a *.conf file in /etc/security/limits.d/ that has settings for memlock, setting it up in /etc/security/limits.conf may not work. Make sure you check the files /etc/security/limits.d/*.conf files.

Step 2: Re-logon to the Oracle product owner account (e.g. 'oracle') and check the memlock limit

$ ulimit -l
60397977

Step 3: If you have Oracle Database 11g or later, the default database created uses the Automatic Memory Management (AMM) feature which is incompatible with HugePages. Disable AMM before proceeding. To disable, set the initialization parameters MEMORY_TARGET and MEMORY_MAX_TARGET to 0 (zero). Please see Document 749851.1 for further information in case you encounter the error below:

ORA-00845: MEMORY_TARGET not supported on this system

 

Note: Starting in 11.2.0.3, AMM is no longer configured by default if DBCA detects that the machine has more than 4GB of RAM. See Document 1453227.1 for details.

Step 4: Make sure that all your database instances are up (including ASM instances) as they would run on production. Use the script hugepages_settings.sh in Document 401749.1 to calculate the recommended value for the vm.nr_hugepageskernel parameter. e.g.:

$ ./hugepages_settings.sh
...
Recommended setting: vm.nr_hugepages = 1496
$

Note: You can also calculate a proper value for the parameter yourself but that is not advised if you do not have extensive experience with HugePages and concepts.

Step 5: Edit the file /etc/sysctl.conf and set the vm.nr_hugepages parameter there:

...
vm.nr_hugepages = 1496
...

This will make the parameter to be set properly with each reboot. 

Step 6: Stop all the database instances and reboot the server 

(Although settings could have been done dynamically they would not be effective to the extent we require before a reboot. The best practice is to do a persistent system configuration and perform a reboot to complete the configuration as presented through the steps above)

What If the Database / SGA Configurations Change?

The performed configuration is basically based on the RAM installed and combined size of SGA of database instances you are running. Based on that when:

• Amount of RAM installed for the Linux OS changed
• New database instance(s) introduced
• SGA size / configuration changed for one or more database instances

you should revise your HugePages configuration to make it suitable to the new memory framework. If not you may experience one or more problems below on the system:

• Poor database performance
• System running out of memory or excessive swapping
• Database instances cannot be started
• Crucial system services failing

Check and Validate the Configuration

After the system is rebooted, make sure that your database instances (including the ASM instances) are started. Automatic startup via OS configuration or CRS, or manual startup (whichever method you use) should have been performed. Check the HugePages state from /proc/meminfo. e.g.:

 

# grep HugePages /proc/meminfo
HugePages_Total:    1496
HugePages_Free:      485
HugePages_Rsvd:      446
HugePages_Surp:        0

The values in the output will vary. To make sure that the configuration is valid, the HugePages_Free value should be smaller than HugePages_Total.

Also there should be some HugePages_Rsvd if PRE_PAGE_SGA is 'false' for all the Oracle database instances.. HugePages_Rsvd counts free pages that are reserved for use (requested for an SGA, but not touched/mapped yet). PRE_PAGE_SGA determines if the all SGA pages are read-in when the instance starts up. If parameter is set to 'true' then the OS page table entries are prebuilt for each page of the SGA, leading to HugePages reservation of those pages. For Oracle database versions before 12.1 the default value for PRE_PAGE_SGA is 'false'. So the HugePages_Rsvd would be higher than 0. But since 12c the PRE_PAGE_SGA defaults to 'true' which would cause the HugePages_Rsvd to be 0.

The sum of Hugepages_Free and HugePages_Rsvd may be smaller than your total combined SGA as instances allocate pages dynamically and proactively as needed.

Troubleshooting

Some of the common problems and how to troubleshoot them are listed in the following table:

附脚本：

oracle@dz-dbepdb01[/oracle]$cat hugepages_settings.sh 
#!/bin/bash
#
# hugepages_settings.sh
#
# Linux bash script to compute values for the
# recommended HugePages/HugeTLB configuration
# on Oracle Linux
#
# Note: This script does calculation for all shared memory
# segments available when the script is run, no matter it
# is an Oracle RDBMS shared memory segment or not.
#
# This script is provided by Doc ID 401749.1 from My Oracle Support 
# http://support.oracle.com

# Welcome text
echo "
This script is provided by Doc ID 401749.1 from My Oracle Support 
(http://support.oracle.com) where it is intended to compute values for 
the recommended HugePages/HugeTLB configuration for the current shared 
memory segments on Oracle Linux. Before proceeding with the execution please note following:
 * For ASM instance, it needs to configure ASMM instead of AMM.
 * The 'pga_aggregate_target' is outside the SGA and 
   you should accommodate this while calculating SGA size.
 * In case you changes the DB SGA size, 
   as the new SGA will not fit in the previous HugePages configuration, 
   it had better disable the whole HugePages, 
   start the DB with new SGA size and run the script again.
And make sure that:
 * Oracle Database instance(s) are up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not setup 
   (See Doc ID 749851.1)
 * The shared memory segments can be listed by command:
     # ipcs -m


Press Enter to proceed..."

read

# Check for the kernel version
KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`

# Find out the HugePage size
HPG_SZ=`grep Hugepagesize /proc/meminfo | awk '{print $2}'`
if [ -z "$HPG_SZ" ];then
    echo "The hugepages may not be supported in the system where the script is being executed."
    exit 1
fi

# Initialize the counter
NUM_PG=0

# Cumulative number of pages required to handle the running shared memory segments
for SEG_BYTES in `ipcs -m | cut -c44-300 | awk '{print $1}' | grep "[0-9][0-9]*"`
do
    MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
    if [ $MIN_PG -gt 0 ]; then
        NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
    fi
done

RES_BYTES=`echo "$NUM_PG * $HPG_SZ * 1024" | bc -q`

# An SGA less than 100MB does not make sense
# Bail out if that is the case
if [ $RES_BYTES -lt 100000000 ]; then
    echo "***********"
    echo "** ERROR **"
    echo "***********"
    echo "Sorry! There are not enough total of shared memory segments allocated for 
HugePages configuration. HugePages can only be used for shared memory segments 
that you can list by command:

    # ipcs -m

of a size that can match an Oracle Database SGA. Please make sure that:
 * Oracle Database instance is up and running 
 * Oracle Database 11g Automatic Memory Management (AMM) is not configured"
    exit 1
fi

# Finish with results
case $KERN in
    '2.2') echo "Kernel version $KERN is not supported. Exiting." ;;
    '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
           echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
    '2.6') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '3.8') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '3.10') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '4.1') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
esac

# End