ORA-00471数据库宕机问题处理

问题排查

检查告警日志：

[oracle@dbhost ~]$ tail -n 2000 $ORACLE_BASE/diag/rdbms/orcldb_0/orcldb/trace/alert_orcldb.log | grep 'error 471' -A3 -B3
Wed Jul 12 10:47:07 2023
Archived Log entry 89914 added for thread 1 sequence 348384 ID 0xe5a511f5 dest 1:
Wed Jul 12 10:47:29 2023
PMON (ospid: 27072): terminating the instance due to error 471
Wed Jul 12 10:47:29 2023
System state dump requested by (instance=1, osid=27072 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /oracle/app/diag/rdbms/orcldb_0/orcldb/trace/orcldb_diag_27083_20230712104729.trc

[oracle@dbhost ~]$ oerr ora 471
00471, 00000, "DBWR process terminated with error"
// *Cause:  The database writer process died
// *Action: Warm start instance

检查大页是否开启：

grep -i hugepage /proc/meminfo
sysctl -a | grep nr_hugepages

发现操作系统未开启大页。

开启大页配置

停库：

shudown immediate;
--shutdown abort;

利用Oracle官方提供的脚本计算合适的大页数量（Doc ID 401749.1）：

sh hugepages_settings.sh
...
Recommended setting: vm.nr_hugepages = 161325

root用户修改大页参数：

sed -i '$ a\vm.nr_hugepages = 161325' /etc/sysctl.conf

使大页参数生效：

#方法一：不重启服务器，停库后执行sysctl -p，会执行很长时间来分配大页（不推荐）
sysctl -p
cat /proc/meminfo | grep -i hugepages

#方法二：重启服务器使得大页参数生效
reboot
cat /proc/meminfo | grep -i hugepages

最后拉起数据库。

计算大页脚本

附hugepages_settings.sh脚本：

#!/bin/bash

# Welcome text
echo "
This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments on Oracle Linux. Before proceeding with the execution please note following:
 * For ASM instance, it needs to configure ASMM instead of AMM.
 * The 'pga_aggregate_target' is outside the SGA and
   you should accommodate this while calculating the overall size.
 * In case you changes the DB SGA size,
   as the new SGA will not fit in the previous HugePages configuration,
   it had better disable the whole HugePages,
   start the DB with new SGA size and run the script again.
And make sure that:
 * Oracle Database instance(s) are up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not setup
   (See Doc ID 749851.1)
 * The shared memory segments can be listed by command:
     # ipcs -m


Press Enter to proceed..."

read

# Check for the kernel version
KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`

# Find out the HugePage size
HPG_SZ=`grep Hugepagesize /proc/meminfo | awk '{print $2}'`
if [ -z "$HPG_SZ" ];then
    echo "The hugepages may not be supported in the system where the script is being executed."
    exit 1
fi

# Initialize the counter
NUM_PG=0

# Cumulative number of pages required to handle the running shared memory segments
for SEG_BYTES in `ipcs -m | cut -c44-300 | awk '{print $1}' | grep "[0-9][0-9]*"`
do
    MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
    if [ $MIN_PG -gt 0 ]; then
        NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
    fi
done

RES_BYTES=`echo "$NUM_PG * $HPG_SZ * 1024" | bc -q`

# An SGA less than 100MB does not make sense
# Bail out if that is the case
if [ $RES_BYTES -lt 100000000 ]; then
    echo "***********"
    echo "** ERROR **"
    echo "***********"
    echo "Sorry! There are not enough total of shared memory segments allocated for
HugePages configuration. HugePages can only be used for shared memory segments
that you can list by command:

    # ipcs -m

of a size that can match an Oracle Database SGA. Please make sure that:
 * Oracle Database instance is up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not configured"
    exit 1
fi

# Finish with results
case $KERN in
    '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;
           echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;
    '2.6') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '3.8') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '3.10') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '4.1') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    '4.14') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;
    *) echo "Kernel version $KERN is not supported by this script (yet). Exiting." ;;
esac

# End

References
【1】Oracle Linux: Shell Script to Calculate Values Recommended Linux HugePages / HugeTLB Configuration (Doc ID 401749.1)
【2】https://docs.oracle.com/en/database/oracle/oracle-database/19/ladbi/restrictions-for-hugepages-and-transparent-hugepages-configurations.html#GUID-D8178896-D00F-4F02-82A7-A44F89D8F103
【3】https://docs.oracle.com/en/database/oracle/oracle-database/19/unxar/administering-oracle-database-on-linux.html#GUID-19BFCC60-BA4C-4750-85EB-7432282A9556