oracle11g 在 16核 64g 的pc server上利用hugePages提升性能

有一个oracle11g服务器，pc server，16核64g内存，性能有些差，准备使用hugepages来提示下oracle11g性能。

1，关于Hugepage

HugePage广泛启用开始于Kernal 2.6,一些版本下2.4内核也可以是用。在Oracle运行环境中开启HugePage是有很多好处的。关于使用hugePages的好处，请参考以前的介绍：http://blog.csdn.net/mchdba/article/details/51030668

 

2，oracle设置hugepage

对于Oracle而言，实例运行环境（Database和ASM）都面对一个HugePage优化的问题。

 

如果是使用11g，Oracle版本，一定需要进行额外的配置，就是将使用的AMM退化为ASMM。在早期的11.2.0.1版本中，这个步骤很重要。因为AMM是不支持HugePage的，如果强在AMM+HugePage模式下打开数据库，是会遇到失败信息。

 

在最新的11.2.0.2版本以及之后，引入了参数use_large_pages，避免了这样的问题。但是AMM与HugePage不兼容的情况，还是存在。所以，需要将ASM切换成AMM，AMM、ASMM切换参考：http://blog.itpub.net/17203031/viewspace-774928/。

 

假如当前是使用AMM的数据库，HugePages_Total,HugePages_Free均为零，表示没有生效，如下所示：

可以看到，HugePages_Total,HugePages_Free均为零，证明没有使用，但可以配置。如果上面什么都没有显示，证明不能配置。

[root@crmtest ~]# cat /proc/meminfo | grep -i huge                                                                                                                                             AnonHugePages:     28672 kB HugePages_Total:       0 HugePages_Free:        0 HugePages_Rsvd:        0 HugePages_Surp:        0 Hugepagesize:       2048 kB [root@crmtest ~]#

 

 

如果碰到这种free比total小，证明已经生效，如下所示：

[root@crmtest oracle]# cat /proc/meminfo | grep -i huge                                                                                                                                    AnonHugePages:     20480 kB HugePages_Total:    2588 HugePages_Free:     2449 HugePages_Rsvd:     2421 HugePages_Surp:        0 Hugepagesize:       2048 kB [root@crmtest oracle]# [root@crmtest oracle]# grep Huge /proc/meminfo AnonHugePages:     20480 kB HugePages_Total:    2588 HugePages_Free:     2449 HugePages_Rsvd:     2421 HugePages_Surp:        0 Hugepagesize:       2048 kB [root@crmtest oracle]#

 

blog来源地址：http://blog.csdn.net/mchdba/article/details/51116234 ，原作者mchdba（黄杉），谢绝转载。

大内存技术，不支持内存自动管理，必须关闭AMM(自动内存管理）特性才能使用hugepage，调整方法如下：

对于OLTP系统：                                                                                                                                                                               SGA_TARGET=(物理内存 x 80%) x 80% SGA_MAX_SIZE=(物理内存 x 80%) x 80% PGA_AGGREGATE_TARGET=(物理内存 x 80%) x 20%   ALTER SYSTEM SET sga_max_size = 50079215104 SCOPE=SPFILE;                                                                                 ALTER SYSTEM SET sga_target =45591424409 SCOPE=SPFILE; ALTER SYSTEM SET PGA_AGGREGATE_TARGET = 11591424409 SCOPE=SPFILE; ALTER SYSTEM SET memory_target = 0 SCOPE=SPFILE; ALTER SYSTEM SET memory_max_target = 0 SCOPE=SPFILE;

 

 

执行后，保存到spfile中，重启oracle实例：

SQL> shutdown immediate                                                                                                                                                                  Database closed. Database dismounted. ORACLE instance shut down. SQL> SQL> SQL> startup ORA-00843: Parameter not taking MEMORY_MAX_TARGET into account ORA-00849: SGA_TARGET 103079215104 cannot be set to more than MEMORY_MAX_TARGET 0. SQL>

 

这个问题的原因是Oracle启动过程中对于参数的内部检查。因为MEMORY_MAX_TARGET被“显示”的赋值，与SGA_TARGET赋值相冲突。

 

解决的方法就是使用参数默认值。创建出pfile之后，将显示赋值为0的MEMORY_TARGET和MEMORY_MAX_TARGET记录行删除掉。再利用pfile启动数据库，重建spfile，如下所示：

SQL> create pfile='/tmp/pfile8.ora' from spfile;                                                                                                                                                                                                                                File created.   SQL> exit Disconnected [oracle@hch_test_pd_121_129 ~]$ vim /tmp/pfile8.ora  # 这里删除掉*.memory_max_target=0和*.memory_target=0这两行 [oracle@hch_test_pd_121_129 ~]$     # 这里开始重新创建spfile，然后用新的spfile启动db，就可以成功了。 [oracle@hch_test_pd_121_129 ~]$ rlwrap sqlplus / as sysdba   SQL*Plus: Release 11.2.0.1.0 Production on Sun Apr 10 18:45:00 2016   Copyright (c) 1982, 2009, Oracle.  All rights reserved.   Connected to an idle instance.   SQL> create spfile from pfile='/tmp/pfile8.ora'   2  ;   File created.   SQL> startup ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance ORACLE instance started.   Total System Global Area 4.9973E+10 bytes Fixed Size              2217424 bytes Variable Size              1.1811E+10 bytes Database Buffers    3.8118E+10 bytes Redo Buffers                42110976 bytes Database mounted. Database opened. SQL>

 

 

3，设置用户内存配置

3.1，修改/etc/security/limits.conf参数文件，添加数据库实例用户的memlock限制

如果不配置锁定内存，在后alert_sid.log中会有类似如下的建议：

RECOMMENDATION:

  Total System Global Area sizeis 5 GB. For optimal performance,

  prior to the next instancerestart:

  1. Large pages areautomatically locked into physical memory.

 Increase the per process memlock(soft) limit to at least 5 GB to lock

 100% System Global Area's largepages into physical memory

并且，hugepage并没有真正使用起来！主要是配置如下2个设置：

oracle soft memlock

oracle hard memlock

注意，这里设置的值均以kb为单位的！

设置用户内存配置，有个计算规则是：实际物理内存 > 锁定内存 >=HugePages_Total*Hugepagesize；

[root@crmtest oracle]# free -t                                                                                                                                                                                                                            total       used       free     shared    buffers     cached Mem:     115742368   76624776   39117592          0     133644   68080568 -/+ buffers/cache:    8410564  107331804 Swap:            0          0          0 Total:   115742368   76624776   39117592 [root@crmtest oracle]#

看到free -t，把total的值取出来，修改/etc/security/limits.conf参数文件，添加数据库实例用户的memlock限制，限制的值就是total值115742368。启用HugePage的第一步就是进行用户参数限制打通，当前内存大小如下：

[root@crmtest oracle]# vim/etc/security/limits.conf

oracle soft memlock 115742368

oracle hard memlock 115742368

注意，这里设置的值均以kb为单位的！

 

遵循原则是：实际物理内存 > 锁定内存 >=HugePages_Total*Hugepagesize，Hugepagesize值从哪里看呢？可以从meminfo里面找到，如下所示： 

[root@pldb1 ~]# cat /proc/meminfo | grep -i huge                                                                                                                                                                                           AnonHugePages:     30720 kB HugePages_Total:   49346 HugePages_Free:    29238 HugePages_Rsvd:    29045 HugePages_Surp:        0 Hugepagesize:       2048 kB [root@pldb1 ~]#

 

 

3.2，oracle中大内存技术的配置建议，oracle针对大内存技术，提供了一个脚本如下：

[root@crmtest oracle]# sh /oracle/hugepages_set.sh                                                                                                                                                                                   This script is provided by Doc ID 401749.1 from My Oracle Support (http://support.oracle.com) where it is intended to compute values for the recommended HugePages/HugeTLB configuration for the current shared  memory segments. Before proceeding with the execution please make sure that:  * Oracle Database instance(s) are up and running  * Oracle Database 11g Automatic Memory Management (AMM) is not setup    (See Doc ID 749851.1)  * The shared memory segments can be listed by command:      # ipcs -m   Press Enter to proceed...   Recommended setting: vm.nr_hugepages = 49346 [root@crmtest oracle]# [root@crmtest oracle]# [root@crmtest oracle]# more /oracle/hugepages_set.sh #!/bin/bash # # hugepages_settings.sh # # Linux bash script to compute values for the # recommended HugePages/HugeTLB configuration # # Note: This script does calculation for all shared memory # segments available when the script is run, no matter it # is an Oracle RDBMS shared memory segment or not. # # This script is provided by Doc ID 401749.1 from My Oracle Support # http://support.oracle.com   # Welcome text echo " This script is provided by Doc ID 401749.1 from My Oracle Support (http://support.oracle.com) where it is intended to compute values for the recommended HugePages/HugeTLB configuration for the current shared memory segments. Before proceeding with the execution please make sure that:  * Oracle Database instance(s) are up and running  * Oracle Database 11g Automatic Memory Management (AMM) is not setup    (See Doc ID 749851.1)  * The shared memory segments can be listed by command:      # ipcs -m   Press Enter to proceed..."   read   # Check for the kernel version KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'`   # Find out the HugePage size HPG_SZ=`grep Hugepagesize /proc/meminfo | awk '{print $2}'`   # Initialize the counter NUM_PG=0   # Cumulative number of pages required to handle the running shared memory segments for SEG_BYTES in `ipcs -m | awk '{print $5}' | grep "[0-9][0-9]*"` do    MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`    if [ $MIN_PG -gt 0 ]; then       NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`    fi done   RES_BYTES=`echo "$NUM_PG * $HPG_SZ * 1024" | bc -q`   # An SGA less than 100MB does not make sense # Bail out if that is the case if [ $RES_BYTES -lt 100000000 ]; then    echo "***********"    echo "** ERROR **"    echo "***********"    echo "Sorry! There are not enough total of shared memory segments allocated for HugePages configuration. HugePages can only be used for shared memory segments that you can list by command:      # ipcs -m   of a size that can match an Oracle Database SGA. Please make sure that:  * Oracle Database instance is up and running  * Oracle Database 11g Automatic Memory Management (AMM) is not configured"    exit 1 fi   # Finish with results case $KERN in    '2.4') HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;           echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;    '2.6') echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;     *) echo "Unrecognized kernel version $KERN. Exiting." ;; esac   # End [root@crmtest oracle]#

 

 

3.3，然后开始运行脚本

[oracle@hch_test_pd_121_129 oracle]$ sh /oracle/hugepages_set.sh                                                                                                                                                    This script is provided by Doc ID 401749.1 from My Oracle Support (http://support.oracle.com) where it is intended to compute values for the recommended HugePages/HugeTLB configuration for the current shared memory segments. Before proceeding with the execution please make sure that:  * Oracle Database instance(s) are up and running  * Oracle Database 11g Automatic Memory Management (AMM) is not setup    (See Doc ID 749851.1)  * The shared memory segments can be listed by command:      # ipcs -m   Press Enter to proceed...   Recommended setting: vm.nr_hugepages = 23938 [oracle@hch_test_pd_121_129 oracle]$

可以看到，运行脚本显示，将vm.nr_hugepages设置为23938

 

3.3，设置/etc/sysctl.conf

修改，添加vm.nr_hugepages = 23938，然后sysctl – p生效：

[root@crmtest oracle]# vim  /etc/sysctl.conf                                                                                                                                                                                                     [root@crmtest oracle]# [root@crmtest oracle]#  sysctl -p net.ipv4.ip_forward = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 error: "net.bridge.bridge-nf-call-ip6tables" is an unknown key error: "net.bridge.bridge-nf-call-iptables" is an unknown key error: "net.bridge.bridge-nf-call-arptables" is an unknown key kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 68719476736 kernel.shmall = 4294967296 fs.aio-max-nr = 1048576 fs.file-max = 6815744 kernel.shmall = 2097152 kernel.shmmax = 536870912 kernel.shmmni = 4096 kernel.sem = 250 32000 100 128 net.ipv4.ip_local_port_range = 9000 65500 net.core.rmem_default = 262144 net.core.rmem_max = 4194304 net.core.wmem_default = 262144 net.core.wmem_max = 1048586 vm.nr_hugepages = 23938 [root@crmtest oracle]#   使用sysctl - p生效设置。

 

 

4，查看hugepage是否生效

设置好后，最好能shutdown–r now重启下服务器，在meminfo文件中，可以查到HugePages的信息：

[root@hch_test_pd_121_129 ~]# grep Huge /proc/meminfo                                                                                                                                                                     AnonHugePages:         0 kB HugePages_Total:   23938 HugePages_Free:    23938 HugePages_Rsvd:        0 HugePages_Surp:        0 Hugepagesize:       2048 kB [root@hch_test_pd_121_129 ~]#

 

 

看到HugePages_Rsvd是0，而且HugePages_Free和HugePages_Total值一样大，这是正常的，因为还没有应用开始应用big page，接下来启动oracle数据库，就会变得不一样了，如下所示：

 

[oracle@hch_test_pd_121_129 ~]$ rlwrap sqlplus / as sysdba                                                                                                                                                                SQL*Plus: Release 11.2.0.1.0 Production on Sun Apr 10 20:35:39 2016   Copyright (c) 1982, 2009, Oracle.  All rights reserved.   Connected to an idle instance.   SQL> startup ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance ORACLE instance started.   Total System Global Area 4.9973E+10 bytes Fixed Size              2217424 bytes Variable Size              1.1811E+10 bytes Database Buffers    3.8118E+10 bytes Redo Buffers                42110976 bytes Database mounted. Database opened. SQL> exit Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options [oracle@hch_test_pd_121_129 ~]$  cat /proc/meminfo | grep -i huge AnonHugePages:     14336 kB HugePages_Total:   23938 HugePages_Free:    22660 HugePages_Rsvd:    22659 HugePages_Surp:        0 Hugepagesize:       2048 kB [oracle@hch_test_pd_121_129 ~]$ exit logout [root@hch_test_pd_121_129 ~]#  cat /proc/meminfo | grep -i huge AnonHugePages:     14336 kB HugePages_Total:   23938 HugePages_Free:    22660 HugePages_Rsvd:    22659 HugePages_Surp:        0 Hugepagesize:       2048 kB [root@hch_test_pd_121_129 ~]#

 

 

现在互联网发展太快，pc server也越来越多，而且内存普遍在64G以上，所以开启hugepage提升oracle性能非常有必要。

 

Oracle11g在64g内存的pc  sever中使用hugePages提升性能：http://blog.csdn.net/mchdba/article/details/51030668

AMM、ASMM切换参考：http://blog.itpub.net/17203031/viewspace-774928/

hugepage参考：http://blog.itpub.net/17203031/viewspace-774843/

linux下hugepage性能：http://blog.itpub.net/29371470/viewspace-1063046/