Implementing Address Windowing Extensions (AWE) or VLM on 32-bit Windows Platforms [ID 225349.1]

 

Applies to:

Oracle Server - Enterprise Edition
Microsoft Windows (32-bit)

Purpose

To address the growing need for use of more memory on 32-Bit Windows platforms, and explain how AWE is implemented by Oracle on Windows.

Scope and Application

• Oracle DBA's running on the Microsoft Windows platform.
• Oracle Support Analysts, Field Engineers troubleshooting problems related to AWE and/or memory issues on Windows.

Implementing Address Windowing Extensions (AWE) or VLM on 32-bit Windows Platforms

AWE Memory implementation on Windows 2000

A common question on the Windows NT/Windows 2000 platform revolves around how to take advantage of systems with more than 4 GB of RAM. As discussed in My Oracle Support Document 46001.1 and Document 46053.1, the 32-Bit process address space for any process on Windows equates to a total of 4GB of addressable virtual memory. Of this, by default, 2GB is reserved for the process itself, and 2GB for the kernel. On systems running either Windows 2000 Advanced Server, or Windows NT 4.0 Enterprise Edition, this ratio can be changed by adding the /3GB switch to the BOOT.INI system configuration file, allowing a process to address 3GB and reserving 1GB for the kernel. However, the total addressable memory for a single process is still only 4GB. See also Document 1036312.6: Utilizing Up to 3GB Virtual Memory on Windows NT Server 4.0

For Windows 2008 the /3GB equivalent is:
    BCDEDIT /Set IncreaseUserVa 3072


What can be done to address memory beyond 4GB?

The answer is to take advantage of Physical Address Extensions (PAE), or Address Windowing Extensions (AWE). These two terms are used interchangeably, so the rest of this document will refer to this simply as AWE. AWE support is available if you are running on a machine with more than 4GB of physical RAM which is running any of the below Windows operating systems:

• Windows 2000 Datacenter Server
• Windows 2000 Advanced Server
• Windows 2003 Data Center Edition (32-Bit)
• Windows 2003 Enterprise Edition (32-Bit)

On the above operating systems, AWE support is built into the OS. No special drivers are needed to take advantage of the additional memory.

AWE CANNOT be used on the following Operating Systems:

• Windows 2000 Server (Standard)
• Windows 2000 Professional
• Windows XP Home Edition
• Windows XP Professional
• Windows 2003 Standard Edition
• Windows 2003 Web Edition

Note: On 64-Bit Windows operating systems, there is no need for AWE implementation support, because the directly addressable memory for a single process on 64-Bit Windows is 8 Terabytes.

Note: AWE is enabled by default on systems which have DEP (Data Execution Prevention) enabled. So even if /PAE is not set for those systems it is still active.

Oracle versions that can use AWE

Oracle can take advantage of AWE in the following 32-Bit RDBMS releases:

• Oracle 8.1.6.x
• Oracle 8.1.7.x
• Oracle 9.2.x
• Oracle 10.x
• Oracle 11.x

Oracle does NOT implement AWE support in release 9.0.1.x.

AWE support is available on both the Enterprise Edition of Oracle and the Standard Edition of Oracle. However, on Standard Edition of 9.2.0.1, you may receive the following error if trying to start the database with USE_INDIRECT_DATA_BUFFERS=TRUE:

ORA-439 - FEATURE NOT ENABLED: VERY LARGE MEMORY

In Standard Edition 9.2.0.2 and 9.2.0.3, you will not receive the above errors, but VLM functionality is still not enabled. Refer to BUG#2945011 for more detail. This BUG is fixed in 9.2.0.3 Patch 2, and is also fixed in 9.2.0.4 and onwards releases.


For RAC environments AWE can be used as well. As it is enabled at the system level, AWE is enabled for instances running on individual nodes.

Enabling support at the OS level

AWE can be enabled at the OS by adding the /PAE switch to the BOOT.INI file as in:

multi(0)disk(0)rdisk(0)partition(1)\WINNT="Microsoft Windows 2000 Advanced Server" /PAE

It is possible to have BOTH the /PAE and /3GB switch in place on the same machine, like in:

multi(0)disk(0)rdisk(0)partition(1)\WINNT="Microsoft Windows 2000 Advanced Server" /3GB /PAE

However, be aware that when /3GB is in place, the server will only be able to recognize up to 16GB of RAM. If you are working with a server with more than 16GB of RAM, you will need to choose between the two.

It is important to note that once either or both of these switches are in place in the BOOT.INI, ALL processes running can take advantage of these switches. Thus, in a case where multiple Oracle instances are running on the same server, ALL instances can take advantage of the additional memory afforded by these switches, up to the amount of physical memory on the box.


Operating System Privileges Needed at the OS Level

In order to take advantage of the additional memory afforded through PAE, the operating system user account which is used to start the OracleService<SID> must be granted the 'Lock Pages in Memory' system privilege at the operating system level. By default, the OracleService<SID> starts as the LocalSystem account. The LocalSystem account has the privilege to Lock Pages in Memory granted to it by default.

However, if you change the OracleService<SID> to logon as a user OTHER than LocalSystem, you may see the following errors when attempting to start the database with USE_INDIRECT_DATA_BUFFERS set to TRUE :

SQL> startup pfile=c:\temp\initscott.ora
ORA-27102: out of memory
OSD-00010: Message 10 not found; product=RDBMS; facility=SOSD

O/S-Error: (OS 1300) Not all privileges referenced are assigned to the caller.


To rectify this, you must grant the 'Lock pages in memory' privilege to the user that the OracleService<SID> starts as. To do this, click on:

Start -> Programs -> Administrative Tools -> Local Security Policy
(on a Domain Controller, click on 'Domain Security Policy' instead of 'Local Security Policy')
Double-click on the 'Lock Pages in memory' policy.
Add the appropriate user and click 'Ok'.
Restart the OracleService<SID>

Understanding the Oracle implementation of AWE support

What the PAE switch allows you to do from the Oracle perspective is to increase the amount of memory that can be used for the Oracle Database Block Buffer Cache. It is important to note that this additional memory can ONLY be used by Oracle in the form of an increased value for DB_BLOCK_BUFFERS.

There is still confusion on the old style of VLM versus AWE on Windows 2000. With VLM on Windows NT 4.0, there was the concept of pointers pointing to the extended memory area, but that is no longer the case on Windows 2000. Instead, the windowing technology as described in these articles is being used. For more information on AWE/PAE implementation on the Windows platform, refer to Microsoft's website.


As mentioned previously, with AWE enabled, this allows the process(es) (in this case ORACLE.EXE) to use memory above and beyond the 4GB mark defined by a 32-Bit Process Address space. The physical location of these blocks does not matter. However, the database blocks must still be accessed from within a ‘window’, which exists (logically) in that regular 3GB process address space. The size of this window is defined by a registry setting in the HOME key for Oracle (HKLM\Software\Oracle\Homex) called AWE_WINDOW_MEMORY. By default, this value is 1GB, so if this value is not set in the registry, AWE_WINDOW_MEMORY will be 1GB. The pages in physical memory above 4GB are then mapped into this window (and paged back into physical memory above 4GB when there is demand for space in thw window for other blocks) for the ORACLE.EXE process to be able to access them in its conventional address space.

If you add the registry key yourself, the datatype should be a string value, or a REG_SZ. The value for AWE_WINDOW_MEMORY must be specified in BYTES.

It is important to realize that any database blocks accessed by Oracle (or any user/background thread within ORACLE.EXE) must first be mapped into the 'window' defined by AWE_WINDOW_MEMORY. In this scenario, it does not matter where the blocks are physically located - there is no need to be concerned with where the blocks are physically residing. The window will be drawn around the block (i.e. the block will be mapped) wherever it is located in memory. If the block is in memory but has not been mapped into the ‘window’, then it may be necessary to unmapped another block that IS in the window, in order to accommodate the new block. While this mapping and unmapping of blocks does add some cost, it is still faster than incurring an I/O operation to read the block from disk. This will be discussed further down in the section on troubleshooting.

Note: Keep in mind that if there are multiple instances on a machine with the /PAE switch enabled, ALL instances can take advantage of the additional memory. However, AWE_WINDOW_MEMORY cannot be set on a per-instance basis, so all databases that are running out of the Oracle Home where AWE_WINDOW_MEMORY is set will inherit the same value.

Enabling AWE Support at the Database/Instance Level

To enable the AWE implementation on Oracle, you must set the following parameter in the instance parameter file (or spfile) used to start the instance:

USE_INDIRECT_DATA_BUFFERS=TRUE

Note again that the buffer cache MUST be defined using the parameter DB_BLOCK_BUFFERS, no matter what version of the RDBMS you are running. The feature allowing for Multiple block sizes in a database will be disabled if you set USE_INDIRECT_DATA_BUFFERS=TRUE, and you CANNOT specify the DB_CACHE_SIZE parameter to define the size of the buffer cache.

If you attempt to startup a database with this combination of parameters:

USE_INDIRECT_DATA_BUFFERS=TRUE
DB_CACHE_SIZE=xxxxx (Any number)

the startup will fail with the following error:

SQL> startup
ORA-00385: cannot enable Very Large Memory with new buffer cache parameters

You must change DB_CACHE_SIZE to use DB_BLOCK_BUFFERS instead.


AWE_WINDOW_MEMORY Within the 3GB Process Address Space

If you are using /PAE and the /3GB switch together, the address space for ORACLE.EXE will be 3GB. The value for AWE_WINDOW_MEMORY must come from the normal address space used by the ORACLE.EXE process. Memory that comes from that 3GB address space addressable by the ORACLE.EXE process includes the following:

• The Value for AWE_WINDOW_MEMORY
• The rest of the SGA (shared pool, large pool, java pool, redo log buffers, etc)
• Overhead for ORACLE.EXE and DLL’s (65-100M depends on version & options)
• Stack space for all threads (defaults to 1MB per thread, unless orastack is used)
• PGA and UGA memory for all user sessions

Therefore, the value for AWE_WINDOW_MEMORY should be tuned such that mapping and unmapping operations are avoided as much as possible, while still allowing enough memory within the 3GB address space for the rest of the process memory that MUST fit within the 3GB (i.e. overhead, remaining SGA components and all user connection memory (stack + UGA + PGA) noted above).

The total size of the buffer cache can then be set to the amount of physical memory remaining above the 4GB barrier, plus AWE_WINDOW_MEMORY. On a machine with 12GB of RAM, using the default value of 1GB for AWE_WINDOW_MEMORY, your total buffer cache could theoretically be as high as 9GB:

(Total RAM - 4GB + AWE_WINDOW_MEMORY) = 12GB - 4GB + 1GB = 9GB

In reality, your maximum buffer cache size will be somewhat less than this, allowing for some overhead and additional processes running on the system.

Attempting to startup the database with a buffer cache larger than the maximum value as calculated above may result in the following errors:

ORA-27102 out of memory
OSD-00034 Message 34 not found; Product=RDBMS;facility =SOSD
O/S Error: (OS 8) Not enough storage is available to process this command

Note: If you are on Release 9.2, another possible cause for these errors is noted further down, in the troubleshooting section.

As mentioned above, the buffer cache must be specified using DB_BLOCK_BUFFERS rather than DB_CACHE_SIZE, so assuming an 8K block size (8192), to get a 9GB buffer cache, you would set the following init parameters:

DB_BLOCK_BUFFERS = 1179648
DB_BLOCK_SIZE = 8192

Troubleshooting AWE_WINDOW_MEMORY implementation


Minimum Value Required for AWE_WINDOW_MEMORY in 9.2 and Above

Here are key points to understand when using AWE_WINDOW_MEMORY:

1. Under Oracle 8.1.7 we do NOT enforce a minimum value for AWE_WINDOW_MEMORY to be able to start the database.
2. This was changed under Oracle9i Release 2, such that we DO enforce a minimum value for AWE_WINDOW_MEMORY. This change was done to help improve performance by enforcing a larger window size.
3. You can alter the minimum required value for AWE_WINDOW_MEMORY under 9.2 by changing/setting the value of the parameter _DB_BLOCK_LRU_LATCHES. Under 8.1.7, this parameter was named DB_BLOCK_LRU_LATCHES. However, under 9.x, this parameter was changed to be a hidden parameter.

The minimum value for AWE_WINDOW_MEMORY starting with 9.2 is calculated as such:

MIN(AWE_WINDOW_MEMORY)=(4096 * DB_BLOCK_SIZE * _DB_BLOCK_LRU_LATCHES)/8

Starting with 9.2, to calculate the value for _DB_BLOCK_LRU_LATCHES, we need this formula:

_DB_BLOCK_LRU_LATCHES = (Max buffer pools * SETS_PER_POOL)

• Max Buffer Pools is a constant = 8
• SETS_PER_POOL is variable, and depends on whether or not VLM is enabled:

SETS_PER_POOL = 2*CPU_COUNT (if VLM is enabled)
SETS_PER_POOL= CPU_COUNT/2 (if VLM is NOT enabled)

Recall that VLM is enabled by setting USE_INDIRECT_DATA_BUFFERS=TRUE.

So, as you can see, the value for _DB_BLOCK_LRU_LATCHES in 9.2 and higher releases of Oracle is dependent on the number of CPUs in the box, and therefore MIN(AWE_WINDOW_MEMORY) is dependent on the # of CPUs as well as the DB_BLOCK_SIZE. The larger the Block Size, and the more CPUs in a system, the higher the value for MIN(AWE_WINDOW_MEMORY). Here are a couple of example configurations and caclulations showing MIN(AWE_WINDOW_MEMORY).

Example #1:

# of CPU's = 8
DB_BLOCK_SIZE = 8192
Total RAM = 8GB

SETS_PER_POOL = 2 * CPU_COUNT = 16
_DB_BLOCK_LRU_LATCHES = (max buffer Pools * sets_per_pool) = 8*16 = 128
MIN(AWE_WINDOW_MEMORY) =(4096*DB_BLOCK_SIZE*_DB_BLOCK_LRU_LATCHES)/8 =
( 4096 * 8192 * 128) / 8 = 536870912 bytes = 512 MB


Example #2:

# of CPU's = 16
DB_BLOCK_SIZE = 8192
Total RAM = 16 GB

SETS_PER_POOL = 2 * CPU_COUNT = 32
_DB_BLOCK_LRU_LATCHES = (max buffer Pools * sets_per_pool) = 8*32 = 256
MIN(AWE_WINDOW_MEMORY) =(4096*DB_BLOCK_SIZE*_DB_BLOCK_LRU_LATCHES)/8 =
( 4096 * 8192 * 256) / 8 = 1073741824 bytes = 1024 MB

The values above are the minimum values required for AWE_WINDOW_MEMORY to be set to, UNLESS you explicitly set _DB_BLOCK_LRU_LATCHES to a lower value. If AWE_WINDOW_MEMORY is not set to the minimum value, you will receive the following errors:

ORA-27102 out of memory
OSD-00034 Message 34 not found; Product=RDBMS;facility =SOSD
O/S Error: (OS 8) Not enough storage is available to process this command

If you receive these errors when trying to start the database under 9.2 or higher, this may be because the AWE_WINDOW_MEMORY value in the registry is set too low for the calculated minimum value. If you cannot increase the value for AWE_WINDOW_MEMORY, then you can explicitly set _DB_BLOCK_LRU_LATCHES to a value lower than the calculated value, and retry the startup.

_DB_BLOCK_LRU_LATCHES must be at least 8 (equal to the maximum number of buffer pools).

Note: Recall from the earlier section that these errors may also occur if you are trying to start up with a buffer cache that is too large for the  physical memory available.

Note: The same errors above have also been observed with a buffer cache that is too small. When USE_INDIRECT_DATA_BUFFERS is set to TRUE the value for DB_BLOCK_BUFFERS should equate to a buffer cache that is AT LEAST equal to AWE_WINDOW_MEMORY. In most cases, the total buffer cache size will be greater than AWE_WINDOW_MEMORY. If you attempt to start up with a buffer cache that is too small (i.e. < AWE_WINDOW_MEMORY) that may also result in the ORA-27102 error.

Note: It has been observed on some systems that you may need to add a few additional meg to AWE_WINDOW_MEMORY to calculate for overhead. Therefore, if you go through the above calculations, and the instance still does not start, try adding an additional 10 Meg or so to the calculated value.

Note: Also, keep in mind that when calculating the # of CPUs in the system, you have to take hyperthreading into account. On a hyperthreaded system, the OS will think that you have double the # of CPUs in the system over what you actually  have, and this is the number that must be used in the calculations.

CPU Spins Possible When Using AWE Implementation

Use caution when setting _DB_BLOCK_LRU_LATCHES or AWE_WINDOW_MEMORY too low. If we are unable to map a requested buffer into the window because all of the space defined by AWE_WINDOW_MEMORY is in use with buffers already actively being accessed, then we spin and wait, checking every so often until an existing buffer in the window can be unmapped, and a new buffer can be mapped in.

This spin will consume CPU cycles until enough buffers can be Mapped/Unmapped to satisfy the request. In some cases, there may be so many buffers needing to be mapped into the window, that DBWR will consume 100% of cycles on all CPUs, effectively locking up the machine. This is normal behavior under some circumstances, and is simply an indication that AWE_WINDOW_MEMORY is too small.


Monitoring Mapping Operations in 9.2 and later releases

Starting with 9.2, we have added additional statistics which can be measured in V$SESSTAT (sesssion level statistics) and V$SYSSTAT (system wide statistics):

STATISTIC# NAME
---------- ------------------------------
154 number of map operations
155 number of map misses


The following query will give you system wide information on map operations and map misses:

SQL> select * from v$sysstat where statistic# in
(select statistic# from v$statname where name like 'number of map %');

If the # of map misses is relatively high, or particularly of the # of map misses increases consistently over time, this may be an indication that the value for AWE_WINDOW_MEMORY is set too low.

Note: The statistic# values change from Oracle version to Oracle version, so you cannot easily query on the statistics 154 and 155. Instead you should check V$STATNAME to see which statistics numbers are associated with these statistics.

Automatic Shared Memory Management and AWE

Oracle10g introduces the concept of Automatic Shared Memory Management, whereby the Oracle kernel will dynamically adjust SGA parameters such as SHARED_POOL_SIZE, JAVA_POOL_SIZE, DB_CACHE_SIZE, etc.

This is enabled by setting the parameter SGA_TARGET to a non-zero value. However, in order for this to work properly, you must use DB_CACHE_SIZE for the buffer cache. When setting USE_INDIRECT_DATA_BUFFERS, you cannot set DB_CACHE_SIZE, as noted above. Therefore, SGA_TARGET should not be set when using AWE - these two features are mutually exclusive. When setting USE_INDIRECT_DATA_BUFFERS=TRUE on Oracle10g and higher Oracle releases, you should also set SGA_TARGET to 0.


Diagnosing Spins Associated With AWE in 8.1.x

The above statistics are not available in 8.1.7, so if you are encountering problems with CPU spins, with AWE_WINDOW_MEMORY enabled, it is more difficult to diagnose.

Any diagnosis can start by identifying and monitoring the thread associated with DBWR via the following query:

SQL> select b.name, p.spid from v$process p, v$bgprocess b
where p.addr=b.paddr;

NAME SPID
----- ---------
PMON 1900
DBW0 1956
LGWR 572
CKPT 1908
SMON 1808
RECO 920
SNP0 1784
SNP1 1892
SNP2 1896
SNP3 1844

10 rows selected.

As you can see, DBWR has an SPID of 1956, which will equate to the thread ID of that thread within the Oracle executable. This thread can then be monitored using Performance Monitor and/or the PSLIST utility, which is available as a free download from http://www.sysinternals.com

If your monitoring shows that DBWR is consuming excessive CPU, you can attempt to get an error stack from that thread using oradebug:

SQL> oradebug setospid 1956
Oracle pid: 3, Windows thread id: 1956, image: ORACLE.EXE
SQL> oradebug unlimit
Statement processed.
SQL> oradebug dump errorstack 3
Statement processed.

This should dump the error stack to the DBWR trace file, generated in the background dump destination directory. If the error stack contains the function SKGMMAP, this is an indication that DBWR is working to map/unmap database block buffers.

Note: In 8.1.7 of the RDBMS, you cannot use DBWR_IO_SLAVES in combination with USE_INDIRECT_DATA_BUFFERS, due to Bug:3042660 and Bug:2215894. You must leave DBWR_IO_SLAVES at its default value - otherwise, buffers are not unmapped and eventually a spin of the process will result. This problem is resolved in 9.2.0.1 - the fix is NOT backported to 8.1.7.

@Known issues:
@


Related documents:

Oracle9i Database Getting Started Release 2 (9.2) for Windows
Part Number A95490-01
Chapter 4: Oracle9i Architecture on Windows
http://www.oracle.com/pls/db92/db92.docindex?remark=homepage

Oracle® Database Platform Guide 10g Release 1 (10.1) for Windows
Part Number B10113-02
Chapter 1: Oracle Database Architecture on Windows
http://www.oracle.com/pls/db10g/portal.portal_demo3?selected=9

Oracle® Database Platform Guide 10g Release 2 (10.2) for Windows
Part Number B14304-05
Chapter 1: Oracle Database Architecture on Windows
http://www.oracle.com/pls/db102/portal.portal_db?selected=9

Oracle® Database Platform Guide 11g Release 1 (11.1) for Windows
Part Number B32010-05
Chapter 1: Oracle Database Architecture on Windows
http://www.oracle.com/pls/db111/portal.portal_db?selected=4&frame=#unix_and_windows_database_administration

Oracle® Database Platform Guide 11g Release 2 (11.2) for Windows
Part Number E10845-03
Chapter 1: Oracle Database Architecture on Windows
http://www.oracle.com/pls/db112/portal.portal_db?selected=4&frame=#unix_and_windows_database_administration