GP Out of memory
ERROR: Out of memory (seg2 slice1 sdw1:40014 pid=16427)
DETAIL: VM Protect failed to allocate 32768 bytes, 0 MB available
主要涉及两个参数：
statement_mem：
ERROR: insufficient memory reserved for statement (memquota.c:228)
当扫描一张分区特别多的表时，会出现该错误，此时需要将默认的
125MB的配置提高，建议在500MB左右或者更高一些
gp_vmem_protect_limit
：
"ERROR","53200","Out of memory.Failed on request of size 156 bytes.(context
'CacheMemoryContext') (aset.c:840)"
"ERROR","53400","Out of memory (seg13 slice13 sdw1-1:40001
pid=10183)","VM Protect failed to allocate 8388608 bytes, 6 MB available"
该错误是Greenplum系统无法从OS申请到所需要的内存导致的错误，因为
gp_vmem_protect_limit配置高于OS所能提供的能力，当Greenplum
申请超过OS能力时，会得到该异常信息，解决办法是降低
gp_vmem_protect_limit配置，建议
gp_vmem_protect_limit
×instance数量等于OS内存总量或者符合Adminguide中的标准，
计算可能涉及OS Cache，对性能有影响，需谨慎
详细可参考Adminguide的相关介绍
http://www.greenplumdba.com/greenplum-database-env---best-practice/avoiding%E2%80%9Coutofmemory%E2%80%9Dissues
DBA's occasionally experience “out of memory” errors that can cause failed queries and degrade system performance. Fortunately, the Greenplum Database provides facilities to avoid this.
We will discuss two of those facilities: the
gp_vmem_protect_limit parameter, and
Greenplum resource queues. The parameters and techniques mentioned here are explained in detail in the Greenplum Database Administrator Guide.
The gp_vmem_protect_limit parameter: The “gp_vmem_protect_limit” parameter sets the amount of memory that all processes of an active segment instance can consume. Queries that cause the limit to be exceeded will be cancelled.
Note that this is a local parameter and must be set for each segment in the system. The system must be restarted for parameter changes to take effect.
How to set the gp_vmem_protect_limit
As a general rule-of-thumb, gp_vmem_protect_limit should be set to:
( X * physical_memory_in_MB ) /#_of_primary_segments
X should be a value between 1.0 and 1.5. A value of X=1.0 offers the best overall system performance; a value of X=1.5 may impact system performance because of swap activity but will result in fewer canceled queries.
For example, to set gp_vmem_protect_limit conservatively (X=1.0) on a segment host with 16GB (16384 MB) of physical memory with 4 primary segment instances, the calculation would be: (1 * 16384) / 4 = 4096.
The
MEMORY_LIMIT parameter for Greenplum Resource Queues:Greenplum resource queues provide a way to manage and prioritize workloads. Resource queues can be created with a MEMORY_LIMIT setting to restrict the total amount of memory that queries can consume in any segment instance. Queries that cause a queue to exceed the MEMORY_LIMIT must wait until queue resources are free before they can execute.
By assigning each user to a queue and limiting the amount of memory queues can consume, administrators can ensure proper resource allocation across the system.
Note that roles with the SUPERUSER attribute are exempt from queue limits.
How to set MEMORY_LIMIT to avoid Out of Memory errors:As a general rule-of-thumb, the sum of all the MEMORY_LIMITs across all the queues should be no more than 90% of the gp_vmem_protect_limit.
Common Out of Memory Errors
The two most common errors are described below. They look similar but have different reasons and solutions.
Error code 53200
Example:
"ERROR","53200","Out of memory. Failed on request of size 156 bytes. (context 'CacheMemoryContext') (aset.c:840)"
Description:
The system canceled a query because a segment server’s OS did not have enough memory to satisfy a postmaster process allocation request.
How to Avoid
1. Set gp_vmem_protect_limit according to the formula above.
2. Adding memory can help greatly if lowering gp_vmem_protect_limit results in too many canceled queries.(Gp_vmem_protect_limit can be raised after adding memory.)
3. Adding swap space may help, although increased swap activity will impact system performance.
Error code 53400
Example
"ERROR","53400","Out of memory (seg13 slice13 sdw1-1:40001 pid=10183)","VM Protect failed to allocate 8388608 bytes, 6 MB available"
Description
The system canceled a query because a postmaster process tried to request more memory than the gp_vmem_protect_limit parameter allows.
How to Avoid
1. Make sure the sum of all MEMORY_LIMITs across all active queues is <= 90% of gp_vmem_protect_limit.
2. Increase gp_vmem_protect_limit, if possible, using the formula described above.
3. Ensure the system is not unbalanced (i.e., some segments down). Use gpstate -e to verify.