postgresql内存配置问题

postgresql内存配置问题

业务环境

操作系统：CentOS Linux release 7.3.1611 (Core)
数据库版本：postgresql 10.6

本环境为一主二从的流复制集群，使用corosync+pacemaker进行高可用管控。一台从库使用同步复制，分担读压力；另一台从库使用异步复制，作为一个实时备份。

场景还原

早上收到监控报警：

Stack: corosync
Current DC: sh01-oscar-cmp-pp-pg03 (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum
Last updated: Wed Oct 16 10:17:02 2019
Last change: Wed Oct 16 10:15:17 2019 by root via crm_attribute on sh01-oscar-cmp-pp-pg03

3 nodes configured
11 resources configured

Online: [ sh01-oscar-cmp-pp-pg01 sh01-oscar-cmp-pp-pg02 sh01-oscar-cmp-pp-pg03 ]

Full list of resources:

fence-sh01-oscar-cmp-pp-pg01	(ocf::heartbeat:fence_check):	Started sh01-oscar-cmp-pp-pg01
fence-sh01-oscar-cmp-pp-pg02	(ocf::heartbeat:fence_check):	Started sh01-oscar-cmp-pp-pg02
fence-sh01-oscar-cmp-pp-pg03	(ocf::heartbeat:fence_check):	Started sh01-oscar-cmp-pp-pg03
Resource Group: master-group
vip-master	(ocf::heartbeat:IPaddr2):	Started sh01-oscar-cmp-pp-pg03
Resource Group: slave-group
vip-slave	(ocf::heartbeat:IPaddr2):	Started sh01-oscar-cmp-pp-pg02
Master/Slave Set: msPostgresql [pgsql]
Masters: [ sh01-oscar-cmp-pp-pg03 ]
Slaves: [ sh01-oscar-cmp-pp-pg02 ]
Stopped: [ sh01-oscar-cmp-pp-pg01 ]
Clone Set: clnPingCheck [pingCheck]
Started: [ sh01-oscar-cmp-pp-pg01 sh01-oscar-cmp-pp-pg02 sh01-oscar-cmp-pp-pg03 ]

Failed Actions:
* pgsql_start_0 on sh01-oscar-cmp-pp-pg01 'unknown error' (1): call=84, status=complete, exitreason='My data may be inconsistent. You have to remove /var/lib/pgsql/tmp/PGSQL.lock file to force start.',
last-rc-change='Wed Oct 16 10:15:06 2019', queued=0ms, exec=167ms

显示01机器出现异常，同事Prometheus收到告警，01节点的服务端口已无法访问。

问题分析

连入操作系统，发现01节点的postgresql已关闭，检查数据库日志发现问题：

2019-10-16 10:15:02.651 CST [55400] LOG:  server process (PID 16342) was terminated by signal 9: Killed

2019-10-16 10:15:02.651 CST [55400] LOG:  terminating any other active server processes
2019-10-16 10:15:02.651 CST [20414] WARNING:  terminating connection because of crash of another server process
2019-10-16 10:15:02.651 CST [20414] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2019-10-16 10:15:02.681 CST [20523] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2019-10-16 10:15:02.694 CST [55400] LOG:  all server processes terminated; reinitializing
2019-10-16 10:15:02.785 CST [20551] LOG:  database system was interrupted; last known up at 2019-10-16 10:12:42 CST
2019-10-16 10:15:02.787 CST [20552] FATAL:  the database system is in recovery mode
2019-10-16 10:15:02.971 CST [20572] FATAL:  the database system is in recovery mode
2019-10-16 10:15:03.050 CST [20551] LOG:  database system was not properly shut down; automatic recovery in progress
2019-10-16 10:15:03.058 CST [20551] LOG:  redo starts at 0/65CBB4A0
2019-10-16 10:15:03.164 CST [20686] FATAL:  the database system is in recovery mode
2019-10-16 10:15:03.190 CST [20688] FATAL:  the database system is in recovery mode
2019-10-16 10:15:03.195 CST [20689] FATAL:  the database system is in recovery mode
2019-10-16 10:15:03.214 CST [20690] FATAL:  the database system is in recovery mode
2019-10-16 10:15:03.260 CST [20551] LOG:  invalid record length at 0/67D224D8: wanted 24, got 0
2019-10-16 10:15:03.260 CST [20551] LOG:  redo done at 0/67D224B0
2019-10-16 10:15:03.260 CST [20551] LOG:  last completed transaction was at log time 2019-10-16 10:15:02.434297+08
2019-10-16 10:15:03.391 CST [55400] LOG:  database system is ready to accept connections
2019-10-16 10:15:03.419 CST [55400] LOG:  received fast shutdown request
2019-10-16 10:15:03.421 CST [55400] LOG:  aborting any active transactions
2019-10-16 10:15:03.423 CST [55400] LOG:  worker process: logical replication launcher (PID 20811) exited with exit code 1
2019-10-16 10:15:03.425 CST [20804] LOG:  shutting down
2019-10-16 10:15:03.462 CST [20859] FATAL:  the database system is shutting down
2019-10-16 10:15:03.465 CST [20860] FATAL:  the database system is shutting down
2019-10-16 10:15:03.491 CST [55400] LOG:  database system is shut down

可观察到，在10:15:02的时候，postgresql的进程被直接kill -9杀死了，，，

查看系统日志（/var/log/messages）：

Oct 16 10:15:02 sh01-oscar-cmp-pp-pg01 kernel: Out of memory: Kill process 16342 (postgres) score 843 or sacrifice child
Oct 16 10:15:02 sh01-oscar-cmp-pp-pg01 kernel: Killed process 16342 (postgres) total-vm:8494044kB, anon-rss:3399704kB, file-rss:400kB, shmem-rss:21080kB
Oct 16 10:15:02 sh01-oscar-cmp-pp-pg01 kernel: postgres: page allocation failure: order:0, mode:0x2015a

原来是内存耗尽，触发了oom-kill，操作系统杀掉了耗费内存最高的postgresql进程。。。

问题解决

检查机器总内存：

[root@sh01-oscar-cmp-pp-pg01 ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.7G        194M        2.6G        149M        971M        3.1G
Swap:          2.0G        290M        1.7G

机器内存只有不到4G

而postgresql.conf文件中配置的share_buffer为3G，再加上各个session占用的内存，导致系统内存不足，后修改share_buffer为1G，问题解决。

具体内存计算方法可参考这个博客：
PostgreSQL消耗的内存计算方法

以下为转载：

• wal_buffers默认值为-1,此时wal_buffers使用的是shared_buffers,wal_buffers大小为shared_buffers的1/32
• autovacuum_work_mem默认值为-1,此时使用maintenance_work_mem的值

1 不使用wal_buffers、autovacuum_work_mem

计算公式为:

max_connections*work_mem 
+ max_connections*temp_buffers 
+ shared_buffers
+ (autovacuum_max_workers * maintenance_work_mem）

假设PostgreSQL的配置如下:

max_connections = 100
temp_buffers=32MB
work_mem=32MB
shared_buffers=19GB
autovacuum_max_workers = 3
maintenance_work_mem=1GB #默认值64MB

则计算出内存为：

select(
	(100*(32*1024*1024)::bigint)
	+ (100*(32*1024*1024)::bigint)
	+ (19*(1024*1024*1024)::bigint)
	+ (3 * (1024*1024*1024)::bigint )
)::float8 / 1024 / 1024 / 1024
--output
28.25

此时pg满载峰值时最多使用28.25GB内存,物理内容为32GB时,还有3.75GB内存给操作系统使用.

2 使用wal_buffers,不使用autovacuum_work_mem

计算公式为:

max_connections*work_mem 
+ max_connections*temp_buffers 
+ shared_buffers+wal_buffers
+ (autovacuum_max_workers * autovacuum_work_mem）

假设PostgreSQL的配置如下:

max_connections = 100
temp_buffers=32MB
work_mem=32MB
shared_buffers=19GB	
wal_buffers=16MB #--with-wal-segsize的默认值
autovacuum_max_workers = 3	
maintenance_work_mem=1GB

则计算出内存为：

select(
	(100*(32*1024*1024)::bigint)
	+ (100*(32*1024*1024)::bigint)
	+ (19*(1024*1024*1024)::bigint)
	+ (16*1024*1024)::bigint
	+ (3 * (1024*1024*1024)::bigint )
)::float8  / 1024 / 1024 / 1024
--output
28.26

此时pg满载峰值时最多使用28.5GB内存,物理内容为32GB,还有3.5GB内存给操作系统使用.

3 同时使用wal_buffers、autovacuum_work_mem[建议使用]

计算公式为:

max_connections*work_mem 
+ max_connections*temp_buffers 
+ shared_buffers+wal_buffers
+ (autovacuum_max_workers * autovacuum_work_mem）
+  maintenance_work_mem

假设PostgreSQL的配置如下:

max_connections = 100
temp_buffers=32MB
work_mem=32MB
shared_buffers=19GB	
wal_buffers=262143kb
autovacuum_max_workers = 3
autovacuum_work_mem=256MB
maintenance_work_mem=2GB

则计算出内存为：

select(
    (100*(32*1024*1024)::bigint)
    + (100*(32*1024*1024)::bigint)
    + (19*(1024*1024*1024)::bigint)
    + (262143*1024)::bigint
    + (3 * (256*1024*1024)::bigint )
    + ( 2 * (1024*1024*1024)::bigint )
)::float8  / 1024 / 1024 / 1024
--output
28.01

此时pg载峰值时最多使用28.25GB内存,物理内容为32GB时,还有3.75GB内存给操作系统使用.建议所有内存消耗根据硬件配置,也就是使用这个配置.