内存泄漏后top&free的小思考

agenda

• 表象
• 解析

表象

之前有一个在Linux下python高耗内存的应用,表象是跑着跑着就显示killed, 没有留下core dump. 依着top/free命令看, free memory一直跌, swap用了将近一半.后来优化程序发现free在程序启动后还是一直跌, buffer/cache很高, swap几乎等于0. 程序不挂了,但是可用内存极少, 为什么优化之后free内存还是极少

解析

究竟top和free输出的free, buff/cache, avali Mem和free的区别在哪里?
存储器系统简单铺垫:

• cpu发出都是逻辑地址,经过TLB硬件机构转成逻物理址访问cache(以core i7为例L1,L2是core独享的,L3是共享 、L1是物理+逻辑地址、L2、L3是物理地址), 在缓存都被击穿之后访问main memory,这里硬件主导, 缺失代价几百时钟周期
• linux main memory划分为段页式, 读缺失以块(64 byte)的形式进入进入cache,这里硬件和操作系统主导. 这里OS针对pid可以检查页表访问/修改的合法性
• 当main memory不够用时就需要不太热的数据dump到磁盘上, 这里是OS主导缺失代价高达数百万时钟周期, 所以swap抖动通常伴随着性能损耗
• main memory +swap构成了虚拟存储器
  之前看到swap占比较高, 伴随震荡,直接反应就是性能急剧下降(在我们的应用里体现在帧率骤降)

但是为什么优化后的程序free memory还是急剧下降呢?
在linux系统设计中free内存就是原罪, free就是真正那些没有只用内存, 道理上有点像手里有好多钱花不完也不去投资做理财买币,实实在在的浪费.
那么linux是如何利用这些暂时不用的内存呢?
可以看到虽然free很低, 但是buff/cache很高, 那么buff和cache是用来做什么的呢

Linux is borrowing unused memory for disk caching. This makes it looks like you are low on memory, but you are not! Everything is fine! Disk caching makes the system much faster and more responsive! There are no downsides, except for confusing newbies.

这些有点像活期理财,需要的时候可以变现使用,虽然临时被占用了大部分在需要的时候可以为程序所用.
buff和cache的区别是啥呢

Short answer: Cached is the size of the page cache. Buffers is the size of in-memory block I/O buffers. Cached matters; Buffers is largely irrelevant.
Long answer: Cached is the size of the Linux page cache, minus the memory in the swap cache, which is represented by SwapCached (thus the total page cache size is Cached + SwapCached). Linux performs all file I/O through the page cache. Writes are implemented as simply marking as dirty the corresponding pages in the page cache; the flusher threads then periodically write back to disk any dirty pages. Reads are implemented by returning the data from the page cache; if the data is not yet in the cache, it is first populated. On a modern Linux system, Cached can easily be several gigabytes. It will shrink only in response to memory pressure. The system will purge the page cache along with swapping data out to disk to make available more memory as needed.
Buffers are in-memory block I/O buffers. They are relatively short-lived. Prior to Linux kernel version 2.4, Linux had separate page and buffer caches. Since 2.4, the page and buffer cache are unified and Buffers is raw disk blocks not represented in the page cache—i.e., not file data. The Buffers metric is thus of minimal importance. On most systems, Buffers is often only tens of megabytes.

cache相对来讲重一些, linux中IO都是基于cache完成的, 面对memory压力, page cache的size会大幅下降. 之前的应用中也确实有重IO.而buffer的单位block,他是作为桥接page cache和kernel的通道, buffer通常mb为单位,相对来讲没那么重

used没有什么歧义,就是程序所用内存, 但是free和available memory就不一样了, free可以理解为飘逸在系统之外的内存, available是真正可用的内存. 一般情况下直接靠buf/cache + free计算不太准确, 因为不是所有的buf/cache都可以释放.这几个名词的关系如下

那么如何看内存状况比较糟糕呢
available memory接近0, swap剧烈抖动, 这时候buf/cache稳定(系统所用之必须),往往伴随着吞吐下降, 性能崩盘.

这里又详细解释为啥buf/cache + free计算不科学的, /proc/meminfo: provide estimated available memory

代码中增加了available memory来衡量相对精确的可用内存

关于linux disk cache的实验也可以参考Experiments and fun with the Linux disk cache
referen link:
https://www.linuxnix.com/find-ram-size-in-linuxunix/
https://www.linuxatemyram.com/
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=34e431b0ae398fc54ea69ff85ec700722c9da773
https://www.linuxatemyram.com/play.html
https://www.quora.com/What-is-the-major-difference-between-the-buffer-cache-and-the-page-cache-Why-were-they-separate-entities-in-older-kernels-Why-were-they-merged-later-on
https://www.quora.com/What-is-the-difference-between-Buffers-and-Cached-columns-in-proc-meminfo-output
http://www.linuxhowtos.org/System/Linux%20Memory%20Management.htm