ceph内存剖析工具--pprof

一、参考文档路径：

http://docs.ceph.org.cn/rados/troubleshooting/memory-profiling/

https://gperftools.github.io/gperftools/heapprofile.html

二、google-perftools工具下载路径：

https://sourceforge.net/projects/goog-perftools/

gperftools-master.zip <见百度云常用工具>

三、工具使用方法

gperftools适合发现运行过程中内存差异，提供泄露的函数范围。而且进程响影响小，适合高压力快速复现。

详细使用方法请参考http://docs.ceph.org.cn/rados/troubleshooting/memory-profiling/，下面仅对该文档做一些补充说明

1、单独查看一个内存堆分析文件

pprof --text /usr/bin/ceph-osd /var/log/ceph/osd.25.profile.0003.heap

2、比较两个内存堆分析文件

pprof --text --base /var/log/ceph/osd.34.profile.0011.heap /usr/bin/ceph-osd /var/log/ceph/osd.34.profile.0029.heap

3、输出结果说明

[root@node4 gperftools-master]# pprof --text /usr/bin/ceph-osd /var/log/ceph/osd.34.profile.0011.heap |less
Using local file /usr/bin/ceph-osd.
Using local file /var/log/ceph/osd.34.profile.0011.heap.
Total: 3.4 MB
3.3 94.8% 94.8% 3.3 94.8% ceph::logging::Log::create_entry
0.1 1.7% 96.6% 0.1 1.7% AsyncConnection::AsyncConnection
0.0 0.7% 97.2% 0.0 0.7% ceph::buffer::list::append@a34000
0.0 0.6% 97.8% 0.0 0.6% ceph::buffer::create_aligned
0.0 0.5% 98.3% 0.1 1.5% AsyncMessenger::add_accept
0.0 0.4% 98.7% 0.0 0.4% EventCenter::create_time_event
0.0 0.3% 99.0% 0.0 1.0% AsyncMessenger::create_connect
0.0 0.3% 99.2% 2.9 83.2% EventCenter::process_events
0.0 0.1% 99.4% 0.0 0.1% ceph::buffer::list::append@a34490
0.0 0.1% 99.5% 0.5 15.3% OSD::heartbeat
0.0 0.1% 99.6% 0.0 0.1% decode_message
0.0 0.1% 99.7% 0.0 0.1% std::basic_string::_Rep::_S_create
0.0 0.0% 99.7% 0.0 0.0% std::_Rb_tree::_M_emplace_hint_unique
0.0 0.0% 99.7% 0.0 0.1% OSD::ms_verify_authorizer
0.0 0.0% 99.8% 0.0 0.0% std::_Rb_tree::_M_copy
0.0 0.0% 99.8% 0.0 0.0% std::_Rb_tree::M_insert
0.0 0.0% 99.8% 1.0 27.9% AsyncConnection::send_message
0.0 0.0% 99.8% 0.0 0.0% PORT_Alloc_Util
0.0 0.0% 99.9% 0.0 0.0% CryptoAES::get_key_handler
0.0 0.0% 99.9% 0.0 0.0% std::vector::_M_emplace_back_aux
0.0 0.0% 99.9% 0.0 0.0% ceph::buffer::create_static
0.0 0.0% 99.9% 0.0 0.0% ceph::buffer::list::append@a2e290
0.0 0.0% 99.9% 0.0 0.0% ceph::buffer::list::append@a2e3c0

Total：后面显示总共申请的内存
第一列值显示的是该函数内，不包括调用的其他函数，总共申请的堆内存
第四列值显示的是该函数内，包括自己申请的堆内存，和调用外部函数申请的堆内存
第二列和第五列，是第一列和第四例相对于Total的比值
第六列显示具体申请堆内存的函数

4、堆内存释放方法

a、首先使用命令：ceph tell {daemon-type}.{daemon-id} heap stats 查看是否有未使用的堆内存。（注意这里的daemon-type可以是mon、osd、mds，daemon-id是对应服务编号）

举例：

 [root@node1 bn_cli]# ceph tell osd.4 heap stats
osd.4 tcmalloc heap stats:------------------------------------------------
MALLOC:       21704640 (   20.7 MiB) Bytes in use by application
MALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist
MALLOC: +      1406560 (    1.3 MiB) Bytes in central cache freelist
MALLOC: +      1310656 (    1.2 MiB) Bytes in transfer cache freelist
MALLOC: +      3316256 (    3.2 MiB) Bytes in thread cache freelists
MALLOC: +      1175712 (    1.1 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =     28913824 (   27.6 MiB) Actual memory used (physical + swap)
MALLOC: +       573440 (    0.5 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =     29487264 (   28.1 MiB) Virtual address space used
MALLOC:
MALLOC:           1039              Spans in use
MALLOC:             36              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.

查看是否有：Bytes in page heap freelist （未释放的堆内存）

b、如果Bytes in page heap freelist 不为0，则执行如下命令，可以释放掉未使用的堆内存

ceph tell osd.4 heap release