内存剖析

Ceph 监视器、 OSD 、和元数据服务器可利用 tcmalloc 生成堆栈剖析,此功能依赖 google-perftools

sudo apt-get install google-perftools

剖析器会把输出保存到 log file 目录(如 /var/log/ceph ),详情见日志记录和调试。剖析器日志可用 Google 性能工具来查看,执行如下命令:

google-pprof --text {path-to-daemon}  {log-path/filename}

例如:

$ ceph tell osd.0 heap start_profiler
$ ceph tell osd.0 heap dump
osd.0 tcmalloc heap stats:------------------------------------------------
MALLOC:        2632288 (    2.5 MiB) Bytes in use by application
MALLOC: +       499712 (    0.5 MiB) Bytes in page heap freelist
MALLOC: +       543800 (    0.5 MiB) Bytes in central cache freelist
MALLOC: +       327680 (    0.3 MiB) Bytes in transfer cache freelist
MALLOC: +      1239400 (    1.2 MiB) Bytes in thread cache freelists
MALLOC: +      1142936 (    1.1 MiB) Bytes in malloc metadata
MALLOC:   ------------
MALLOC: =      6385816 (    6.1 MiB) Actual memory used (physical + swap)
MALLOC: +            0 (    0.0 MiB) Bytes released to OS (aka unmapped)
MALLOC:   ------------
MALLOC: =      6385816 (    6.1 MiB) Virtual address space used
MALLOC:
MALLOC:            231              Spans in use
MALLOC:             56              Thread heaps in use
MALLOC:           8192              Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
$ google-pprof --text \
               /usr/bin/ceph-osd  \
               /var/log/ceph/ceph-osd.0.profile.0001.heap
 Total: 3.7 MB
 1.9  51.1%  51.1%      1.9  51.1% ceph::log::Log::create_entry
 1.8  47.3%  98.4%      1.8  47.3% std::string::_Rep::_S_create
 0.0   0.4%  98.9%      0.0   0.6% SimpleMessenger::add_accept_pipe
 0.0   0.4%  99.2%      0.0   0.6% decode_message
 ...

再次转储堆栈会生成另外一个文件,这样便于和前面的堆栈转储相比较,看看这段时间发生了什么。例如:

$ google-pprof --text --base out/osd.0.profile.0001.heap \
      ceph-osd out/osd.0.profile.0003.heap
 Total: 0.2 MB
 0.1  50.3%  50.3%      0.1  50.3% ceph::log::Log::create_entry
 0.1  46.6%  96.8%      0.1  46.6% std::string::_Rep::_S_create
 0.0   0.9%  97.7%      0.0  26.1% ReplicatedPG::do_op
 0.0   0.8%  98.5%      0.0   0.8% __gnu_cxx::new_allocator::allocate

详情见 Google 堆栈剖析器

安装堆栈剖析器后,启动集群就可以开始使用了。你可以在运行时启用、或禁用堆栈剖析器,或确保它在持续运行。在随后的几个命令行用法中,用 monosdmds 替换掉 {daemon-type} ,用 OSD 号、监视器或元数据服务器的 ID 替换掉 {daemon-id}

启动剖析器

要启动堆栈剖析器,执行如下命令:

ceph tell {daemon-type}.{daemon-id} heap start_profiler

例如:

ceph tell osd.1 heap start_profiler

另外,在启动守护进程时若设置了 CEPH_HEAP_PROFILER_INIT=true 环境变量,剖析器也会启动。

打印统计信息

用下列命令打印统计状态:

ceph  tell {daemon-type}.{daemon-id} heap stats

例如:

ceph tell osd.0 heap stats

Note

打印状态不要求剖析器在运行,也不会把堆栈分配信息转储到文件。

转储堆栈信息

用下列命令转储堆栈信息:

ceph tell {daemon-type}.{daemon-id} heap dump

例如:

ceph tell mds.a heap dump

Note

只能在剖析器运行的时候转储堆栈信息。

释放内存

要释放由 tcmalloc 分配、但不是被 Ceph 守护进程使用的内存,用下列命令:

ceph tell {daemon-type}{daemon-id} heap release

例如:

ceph tell osd.2 heap release

停止剖析器

要停止堆栈剖析器,执行下列命令:

ceph tell {daemon-type}.{daemon-id} heap stop_profiler

例如:

ceph tell osd.0 heap stop_profiler