system.cpu Metrics from DataDog Agent

Name

• system.cpu.guest

• Percent (Unit)
• The percent of time the CPU spent running the virtual processor. Only applies to hypervisors.

• system.cpu.idle

• Percent (Unit)
• Percent of time the CPU spent in an idle state.

• system.cpu.iowait

• Percent (Unit)
• The percent of time the CPU spent waiting for IO operations to complete.

• system.cpu.stolen

• Percent (Unit)
• The percent of time the virtual CPU spent waiting for the hypervisor to service another virtual CPU. Only applies to virtual machines.

• system.cpu.system

• Percent (Unit)
• The percent of time the CPU spent running the kernel.

• system.cpu.user

• Percent (Unit)
• The percent of time the CPU spent running user space processes.

计算公式

system.cpu.guest = %guest
system.cpu.idle = %idle
system.cpu.iowait = %iowait
system.cpu.stolen = %steal
system.cpu.system = %sys + %irq + %soft
system.cpu.user = %usr + %user + %nice

数据来源

针对Linux系统，DataDog Agent使用mpstat实时监控工具，获取CPU相关的Metrics。DataDog Agent并没有获取各个CPU的详细信息，而是所有CPU的平均值。

mpstat是Multiprocessor Statistics的缩写，是实时系统CPU监控工具。监控信息来自内核文件：/proc/stat

mpstat输出内容示例

# mpstat 1 3

Linux 3.10.0-693.21.1.el7.x86_64 (ip-172-31-22-228.cn-north-1.compute.internal)     08/29/2018  _x86_64_    (8 CPU)

05:59:14 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
05:59:15 AM  all    8.06    0.13    7.15    0.00    0.00    0.78    0.13    0.00    0.00   83.75
05:59:16 AM  all   14.61    0.00    8.16    0.00    0.00    1.05    0.26    0.00    0.00   75.92
05:59:17 AM  all   18.69    0.13    5.75    0.00    0.00    0.78    0.13    0.00    0.00   74.51
Average:     all   13.78    0.09    7.02    0.00    0.00    0.87    0.17    0.00    0.00   78.07

各个字段含义如下(internal时间段内的平均值)

CPU       Processor number. The keyword all indicates that statistics are calculated as averages among all processors.
%usr      Show the percentage of CPU utilization that occurred while executing at the user level (application).
%nice     Show the percentage of CPU utilization that occurred while executing at the user level with nice priority.
%sys      Show  the  percentage  of  CPU utilization that occurred while executing at the system level (kernel). Note that this does not include time spent servicing hardware and software interrupts.
%iowait   Show the percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
%irq      Show the percentage of time spent by the CPU or CPUs to service hardware interrupts.
%soft     Show the percentage of time spent by the CPU or CPUs to service software interrupts.
%steal    Show  the  percentage of time spent in involuntary wait by the virtual CPU or CPUs while the hypervisor was servicing another virtual processor.
%guest    Show the percentage of time spent by the CPU or CPUs to run a virtual processor.
%gnice    Show the percentage of time spent by the CPU or CPUs to run a niced guest.
%idle     Show the percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.