Linux Performance Tools

copy: http://www.brendangregg.com/linuxperf.html

Linux Performance

hi-res: observability + static + perf-tools/bcc (svg)

slides: observability

slides: static, benchmarking, tuning

sar, perf-tools, bcc/BPF:

Images license: creative commons Attribution-ShareAlike 4.0.

This page links to various Linux performance material I've created, including the tools maps on the right. The first is a hi-res version combining observability, static performance tuning, and perf-tools/bcc (see discussion). The remainder were designed for use in slide decks and have larger fonts and arrows, and show: Linux observability tools, Linux benchmarking tools, Linux tuning tools, and Linux sar. For even more diagrams, see my slide decks below.

Tools

• perf_events: perf one-liners, examples, visualizations.
• eBPF tools: eBPF tracing tools and examples with bcc.
• perf-tools: perf analysis tools using ftrace (github).
• bcc: perf analysis tools using eBPF (github).
• ktap: one-liners, examples, and scripts.
• Flame Graphs: using perf, SystemTap, and ktap.

Documentation

• Linux Performance Analysis in 60,000 Milliseconds shows the first ten commands to use in an investigation (video, PDF). Written by myself and the performance engineering team at Netflix (2015).
• My post Performance Tuning Linux Instances on EC2 includes the tunables we're using at Netflix (2015).
• A post on Linux Load Averages: Solving the Mystery, explaining what they are and why they include the uninterruptible sleep state (2017).
• A gdb Debugging Full Example (Tutorial), including the use of some perf/debugging tools (2016).
• Generating flame graphs on Linux using perf & eBPF:
  • CPU Flame Graphs
    Off-CPU Flame Graphs
    Memory Flame Graphs
• Posts about eBPF, bcc, and bpftrace (2015-8):
  • Linux eBPF (2015)
    bcc: Taming Linux 4.3+ Tracing Superpowers
    tcpconnect and tcpaccept for Linux (bcc)
    Linux eBPF Stack Trace Hack (bcc) (2016)
    Linux eBPF Off-CPU Flame Graph (bcc)
    Linux Wakeup and Off-Wake Profiling (bcc)
    Linux chain graph prototype (bcc)
    Linux eBPF/bcc uprobes
    Linux BPF/bcc Road Ahead
    Ubuntu Xenial bcc/BPF
    Linux bcc/BPF Tracing Security Capabilities
    Linux MySQL Slow Query Tracing with bcc/BPF
    Linux bcc/BPF ext4 Latency Tracing
    Linux bcc/BPF Run Queue (Scheduler) Latency
    Linux bcc/BPF Node.js USDT Tracing
    Linux bcc tcptop
    Linux 4.9's Efficient BPF-based Profiler
    DTrace for Linux 2016
    Linux 4.x Tracing Tools: Using BPF Superpowers
    Linux bcc/BPF tcplife: TCP Lifespans
    Golang bcc/BPF Function Tracing (2017)
    7 BPF tools for performance analysis on Fedora
    TCP Tracepoints (2018)
    Linux bcc/eBPF tcpdrop
    bpftrace (DTrace 2.0) for Linux 2018
    
     
• My lwn.net article Ftrace: The Hidden Light Switch shows a use case for Linux ftrace (Aug, 2014).
• Posts about ftrace-based perf-tools (2014-5):
  • iosnoop for Linux, iosnoop Latency Heat Maps, opensnoop for Linux, execsnoop for Linux, ftrace: The Hidden Light Switch, tcpretrans
    Page Cache Hit Ratio, uprobe: User-Level Dynamic Tracing, Hacking Linux USDT
• Posts about perf-based perf-tools: perf Hacktogram.
• Posts about perf_events (2014-7):
  • perf CPU Sampling, perf Static Tracepoints, perf Heat Maps, perf Counting, perf Kernel Line Tracing,
    perf Off-CPU Time Flame Graphs, Linux Profiling at Netflix, Java Mixed-Mode Flame Graphs (PDF), Linux 4.5 perf folded format,
    perf sched for Linux CPU scheduler analysis
• A page on Working Set Size Estimation for Linux (2018+).
• A post on KPTI/KAISER Meltdown Initial Performance Regressions (2018).
• In The PMCs of EC2: Measuring IPC I showed the new Performance Monitoring Counter (PMC) support in the AWS EC2 cloud (2017).
• CPU Utilization is Wrong: a post explaining the growing problem of memory stall cycles dominating the %CPU metric (2017).
• A post about Linux 4.7 Hist Triggers (2016).
• The blog post strace Wow Much Syscall discusses strace(1) for production use, and compares it to advanced tracing tools (2014).
• USE Method: Linux Performance Checklist; also see the USE Method page for the description of this methodology.
• Off-CPU Analysis Method, where I demonstrate this methodology on Linux.
• Systems Performance: Enterprise and the Cloud (Prentice Hall, 2013) uses Linux distributions as the primary example.