Perf火焰图(by quqi99)

版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明 (http://blog.csdn.net/quqi99)

Install perf

sudo add-apt-repository ppa:canonical-kernel-team/ppa
sudo apt install linux-tools-$(uname -r)

注意点

• 火焰图看最宽的那条, 最宽的显示最耗时的, 上下最高则表示调用深度最长
• 如若抓应用态的(如ovs)则应添加" --call-graph dwarf " 参数, ovs最好也安装符号表
• 如ovs隔六七秒就死, 可以使用它来抓这六七秒内的调用栈 - sudo perf record -a --call-graph dwarf – sleep 12
• ideal的占一成的话, 那cpu占用率就是9成左右. 如果所有进程耗时都比较平均, 那只能说明整体性能不够
• 使用Ctrl + F可以搜索
• offcpuflamegraphs.html 可以用来分析cpu为什么会睡那么久soft-lock相关的问题
• 之前使用systemtap, 但这东西需要脚本, 同时也不方便online debug session时安装. 现在主要使用perf, 另外也可以学习ebpf

Perf Flame Graph

git clone https://github.com/cobblau/FlameGraph
sudo perf record --call-graph dwarf -p $(pidof qemu-system-x86_64)
sudo perf script | FlameGraph/stackcollapse-perf.pl | FlameGraph/flamegraph.pl > process.svg

分解

sudo perf record -F 99 -g -- sleep 30 $(pidof qemu-system-x86_64)
sudo perf report -n --stdio
sudo perf script | gzip > out.perf01.gz
git clone https://github.com/cobblau/FlameGraph
; http://svgshare.com can be used to share svg
sudo perf script | ./FlameGraph/stackcollapse-perf.pl | sed '/cpu_idle/d;s/\[unknown\];//g' | ./FlameGraph/flamegraph.pl --width 1000 > perf.svg
google-chrome-stable ./perf.svg

其他一

参考： https://github.com/bboymimi/perf-utils

# To get the all CPUs profiling callstack from user space to kernel space (both kernel and user space)
sudo perf record -a --call-graph dwarf
# To get the all CPUs profiling of kernel space callstack (just kernel space)
sudo perf record -a -g
git clone https://github.com/bboymimi/perf-utils.git
sudo ./perf-utils/easy-flamegraph.sh -i perf.data
google-chrome-stable ./perf-output/2018-01-04_14:18:15.perf.data.svg

其他二##

# Enable the sysrq 
sudo sysctl -w kernel.sysrq=1 
# Dump all active tasks stack - https://www.kernel.org/doc/html/latest/admin-guide/sysrq.html
echo l | sudo tee /proc/sysrq-trigger 
# Dumps tasks that are in uninterruptable (blocked) state. 
echo p | sudo tee /proc/sysrq-trigger 
# Performance profiling and get the flame-graph to see if there is 
any performance bottleneck when the problem happened
git clone git://kernel.ubuntu.com/gavinguo/perf-production.git 
sudo perf record -a -g sleep 5 
sudo ./easy-flamegraph.sh -i ./perf.data -t 

Profile Java with Perf
默认的perf只能分析c与c++程序, 其他用户态的perf-java, perf-python, perf-go应当分别学习, 这里说的是perf-java

Profile Java with Perf
# https://cloud.tencent.com/developer/article/1416234
# https://my.oschina.net/u/150599/blog/3023394
sudo apt install -y linux-tools-common linux-tools-generic bindfs
mkdir /tmp/containerrootfs
PID=$(docker inspect --format {{.State.Pid}} cas1)
sudo perf record -g -p $PID
sudo perf report
#fix symbols in container
cp perf.data /tmp/containerrootfs/
bindfs /proc/$PID/root /tmp/containerrootfs
perf report --symfs /tmp/containerrootfs
umount /tmp/containerrootfs

# install java
sudo apt install -y openjdk-8-jdk
dpkg-query -L openjdk-8-jdk
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
# install perf-map-agent to help generate symbols with FlameGraph
sudo apt install -y cmake pkg-config libfuse-dev libfuse2 autoconf
git clone https://github.com/brendangregg/FlameGraph.git
export FLAMEGRAPH_DIR=/home/ubuntu/FlameGraph
git clone https://github.com/jvm-profiling-tools/perf-map-agent.git
cd perf-map-agent/
cmake .
make
sudo bin/create-links-in /usr/local/bin

# Use perf-java-flames (
# Need to add one jvm option in java process: -XX:+PreserveFramePointer
perf-java-flames $PID -g
#perf-java-record-stack $PID -g
chromium-browser /bak/work/perf-map-agent/flamegraph-10286.svg

上下文切换高
启用sar，

sudo apt install sysstat
sudo sed -i 's/^ENABLED="false"/ENABLED="true"/' /etc/default/sysstat
sudo sed -i 's/^5-55\/10/*\/2/' /etc/cron.d/sysstat
sudo systemctl restart sysstat

后可以观测如上文文切换过高

00:00:01       proc/s   cswch/s
Average:       143.82  59339.01

可以搜索火焰图（用户态内核态一起收集)

git clone git://kernel.ubuntu.com/gavinguo/perf-production.git
cd perf-production
sudo perf record -a --call-graph dwarf sleep 5
sudo ./easy-flamegraph.sh -i ./perf.data -t

也可以使用pidstat -w 1来确定哪个进程的上下文切换高， 或者:

# https://serverfault.com/questions/190049/find-out-which-task-is-generating-a-lot-of-context-switches-on-linux
sudo perf record -e context-switches -a -g
# then ctrl+c
sudo perf report # inspect the result

**Appendix - error - ERROR: No stack counts found **
遇到下面error是需要: sudo chown -R root:root perf.data, 必须使用root用户, 而非它说的普通用户.

hua@t440p:/bak/work/perf-utils$ sudo ./easy-flamegraph.sh perf.data  -f
Use the /bak/work/perf-utils/perf.data as the source of the FlameGraph.
File /bak/work/perf-utils/perf.data not owned by current user or root (use -f to override)
ERROR: No stack counts found

tool - poormansprofiler

#see what [multithreaded] programs are blocking on - https://poormansprofiler.org/
sudo gdb -ex “set pagination 0” -ex “thread apply all bt” --batch -p $(pidof ovs-vswitchd)

mkdir gdb_ovs; cd gdb_ovs; for i in $(seq 1 600); do gdb -ex “set pagination 0” -ex “thread apply all bt” -batch -p KaTeX parse error: Expected 'EOF', got '&' at position 5: pid &̲> gdb.ovs-vswit…(date -Is) ; sleep 0.1; done

tool - tmate

tmate is used to share ssh with others

收集符号表

• 内核符号表可以通过’sudo cat /proc/kallsyms | tee $(uname -r).kallsyms’命令收集,然后使用’per script --kallsyms’命令使用它.
• 用户态符号表在gavin的工具中已得到处理, 非要人工安装的话可以安装到/usr/lib/debug目录, 这也是ddeb包安装的目录
• libunwind是一种符号表新标准.在build包时指定它生成的包体积小, 同时也能找到符号表

wget https://raw.githubusercontent.com/torvalds/linux/master/tools/perf/perf-archive.sh -O perf-archive.sh 
chmod +x perf-archive.sh 
./perf-archive.sh # this can take a while, a few minutes

#sudo cat /proc/kallsyms | tee $(uname -r).kallsyms

新工具

• hostspot, 把perf数据用时间轴的方式图像化, 细致到时间点, 能以"Bottom Up" “top Down” Flame Graph", "Caller/Callee"的方式看, 且能放大 hotspot --kallsyms …/xx-kallsyms
• flamescope, 把perf数据用时间轴时间用热点的方式表示
• speedscope, 把perf数据用

perf debian

#分文件存放, cpu_50p(cpu > 50%), mem_5G(mem > 5G)
sudo add-apt-repository ppa:mimi0213kimo/easyflamegraph
sudo apt-get update
ls /var/log/easy-flamegraph/mem/
grep -r 'USE_EASY_FLAMEGRAPH' /etc/default/easy-flamegraph 

Reference

[1] http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html
[2] http://www.brendangregg.com/flamegraphs.html