Analyzing CPU Usage with Perf (1) - preparation

About Perf

Perf is a profiling utility shipped with Linux kernel.

Perf based profiling is split in to 3 parts:

1. Data sources (tracing data that comes from kernel/application)
2. Data extractor (collection/recording of profiled data)
3. Data presentation (front-end tools to generate statistics and graphs)

Building Perf-enabled Firmware

Building Perf-enabled Firmware

1. Clean built binaries, rootfs and Buildroot: (in $(SDK_DIR)/build/):

$ make clean; make rootfs-distclean buildroot-clean

2. Enable the following configs in menuconfig (in $(SDK_DIR)/buildroot/src/ , which is later referred to as $(BUILDROOT_DIR)):

BR2_OPTIMIZE_G=y
BR2_STRIP_NONE=y

3. Rebuild buildroot and rootfs

The generated debugging version of rootfs is in $(ROOTFS_DIR)/output/target_debug.
User should confirm that libraries within are not stripped.

Enable Perf in Linux Kernel Config

Edit Linux configuration file.

$ cd $SDK_DIR/linux
$ make menuconfig

Turn on perf.

1. Go to “General setup” and then “Kernel Performance Events And Counters”.
2. Select “Kernel performance events and counters”.
3. steps to enable frame pointer
   3.1 in menuconfig, press / to open search window
   3.2 enter CONFIG_FRAME_POINTER and press enter
   3.3 Check first line is Symbol: FRAME_POINTER [=y]
4. Save the config and exit.

Note: CONFIG_FRAME_POINTER might not show up in menuconfig.
Force to turn on frame pointer (Eg. Add “select ARCH_WANT_FRAME_POINTERS” to arch/arm/Kconfig) in this case will fail during compilation.

check the result

You can double check the .config file.

$ vi $SDK_DIR/linux/.config
You will see the following options enabled.

CONFIG_PERF_EVENTS=y
CONFIG_HW_PERF_EVENTS=y
CONFIG_HAVE_PERF_EVENTS=y
CONFIG_FRAME_POINTER=y

Save the config and exit.

$ cd SDK_DIR/linux/
$ make savedefconfig

Update from top level.
The kernel with perf enabled will be built during the process.

$ cd $SDK_DIR/build
$ make clean;  make <prod config> ; make 

Building and Installing Perf

If you need to compile from source, just follow these steps.

1. Unload japan-linu-sdk.lua if loaded
   On build server, default japan-linux-sdk env. setting would use Augentix toolchain in /tool. We are to disable it first:

$ module unload japan-linux-sdk

2. Downlaod zlib

wget http://downloads.sourceforge.net/project/libpng/zlib/1.2.8/zlib-1.2.8.tar.gz

3. Configure and install zlib

$ tar -zxvf zlib-1.2.8.tar.gz
$ cd  zlib-1.2.8
$ ./configure --host=arm-xxxx-linux-gnueabi --prefix $SDKSRC_DIR/toolchain/arm-xxxx-linux-gnueabi/arm-xxxx-linux-gnueabi/sysroot/usr/
$ make && make install

4. Download elfutils

$ wget https://fedorahosted.org/releases/e/l/elfutils/0.161/elfutils-0.161.tar.bz2 --no-check-certificate
$ tar -jxvf elfutils-0.161.tar.bz2

5. Configure and compile elfutils and libelf:

$ cd elfutils-0.161
$ ./configure --host=arm-xxxx-linux-gnueabi --prefix $SDKSRC_DIR/toolchain/arm-xxxx-linux-gnueabi/arm-xxxx-linux-gnueabi/sysroot/usr/
$ make && make install

6. Apply env. setting for toolchain in SDK

export PATH=$(SDKSRC_DIR)/toolchain/arm-xxxx-linux-gnueabi/bin:$PATH

7. Compile perf
   (In $(SDKSRC_DIR)/linux/tool/perf)

$ make LDFLAGS="-static -L$(SDKSRC_DIR)/toolchain/arm-xxxx-linux-gnueabi/arm-xxxx-linux-gnueabi/sysroot/usr/lib -lelf -IL$SDKSRC_DIR/toolchain/arm-xxxx-linux-gnueabi/arm-xxxx-linux-gnueabi/sysroot/usr/include" ARCH=arm CROSS_COMPILE=arm-xxxx-linux-gnueabi-

Add Thread Names to Application

If you are profiling an application with threads, adding thread name is highly recommended.

The following code illustrates how to set thread name using the pthread_setname_np() function.

if (pthread_create(&ctx->tid_movdet, NULL, runOd, NULL) != 0) {
        fprintf(stderr, "Create thread to run IVA object failed.\n");
        goto err_getchn;
}

char *name = "object detect"; /* max. length is 16 */
err = pthread_setname_np(ctx->tid_movdet, name);
if (err != 0) {
        printf("Create object detection thread to config failed. err = %d\n", err);
}

Compiling Application for Stack Tracing

Compile the application with frame pointers preserved allows you to track calling stack when profiling.
Set environment variable before you build firmware.

$ export PERF_NO_OMIT_FP=-fno-omit-frame-pointer

Downloading Perf-enabled Firmware

Apply Un-Stripped Version of Target Libraries

After boot up, user should selectively copy un-stripped library to target platform via NFS or SD card.
The unstripped binaries, modules and libraries can be found in

$(SDK_DIR)/fs/rootfs/output/target_debug

Copy applications, modules or libraries of interest from above location to corresponding directory on target platform.
Note that if you are to copy libraries from NFS over Windows, you must:

1. Create a copy of target_debug directory in Linux environment
2. Remove symbolic links first (in the copied target_debug directory):

$ find . -type l -delete

3. Copy the directory to NFS directory in Windows.

Debugging (unstripped) version of following libs and binaries are required:

• libc and libgcc in /lib libpthread in /lib
• libstdc++ in /lib
• libsensor_*.so in /system/lib
• target binary in /system/bin

Since executable binaries are copied from Windows NFS, we have to change access permission to make it executable:

$ chmod a+x /system/bin/<target bin>

Note that you must not copy /lib/ld-2.22.so at runtime, or the system will fail.