Detecting Memory Management and Threading Bugs with Valgrind

About Valgrind

Valgrind is an instrumentation framework for building dynamic analysis tools. There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. You can also use Valgrind to build new tools.

According to the Valgrind FAQ, Val- is supposed to be pronounced like value and -grind is supposed to be pronounced like grinned.

Installing Valgrind

From Source

1. Download Valgrind and checkout tag VALGRIND_3_14_0

$ git config --global http.sslverify false
$ git clone https://sourceware.org/git/valgrind.git
$ cd valgrind
$ git checkout VALGRIND_3_14_0

2. Apply patch to support ARMv5. (armv5.patch published by https://bugs.kde.org/show_bug.cgi?id=248998)

 $patch -p1 < armv5.patch

3. Follow the steps in README to use autogen.sh

$ ./autogen.sh

4. select your toolchain

$ export MYTOOLCHAIN=arm-linux-gnueabi << or other case
$ CC=$MYTOOLCHAIN-gcc  
$ ./configure --host=$MYTOOLCHAIN --target=$MYTOOLCHAIN --prefix=$HOME/valgrind.armv5 CFLAGS="-fPIC" 
$ make
$ make install

From Pre-compiled Binary

1. This build is for arm-linux-gnueabi toolchain. (E.g. tootlchain or IC name)
   Unpack it to a location where accessible from target via NFS,
   and change the access mode of the binaries

chmod 755 /mnt/nfs/ethnfs/valgrind.armv5/bin/*
chmod 755 /mnt/nfs/ethnfs/valgrind.armv5/lib/valgrind/memcheck-arm-linux

2. This build is for arm-linux-gnueabihf toolchain
   Copy this directory directly into your NFS folder.
   Following symbols should be used instead of the ones in “Running Valgrind” section later.
   (Assuming that this directory is named /mnt/nfs/ethnfs/valgrind.kyotofpu)

export VALGRIND_LIB=/mnt/nfs/ethnfs/valgrind.kyotofpu
export PATH=/mnt/nfs/ethnfs/valgrind.kyotofpu:$PATH

Prepare FW and standard library with symbol

Prepare FW

A normal build without symbols will do, FW image with symbols are usually too large to be written on a platform.
Notice: valgrind requires a large quantity of memory, you might not be able to run it on platforms with small memory size.
Use platforms with larger memory size if possible.

Prepare sysroot on NFS

Linaro sysroot need to rename libc, $XTOOL_PATH also different

$ module load XXXX << prepare your toolchain
copy $XTOOL_PATH/arm-linux-gnueabi/libc/lib to your NFS

There is no need to copy the entire sysroot;
just prepare the essential libraries.

The following steps assume that this directory will be copied to /sysroot/lib. After copying, the /sysroot/lib directory will contain libraries such as ld-xxx.so, libc-xxx.so, libstdc++.so, and others.

Set LD_LIBRARY_PATH and create symbolic links

valgrind need symbols in standard libraries, especially libc.so and ld.so, so we need to link it to the standard library with symbols.

Here provide a simplest method, which utilize the linking priority “LD_PRELOAD > LD_LIBRARY_PATH > /etc/ld.so.cache > /lib > /usr/lib”

$ export LD_LIBRARY_PATH=<nfs path>/sysroot/lib
$ cd <nfs path>/sysroot/lib
$ # replace the following commands with the real filenames in sysroot/lib
$ ln -sf ld-2.xx-xxx.so ld-linux.so.3
$ ln -sf libc-2.xx-xxx.so libc.so.6

Where sysroot/lib contains shared libraries with symbols.
Notice that the version of libc.so and ld.so may vary.

Prepare self-implement libraries and applications with symbols

If the process need application libraries like libaftr.so.3, libvftr.so.3, libmpp.so.3, etc.
You can find copies of them with symbols in /fs/rootfs/output/target_debug/system/lib (which should be in somewhere like the build servers).
Put them into /sysroot/lib since LD_LIBRARY_PATH is now there.

Similarly, reference and sample applications with symbols can be found in /fs/rootfs/output/target_debug/system/bin.
Copy them into the NFS if needed.

Running Valgrind

Valgrind is too large to put into target, so we need to run it from NFS.
In the following example, valgrind is accessible from /mnt/nfs/ethnfs/valgrind.armv5.

1. Define VALGRIND_LIB environment variable

$ export VALGRIND_LIB=/mnt/nfs/ethnfs/valgrind.armv5/lib/valgrind

2. Add to PATH

$ export PATH=/mnt/nfs/ethnfs/valgrind.armv5/bin:$PATH

3. Run valgrind for a full analysis of memory management on a program

$ valgrind --leak-check=full --error-limit=no <program to be analyzed>
Eg. valgrind --leak-check=full --error-limit=no mpi_stream -d /system/mpp/case_config/case_config_1001

Verification

Put the test programs on DUT and set environment correctly according to previous sections.

Here is the results:

[root@~/usrdata]# valgrind --leak-check=full ./test1
==441== Memcheck, a memory error detector
==441== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==441== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==441== Command: ./test1
==441==
Hey Hey Hey!
Hey Hey Hey!
==441==
==441== HEAP SUMMARY:
==441==     in use at exit: 128 bytes in 3 blocks
==441==   total heap usage: 4 allocs, 1 frees, 132 bytes allocated
==441==
==441== 28 bytes in 2 blocks are definitely lost in loss record 1 of 2
==441==    at 0x4835880: malloc (vg_replace_malloc.c:299)
==441==    by 0x48C2C93: strdup (in /mnt/nfs/ethnfs/sysroot/lib/libc-2.22.so)
==441==
==441== 100 bytes in 1 blocks are definitely lost in loss record 2 of 2
==441==    at 0x4835880: malloc (vg_replace_malloc.c:299)
==441==    by 0x84C7: leak1 (test1.c:11)
==441==    by 0x85CB: main (test1.c:47)
==441==
==441== LEAK SUMMARY:
==441==    definitely lost: 128 bytes in 3 blocks
==441==    indirectly lost: 0 bytes in 0 blocks
==441==      possibly lost: 0 bytes in 0 blocks
==441==    still reachable: 0 bytes in 0 blocks
==441==         suppressed: 0 bytes in 0 blocks
==441==
==441== For counts of detected and suppressed errors, rerun with: -v
==441== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 0 from 0)
[root@Augentix:/usrdata]# valgrind --leak-check=full ./test2
==442== Memcheck, a memory error detector
==442== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==442== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==442== Command: ./test2
==442==
==442== Invalid write of size 1
==442==    at 0x8448: main (test2.c:8)
==442==  Address 0x4988032 is 0 bytes after a block of size 10 alloc'd
==442==    at 0x4835880: malloc (vg_replace_malloc.c:299)
==442==    by 0x8433: main (test2.c:7)
==442==
==442==
==442== HEAP SUMMARY:
==442==     in use at exit: 0 bytes in 0 blocks
==442==   total heap usage: 1 allocs, 1 frees, 10 bytes allocated
==442==
==442== All heap blocks were freed -- no leaks are possible
==442==
==442== For counts of detected and suppressed errors, rerun with: -v
==442== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

Q&A

Failed to start ‘memcheck’ : No such file or directory

valgrind lib export的路徑有問題，檢查VALGRIND_LIB是否設定正確
There is an issue with the path exported for VALGRIND_LIB. Please check if VALGRIND_LIB is set correctly to ensure it points to the proper directory.

export VALGRIND_LIB=/mnt/nfs/ethnfs/valgrind.armv5/lib/valgrind

Compile error

can not be used when making a shared object; recompile with -fPIC

configure時不要套用static參數，改用動態連接
VALGRIND_LIB有可能需要改成連接到libexec下，根據板本的不同
Ans: When configuring, avoid applying static parameters and switch to dynamic linking. The VALGRIND_LIB may need to be changed to link to the libexec directory, depending on the version.

export VALGRIND_LIB=/mnt/nfs/ethnfs/valgrind/libexec/valgrind
./configure --host=arm-augentix-linux-gnueabi --target=arm-augentix-linux-gnueabi --prefix=$HOME/valgrind.armv5

Runtime error

VEX: NEON instructions are not supported for ARMv5.

Ans: Do not apply the armv5.patch during compilation.

VEX: NEON instructions are not supported for ARMv5.
Found: ARMv5-vfp
Cannot continue. Good-bye

valgrind: No such file or directory

請確定 $PATH 所提供的路徑下包含了 valgrind，或改用絕對路徑來執行 valgrind
如果確認程式路徑沒問題之後，還是有問題的話，可能是用了其他 toolchain 編譯的 valgrind，請確認 FW 使用的 toolchain 與編譯 valgrind 的 toolchain 是否相同。

Please ensure that the path provided in $PATH includes valgrind, or use the absolute path to run valgrind .
If the program path is correct but issues still persist, it could be that valgrind was compiled using a different toolchain. Please verify that the toolchain used by the firmware (FW) matches the one used to compile valgrind.

Insufficient memory

If there is insufficient memory for Valgrind to run, it is recommended to switch to a platform with more abundant memory for testing.
Generally, memory leak issues should not be platform-dependent.

Alternatively, you may need to reduce the size of VB (Video Buffer) and CMA (Contiguous Memory Allocator).

如果 Memory 不夠 Valgrind 執行的話，建議改用其他 memory 較充裕的平台測試，一般而言 memory leak 問題應該跟不同平台無關