How to Analyze GStreamer with Valgrind

Initial Preparation

Install Debugging Symbols

Debugging symbols map compiled instructions to the source code. These can be embedded into the same library/application binary or in a separate file. Tools like Valgrind won't be able to provide a thorough analysis unless debugging symbols for the application and its runtime dependencies are available. For Debian based projects, debugging symbols may be installed using APT.

To run a Gstreamer based application through Valgrind, you need to install GStreamer and GLib symbols:

GStreamer

sudo apt install gstreamer1.0-.*dbg gstreamer1.0-tools

Glib

sudo apt install libglib2.0-0-dbg

If the package with debug symbols isn´t available in your system, you should follow these steps to enable it (More info at https://wiki.ubuntu.com/Debug%20Symbol%20Packages):

• Create an /etc/apt/sources.list.d/ddebs.list:

echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list

• Import the debug symbol archive signing key from the Ubuntu server. On Ubuntu 18.04 LTS and newer:

sudo apt install ubuntu-dbgsym-keyring

• Update your package list

sudo apt update

• Install the GLIB packages with debug symbols:

sudo apt install libglib2.0-bin-dbgsym libglib2.0-0-dbgsym libglib2.0-dev-bin-dbgsym

Building your Application with Debug Symbols

Similarly, you need build you application under analysis with debug symbols. Here are a couple of examples for Autotools and Meson:

Meson

# In the following line:
# --prefix /usr
#     This is required when building GStreamer plug-ins. Other applications may ignore this
# c_args
#     If using C
# cpp_args
#     If using C++

meson --prefix /usr -Dc_args="-g -O0" -Dcpp_args="-g -O0" build
ninja -C build
sudo ninja -C build install

Autotools

# In the following line:
# --prefix /usr --libdir /usr/lib/aarch64-linux-gnu
#     This is required when building GStreamer plug-ins. Other applications may ignore this
#     Note that aarch64-linux-gnu is for AARCH64, change accordingly
# CFLAGS
#     If using C
# CXXFLAGS
#     If using C++

./configure --prefix /usr --libdir /usr/lib/aarch64-linux-gnu CFLAGS="-g -O0" CXXFLAGS="-g -O0"
make
sudo make install

Suppressions File

The suppression file is a special file used by Valgrind/Memcheck to ignore certain leaks errors that are either expected or not real. GStreamer provides a suppression file that must be used in order to produce a clean, real report.

Download and save the following file as gst.supp somewhere in your system:

https://gitlab.freedesktop.org/gstreamer/common/-/blob/master/gst.supp

Also, Glib provides a suppression file that must be used alongside the GStreamer suppression file. Download and save the following file as glib.supp somewhere in your system:

https://github.com/GNOME/glib/blob/master/glib.supp

Running Valgrind with GStreamer

This is the recommended recipe to run Valgrind with a GStreamer application:

G_SLICE=always-malloc G_DEBUG=gc-friendly valgrind \
--leak-check=full --leak-resolution=high --num-callers=20 --trace-children=yes \
--suppressions=path/to/gst.supp \
--suppressions=path/to/glib.supp \
gst-launch-0.10 videotestsrc num-buffers=10 ! fakesink

From the command above, these are of special importance:

G_SLICE=always-malloc

Force GLib to use malloc instead of the internal slice allocator.

G_DEBUG=gc-friendly

Initialize all memory blocks in a way that helps memory analyzers.

--suppressions=path/to/gst.supp

Path to the GStreamer suppression file discussed above.

--suppressions=path/to/glib.supp

Path to the Glib suppression file discussed above.

The command above will produce the following output. Note the expected report of 0 bytes definitely lost.

==14217== Memcheck, a memory error detector
==14217== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==14217== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for copyright info
==14217== Command: /opt/gstreamer-0.10/libexec/gstreamer-0.10/gst-plugin-scanner -l
==14217== 
GStreamer has detected that it is running inside valgrind.
It might now take different code paths to ease debugging.
Of course, this may also lead to different bugs.
==14217== 
==14217== HEAP SUMMARY:
==14217==     in use at exit: 74,096 bytes in 1,940 blocks
==14217==   total heap usage: 3,437 allocs, 1,497 frees, 166,891 bytes allocated
==14217== 
==14217== LEAK SUMMARY:
==14217==    definitely lost: 0 bytes in 0 blocks
==14217==    indirectly lost: 120 bytes in 10 blocks
==14217==      possibly lost: 0 bytes in 0 blocks
==14217==    still reachable: 10,158 bytes in 364 blocks
==14217==         suppressed: 63,818 bytes in 1,566 blocks
==14217== Reachable blocks (those to which a pointer was found) are not shown.
==14217== To see them, rerun with: --leak-check=full --show-reachable=yes
==14217== 
==14217== For counts of detected and suppressed errors, rerun with: -v
==14217== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 188 from 139)
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 19212166 ns.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
==14212== 
==14212== HEAP SUMMARY:
==14212==     in use at exit: 285,981 bytes in 3,333 blocks
==14212==   total heap usage: 32,952 allocs, 29,619 frees, 1,766,634 bytes allocated
==14212== 
==14212== LEAK SUMMARY:
==14212==    definitely lost: 0 bytes in 0 blocks
==14212==    indirectly lost: 120 bytes in 10 blocks
==14212==      possibly lost: 0 bytes in 0 blocks
==14212==    still reachable: 14,335 bytes in 480 blocks
==14212==         suppressed: 271,526 bytes in 2,843 blocks
==14212== Reachable blocks (those to which a pointer was found) are not shown.
==14212== To see them, rerun with: --leak-check=full --show-reachable=yes
==14212== 
==14212== For counts of detected and suppressed errors, rerun with: -v
==14212== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 196 from 145)

Valgrind on NVIDIA Jetson and Drive platforms

Based on our experience, we found that the Valgrind package installed with sudo apt install valgrind command reports a leak of definitely-lost 16384 bytes with the above baseline test example. This reported leak report is not reliable, since that basic pipeline should report 0 memory leak, which indicates that there is something wrong with the test setup. This problem was observed on NVIDIA Jetson and NVIDIA DRIVE platforms which runs Ubuntu 18.04 with GStreamer 1.14.5.

To fix this problem is necessary to build and install Valgrind from source code. Below you will find the required steps:

1) Uninstall Valgrind from APT sources:

#Execute the below commands if you already have Valgrind installed in your system
sudo apt-get remove valgrind
sudo apt-get purge valgrind

2) Build and install Valgrind from source code:

git clone git://sourceware.org/git/valgrind.git
cd valgrind
./autogen.sh
./configure --prefix=...
make
sudo make install

If you want a specific version download the tarball and follow the steps in the following examples for version 3.16.1:

wget https://sourceware.org/pub/valgrind/valgrind-3.16.1.tar.bz2
tar xvfj valgrind-3.16.1.tar.bz2
cd valgrind-3.16.1
./configure 
make
sudo make install