GStreamer Debugging Using GDB

From RidgeRun Developer Connection
Revision as of 00:55, 15 September 2010 by Tfischer (talk | contribs) (→‎Example Debugging Session)
Jump to: navigation, search

Overview

The focus of the information on this page is to communicate GStreamer shared library debugging, not describing how to build GStreamer or how to use GDB. For that reason, there is a large cut-and-paste set of commands that are not well explained except for the parts that are of interest to GStreamer shared library debugging.

The debugging session is done on a Ubuntu host. The inspiration of the debugging session was to watch a buffer get passed down the GStreamer pipeline. To have GStreamer shared libraries that are built with debug symbols, the libraries had to be rebuilt.

Complexities

Several unexpected challenges were encountered that are worth documenting.

GStreamer Libraries with Debugging Symbols

The Ubuntu Workstation GStreamer shared libraries are not built with debug symbols. In order to set breakpoints and step though the GStreamer shared libraries, they need to be rebuilt.

Not Corrupting Ubuntu Workstation

When we build and install the debug versions of the GStreamer library, we don't want to have that library affecting any other Ubuntu applications, like VLC. To avoid polluting the Ubuntu workstation, we install the custom build in a locale directory using the --prefix configure parameter and then we have to tell the GStreamer application where to find the shared GStreamer libraries from the custom build using the GST_PLUGIN_SYSTEM_PATH shell environment variable.

Breakpoints in Shared Libraries

You can't set a breakpoint in a shared library unless the library is already loaded. To address this issue, the LD_PRELOAD environment variable is set to the list of shared libraries we want pre-loaded for debugging.

Identifying Shared Libraries

You can't list the shared libraries to preload unless you can identify them. A simple way to do that is using 'strace and grabbing just of system calls of interest - namely open() calls.

In a simplistic example (using Ubuntu's normal GStreamer components): 

strace gst-launch videotestsrc num-buffers=30 ! ximagesink 2>&1 | grep '^open.*gst' | fgrep -v ENOENT

Which produces output like: 

open("/usr/lib/libgstreamer-0.10.so.0", O_RDONLY) = 3
open("/home/tfischer/.gstreamer-0.10/registry.i486.bin", O_RDONLY|O_LARGEFILE) = 3
open("/usr/lib/gstreamer-0.10/libgstvideotestsrc.so", O_RDONLY) = 3
open("/usr/lib/libgstbase-0.10.so.0", O_RDONLY) = 3
open("/usr/lib/gstreamer-0.10/libgstximagesink.so", O_RDONLY) = 3
open("/usr/lib/libgstinterfaces-0.10.so.0", O_RDONLY) = 3
open("/usr/lib/libgstvideo-0.10.so.0", O_RDONLY) = 3

GStreamer Registry Cache

Since you will have two sets of GStreamer shared libraries on your system, be careful what ends up in your GStreamer Registry Cache. You can stop the cache from being updated via: 

GST_REGISTRY_UPDATE=no

For this debugging session, I am not sure it makes any difference.

Shell Environment Variables

The syntax of a simple shell command is defined (man bash) as 

 A simple command is a sequence of optional variable assignments followed by blank-separated words and redirections, and terminated  by a control operator.

I used something like 

LD_PRELOAD="$DEVDIR/fs/lib/libgstreamer-0.10" gst-launch testsrc | testsink

Successfully, but when I went to use the debugger, like 

LD_PRELOAD="$DEVDIR/fs/lib/libgstreamer-0.10" gdb gst-launch testsrc | testsink

I found the LD_PRELOAD had no effect.

Instead, I found I had to first set the variable, then on a separate command line, invoke gdb (again, simplified example): 

LD_PRELOAD="$DEVDIR/fs/lib/libgstreamer-0.10"
gdb gst-launch testsrc | testsink

Setup Host Development Directory

# Install host tools

sudo apt-get install liboil0.3-dev strace libdbus-1-dev libdbus-glib-1-dev libreadline-dev unzip


# Create DEVDIR and get needed packages and a movie file

DEVDIR=$HOME/projects/gst

mkdir -p $DEVDIR/downloads $DEVDIR/fs
cd $DEVDIR/downloads
wget http://gstreamer.freedesktop.org/src/gstreamer/gstreamer-0.10.23.tar.bz2
wget http://gstreamer.freedesktop.org/src/gstreamer/gstreamer-0.10.23.tar.bz2.md5
wget http://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-good-0.10.14.tar.bz2
wget http://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-good-0.10.14.tar.bz2.md5
wget http://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-0.10.23.tar.bz2
wget http://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-0.10.23.tar.bz2.md5
md5sum -c *.md5
wget http://downloads.dvdloc8.com/trailers/divxdigest/serenity_hd_dvd-trailer.zip
unzip serenity_hd_dvd-trailer.zip
rm serenity_hd_dvd-trailer.zip
mv "Serenity - HD DVD Trailer.mp4" `echo Serenity - HD DVD Trailer.mp4 | tr ' ' '_'`


# Unpack packages

cd $DEVDIR
tar -xjf downloads/gstreamer-0.10.23.tar.bz2
tar -xjf downloads/gst-plugins-base-0.10.23.tar.bz2
tar -xjf downloads/gst-plugins-good-0.10.14.tar.bz2
git clone git://gstd.git.sourceforge.net/gitroot/gstd/gstd gstd


# Configure and build GStreamer

cd $DEVDIR/gstreamer-0.10.23
CFLAGS=-g \
  ./configure -prefix=$DEVDIR/fs --enable-debug --disable-examples --disable-tests
make -j 3
make install


# Configure and build base plug-ins

cd $DEVDIR/gst-plugins-base-0.10.23
GST_PLUGIN_SYSTEM_PATH=$DEVDIR/fs/lib \
  GST_PLUGIN_PATH= \
  CFLAGS=-g \
  ./configure --prefix=$DEVDIR/fs --enable-debug --disable-examples --disable-tests
make -j 3
make install


# Configure and build good plug-ins

cd $DEVDIR/gst-plugins-good-0.10.14
GST_PLUGIN_SYSTEM_PATH=$DEVDIR/fs/lib \
  GST_PLUGIN_PATH= \
  CFLAGS=-g \
  ./configure --prefix=$DEVDIR/fs --enable-debug --disable-examples --disable-tests
make -j 3
make install


# Configure and build GStreamer Daemon

cd $DEVDIR/gstd
GST_PLUGIN_SYSTEM_PATH=$DEVDIR/fs/lib \
  GST_PLUGIN_PATH= \
  CFLAGS=-g \
  ./configure --prefix=$DEVDIR/fs
make -j 3
sudo make install # need to be root due to connection to system dbus
sudo /etc/init.d/dbus force-reload # activate GStreamer Daemon D-Bus Configuration

Exploring and Verifying Custom GStreamer Build

Setup shell environment

I gathered all the shell environment variables that need to be set, as described in the Complixities section and a few more used for the example pipeline. 

DEVDIR=$HOME/projects/gst
GST_PLUGIN_SYSTEM_PATH=$DEVDIR/fs/lib
GST_REGISTRY_UPDATE=no
LD_PRELOAD="$DEVDIR/fs/lib/libgstreamer-0.10.so.0.20.0 $DEVDIR/fs/lib/gstreamer-0.10/libgstcoreelements.so $DEVDIR/fs/lib/libgstbase-0.10.so.0.20.0 $DEVDIR/fs/lib/gstreamer-0.10/libgstqtdemux.so $DEVDIR/fs/lib/libgstrtp-0.10.so.0.16.0 $DEVDIR/fs/lib/libgsttag-0.10.so.0.16.0 $DEVDIR/fs/lib/libgstaudio-0.10.so.0.16.0"

AV_FILE=$DEVDIR/downloads/Serenity_-_HD_DVD_Trailer.mp4
MULTICAST_IP_ADDR=224.1.1.1
VIDEO_UDP_PORT=5000
PIPELINE="filesrc ! queue ! qtdemux name=dem dem. ! queue ! rtph264pay ! udpsink host=$MULTICAST_IP_ADDR port=$VIDEO_UDP_PORT auto-multicast=true"

List available elements

Should see around 308 features. 

$DEVDIR/fs/bin/gst-inspect

Simple gst-launch video test

Make sure our custom build works. 

$DEVDIR/fs/bin/gst-launch videotestsrc num-buffers=30 ! ximagesink

Verify libraries contain debugging symbols

objdump -h $DEVDIR/fs/lib/libgstreamer-0.10.so.0.20.0

You should see around nine headers that start with .debug_.

Identify which custom plug-ins are being used

strace $DEVDIR/fs/bin/gst-launch $PIPELINE 2>&1 |  grep '^open.*gst' | fgrep -v ENOENT

Verify the output showing GStreamer shared libraries being used only come from $DEVDIR/fs/lib.

Verify GStreamer Daemon is working

killall -9 gstd
$DEVDIR/fs/bin/gstd &

$DEVDIR/fs/bin/gst-client create "videotestsrc ! ffmpegcolorspace ! ximagesink"
$DEVDIR/fs/bin/gst-client -p 0 play
sleep 1
$DEVDIR/fs/bin/gst-client -p 0 destroy
killall -9 gstd

If you see an error like any of the following 

GStreamer-CRITICAL **: gst_type_find_register: assertion `func != NULL' failed
GLib-CRITICAL **: g_once_init_leave: assertion `initialization_value != 0' failed
Streamer-CRITICAL **: gst_element_register: assertion `g_type_is_a (type, GST_TYPE_ELEMENT)' failed

I found they went away if I tried a few more times (weird).

Example Debugging Session

# Start gdb debugging session

killall -9 gstd
gdb $DEVDIR/fs/bin/gstd

# set a gdb breakpoint in filesrc element

b gst_file_src_set_location

# In another windows, create pipeline using gst-client (first remembering to setup the environment)

DEVDIR=$HOME/projects/gst
GST_PLUGIN_SYSTEM_PATH=$DEVDIR/fs/lib
GST_REGISTRY_UPDATE=no
LD_PRELOAD="$DEVDIR/fs/lib/libgstreamer-0.10.so.0.20.0 $DEVDIR/fs/lib/gstreamer-0.10/libgstcoreelements.so $DEVDIR/fs/lib/libgstbase-0.10.so.0.20.0 $DEVDIR/fs/lib/gstreamer-0.10/libgstqtdemux.so $DEVDIR/fs/lib/libgstrtp-0.10.so.0.16.0 $DEVDIR/fs/lib/libgsttag-0.10.so.0.16.0 $DEVDIR/fs/lib/libgstaudio-0.10.so.0.16.0"

AV_FILE=$DEVDIR/downloads/Serenity_-_HD_DVD_Trailer.mp4
MULTICAST_IP_ADDR=224.1.1.1
VIDEO_UDP_PORT=5000
PIPELINE="filesrc name=src ! queue ! qtdemux name=dem dem. ! queue ! rtph264pay ! udpsink host=$MULTICAST_IP_ADDR port=$VIDEO_UDP_PORT auto-multicast=true"

$DEVDIR/fs/bin/gst-client create "$PIPELINE"

# Still in the other window, set the name of the file to be read by the filesrc element

$DEVDIR/fs/bin/gst-client set src location $AV_FILE
Retrieved from "http://developer.ridgerun.com/wiki/index.php?title=GStreamer_Debugging_Using_GDB&oldid=1460"