Difference between revisions of "GStreamer Debugging"
m (Created page with '= Links = * Good collection of useful hints (including core dumps, using gdb) for GStreamer applications -http://www.buzztard.org/index.php/Debugging * GStreamer application deb...') |
m (→The GStreamer debugging levels include:) |
||
(88 intermediate revisions by 8 users not shown) | |||
Line 1: | Line 1: | ||
− | = | + | <seo title="GStreamer Debugging | GStreamer Element Debugging | Gst debug" titlemode="replace" metakeywords="GStreamer, Linux SDK, Linux BSP, Embedded Linux, Device Drivers, Nvidia, Xilinx, TI, NXP, Freescale, Embedded Linux driver development, Linux Software development, Embedded Linux SDK, Embedded Linux Application development, GStreamer Multimedia Framework, GStreamer Debugging, GStreamer Element Debugging, Gst debug, GStreamer debugging levels, gst tracelib, GStreamer Daemon, GStreamer debug, GST_CAPS, GStreamer buffers" metadescription="This Wiki guide from RidgeRun provides a very useful information about the GStreamer Debugging with different debugging approaches."></seo> |
− | + | == GStreamer debugging approaches == | |
− | |||
− | |||
− | |||
− | + | Some of these GStreamer debugging approaches are only useful when you are running a pipeline and audio and/or video stops at an unexpected place in the data stream. | |
− | + | === Use standard GStreamer debug output with filter === | |
− | + | To see what debug can be enabled, add <tt>--gst-debug-help</tt> to your GStreamer application arguments, such as: | |
− | < | + | <syntaxhighlight lang='bash'> |
− | gst-launch videotestsrc num-buffers=3 ! fakesink --gst-debug=GST_REFCOUNTING:5 --gst-debug-no-color=1 2>&1 | grep "\->0" > log.txt | + | gst-launch --gst-debug-help |
− | </ | + | </syntaxhighlight> |
+ | |||
+ | Then you can see which debug you are interested in, such as reference counting and buffer movement, and enable all debug output (level output 5). | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | gst-launch videotestsrc num-buffers=3 ! fakesink --gst-debug=GST_REFCOUNTING:5,GST_BUFFER:5 --gst-debug-no-color=1 2>&1 | grep "\->0" > log.txt | ||
+ | </syntaxhighlight> | ||
Gets useful data, but typically slows pipeline performance to the point of being not usable. | Gets useful data, but typically slows pipeline performance to the point of being not usable. | ||
− | == Use gst-tracelib library to log key pipeline behavior == | + | Another way you can generate the same output is set the <tt>GST_DEBUG</tt> shell variable: |
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | GST_DEBUG=GST_REFCOUNTING:5 gst-launch videotestsrc num-buffers=3 ! fakesink | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | Here is a simple way to count the number of frames in a (h264/quicktime) video file: | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | MOVIE=slomo_1425510055_2.mov | ||
+ | gst-launch -v filesrc location=$MOVIE ! qtdemux ! ffdec_h264 ! fakesink | fgrep chain | wc -l | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | ==== The GStreamer debugging levels include: ==== | ||
+ | <!------------------- | ||
+ | {| border=2 | ||
+ | | 0 || style="text-align:center" | none || No debug information is output. | ||
+ | |- | ||
+ | | 1 || style="text-align:center; background:black; color:red" | ERROR || Logs all fatal errors. These are errors that do not allow the core or elements to perform the requested action. The application can still recover if programmed to handle the conditions that triggered the error. | ||
+ | |- | ||
+ | | 2 || style="text-align:center; background:black; color:yellow" | WARNING || Logs all warnings. Typically these are non-fatal, but user-visible problems are expected to happen. | ||
+ | |- | ||
+ | | 3 || style="text-align:center; background:black; color:green" | INFO || Logs all informational messages. These are typically used for events in the system that only happen once, or are important and rare enough to be logged at this level. | ||
+ | |- | ||
+ | | 4 || style="text-align:center; background:black; color:aqua" | DEBUG || Logs all debug messages. These are general debug messages for events that happen only a limited number of times during an object's lifetime; these include setup, teardown, and change of parameters, ... | ||
+ | |- | ||
+ | | 5 || style="text-align:center; background:black; color:gray" | LOG || Logs all log messages. These are messages for events that happen repeatedly during an object's lifetime; these include streaming and steady-state conditions. | ||
+ | |- | ||
+ | | 9 || style="text-align:center" | buffer<br>dump || Hex dump of buffer contents. | ||
+ | |} | ||
+ | -------------> | ||
+ | |||
+ | {{GStreamer Debugging Levels}} | ||
+ | |||
+ | ==== Resolving erroneous pipeline could not link ==== | ||
+ | |||
+ | If you get an error like: | ||
+ | |||
+ | WARNING: erroneous pipeline: could not link capsfilter0 to ffenc_mjpeg0 | ||
+ | |||
+ | That means the two elements couldn't exchange GStreamer buffers because they couldn't find a common format they both support. To get an insight on what formats are supported, you can add | ||
+ | |||
+ | --gst-debug=GST_CAPS:4 | ||
+ | |||
+ | === Create DOT file to see exact pipeline and capabilities used === | ||
+ | |||
+ | In order to generate a DOT file, set the GST_DEBUG_DUMP_DOT_DIR environment variable to point to the folder where you want the .dot files to be stored. | ||
+ | |||
+ | 1) Install "dot" tool on your host machine...to do so simply install graphviz: | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | sudo apt-get install graphviz | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | 2) On your target board, set the environment dot dump variable: | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | export GST_DEBUG_DUMP_DOT_DIR=/root | ||
+ | mkdir -p $GST_DEBUG_DUMP_DOT_DIR | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | 3) Run your pipeline, a .dot file will be generated according to each of the state changes that occur showing how the caps are negotiated during the process. | ||
+ | |||
+ | 4) Convert the dot files to PNG image files or SVG graphics files | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | dot -Tpng input.dot > output.png | ||
+ | |||
+ | dot -Tsvg input.dot > output.svg | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | Using SVG allows you to infinitely zoom to be able to read all the text. You can use inkscape to view and edit SVG files. | ||
+ | |||
+ | === Use gst-tracelib library to log key pipeline behavior === | ||
− | + | The gst-tracelib library hooks into GStreamer key functions and logs the function usage behavior. When the application exits it displays some general statistics. Further analysis can be done based on the data written to the log file. Detailed usage description in the source code [http://cgit.freedesktop.org/~ensonic/gst-tracelib/tree/README README] file. | |
gst-tracelib logs | gst-tracelib logs | ||
− | * dataflow, messages, queries and events | + | * dataflow, messages, queries, and events |
− | * caps set | + | * caps set and get activity |
* pipeline topology changes | * pipeline topology changes | ||
* resource usage | * resource usage | ||
Line 30: | Line 106: | ||
Example usage: | Example usage: | ||
− | < | + | <syntaxhighlight lang='bash'> |
export GSTTL_HIDE="caps;chk;topo" | export GSTTL_HIDE="caps;chk;topo" | ||
export GSTTL_LOG_SIZE=1048576 | export GSTTL_LOG_SIZE=1048576 | ||
AV_FILE=/SD/content/MoMen-dm365.mov | AV_FILE=/SD/content/MoMen-dm365.mov | ||
− | LD_PRELOAD=/usr/lib/gst-tracelib/libgsttracelib.so | + | LD_PRELOAD=/usr/lib/gst-tracelib/libgsttracelib.so gst-launch filesrc location = $AV_FILE ! qtdemux name=demux ! queue ! dmaidec_h264 numOutputBufs=12 ! \ |
priority nice=-10 ! queue ! priority nice=-10 ! dmaiperf ! TIDmaiVideoSink accelFrameCopy=true \ | priority nice=-10 ! queue ! priority nice=-10 ! dmaiperf ! TIDmaiVideoSink accelFrameCopy=true \ | ||
videoOutput=DVI videoStd=720P_60 demux.audio_00 ! queue ! priority nice=-5 ! dmaidec_aac ! alsasink | videoOutput=DVI videoStd=720P_60 demux.audio_00 ! queue ! priority nice=-5 ! dmaidec_aac ! alsasink | ||
− | </ | + | </syntaxhighlight> |
− | By default, the log file is named <tt>/tmp/gsttl.log</tt>. | + | By default, the log file is named <tt>/tmp/gsttl.log</tt>. Copy the log file somewhere on your host PC where the scripts gsttl_splitlog.py and gsttl_plt.sh are available (or include the path where the scripts are on $PATH). You can use the following command to create a png file with the information gathered by gst-tracelib: |
− | == Watch system interrupts == | + | rm -rf gsttl gsttl.png; gsttl_splitlog.py; gsttl_plot.sh | gnuplot; evince gsttl.png |
+ | |||
+ | === Watch system interrupts === | ||
Run the telnet daemon on the target - likely: | Run the telnet daemon on the target - likely: | ||
− | < | + | <syntaxhighlight lang='bash'> |
/etc/init.d/inetd start | /etc/init.d/inetd start | ||
Line 52: | Line 130: | ||
inetd | inetd | ||
− | </ | + | </syntaxhighlight> |
telnet into the target hardware and watch the interrupt count | telnet into the target hardware and watch the interrupt count | ||
− | < | + | <syntaxhighlight lang='bash'> |
while sleep 1 ; do cat /proc/interrupts ; done | while sleep 1 ; do cat /proc/interrupts ; done | ||
− | </ | + | </syntaxhighlight> |
− | If the pipeline is | + | If the pipeline is supposed to be running, the changes in the interrupt count may provide clues as to what is going on. |
− | == Capturing a core dump == | + | === Capturing a core dump === |
If your GStreamer application is crashing with a segfault or similar condition, enable saving a core dump before running the application. | If your GStreamer application is crashing with a segfault or similar condition, enable saving a core dump before running the application. | ||
− | < | + | <syntaxhighlight lang='bash'> |
ulimit -c 10000 | ulimit -c 10000 | ||
mkdir -m 777 /root/dumps | mkdir -m 777 /root/dumps | ||
echo "/root/dumps/%e.core" > /proc/sys/kernel/core_pattern | echo "/root/dumps/%e.core" > /proc/sys/kernel/core_pattern | ||
− | </ | + | </syntaxhighlight> |
− | Once you have a <tt>/root/dumps/*.core</tt> file, copy it to your and and inspect it with | + | Once you have a <tt>/root/dumps/*.core</tt> file, copy it to your workstation and and inspect it with |
− | < | + | <syntaxhighlight lang='bash'> |
ddd -debugger arm-linux-gnueabi-gdb $GSTREAMER_APPLIATION | ddd -debugger arm-linux-gnueabi-gdb $GSTREAMER_APPLIATION | ||
− | </ | + | </syntaxhighlight> |
Then in gdb, | Then in gdb, | ||
− | < | + | <syntaxhighlight lang='bash'> |
target core <core file> | target core <core file> | ||
bt | bt | ||
− | </ | + | </syntaxhighlight> |
and see what function caused the core dump. | and see what function caused the core dump. | ||
− | == | + | === Building application and GStreamer libraries with debug enabled === |
− | + | Before using gdb, you need to build your application and the GStreamer libraries with debug enabled. | |
+ | |||
+ | * Make sure you optimization level is not set to <tt>-O3</tt> | ||
+ | * Add <tt>-ggdb</tt> to your <tt>CFLAGS</tt> | ||
+ | * Remove <tt>-fomit-frame-pointer</tt> flag from <tt>CFLAGS</tt> | ||
+ | |||
+ | For TI DMAI plugin and other gstreamer components in the RidgeRun SDK, the can be done via: | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | cd $DEVDIR/proprietary/gst-dmai-plugins | ||
+ | make clean | ||
+ | make APPS_CFLAGS=-ggdb build install | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | === Use gdb to attach to GStreamer application and examine all the threads === | ||
+ | |||
+ | The [[Debug and Profiling Guide#Software debugging with GDB | SDK Debugging Guide]] provides detailed instructions. | ||
Built your application with symbols (-g) and no optimization (-O0). Use GStreamer libraries that are built with symbols. | Built your application with symbols (-g) and no optimization (-O0). Use GStreamer libraries that are built with symbols. | ||
Line 95: | Line 189: | ||
Attach to your running GStreamer application using gdbserver | Attach to your running GStreamer application using gdbserver | ||
− | < | + | <syntaxhighlight lang='bash'> |
ps | ps | ||
PID=4512 # set the right value based on your application's PID | PID=4512 # set the right value based on your application's PID | ||
gdbserver :2345 --attach $PID | gdbserver :2345 --attach $PID | ||
− | </ | + | </syntaxhighlight> |
− | Then start the cross compile version of the GNU debugger, like | + | Then start the cross-compile version of the GNU debugger, like |
− | < | + | <syntaxhighlight lang='bash'> |
− | + | arm-linux-gnueabi-gdb -tui $GSTREAMER_APP | |
− | </ | + | </syntaxhighlight> |
and get the gdb debugger connected to gdb server on the target | and get the gdb debugger connected to gdb server on the target | ||
− | < | + | <syntaxhighlight lang='bash'> |
set solib-absolute-prefix <path to devdir>/fs/fs | set solib-absolute-prefix <path to devdir>/fs/fs | ||
file <path to file> | file <path to file> | ||
target remote <ip address>:2345 | target remote <ip address>:2345 | ||
+ | </syntaxhighlight> | ||
− | + | List the threads and do a backtrace on each one | |
− | |||
− | List the threads and do a | ||
− | < | + | <syntaxhighlight lang='bash'> |
info threads | info threads | ||
bt | bt | ||
thread 2 | thread 2 | ||
bt | bt | ||
− | </ | + | </syntaxhighlight> |
+ | |||
+ | === Use gdbserver in multi-process mode to attach to GStreamer application and examine all the threads === | ||
+ | |||
+ | If the <tt>info threads</tt> command only shows one thread running, you can see if using gstserver in multi-process mode will work better. | ||
+ | |||
+ | On the target (assuming your program's name is <tt>gstd</tt>), first list the PIDs for all the threads in your application: | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | PROG=gstd | ||
+ | ls /proc/`pidof $PROG`/task | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | Then start gdbserver in multi-process mode: | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | gdbserver --multi :2345 | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | Then on the host start the cross-compile version of the GNU debugger, like | ||
− | == Exit locked GStreamer application and see what works stand alone == | + | <syntaxhighlight lang='bash'> |
+ | arm-linux-gnueabi-gdb -tui $GSTREAMER_APP | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | and get the gdb debugger connected to gdb server on the target | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | set solib-absolute-prefix <path to devdir>/fs/fs | ||
+ | file <path to file> | ||
+ | target extended-remote <ip address>:2345 | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | Now you can attach to a specific PID, do a backtrack, snoop around, and detach from that PID. Once detached, you can do the same thing again using a different PID from the output of the <tt>ls</tt> command above. | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | attach <pid> | ||
+ | bt | ||
+ | detach | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | Once you detach, the PID will run normally. | ||
+ | |||
+ | === Exit locked GStreamer application and see what works stand-alone === | ||
If you have a GStreamer application that locks up and doesn't run correctly | If you have a GStreamer application that locks up and doesn't run correctly | ||
− | even after you exit the program (possibly with | + | even after you exit the program (possibly with Ctrl-C) and restart, then |
− | it is possible some kernel provided resource is the | + | it is possible some kernel provided resource is the culprit. For example, |
if you are using a defective ALSA audio out driver, you might find the | if you are using a defective ALSA audio out driver, you might find the | ||
GStreamer pipeline locks up in the middle. If you exit the GStreamer application | GStreamer pipeline locks up in the middle. If you exit the GStreamer application | ||
Line 135: | Line 269: | ||
to identify the source of your problem. | to identify the source of your problem. | ||
− | == Examine | + | === Examine history of what transpired just prior to lockup === |
If you have a GStreamer application that locks up and can change the pipeline | If you have a GStreamer application that locks up and can change the pipeline | ||
− | to include a (at this point mythical) recent activity history logger | + | to include a (at this point mythical) recent activity history logger, such a |
logger element could be put anywhere in the pipeline. The logger would have | logger element could be put anywhere in the pipeline. The logger would have | ||
circular buffers to keep track of all potentially interesting recent history, | circular buffers to keep track of all potentially interesting recent history, | ||
Line 147: | Line 281: | ||
This idea is after the pipeline locks up, you could cause the history logger | This idea is after the pipeline locks up, you could cause the history logger | ||
− | to dump | + | to dump its data, and then get an idea of what is supposed to be happening that |
− | isn't | + | isn't occurring. |
+ | |||
+ | === Activate verbose from application === | ||
+ | |||
+ | Some elements (like fakesink and identity) print out information when you enable the verbose output (running the pipeline through gst-launch with the option -v), in order to activate this output manually from a C application you can use: | ||
+ | |||
+ | g_signal_connect( pipeline, "deep-notify", G_CALLBACK( gst_object_default_deep_notify ), NULL ); | ||
+ | |||
+ | === Enable DMAI Davinci Multimedia Application Interface debug output === | ||
+ | |||
+ | If you are using the TI DMAI GStreamer plug-in for the TI Davinci and OMAP SoCs, then you can enable DMAI debug to get more information if a hardware-accelerated encoder or decoder is throwing an error. | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | DMAI_DEBUG=2 gst-launch -e alsasrc num-buffers=50 ! 'audio/x-raw-int,rate=(int)44100,channels=(int)2' ! queue ! dmaienc_aac bitrate=128000 ! qtmux ! filesink location=gstaudio_aac_2ch_128k.mp4 | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | === Enable CE Codec Engine debug output === | ||
+ | |||
+ | If you are using the TI DMAI GStreamer plug-in for the TI Davinci and OMAP SoCs, then you can enable CE debug to get more information if a hardware-accelerated encoder or decoder is throwing an error. Note that the stack is GStreamer DMAI plug-in which uses DMAI which uses CE which uses codecs that take advantage of the hardware accelerators. | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | CE_DEBUG=3 gst-launch -e alsasrc num-buffers=50 ! 'audio/x-raw-int,rate=(int)44100,channels=(int)2' ! queue ! dmaienc_aac bitrate=128000 ! qtmux ! filesink location=gstaudio_aac_2ch_128k.mp4 | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | And you can combine DMAI and CE debug | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | CE_DEBUG=3 DMAI_DEBUG=2 gst-launch -e alsasrc num-buffers=50 ! 'audio/x-raw-int,rate=(int)44100,channels=(int)2' ! queue ! dmaienc_aac bitrate=128000 ! qtmux ! filesink location=gstaudio_aac_2ch_128k.mp4 | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | To write the debug to a file on tmpfs, which should reduce the performance impact of enabling debug, add | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | 2>&1 | cat - > /tmp/gst-debug.txt | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | to the end of the gst-launch command. | ||
+ | |||
+ | === Use NFS mount to make it easy to transfer files === | ||
+ | |||
+ | If you are creating files on a network-connected embedded device, you can use NFS mount (which is different than NFS root mount) so that once you generate a file, it is easy to get it on your desktop machine. For any real-time debugging, don't use NFS root mount as the network activity will affect the timing of your pipelines. See [[Setting Up A NFS Service]] for instructions on configuring your desktop machine. | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | HOST_IP=10.0.1.1 | ||
+ | HOST_NFS_SHARE_DIR=/local/home/tfischer/work/sdk/fs/fs | ||
+ | |||
+ | mkdir /tmp/nfs | ||
+ | mount -t nfs -n -o nolock,rsize=1024,wsize=1024 $HOST_IP:$HOST_NFS_SHARE_DIR /tmp/nfs | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | Now you can copy files between host and target over the network. If you have a file like <tt>/tmp/debug-out.txt</tt> on the target, you can copy it to the $HOST_NFS_SHARE_DIR on the host using | ||
+ | |||
+ | cp /tmp/debug-out.txt /tmp/nfs | ||
+ | |||
+ | === Look for memory leaks === | ||
+ | |||
+ | Looking for memory leaks in programs is a very challenging problem. This is even more true for embedded systems, since some of the best memory analysis tools aren't always available, or are difficult to use in this environment. | ||
+ | |||
+ | An easy way to debug the memory leak of a GStreamer program that is running is by looking at the reference counters. Memory leaks on GStreamer happened because reference counters on one or more objects never go back to zero. | ||
+ | |||
+ | References: | ||
+ | |||
+ | * [http://www.linuxjournal.com/article/6059 Linux Journal article - Memory Leak Detection in Embedded Systems] | ||
+ | |||
+ | ==== GStreamer <tt>GST_REFCOUNTING</tt> Debug Variable ==== | ||
+ | |||
+ | One of the debug variables is <tt>GST_REFCOUNTING</tt>. With that, you can get a log of all the refcount changes that happened inside a gst application. | ||
+ | |||
+ | ==== Setting <tt>GST_REFCOUNTING</tt> ==== | ||
+ | |||
+ | Add the following to your application's command line parameters: | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | --gst-debug=GST_REFCOUNTING:5 | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | Another option is to set the GStreamer debugging shell variable, then run your program: | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | export GST_DEBUG=GST_REFCOUNTING:5 | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | ==== Filtering debug output ==== | ||
+ | |||
+ | Here are several examples of how to filter the debug output. | ||
+ | |||
+ | You can see which objects disappear, meaning their reference count reaches zero: | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | gst-launch --gst-debug=GST_REFCOUNTING:5 videotestsrc num-buffers=3 ! fakesink 2>&1 | grep "\->0" | cut -d' ' -f 19 | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | In useful applications, there are always some objects that aren't freed until the program exists. With a simple gst-hello example all the objects are always freed. If you set the pipe to run, let's say 13 times, and in the log, you see one object that wasn't freed and the count reached 13 or more, then you have a suspicious object or if you see that the number of non-freed objects increases as you increase the number of runs, then you have an indication of a problem. | ||
+ | |||
+ | === Use remote syslog === | ||
+ | |||
+ | If your application logs warnings and errors, use [[How to Configure Remote Syslog Logging | remote syslog]] to make the information more easily available. | ||
+ | |||
+ | If your application generates output, you can redirect it to Syslog as follows: | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | MY_APP=gst-render | ||
+ | |||
+ | $MY_APP | logger -t "$MY_APP:" | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | === Generated files debug === | ||
+ | |||
+ | There are some tools that can be used for getting information from generated files by GStreamer: | ||
+ | |||
+ | *libmp4v2 | ||
+ | |||
+ | *MP4BOX | ||
+ | |||
+ | *mediainfo | ||
+ | |||
+ | With MP4Box you can use the -info in order to get more info from a generated mp4 file: | ||
+ | <syntaxhighlight lang='bash'> | ||
+ | MP4Box -info test.mp4 | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | == GStreamer Element Debugging == | ||
+ | |||
+ | If you are creating an element you plan to use in an embedded device, you might find it helpful to build and test the element on your development machine first. | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | ./autogen.sh | ||
+ | CFLAGS=-ggdb ./configure | ||
+ | make | ||
+ | objdump -x src/.libs/*.so | fgrep DEBUG # verify library is built with debugging symbols enabled | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | Use GST_PLUGIN_PATH to enable GStreamer to find your element library: | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | export GST_PLUGIN_PATH=$HOME/projects/gst-emboverlay/src/.libs | ||
+ | gst-inspect emboverlay | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | Test out your element. Better to use GStreamer daemon than gst-launch. | ||
+ | |||
+ | <syntaxhighlight lang='bash'> | ||
+ | TEXT_OVERLAY="text=Hello\ There text-offseth=50 text-offsetv=100 text-font-height=50 text-color=0x800040 text-enable=true" | ||
+ | TEXT2_OVERLAY="text2=World text2-offseth=50 text2-offsetv=150 text2-font-height=50 text2-color=0x800040 text2-enable" | ||
+ | TIME_OVERLAY="time-param=%b\ %d\ %Y\ %H:%M:%S time-offseth=50 time-offsetv=200 time-font-height=50 time-color=0x800040 time-enable=true" | ||
+ | LOGO_OVERLAY="logo=$HOME/projects/gst-emboverlay/RidgeRunLogo312x84-with-alpha-channel.png logo-offseth=-1 logo-offsetv=-1 logo-transp=1 logo-scale=1.0 logo-autofit=true time-enable=true" | ||
+ | FRAME_OVERLAY="frame-number=-20 frame-offseth=50 frame-offsetv=250 frame-font-height=50 frame-color=0x800040 frame-enable=true" | ||
+ | |||
+ | CAPS="video/x-raw-yuv, format=(fourcc)YV12, width=(int)1280, height=(int)720, interlaced=(boolean)false, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)10/1" | ||
+ | |||
+ | gst-launch --gst-debug=emboverlay:4 -v v4l2src num-buffers=60 ! capsfilter caps="$CAPS" ! emboverlay $TIME_OVERLAY $TEXT_OVERLAY $TEXT2_OVERLAY $LOGO_OVERLAY $FRAME_OVERLAY ! xvimagesink | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | == GStreamer Performance Tuning == | ||
+ | |||
+ | This is worthy of its own [[Embedded GStreamer Performance Tuning]] topic. | ||
+ | |||
+ | == GStreamer Application Development == | ||
+ | |||
+ | === GStreamer Daemon === | ||
+ | |||
+ | If you can use [https://github.com/RidgeRun/gstd/wiki GStreamer Daemon] as your streaming media server, then avoid writing a custom application in the first place! | ||
− | == | + | === Developing on host computer === |
− | If you have a | + | If you have a pipeline where source pads and sink pads are getting connected and disconnected, then you will likely need to write a custom streaming media server. Most people find it easier to build and debug by running the application on the host before trying it on the target hardware. |
− | + | == See also == | |
− | + | * '''GStreamer Basic tutorial 11: Debugging Tools''' gstreamer.freedesktop.org tutorial documentation.URL: https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html | |
− | + | * '''Running GStreamer Applications — How to run and debug your GStreamer application''' gstreamer.freedesktop.org documentation. URL: http://www.gstreamer.net/data/doc/gstreamer/head/gstreamer/html/gst-running.html | |
+ | * '''Things to check when writing an element''' gstreamer.freedesktop.org plugin development documentation.URL: https://gstreamer.freedesktop.org/documentation/plugin-development/appendix/checklist-element.html | ||
+ | * '''GstInfo - Debugging and logging facilities''' gstreamer.freedesktop.org GStreamer 1.0 Core Reference Manual.URL: http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gstreamer-GstInfo.html | ||
+ | * '''gst-tracelib - Performance tracing library for GStreamer''' GStreamer tracing library.URL: http://cgit.freedesktop.org/~ensonic/gst-tracelib/tree/README | ||
+ | * '''GStreamer Debugging Using GDB'''. URL: https://developer.ridgerun.com/wiki/index.php/GStreamer_Debugging_Using_GDB | ||
+ | {{ContactUs}} | ||
− | [[Category:GStreamer]] | + | [[Category:GStreamer]][[Category:Debug]][[Category:HowTo]][[Category:Whitepaper]] |
Latest revision as of 21:48, 5 March 2023
Contents
- 1 GStreamer debugging approaches
- 1.1 Use standard GStreamer debug output with filter
- 1.2 Create DOT file to see exact pipeline and capabilities used
- 1.3 Use gst-tracelib library to log key pipeline behavior
- 1.4 Watch system interrupts
- 1.5 Capturing a core dump
- 1.6 Building application and GStreamer libraries with debug enabled
- 1.7 Use gdb to attach to GStreamer application and examine all the threads
- 1.8 Use gdbserver in multi-process mode to attach to GStreamer application and examine all the threads
- 1.9 Exit locked GStreamer application and see what works stand-alone
- 1.10 Examine history of what transpired just prior to lockup
- 1.11 Activate verbose from application
- 1.12 Enable DMAI Davinci Multimedia Application Interface debug output
- 1.13 Enable CE Codec Engine debug output
- 1.14 Use NFS mount to make it easy to transfer files
- 1.15 Look for memory leaks
- 1.16 Use remote syslog
- 1.17 Generated files debug
- 2 GStreamer Element Debugging
- 3 GStreamer Performance Tuning
- 4 GStreamer Application Development
- 5 See also
- 6 Contact Us
GStreamer debugging approaches
Some of these GStreamer debugging approaches are only useful when you are running a pipeline and audio and/or video stops at an unexpected place in the data stream.
Use standard GStreamer debug output with filter
To see what debug can be enabled, add --gst-debug-help to your GStreamer application arguments, such as:
gst-launch --gst-debug-help
Then you can see which debug you are interested in, such as reference counting and buffer movement, and enable all debug output (level output 5).
gst-launch videotestsrc num-buffers=3 ! fakesink --gst-debug=GST_REFCOUNTING:5,GST_BUFFER:5 --gst-debug-no-color=1 2>&1 | grep "\->0" > log.txt
Gets useful data, but typically slows pipeline performance to the point of being not usable.
Another way you can generate the same output is set the GST_DEBUG shell variable:
GST_DEBUG=GST_REFCOUNTING:5 gst-launch videotestsrc num-buffers=3 ! fakesink
Here is a simple way to count the number of frames in a (h264/quicktime) video file:
MOVIE=slomo_1425510055_2.mov
gst-launch -v filesrc location=$MOVIE ! qtdemux ! ffdec_h264 ! fakesink | fgrep chain | wc -l
The GStreamer debugging levels include:
The GStreamer debugging levels are the following ones:
0 | none | No debug information is output. |
1 | ERROR | Logs all fatal errors. These are errors that do not allow the core or elements to perform the requested action. The application can still recover if programmed to handle the conditions that triggered the error. |
2 | WARNING | Logs all warnings. Typically these are non-fatal, but user-visible problems are expected to happen. |
3 | FIXME | Logs all fixme messages. Fixme messages are messages that indicate that something in the executed code path is not fully implemented or handled yet. The purpose of this message is to make it easier to spot incomplete/unfinished pieces of code when reading the debug log. |
4 | INFO | Logs all informational messages. These are typically used for events in the system that only happen once, or are important and rare enough to be logged at this level. |
5 | DEBUG | Logs all debug messages. These are general debug messages for events that happen only a limited number of times during an object's lifetime; these include setup, teardown, change of parameters, ... |
6 | LOG | Logs all log messages. These are messages for events that happen repeatedly during an object's lifetime; these include streaming and steady-state conditions. |
7 | TRACE | Logs all trace messages. These messages for events that happen repeatedly during an object's lifetime such as the ref/unref cycles. |
9 | MEMDUMP | Log all memory dump messages. Memory dump messages are used to log (small) chunks of data as memory dumps in the log. They will be displayed as hexdump with ASCII characters. |
Resolving erroneous pipeline could not link
If you get an error like:
WARNING: erroneous pipeline: could not link capsfilter0 to ffenc_mjpeg0
That means the two elements couldn't exchange GStreamer buffers because they couldn't find a common format they both support. To get an insight on what formats are supported, you can add
--gst-debug=GST_CAPS:4
Create DOT file to see exact pipeline and capabilities used
In order to generate a DOT file, set the GST_DEBUG_DUMP_DOT_DIR environment variable to point to the folder where you want the .dot files to be stored.
1) Install "dot" tool on your host machine...to do so simply install graphviz:
sudo apt-get install graphviz
2) On your target board, set the environment dot dump variable:
export GST_DEBUG_DUMP_DOT_DIR=/root
mkdir -p $GST_DEBUG_DUMP_DOT_DIR
3) Run your pipeline, a .dot file will be generated according to each of the state changes that occur showing how the caps are negotiated during the process.
4) Convert the dot files to PNG image files or SVG graphics files
dot -Tpng input.dot > output.png
dot -Tsvg input.dot > output.svg
Using SVG allows you to infinitely zoom to be able to read all the text. You can use inkscape to view and edit SVG files.
Use gst-tracelib library to log key pipeline behavior
The gst-tracelib library hooks into GStreamer key functions and logs the function usage behavior. When the application exits it displays some general statistics. Further analysis can be done based on the data written to the log file. Detailed usage description in the source code README file.
gst-tracelib logs
- dataflow, messages, queries, and events
- caps set and get activity
- pipeline topology changes
- resource usage
Example usage:
export GSTTL_HIDE="caps;chk;topo"
export GSTTL_LOG_SIZE=1048576
AV_FILE=/SD/content/MoMen-dm365.mov
LD_PRELOAD=/usr/lib/gst-tracelib/libgsttracelib.so gst-launch filesrc location = $AV_FILE ! qtdemux name=demux ! queue ! dmaidec_h264 numOutputBufs=12 ! \
priority nice=-10 ! queue ! priority nice=-10 ! dmaiperf ! TIDmaiVideoSink accelFrameCopy=true \
videoOutput=DVI videoStd=720P_60 demux.audio_00 ! queue ! priority nice=-5 ! dmaidec_aac ! alsasink
By default, the log file is named /tmp/gsttl.log. Copy the log file somewhere on your host PC where the scripts gsttl_splitlog.py and gsttl_plt.sh are available (or include the path where the scripts are on $PATH). You can use the following command to create a png file with the information gathered by gst-tracelib:
rm -rf gsttl gsttl.png; gsttl_splitlog.py; gsttl_plot.sh | gnuplot; evince gsttl.png
Watch system interrupts
Run the telnet daemon on the target - likely:
/etc/init.d/inetd start
# or
inetd
telnet into the target hardware and watch the interrupt count
while sleep 1 ; do cat /proc/interrupts ; done
If the pipeline is supposed to be running, the changes in the interrupt count may provide clues as to what is going on.
Capturing a core dump
If your GStreamer application is crashing with a segfault or similar condition, enable saving a core dump before running the application.
ulimit -c 10000
mkdir -m 777 /root/dumps
echo "/root/dumps/%e.core" > /proc/sys/kernel/core_pattern
Once you have a /root/dumps/*.core file, copy it to your workstation and and inspect it with
ddd -debugger arm-linux-gnueabi-gdb $GSTREAMER_APPLIATION
Then in gdb,
target core <core file>
bt
and see what function caused the core dump.
Building application and GStreamer libraries with debug enabled
Before using gdb, you need to build your application and the GStreamer libraries with debug enabled.
- Make sure you optimization level is not set to -O3
- Add -ggdb to your CFLAGS
- Remove -fomit-frame-pointer flag from CFLAGS
For TI DMAI plugin and other gstreamer components in the RidgeRun SDK, the can be done via:
cd $DEVDIR/proprietary/gst-dmai-plugins
make clean
make APPS_CFLAGS=-ggdb build install
Use gdb to attach to GStreamer application and examine all the threads
The SDK Debugging Guide provides detailed instructions.
Built your application with symbols (-g) and no optimization (-O0). Use GStreamer libraries that are built with symbols.
Attach to your running GStreamer application using gdbserver
ps
PID=4512 # set the right value based on your application's PID
gdbserver :2345 --attach $PID
Then start the cross-compile version of the GNU debugger, like
arm-linux-gnueabi-gdb -tui $GSTREAMER_APP
and get the gdb debugger connected to gdb server on the target
set solib-absolute-prefix <path to devdir>/fs/fs
file <path to file>
target remote <ip address>:2345
List the threads and do a backtrace on each one
info threads
bt
thread 2
bt
Use gdbserver in multi-process mode to attach to GStreamer application and examine all the threads
If the info threads command only shows one thread running, you can see if using gstserver in multi-process mode will work better.
On the target (assuming your program's name is gstd), first list the PIDs for all the threads in your application:
PROG=gstd
ls /proc/`pidof $PROG`/task
Then start gdbserver in multi-process mode:
gdbserver --multi :2345
Then on the host start the cross-compile version of the GNU debugger, like
arm-linux-gnueabi-gdb -tui $GSTREAMER_APP
and get the gdb debugger connected to gdb server on the target
set solib-absolute-prefix <path to devdir>/fs/fs
file <path to file>
target extended-remote <ip address>:2345
Now you can attach to a specific PID, do a backtrack, snoop around, and detach from that PID. Once detached, you can do the same thing again using a different PID from the output of the ls command above.
attach <pid>
bt
detach
Once you detach, the PID will run normally.
Exit locked GStreamer application and see what works stand-alone
If you have a GStreamer application that locks up and doesn't run correctly even after you exit the program (possibly with Ctrl-C) and restart, then it is possible some kernel provided resource is the culprit. For example, if you are using a defective ALSA audio out driver, you might find the GStreamer pipeline locks up in the middle. If you exit the GStreamer application and try a simple audio application, like aplay, you might be able to identify the source of your problem.
Examine history of what transpired just prior to lockup
If you have a GStreamer application that locks up and can change the pipeline to include a (at this point mythical) recent activity history logger, such a logger element could be put anywhere in the pipeline. The logger would have circular buffers to keep track of all potentially interesting recent history, such as pad activity, bus activity, and any other relevant information. The circular buffer entries would all be timestamped. When some event occurs (a file exists, a message/signal is received, etc), the element would dump the history, and continue capturing new data.
This idea is after the pipeline locks up, you could cause the history logger to dump its data, and then get an idea of what is supposed to be happening that isn't occurring.
Activate verbose from application
Some elements (like fakesink and identity) print out information when you enable the verbose output (running the pipeline through gst-launch with the option -v), in order to activate this output manually from a C application you can use:
g_signal_connect( pipeline, "deep-notify", G_CALLBACK( gst_object_default_deep_notify ), NULL );
Enable DMAI Davinci Multimedia Application Interface debug output
If you are using the TI DMAI GStreamer plug-in for the TI Davinci and OMAP SoCs, then you can enable DMAI debug to get more information if a hardware-accelerated encoder or decoder is throwing an error.
DMAI_DEBUG=2 gst-launch -e alsasrc num-buffers=50 ! 'audio/x-raw-int,rate=(int)44100,channels=(int)2' ! queue ! dmaienc_aac bitrate=128000 ! qtmux ! filesink location=gstaudio_aac_2ch_128k.mp4
Enable CE Codec Engine debug output
If you are using the TI DMAI GStreamer plug-in for the TI Davinci and OMAP SoCs, then you can enable CE debug to get more information if a hardware-accelerated encoder or decoder is throwing an error. Note that the stack is GStreamer DMAI plug-in which uses DMAI which uses CE which uses codecs that take advantage of the hardware accelerators.
CE_DEBUG=3 gst-launch -e alsasrc num-buffers=50 ! 'audio/x-raw-int,rate=(int)44100,channels=(int)2' ! queue ! dmaienc_aac bitrate=128000 ! qtmux ! filesink location=gstaudio_aac_2ch_128k.mp4
And you can combine DMAI and CE debug
CE_DEBUG=3 DMAI_DEBUG=2 gst-launch -e alsasrc num-buffers=50 ! 'audio/x-raw-int,rate=(int)44100,channels=(int)2' ! queue ! dmaienc_aac bitrate=128000 ! qtmux ! filesink location=gstaudio_aac_2ch_128k.mp4
To write the debug to a file on tmpfs, which should reduce the performance impact of enabling debug, add
2>&1 | cat - > /tmp/gst-debug.txt
to the end of the gst-launch command.
Use NFS mount to make it easy to transfer files
If you are creating files on a network-connected embedded device, you can use NFS mount (which is different than NFS root mount) so that once you generate a file, it is easy to get it on your desktop machine. For any real-time debugging, don't use NFS root mount as the network activity will affect the timing of your pipelines. See Setting Up A NFS Service for instructions on configuring your desktop machine.
HOST_IP=10.0.1.1
HOST_NFS_SHARE_DIR=/local/home/tfischer/work/sdk/fs/fs
mkdir /tmp/nfs
mount -t nfs -n -o nolock,rsize=1024,wsize=1024 $HOST_IP:$HOST_NFS_SHARE_DIR /tmp/nfs
Now you can copy files between host and target over the network. If you have a file like /tmp/debug-out.txt on the target, you can copy it to the $HOST_NFS_SHARE_DIR on the host using
cp /tmp/debug-out.txt /tmp/nfs
Look for memory leaks
Looking for memory leaks in programs is a very challenging problem. This is even more true for embedded systems, since some of the best memory analysis tools aren't always available, or are difficult to use in this environment.
An easy way to debug the memory leak of a GStreamer program that is running is by looking at the reference counters. Memory leaks on GStreamer happened because reference counters on one or more objects never go back to zero.
References:
GStreamer GST_REFCOUNTING Debug Variable
One of the debug variables is GST_REFCOUNTING. With that, you can get a log of all the refcount changes that happened inside a gst application.
Setting GST_REFCOUNTING
Add the following to your application's command line parameters:
--gst-debug=GST_REFCOUNTING:5
Another option is to set the GStreamer debugging shell variable, then run your program:
export GST_DEBUG=GST_REFCOUNTING:5
Filtering debug output
Here are several examples of how to filter the debug output.
You can see which objects disappear, meaning their reference count reaches zero:
gst-launch --gst-debug=GST_REFCOUNTING:5 videotestsrc num-buffers=3 ! fakesink 2>&1 | grep "\->0" | cut -d' ' -f 19
In useful applications, there are always some objects that aren't freed until the program exists. With a simple gst-hello example all the objects are always freed. If you set the pipe to run, let's say 13 times, and in the log, you see one object that wasn't freed and the count reached 13 or more, then you have a suspicious object or if you see that the number of non-freed objects increases as you increase the number of runs, then you have an indication of a problem.
Use remote syslog
If your application logs warnings and errors, use remote syslog to make the information more easily available.
If your application generates output, you can redirect it to Syslog as follows:
MY_APP=gst-render
$MY_APP | logger -t "$MY_APP:"
Generated files debug
There are some tools that can be used for getting information from generated files by GStreamer:
- libmp4v2
- MP4BOX
- mediainfo
With MP4Box you can use the -info in order to get more info from a generated mp4 file:
MP4Box -info test.mp4
GStreamer Element Debugging
If you are creating an element you plan to use in an embedded device, you might find it helpful to build and test the element on your development machine first.
./autogen.sh
CFLAGS=-ggdb ./configure
make
objdump -x src/.libs/*.so | fgrep DEBUG # verify library is built with debugging symbols enabled
Use GST_PLUGIN_PATH to enable GStreamer to find your element library:
export GST_PLUGIN_PATH=$HOME/projects/gst-emboverlay/src/.libs
gst-inspect emboverlay
Test out your element. Better to use GStreamer daemon than gst-launch.
TEXT_OVERLAY="text=Hello\ There text-offseth=50 text-offsetv=100 text-font-height=50 text-color=0x800040 text-enable=true"
TEXT2_OVERLAY="text2=World text2-offseth=50 text2-offsetv=150 text2-font-height=50 text2-color=0x800040 text2-enable"
TIME_OVERLAY="time-param=%b\ %d\ %Y\ %H:%M:%S time-offseth=50 time-offsetv=200 time-font-height=50 time-color=0x800040 time-enable=true"
LOGO_OVERLAY="logo=$HOME/projects/gst-emboverlay/RidgeRunLogo312x84-with-alpha-channel.png logo-offseth=-1 logo-offsetv=-1 logo-transp=1 logo-scale=1.0 logo-autofit=true time-enable=true"
FRAME_OVERLAY="frame-number=-20 frame-offseth=50 frame-offsetv=250 frame-font-height=50 frame-color=0x800040 frame-enable=true"
CAPS="video/x-raw-yuv, format=(fourcc)YV12, width=(int)1280, height=(int)720, interlaced=(boolean)false, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)10/1"
gst-launch --gst-debug=emboverlay:4 -v v4l2src num-buffers=60 ! capsfilter caps="$CAPS" ! emboverlay $TIME_OVERLAY $TEXT_OVERLAY $TEXT2_OVERLAY $LOGO_OVERLAY $FRAME_OVERLAY ! xvimagesink
GStreamer Performance Tuning
This is worthy of its own Embedded GStreamer Performance Tuning topic.
GStreamer Application Development
GStreamer Daemon
If you can use GStreamer Daemon as your streaming media server, then avoid writing a custom application in the first place!
Developing on host computer
If you have a pipeline where source pads and sink pads are getting connected and disconnected, then you will likely need to write a custom streaming media server. Most people find it easier to build and debug by running the application on the host before trying it on the target hardware.
See also
- GStreamer Basic tutorial 11: Debugging Tools gstreamer.freedesktop.org tutorial documentation.URL: https://gstreamer.freedesktop.org/documentation/tutorials/basic/debugging-tools.html
- Running GStreamer Applications — How to run and debug your GStreamer application gstreamer.freedesktop.org documentation. URL: http://www.gstreamer.net/data/doc/gstreamer/head/gstreamer/html/gst-running.html
- Things to check when writing an element gstreamer.freedesktop.org plugin development documentation.URL: https://gstreamer.freedesktop.org/documentation/plugin-development/appendix/checklist-element.html
- GstInfo - Debugging and logging facilities gstreamer.freedesktop.org GStreamer 1.0 Core Reference Manual.URL: http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/gstreamer-GstInfo.html
- gst-tracelib - Performance tracing library for GStreamer GStreamer tracing library.URL: http://cgit.freedesktop.org/~ensonic/gst-tracelib/tree/README
- GStreamer Debugging Using GDB. URL: https://developer.ridgerun.com/wiki/index.php/GStreamer_Debugging_Using_GDB
RidgeRun Resources | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||
Contact Us
|