GStreamer DispTEC Tracker Plugin

Overview

GstDispTEC object tracker is a GStreamer element which integrates DispTEC library's algorithms for object tracking, making it possible to incorporate their functionalities into GStreamer pipelines, such as showing where the object is located on a frame! This allows the user to strengthen image analysis, by connecting its own applications to the pipeline for receiving the analysis data and much more amazing things, as shown in the following demo.

Description

Object tracking has been implemented based on Kernelized Correlation Filters (KCF). Each tracker element is initialized with the coordinates of the object in the current frame, snipping the object from the frame and taking it as a template. Afterwards, the tracker element updates the filter every time that there is an object hit, adapting itself to the possible changes which an object might have during the video, making it robust enough to cope with potential perspective changes.

This object tracking technique gives the following features:

• Non-stationary camera resilient.
• Tolerates object scaling, rotation and transformation.
• Resilient to changes in colour and light.

GstDispTEC object tracker goes even further, making possible to reset the object position at run-time, even if the pipeline is in Play state. Let's look nearer:

• Possibility of tracker cascading.
• Tracker reinitialization at run-time.
• Application integrability.
• Visual feedback of object position overlaid to frames.
• Multi-tracker deployment in the same element (coming soon).

The tracker element gets some parameters to surround the object to get a template and start tracking. Also, it is possible to enable the visual feedback overlaid on the frame highlighting the current position. These parameters are described below:

Parameter	Description
x	(Unsigned int) Object x coordinate at the top-left corner in pixels. Range: 0 - 4294967295. Default: 0
y	(Unsigned int) Object y coordinate at the top-left corner in pixels. Range: 0 - 4294967295. Default: 0
width	(Unsigned int) Object width in pixels. Range: 0 - 4294967295. Default: 100
height	(Unsigned int) Object height in pixels. Range: 0 - 4294967295. Default: 100
render-bbox	(Boolean) Render a bounding box which gives a visual representation where the object is currently located. Default: TRUE

The tracker element also generates a signal for third-party applications for retrieving the current object position (coordinates), whose coordinates are expressed as integers in the following sequence: x, y, width, and height.

Architecture

GstDispTEC object tracker has been created based on the DispTEC library. However, it is independent of the DispTEC's dependencies and it can be easily integrated whichever library that DispTEC is based on (OpenCV, LTI-lib, etc). This is achieved by an Abstraction layer just on the base of the plug-in, which can adapt it to several DispTEC variations.

By using DispTEC as the base, it is possible to perform and run Computer Vision algorithms easily on CPU, accelerating the performance and delivery time of video analyses. Nevertheless, joining DispTEC with GStreamer, we offer the best of both worlds: the powerful algorithms available in DispTEC and the flexibility of GStreamer to process videos, allowing live-streaming, pipelining and more.

Error creating thumbnail: Unable to save thumbnail to destination

GstDispTEC architecture diagram.

GstDispTEC Arquitecture Description illustrates the general architecture of the plug-in and what makes it one of the best solutions for image analysis for GStreamer.

Supported platforms

GstDispTEC object tracker currently supports platforms capable to run Open Computer Vision Library (OpenCV). Some of the platforms already tested are:

Platform	Operating System
Nvidia Jetson TX1	Ubuntu 16.04 64 bits
Personal Computer (PC)	Ubuntu 16.04 64 bits

Both platforms were running OpenCV 3.3.

Building GstDispTEC

Dependencies

Before trying to install, make sure of having the following dependencies already installed on the target platform:

• OpenCV Core >= 3.3
• OpenCV Contribution Modules >= 3.3
• DispTEC >= 0.2
• GStreamer >= 1.8.1
• Autotools

Installing dependencies

DispTEC

sudo apt install cmake
git clone https://github.com/RidgeRun/dispTEC_2017
cmake .
make
sudo make install

OpenCV

For installing OpenCV, please, follow the steps in the link: OpenCV Installation

GStreamer

sudo apt-get install \
gstreamer1.0-x \
libgstreamer1.0-dev \
libgstreamer-plugins-base1.0-dev

Autotools

sudo apt install \
autotools-dev \
autoconf

Installing GstDispTEC object tracker

When you purchase the gst-dttracker plugin, you will get a Gitlab repository with the source code inside. You need it to build it in your system and add it to your GStreamer plug-ins catalogue. So, in a temporary path, clone the repository using:

git clone git://git@gitlab.com/RidgeRun/orders/${CUSTOMER_DIRECTORY}/gst-disptec.git

Where ${CUSTOMER_DIRECTORY} contains the name of your customer directory given after the purchase.

Afterwards, find the path where GStreamer looks for plug-ins and libraries (LIBDIR). Take as a reference to the following table:

Platform	LIBDIR path
PC 32-bits/x86	/usr/lib/i386-linux-gnu/
PC 64-bits/x86	/usr/lib/x86_64-linux-gnu/
Nvidia Jetson	/usr/lib/aarch64-linux-gnu/

According to the table, define an environment variable which stores the path accordingly to your system. For example, for an x64 PC, it will be:

LIBDIR=/usr/lib/x86_64-linux-gnu/

Then, do the installation using:

./autogen.sh
./configure --libdir $LIBDIR
make
sudo make install

Test installation

For testing if everything is properly installed, run:

gst-inspect-1.0 dttracker

It should show:

Factory Details:
  Rank                     none (0)
  Long-name                DispTEC tracker element
  Klass                    Generic
  Description              DispTEC object tracking element over a video stream
  Author                   <luis.leon@ridgerun.com, carolina.trejos@ridgerun.com>

Plugin Details:
  Name                     disptec
  Description              DispTEC Image processing library plugin
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstdisptec.so
  Version                  0.3.0
  License                  Proprietary
  Source module            gst-disptec
  Binary package           RidgeRun
  Origin URL               https://www.ridgerun.com

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseTransform
                         +----GstVideoFilter
                               +----GstDttracker

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: BGR
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]

  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: BGR
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]


Element Flags:
  no flags set

Element Implementation:
  Has change_state() function: gst_element_change_state_func

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "dttracker0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  qos                 : Handle Quality-of-Service events
                        flags: readable, writable
                        Boolean. Default: true
  x                   : Initial left corner X position
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0 
  y                   : Initial left corner Y position
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0 
  width               : Initial object width
                        flags: readable, writable
                        Unsigned Integer. Range: 1 - 4294967295 Default: 100 
  height              : Initial object height
                        flags: readable, writable
                        Unsigned Integer. Range: 1 - 4294967295 Default: 100 
  render-bbox         : Enables or disables the bounding box around the tracked object
                        flags: readable, writable
                        Boolean. Default: true

Element Signals:
  "coordinates" :  void user_function (GstElement* object,
                                       guint arg0,
                                       guint arg1,
                                       guint arg2,
                                       guint arg3,
                                       gpointer user_data);

Examples

Example 1: Ball pattern with visual feedback

gst-launch-1.0 videotestsrc pattern=ball ! videoconvert ! dttracker name=tracker x=130 y=90 width=50 height=50 ! videoconvert ! xvimagesink

For this case, the ball always starts at x=130, y=90 and has a width around 50x50 px.

Example 2: Video loaded from a file

INITIAL_OBJ_POSITION="x=100 y=100 width=100 height=100"
gst-launch-1.0 filesrc location=$PATH_TO_FILE ! videoconvert ! dttracker name=tracker $INITIAL_OBJ_POSITION ! videoconvert ! xvimagesink

Set the object initial position into INITIAL_OBJ_POSITION

Example 3: Capturing from camera

INITIAL_OBJ_POSITION="x=100 y=100 width=100 height=100"
gst-launch v4l2src device=$CAPTURE_DEVICE ! $CAPS ! dttracker name=tracker $INITIAL_OBJ_POSITION ! videoconvert ! xvimagesink