Gstreamer QT Overlay

Overview

GstQtOverlay is a GStreamer plug-in that renders QT graphics on top of a video stream. This rendering occurs directly into the GStreamer video frame buffer, rather than in physical graphics memory, with the resultant video having the QT graphics image merged into the video image. This brings together the best of the two worlds:

• Modularity and extensibility of GStreamer
• Beauty and flexibility of QT

With the RidgeRun GstQtOverlay element, GUIs will be available not only at the video display, but also embedded in file recordings and network streaming, for example.

GstQtOverlay makes heavy use of OpenGL in order to achieve high performance overlay rendering. The pixel blitting and memory transfers are done by GPU and DMA hardware accelerators, allowing real time operation at HD resolutions. As a consequence, the CPU remains free for other processing.

Graphics are modeled using QT's QML, which enables fast, powerful and dynamic GUI implementation. QML is loaded at runtime so there is no need to recompile the GstQtOverlay plug-in in order to change the GUI. Having QML independent of the GStreamer element speeds up development time, eases quick prototyping and reduces time-to-market without impacting performance. A single GstQtOverlay element is able to handle many different GUIs.

Architecture

GstQtOverlay element's architecture is summarized in Figure 1.

Figure 1. GstQtOverlay Element Architecture

• GStreamer

This layer provides the necessary classes to expose GstQtOverlay as a GStreamer element. As such, GstQtOverlay can be linked into any compatible GStreamer pipeline and will participate in the standard pipeline lifecycle. This includes among other, caps and allocator negotiation, and pipeline state changes.

• QT/QML

On top of GStreamer, a QT powered class is instantiated in order to manage graphic resources in a decoupled way. Here, QT event loop is specially processed in order to have both Glib's and QT's event sources active. Finally, a QML engine is created to read and render the user's QML file.

• HAL

The Hardware Abstraction Layer (or simply HAL) is a thin layer that acts as an adapter between QT and OpenGL in order to leverage the platform specific GPU utilities and maximize performance. This layer is the only one that is HW dependent and will be conditionally built via configuration parameters. The HAL ensures that data transfers are HW accelerated so that there is no performance penalty.

• OpenGL

The OpenGL layer will render QT content into the image data using the GPU's parallel power. The blitting to merge the QT image onto a video frame is performed into a copy of the frame image, so that the integrity of the original data is preserved.

Supported Hardware

Currently, the following hardware is supported:

• X86 - Linux Ubuntu 16.04
• X86 - Apple Mac OSX
• IMX6

More platforms are under development. If you wish to sponsor the port for a custom platform or accelerate development for another SoC not listed above, please contact us to support@ridgerun.com

Interacting with the GUI

Supported Capabilities

Currently, the GstQtOverlay element supports the following formats at the Sink and Source pads respectively. More formats may be supported in the future, subject to platform capabilities.

Input

RGBA.

Output

RGBA.

GStreamer Properties

GStreamer properties are used to customize the QT image overlay and to interact with QML.

qml property

The qml property specifies the location of the QML source file to use. This path can be either relative or absolute and must point to a valid/existing file at the moment the pipeline starts up. Some examples include:

 gst-launch-1.0 videotestsrc ! qtoverlay qml=/mnt/gui/sources/main.qml ! autovideosink

 gst-launch-1.0 videotestsrc ! qtoverlay qml=../../sources/main.qml ! autovideosink

If not specified, the element will attempt to load main.qml from the current working directory.

qml-attribute property

QML attributes may be configured at runtime using the qml-attribute property. The property syntax is as follows:

 <item name>.<attribute name>:<value>

Item name refers to the QML objectName attribute of an item, rather than its ID. The following QML contains an item named labelMain:

 Label {
     text: "Way cool QT imaging with GStreamer!"
     font.pixelSize: 22
     font.italic: true
     color: "steelblue"
     objectName: "labelMain"
 }

For example, to modify the text attribute of a label named labelMain to display GStreamer with QT imaging is way cool! you would run:

 gstd-client element_set qtoverlay qml-attribute "labelMain.text:GStreamer with QT imaging is way cool!"

Nested items are allowed and also available via the property:

 <item name>.<child name>.<grand child name>.<etc...>.<attribute>:<value>

The objectName attribute is not required to be unique, so multiple items can be modified as once by assigning them the same objectName.

qml-action property

External events can also trigger actions on the QML by using the qml-action property. Similarly, the syntax goes as the following:

 <item name>.<method name>()

Again, invoking actions in nested items is allowed by running:

 <item name>.<child name>.<grand child name>.<etc...>.<method name>()

For example, consider the following QML snippet:

import QtQuick 2.0

Item {
    objectName:i1
    function sayHi() {
        console.log("Hello World!")
    }
}

To invoke sayHi you would run:

 gstd-client element_set  qtoverlay qml-action "i1.sayHi()"

At this time, passing parameters to the invoked functions is not supported. Instead, first set an attribute value, then invoke the function which can access the attribute. There are future plans to support function parameters via Variants or some similar mechanism.

Example

imx6

Example pipeline:

   gst-launch-1.0 videotestsrc is-live=true ! 'video/x-raw, width=640, height=480, framerate=30/1' ! queue ! imxg2dvideotransform ! \
   queue ! qtoverlay qml=/main.qml ! queue ! perf print-arm-load=true ! imxeglvivsink qos=false sync=false enable-last-sample=false

FAQ

• Where can I get GstQtOverlay?

You can purchase it at RidgeRun' Store (TODO: add link) or contacting our engineering team via support@ridgerun.com

• Is the source code delivered with the purchase?

Yes. After the purchase the complete source code is delivered.

• Can I use external events to interact with my GUI?

Yes. See Gstreamer Properties

• Am I limited to a single QML?

No. As with any regular QML powered application, events can fire up the load of other QML files. This allows the user to build state-pattern like apps with ease.

• Can I use mouse and/or keyboard events?

Not yet. At least not out-of-the-box.

• Is QML enough for my GUI?

Yes (most likely). QML support is great, there's a lot of documentation and examples available. QML is competitive with HTML5 in cross-platform app development.

• Can I create my own custom QML components?

Yes. See the QT QML module guide.

• Can I use animations?

Not yet. We expect to support this in future releases.

• Does my HW support GstQtOverlay?

Check Supported Hardware