GstExtractor

Overview

GstExtractor is a GStreamer plug-in that extracts data embedded at the end of a video frame and allows the user to do customized parsing to send the embedded data downstream as KLV, H264/5 SEI metadata, or expose the data as a readable GstExtractor parameter, thus allowing the controlling application to retrieve the values of interest. GstExtractor takes care of separating the embedded data from the actual video frame and output them to different pads, so that downstream video-processing elements will not know about the incoming extra data region. This separation is performed in a way that the video frame isn't copied so processing time is mainly due to the customized data parsing.

Figure 1 shows the basic functionality of GstExtractor. Here we have an input extended video frame (video frame + embedded data in a data region at the end of the frame) and two outputs:

1. A normal video frame that might or might not carry H264/5 SEI metadata as a GstMeta structure.

2. KLV metadata.

Figure 1. GstExtractor General Block Diagram

It is the responsibility of the user to implement an appropriate customized parser and add the parser to the GstExtractor. The parser generates the binary SEI and KLV encoded data.

Note: In order to use the SEI metadata feature you need to also acquire RidgeRun's GstSEIMetadata plugin.

Getting the Code

When you purchase GstExtractor, you will get a git repository with the source code inside. You need to build the plug-in in your system (since you are adding a customized parser) then add the plug-in to your GStreamer catalogue. Clone the repository using something similar to:

git clone git://git@gitlab.com/RidgeRun/orders/${CUSTOMER_DIRECTORY}/gst-extractor.git
cd gst-extractor
GST_EXTRACTOR_DIR=${PWD}

Where ${CUSTOMER_DIRECTORY} contains the name of your customer directory given by RidgeRun after purchase.

How to Use GstExtractor

A deeper view of the GstExtractor functionality can be seen in Figure 2.

Figure 2. GstExtractor Processing

All the GStreamer functionality for the pads, buffer creation, caps negotiation and others is already taken care of by GstExtractor, you only need to add the custom embedded data parser.

In order to use GstExtractor you need to set up three main things:

1. An input video stream with an extended video frame with a data region to hold metadata.
2. Define GstExtractor element properties according to any data you want to make available to the controlling application.
3. Implement a custom embedded parser for the extracting embedded data.

Extended Video Frame Structure

The extended video frame shall have the following structure:

Error creating thumbnail: Unable to save thumbnail to destination

Figure 3. GstExtractor extended video frame structure

Where we have first, the video frame memory, and just after it ends, the embedded data memory. However, our extended video frame can't just have a size of (video frame size + data size) because it would not be compliant with any video format and we wouldn't be able to pass it through a Gstreamer pipeline. Instead, we need to define a height padding in pixels big enough to contain our embedded metadata. For example, if we have a YUY2 video frame of 1280x720 and an embedded data of 34 bytes, then 1 pixel of height padding would be enough because (1280 * 2 bytes per pixel * 1 pixel) > 34. Then our extended video frame would need to have a resolution of 1280x721.

GstExtractor Element Properties

The GstExtractor element has the following properties:

  value               : Binary value to query some custom information included in the frame embedded data.
                        flags: readable
                        Boxed pointer of type "GByteArray"
  data-size           : Custom data size in bytes.
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0 
  height-padding      : Extra height padding added in pixel units.
                        flags: readable, writable
                        Integer. Range: 0 - 2147483647 Default: 0
  filepath            : Path to dummy frame to use in replacement of input buffer when a custom condition is met.
                        flags: readable, writable
                        String. Default: null

• Where data-size and height-padding must be consistent with the extended video frame structure. So following our last example, you would need to assign data-size=34 and height-padding=1.

• The value property is intended for the user to query it from an user application perspective, so you don't have to set it on the pipeline. However, it is also part of the custom data parsing implementation to assign the appropriate binary value to it.

• The filepath property is optional in case you want to show a dummy image if a given condition is met. This image should be raw (not encoded) and of the same format and size of the normal stream.

Note: Output caps will be set based on height-padding, so this property value will only be valid during caps negotiation and further writes to it won't have effect.

Custom Data Parsing Implementation

GstExtractor will be in charge of separating the embedded data from the video frame and send the H264/5 SEI metadata and KLV metadata buffer, however you need to provide a pointer for both the SEI and KLV data to be sent. In order to do this we prepared a special code section located at gst/gstextractor.c for you to modify. This section currently has a basic example of how this could be done, but you must modify it according to your needs:

  {
    /* TODO: Add your custom parsing here. This is just an example that
       simulates that the first sei_size bytes are a timestamp and the
       following bytes are some other KLV data. In this example the last
       bin_size bytes of KLV data are the value we would like to query from the
       user point of view. Also if those bin_size bytes represent a number less
       than bin_threshold the dummy frame will be used instead of the input
       buffer */
    const gint bin_size = 5;
    const gint bin_threshold = 50;
    sei_size = 20;
    klv_size = 14;

    sei_data = (guint8 *) g_memdup (data, sei_size);
    klv_data = (guint8 *) g_memdup (data + sei_size, klv_size);

    GST_OBJECT_LOCK (self);
    if (self->value) {
      g_byte_array_free (self->value, TRUE);
    }
    self->value = g_byte_array_sized_new (bin_size);
    self->value->len = bin_size;
    memcpy (self->value->data, klv_data + (klv_size - bin_size), bin_size);

    if ((atoi ((char *) self->value->data) < bin_threshold) && self->filepath) {

      /* Replace input buffer with our dummy buffer */
      GST_BUFFER_PTS (self->dummy_buf) = GST_BUFFER_PTS (inbuf);
      gst_buffer_unref (inbuf);
      gst_buffer_ref (self->dummy_buf);
      inbuf = self->dummy_buf;
      inbuf = gst_buffer_make_writable (inbuf);
    }
    GST_OBJECT_UNLOCK (self);
  }

Example Data Parsing Implementation Walkthrough

1. First we declare and assign some variables for the parsing:

const gint bin_size = 5;
const gint bin_threshold = 50;
sei_size = 20;
klv_size = 14;

It's mandatory that you assign the sei_size and klv_size to their corresponding value, otherwise they will be set to 0 and the further buffer/metadata creation will fail. bin_size and bin_threshold on the other side, are local variables for the parsing to set the value property and to replace the input buffer with a dummy frame respectively, so they are optional.

2. "Parse" and assign the result to the provided pointers:

sei_data = (guint8 *) g_memdup (data, sei_size);
klv_data = (guint8 *) g_memdup (data + sei_size, klv_size);

In this example we are not really parsing anything, just copying the data into the sei_data and klv_data pointers, but this should be the last step after you parse/encode your data. Keep in mind that sei_data and klv_data are null pointers at that point, so you need to allocate them with sei_size and klv_size respectively (in this case the g_memdup function takes care of it).

Also note that the SEI and KLV data does not necessarily need to be encoded for the pipeline to work, it is just simple binary data that your applications should know how to interpret when it is received.

3. Lock the object since we will modify data that can be accessed from another thread:

GST_OBJECT_LOCK (self);

4. Assign the appropriate binary data to the value property:

if (self->value) {
  g_byte_array_free (self->value, TRUE);
}
self->value = g_byte_array_sized_new (bin_size);
self->value->len = bin_size;
memcpy (self->value->data, klv_data + (klv_size - bin_size), bin_size);

If you don't need this step, just remove this code.

5. Create a proper custom condition to know when to replace the input buffer with the dummy frame. You should only replace the (atoi ((char *) self->value->data) < bin_threshold) condition. Keep in mind that for this piece of code to work you need to assign the filepath property to extractor element.

if ((atoi ((char *) self->value->data) < bin_threshold) && self->filepath) { /* Replace with custom condition */
      /* Replace input buffer with our dummy buffer */
      GST_BUFFER_PTS (self->dummy_buf) = GST_BUFFER_PTS (inbuf);
      gst_buffer_unref (inbuf);
      gst_buffer_ref (self->dummy_buf);
      inbuf = self->dummy_buf;
      inbuf = gst_buffer_make_writable (inbuf);
}

If you don't need this step, just remove this code.

6. Unlock the object.

GST_OBJECT_UNLOCK (self);

7. Follow the next section to install and test your changes.

Building GstExtractor

Install Dependencies

GstExtractor has the following dependencies:

• Gstreamer
• Gstreamer Plugins Base
• Meson
• GstSEIMetadata

In order to install the first three run the following commands:

# Gstreamer
sudo apt install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev gstreamer1.0-plugins-base
# Meson
sudo apt install python3 python3-pip python3-setuptools python3-wheel ninja-build
sudo -H pip3 install git+https://github.com/mesonbuild/meson.git

Then, GstSEIMetadata is a proprietary product from RidgeRun, so check this link to learn how to get and install it.

Compile and Install

First, find the path where Gstreamer looks for plug-ins and libraries (LIBDIR). Take as a reference to the following table:

Platform	LIBDIR path
PC 64-bits/x86	/usr/lib/x86_64-linux-gnu/
NVIDIA Jetson	/usr/lib/aarch64-linux-gnu/

Then, run:

cd $GST_EXTRACTOR_DIR # Path to your gst-extractor cloned repository
meson build --prefix /usr --libdir $LIBDIR
ninja -C build
sudo ninja -C build install

Examples

Here are some examples for you to test the gst-extractor.

Simple Examples

Display

This pipeline will allow you to display the normal input video (without metadata) and will send the KLV metadata to a file.

• NVIDIA Elements:

INPUT="<path-to-yuy2-1280-722-file-with-metadata>"
gst-launch-1.0 filesrc location=${INPUT} ! videoparse width=1280 height=722 format=yuy2 framerate=30 ! extractor data-size=34 height-padding=2 name=ext ext.video ! queue ! nvvidconv ! "video/x-raw(memory:NVMM),format=NV12" ! nvoverlaysink sync=false ext.klv ! queue ! filesink location=meta.klv

• Generic Elements:

INPUT="<path-to-yuy2-1280-722-file-with-metadata>"
gst-launch-1.0 filesrc location=${INPUT} ! videoparse width=1280 height=722 format=yuy2 framerate=30 ! extractor data-size=34 height-padding=2 name=ext ext.video ! queue ! videoconvert ! ximagesink sync=false ext.klv ! queue ! filesink location=meta.klv

Display Using Dummy Frame

This pipeline will allow you to display the normal input video (without metadata), but will replace the input buffer with the dummy frame pointed by filepath. Finally, it will send the KLV metadata to a file.

• NVIDIA Elements:

INPUT="<path-to-yuy2-1280-722-file-with-metadata>"
FILEPATH="<path-to-dummy-frame-yuy2>"
gst-launch-1.0 filesrc location=${INPUT} ! videoparse width=1280 height=722 format=yuy2 framerate=30 ! extractor data-size=34 height-padding=2 name=ext filepath=${FILEPATH} ext.video ! queue ! videoconvert ! ximagesink sync=false ext.klv ! queue ! filesink location=meta.klv

• Generic Elements:

INPUT="<path-to-yuy2-1280-722-file-with-metadata>"
FILEPATH="<path-to-dummy-frame-yuy2>"
gst-launch-1.0 filesrc location=${INPUT} ! videoparse width=1280 height=722 format=yuy2 framerate=30 ! extractor data-size=34 height-padding=2 filepath=${FILEPATH} name=ext ext.video ! queue ! nvvidconv ! "video/x-raw(memory:NVMM),format=NV12" ! nvoverlaysink sync=false ext.klv ! queue ! filesink location=meta.klv

Advanced Examples

This section will show some more real use case examples that will demonstrate how the extractor can be used inside more complex pipelines.

Sending Split Video and Metadata through UDP Using MPEG-TS

One typical use case for the KLV and SEI metadata is to be sent across UDP. The following image shows a diagram of this.

GstExtractor typical advanced use case

For a case like this we could use the following pipelines:

• Sender Side

IP="224.1.1.1"
PORT="12345"
INPUT="<path-to-yuy2-1280-722-file-with-metadata>"
FILEPATH="<path-to-dummy-frame-yuy2>"
gst-launch-1.0 filesrc location=${INPUT} ! videoparse width=1280 height=722 format=yuy2 framerate=30 ! extractor name=ext data-size=34 height-padding=2 filepath=${FILEPATH} ext.video ! queue ! nvvidconv ! nvv4l2h265enc name=encoder bitrate=2000000 iframeinterval=300 vbv-size=33333 insert-sps-pps=true control-rate=constant_bitrate profile=Main num-B-Frames=0 ratecontrol-enable=true preset-level=UltraFastPreset EnableTwopassCBR=false maxperf-enable=true ! seiinject ! h265parse ! queue ! mux. ext.klv ! meta/x-klv ! queue ! mpegtsmux name=mux alignment=7 ! udpsink host=${IP} port=${PORT} auto-multicast=true

• Receiving Side

IP="224.1.1.1"
PORT="12345"
GST_DEBUG=2,*seiextract*:MEMDUMP gst-launch-1.0 udpsrc port=12345 address=224.1.1.1 ! tsdemux name=demux demux. ! queue !  h265parse ! seiextract ! avdec_h265 ! autovideosink sync=false demux. ! queue ! 'meta/x-klv' ! fakesink sync=false

Note that to use this pipeline you should have installed GstSEI on the receiving side too. The debug flags GST_DEBUG=2,*seiextract*:MEMDUMP will allow you to see the extracted SEI metadata into the Gstreamer debug. Also, if you would like to see the KLV metadata in the terminal output too, you can check our GStreamer In-Band Metadata for MPEG Transport Stream plugin and replace the fakesink element with metasink. Then you should see something similar to this on the receiving side:

# KLV meta
00000000 (0x7f893402e590): 73 6f 6d 65 20 6b 6c 76 20 30 31 34 39 00        some klv 0149.  
# SEI meta
0:00:43.188390365 146210 0x55b6049b0f60 LOG               seiextract gstseiextract.c:386:gst_sei_extract_prepare_output_buffer:<seiextract0> prepare_output_buffer
0:00:43.188486504 146210 0x55b6049b0f60 MEMDUMP           seiextract gstseiextract.c:356:gst_sei_extract_extract_h265_data:<seiextract0> ---------------------------------------------------------------------------
0:00:43.188530839 146210 0x55b6049b0f60 MEMDUMP           seiextract gstseiextract.c:356:gst_sei_extract_extract_h265_data:<seiextract0> The extracted data is: 
0:00:43.188578709 146210 0x55b6049b0f60 MEMDUMP           seiextract gstseiextract.c:356:gst_sei_extract_extract_h265_data:<seiextract0> 00000000: 30 35 2d 31 31 2d 32 30 32 31 2d 31 35 3a 34 33  05-11-2021-15:43
0:00:43.188620166 146210 0x55b6049b0f60 MEMDUMP           seiextract gstseiextract.c:356:gst_sei_extract_extract_h265_data:<seiextract0> 00000010: 3a 30 30 20                                      :00             
0:00:43.188650823 146210 0x55b6049b0f60 MEMDUMP           seiextract gstseiextract.c:356:gst_sei_extract_extract_h265_data:<seiextract0> ---------------------------------------------------------------------------

Where the injected metadata was some klv 0149 and 05-11-2021-15:43:00 for KLV and SEI, respectively.

RidgeRun Resources

Quick Start Client Engagement Process RidgeRun Blog Homepage Technical and Sales Support RidgeRun Online Store RidgeRun Videos Contact Us

Contact Us Visit our Main Website for the RidgeRun Products and Online Store. RidgeRun Engineering informations are available in RidgeRun Professional Services, RidgeRun Subscription Model and Client Engagement Process wiki pages. Please email to support@ridgerun.com for technical questions and contactus@ridgerun.com for other queries. Contact details for sponsoring the RidgeRun GStreamer projects are available in Sponsor Projects page.