GstInference - Metadatas - GstInferenceMeta

GstInference
Introduction
Getting started
Getting the code Building the plugin Quick Starting Guide
Supported architectures
InceptionV1 InceptionV2 InceptionV3 InceptionV4 TinyYoloV2 TinyYoloV3 YoloV2 ResNetV1 MobileNetV2 MobileNetV2 + SSD AlexNet
Supported backends
TensorFlow Tensorflow-Lite Coral from Google Caffe TensorRT ONNXRT ONNXRT ACL ONNXRT OpenVINO
Metadata and Signals
GstClassificationMeta GstInferenceMeta Signals
Overlay Elements
Inference Overlay
Utils Elements
InferenceFilter InferenceCrop InferenceDebug InferenceBin
Legacy pipelines
PC NANO TX2 Xavier IMX8
Example pipelines
Example applications
Classification Detection Hand Pose Recognition
Benchmarks
Model Zoo
Project Status
Releases Roadmap
Contact Us

GstInferenceMeta is a new hierarchical approach towards storing inferred data on a buffer. It follows a n-ary tree structure, in which each node contains a prediction, a pointer to struct of type GstInferencePrediction, which is explained in the sections below.

Fields

field	type	description
meta	GstMeta	Buffer metadata
prediction	GstInferencePrediction *	Contains all the predictions associated to this node.

GstInferencePrediction

Each prediction on each inference meta node contains the following information:

Fields

field	type	description
prediction_id	guint64	A unique id for this specific prediction.
enabled	gboolean	This flag indicates whether or not this prediction should be used for further inference.
bbox	BoundingBox	Bouding box for this specific prediction.
classifications	GList *	a linked list of GstInfereferenceClassification associated to this prediction.
predictions	GNode *	a n-ary tree of child predictions within this specific prediction. It is recommended to to access the tree directly, but to use this module's API to interact with the children.

Each bounding box is a struct defined as follows:

field	type	description
x	guint	horizontal coordinate of the upper left position of the bounding box in pixels
y	guint	vertical coordinate of the upper left position of the bounding box in pixels
width	guint	width of the bounding box in pixels
height	guint	height of the bounding box in pixels

GstInferenceClassification

Each prediction can contain a number of classifications. The GstInferenceClassification struct holds all the potentially relevant information regarding such classifications in the fields specified as follows.

Fields

field	type	description
class_id	gint	The numerical id associated to the assigned class.
class_prob	gdouble	The resulting probability of the assigned class. Typically ranges between 0 and 1.
class_label	gchar *	The label associated to this class or NULL if not available.
num_classes	gint	The total amount of classes of the entire prediction.
probabilities	gdouble	The entire array of probabilities of the prediction.
labels	gchar**	The entire array of labels of the prediction. NULL if not available.

Access metadata

If you want to access this metadata from your custom Gstreamer element instead the process is fairly easy:

1. Include GstInference metadata header: #include "gst/r2inference/gstinferencemeta.h"
2. Get a GstInferenceMeta object from the buffer: inference_meta = (GstInferenceMeta *) gst_buffer_get_meta (frame->buffer, GST_INFERENCE_META_API_TYPE);

In the following section of code, we add the include like in point 2 and safe the GstInferenceMeta in the inference_meta variable like in point 3.

#include "gst/r2inference/gstinferencemeta.h"
static void 
get_buffer(GstPadProbeInfo * info)
{
  GstBuffer *buffer;
  GstInferenceMeta *meta;
  buffer = gst_pad_probe_info_get_buffer (info);
  meta = (GstInferenceMeta *) gst_buffer_get_meta (buffer,
                  GST_INFERENCE_META_API_TYPE);
}