GstInference and GstInferenceMeta metadata
 | Make sure you also check GstInference's companion project: R2Inference |
Contents
Overview
GstInferenceMeta is a new hierarchical approach towards storing inferred data on a buffer. This meta replace the old GstDetectionMeta and GstClassificationMeta combining the data generated by detection and classification elements in a single structure that simplifies the pipeline.
GstInferenceMeta follow this structure:
- The GstInferenceMeta is initialized with a root prediction (the complete image).
- Root prediction has a list of GstInferenceClassification.
- Root prediction contains n-ary tree structure, in which each node contains a prediction, a pointer to struct of type GstInferencePrediction.
- Each prediction child has an associated bounding box (Filled by detection elements).
- Each prediction child has a list of GstInferenceClassification (Filled by classification and detection elements).
- Finally each prediction child could have child predictions of type GstInferencePrediction.
The following section shows how this structure is used in a real scenario.
Real Example
Consider the following scene with a cat and a dog.
Graphical Representation
Prediction Metadata Diagram
The information in the scene is summarized as in the following diagram:
Serialized Prediction
GstInference will serialize the structure above as a recursive JSON object as the following:
{
id : 597,
enabled : True,
bbox : {
x : 0
y : 0
width : 416
height : 416
},
classes : [
],
predictions : [
{
id : 598,
enabled : True,
bbox : {
x : 30
y : 73
width : 161
height : 318
},
classes : [
{
Id : 648
Class : 7
Label : cat
Probability : 16,072840
Classes : 20
},
],
predictions : [
]
}, {
id : 599,
enabled : True,
bbox : {
x : 211
y : 38
width : 182
height : 406
},
classes : [
{
Id : 649
Class : 11
Label : dog
Probability : 16,978619
Classes : 20
},
],
predictions : [
]
},
]
}
Detailed Description
The following tables provide the specific details of each structure within the prediction tree.
GstInferenceMeta
It is the GStreamer meta structure attached to the buffer. It contains the root prediction.
| field | type | description |
|---|---|---|
| meta | GstMeta | Buffer metadata |
| prediction | GstInferencePrediction * | Contains all the predictions associated to this node. |
GstInferencePrediction
Each prediction on each inference meta node contains the following information:
| field | type | description |
|---|---|---|
| prediction_id | guint64 | A unique id for this specific prediction. |
| enabled | gboolean | This flag indicates whether or not this prediction should be used for further inference. |
| bbox | BoundingBox | Bouding box for this specific prediction. |
| classifications | GList * | a linked list of GstInfereferenceClassification associated to this prediction. |
| predictions | GNode * | a n-ary tree of child predictions within this specific prediction. It is recommended to to access the tree directly, but to use this module's API to interact with the children. |
Each bounding box is a struct defined as follows:
| field | type | description |
|---|---|---|
| x | gint | horizontal coordinate of the upper left position of the bounding box in pixels |
| y | gint | vertical coordinate of the upper left position of the bounding box in pixels |
| width | guint | width of the bounding box in pixels |
| height | guint | height of the bounding box in pixels |
GstInferenceClassification
Each prediction can contain a number of classifications. The GstInferenceClassification struct holds all the potentially relevant information regarding such classifications in the fields specified as follows.
| field | type | description |
|---|---|---|
| classification_id | guint64 | A unique id associated to this classification. |
| class_id | gint | The numerical id associated to the assigned class. |
| class_prob | gdouble | The resulting probability of the assigned class. Typically ranges between 0 and 1. |
| class_label | gchar * | The label associated to this class or NULL if not available. |
| num_classes | gint | The total amount of classes of the entire prediction. |
| probabilities | gdouble* | The entire array of probabilities of the prediction. |
| labels | gchar** | The entire array of labels of the prediction. NULL if not available. |
Programmatic Metadata Access
If you want to access this metadata from your custom Gstreamer element instead the process is fairly easy:
- Include GstInference metadata header:
#include "gst/r2inference/gstinferencemeta.h" - Get a GstInferenceMeta object from the buffer:
inference_meta = (GstInferenceMeta *) gst_buffer_get_meta (frame->buffer, GST_INFERENCE_META_API_TYPE); - Grab the prediction from the meta:
pred = inference_meta->prediction;
The following snippet exemplifies how to interact with the prediction tree.
1 #include "gst/r2inference/gstinferencemeta.h"
2
3 static void
4 get_buffer(GstPadProbeInfo * info)
5 {
6 GstBuffer *buffer;
7 GstInferenceMeta *meta;
8 GstInferencePrediction *root, *child;
9 GstInferenceClassification *classification;
10 GSList *child_predictions;
11 GList *classes;
12
13 gchar *str;
14
15 buffer = gst_pad_probe_info_get_buffer (info);
16 meta = (GstInferenceMeta *) gst_buffer_get_meta (buffer,
17 GST_INFERENCE_META_API_TYPE);
18 root = meta->prediction;
19
20 GST_INFERENCE_PREDICTION_LOCK (root);
21
22 /* Print the entire prediction tree */
23 str = gst_inference_prediction_to_string (root);
24 GST_INFO ("Prediction tree: %s", str);
25 g_free (str);
26
27 /* Iterate through the immediate child predictions */
28 for (child_predictions = gst_inference_prediction_get_children(root);
29 child_predictions; child_predictions = g_slist_next (child_predictions)) {
30 child = (GstInferencePrediction *)child_predictions->data;
31
32 /* On each children, iterate through the different associated classes */
33 for (classes = child->classifications;
34 classes;
35 classes = g_list_next (classes)) {
36 classification = (GstInferenceClassification *)classes->data;
37 }
38 }
39
40 GST_INFERENCE_PREDICTION_UNLOCK (root);
41
42 }
