GstInference/Example Applications/Classification

GstInference
Introduction
Getting started
Getting the code Building the plugin Quick Starting Guide
Supported architectures
InceptionV1 InceptionV2 InceptionV3 InceptionV4 TinyYoloV2 TinyYoloV3 YoloV2 ResNetV1 MobileNetV2 MobileNetV2 + SSD AlexNet
Supported backends
TensorFlow Tensorflow-Lite Coral from Google Caffe TensorRT ONNXRT ONNXRT ACL ONNXRT OpenVINO
Metadata and Signals
GstClassificationMeta GstInferenceMeta Signals
Overlay Elements
Inference Overlay
Utils Elements
InferenceFilter InferenceCrop InferenceDebug InferenceBin
Legacy pipelines
PC NANO TX2 Xavier IMX8
Example pipelines
Example applications
Classification Detection Hand Pose Recognition
Benchmarks
Model Zoo
Project Status
Releases Roadmap
Contact Us

This example receives an input video file and classifies each frame into one of 1000 possible classes. For each classified frame the application captures the signal emitted by GstInference, forwarding the prediction to a placeholder for external logic. Simultaneously, the pipeline displays the captured frames with the associated label in a window. Note that the image currently being displayed not necessarily matches the one being handled by the signal. The display is meant for visualization and debugging purposes.

The classification architecture being used by the example is InceptionV4 trained using the Large Scale Visual Recognition Challenge 2012 (ILSVRC2012) Dataset. A pre-trained model can be downloaded from the GstInference Model Zoo

This examples serves both as an example and as a starting point for a classification application.

Building the Example

The example builds along the GstInference project. Make sure you follow the instructions in Building the Plug-In to make sure all the dependencies are correctly fulfilled.

Once the project is built the example may be built independently by running make within the example directory.

1 cd tests/examples/classification
2 make

The example is not meant to be installed.

Running the Example

The classification application provides a series of cmdline options to control de behavior of the example. The basic usage is:

./classification -m MODEL -f FILE -b BACKEND [-v]

-m|--model

Mandatory. Path to the InceptionV4 trained model

-f|--file

Mandatory. Path to the video file to be used

-b|--backed

Mandatory. Name of the backed to be used. See Supported Backends for a list of possible options.

-v

Optional. Run verbosely.

You may always run --help for further details.

./classification --help
Usage:
  classification [OPTIONS] - GstInference Classification Example

Help Options:
  -h, --help                        Show help options
  --help-all                        Show all help options
  --help-gst                        Show GStreamer Options

Application Options:
  -v, --verbose                     Be verbose
  -m, --model                       Model path
  -f, --file                        File path
  -b, --backend                     Backend used for inference, example: tensorflow

Extending the Application

The example is composed by the following main files:

gstclassification.c

Source file with GStreamer logic

customlogic.c

Placeholder for custom logic

customlogic.h

Header file for custom logic source

Makefile

Build script

classification

Executable generated on build

The application can be extended by filling in the placeholders in customlogic.c. In the most simple cases, the information provided in handle_predictionshould suffice to react to the prediction.

 1 void
 2 handle_prediction (unsigned char *image, 
 3                    int width, 
 4                    int height,
 5                    int size,
 6                    double *probabilities,
 7                    int num_probabilities)
 8 {
 9     /* FILLME: Handle image and prediction here */
10 }

image

Image data in RGB raster format.

width

The width of the image, in pixels

height

The height of the image, in pixels

size

The size of the total image, in bytes

probabilities

An array of probabilities for each class

num_probabilities

The length of the probabilities array

NOTE: The signal will block buffer streaming. Long processing operations should perform asynchronously to avoid blocking the pipeline

Troubleshooting

The first debug level is GStreamer debug. You may execute the application with debug enabled by running:

1 ./classification --gst-debug=2

For advanced support, please contact [1] with the output of the following command:

1 ./classification --gst-debug=2,videoinference:6,inceptionv4:6

Reporting a Bug

Please feel free to report bugs using GitHub's issue tracker.