GstInference/Supported backends/NCSDK

GstInference
Introduction
Getting started
Getting the code Building the plugin Quick Starting Guide
Supported architectures
InceptionV1 InceptionV2 InceptionV3 InceptionV4 TinyYoloV2 TinyYoloV3 YoloV2 ResNetV1 MobileNetV2 MobileNetV2 + SSD AlexNet
Supported backends
TensorFlow Tensorflow-Lite Coral from Google Caffe TensorRT ONNXRT ONNXRT ACL ONNXRT OpenVINO
Metadata and Signals
GstClassificationMeta GstInferenceMeta Signals
Overlay Elements
Inference Overlay
Utils Elements
InferenceFilter InferenceCrop InferenceDebug InferenceBin
Legacy pipelines
PC NANO TX2 Xavier IMX8
Example pipelines
Example applications
Classification Detection Hand Pose Recognition
Benchmarks
Model Zoo
Project Status
Releases Roadmap
Contact Us

The NCSDK Intel® Movidius™ Neural Compute SDK (Intel® Movidius™ NCSDK) enables deployment of deep neural networks on compatible devices such as the Intel® Movidius™ Neural Compute Stick. The NCSDK includes a set of software tools to compile, profile, and validate DNNs (Deep Neural Networks) as well as APIs on C/C++ and Python for application development.

To use the ncsdk on Gst-Inference be sure to run the R2Inference configure with the flag --enable-ncsdk and use the property backend=ncsdk on the Gst-Inference plugins.

Installation

You can install the NCSDK on a system running Linux directly, downloading a Docker container, on a virtual machine or using a Python virtual environment. All the possible installation paths are documented on the official installation guide.

We also provide an installation guide with troubleshooting on the Intel Movidius Installation wiki page

Note: It is recommended to take the docker container route on the NCSDK installation. Other routes may affect your python environment because it sometimes uninstalls and reinstalls python and some common plugins such as numpy or TensorFlow. Docker installation is actually very simple and it doesn't affect your environment at all. Use this link to jump directly to the docker section on the installation guide.

Enabling the backend

To enable NCSDK as a backend for GstInference you need to install R2Inference with NCSDK support. To do this, use the option --enable-ncsdk during R2Inference configure:

./autogen.sh --enable-ncsdk

Generating a graph

When you use the ncsdk backend you will need a compiled ncs graph file. You can obtain this file from tensorflow's protobuff and weights filer; or caffe's prototxt and caffemodel files. mvNCCompile is a tool included with the ncsdk installation that compiles a network and produces a graph file that is compatible with the NCAPI and the Gst-Inference plugins using the ncsdk backend.

For example, given a caffe model (googlenet.caffemodel) and a network description (deploy.prototxt):

mvNCCompile -w googlenet.caffemodel -s 12 deploy.prototxt

This command will output the graph and output_expected.npy files, that can be used later with the googlenet plugin.

Options

You can find the full documentation of the C API here and the Python API here. Gst-Inference uses only the C API and R2Inference takes care of devices, graphs, models and fifos. Because of this, we will only take a look at the options that you can change when using the C API through R2Inference.

The following syntax is used to change backend options on Gst-Inference plugins:

backend::<property>

For example to change the NCSDK API log level of the googlenet plugin you need to run the pipeline like this:

gst-launch-1.0 \
googlenet name=net model-location=/root/r2inference/examples/r2i/ncsdk/graph_googlenet backend=ncsdk backend::log-level=1 \
videotestsrc ! tee name=t \
t. ! queue ! videoconvert ! videoscale ! net.sink_model \
t. ! queue ! net.sink_bypass \
net.src_bypass ! fakesink

The backend::log-level=1 section of the pipeline sets the NC_RW_LOG_LEVEL option of the NCSDK C API to 1.

To learn more about the NCSDK C API option, please check the NCSDK wiki page on the R2Inference subwiki.

Device Options

All the device options are read only.

Property	C API Counterpart	Value	Description
thermal-throttling-level	NC_RO_THERMAL_THROTTLING_LEVEL	Integer (0,1,2)	0: No limit reached. 1: Lower temperature guard threshold reached. 2: Upper temperature guard threshold reached.
device-state	NC_RO_DEVICE_STATE	Integer (0,1,2,3)	The current state of the device: 0: CREATED: The struct has been initialized. 1: OPENED: The device communication has been opened. 2: CLOSED: Communication with the device has been closed. 3: DESTROYED: The device handler has been freed.
current-memory-used	NC_RO_DEVICE_CURRENT_MEMORY_USED	Integer	Current memory used on the device.
memory-size	NC_RO_DEVICE_MEMORY_SIZE	Integer	Total memory available on the device.
max-fifo-num	NC_RO_DEVICE_MAX_FIFO_NUM	Integer	Max number of fifos.
allocated-fifo-num	NC_RO_DEVICE_ALLOCATED_FIFO_NUM	Integer	Number of fifos currently allocated.
max-graph-num	NC_RO_DEVICE_MAX_GRAPH_NUM	Integer	Max number of graphs.
allocated-graph-num	NC_RO_ALLOCATED_GRAPH_NUM	Integer	Number of graphs currently allocated.
option-class-limit	NC_RO_DEVICE_OPTION_CLASS_LIMIT	Integer	Highest option class supported.
device-name	NC_RO_DEVICE_NAME	String	Device name.

Fifo Options

Most of the R/W options on the FIFO can only be modified between creation and allocation, and R2Inference does both in a single method (Engine->Start()), so it is impossible to write on these options. R2Inference also fixates those options to our specific implementation, so they are not exposed on the plugin.

Global Options

Pay special attention to the log level enumeration, because it is ordered counter intuitively. 1 is actually the highest log level, 4 is the lowest and 0 the default.

Property	C API Counterpart	Value	Description
log-level	NC_RW_LOG_LEVEL	Integer	NCSDK debug log level from ncLogLevel_t enum 0: NC_LOG_DEBUG: Debug, warning, error and fatal. 1: NC_LOG_INFO: Info, debug, warning, error and fatal. 2: NC_LOG_WARN: Warning, error and fatal. 3: NC_LOG_ERROR: Error and fatal. 4: NC_LOG_FATAL: Fatal only.

Graph Options

Property	C API Counterpart	Value	Description
graph-state	NC_RO_GRAPH_STATE	Integer	The current state of the graph from ncGraphState_t enum 0: NC_GRAPH_CREATED: The struct has been initialized. 1: NC_GRAPH_ALLOCATED: The graph has been allocated. 2: NC_GRAPH_WAITING_FOR_BUFFERS: The graph is waiting for input. 3: NC_GRAPH_RUNNING: The graph is currently running an inference.
graph-input-count	NC_RO_GRAPH_INPUT_TENSOR_DESCRIPTORS	Integer	Array of graph inputs. Returns the size of the array instead of the array itself.
graph-output-count	NC_RO_GRAPH_OUTPUT_TENSOR_DESCRIPTORS	Integer	Array of graph outputs. Returns the size of the array instead of the array itself.
graph-debug-info	NC_RO_GRAPH_DEBUG_INFO	String	Debug information.
graph-name	NC_RO_GRAPH_NAME	String	Graph name.
graph-option-class-limit	NC_RO_GRAPH_OPTION_CLASS_LIMIT	Integer	The highest option class supported.
graph-version	NC_RO_GRAPH_VERSION	String	The version ([major, minor]) of the compiled graph.

NCSDK Tools

The NCSDK installation include some useful tools to analyze, optimize and compile models. We will mention these tools here, but if you want some examples and a more complete description please check the NCSDK wiki page on the R2Inference subwiki.

• mvNCCheck: Checks the validity of a Caffe or TensorFlow model on a neural compute device. The check is done by running an inference on both the device and in software and then comparing the results to determine a if the network passes or fails.
• mvNCCompile: Compiles a network and weights files from Caffe or TensorFlow models into a graph file that is compatible with the NCAPI.
• mvNCProfile: Compiles a network, runs it on a connected neural compute device, and outputs profiling info on the terminal and on an HTML file. The profiling data contains layer performance and execution time of the model. The html version of the report also contains a graphical representation of the model.