R2Inference - ONNXRT OpenVINO
 | Make sure you also check R2Inference's companion project: GstInference |
| R2Inference |
|---|
 |
| Introduction |
| Getting started |
| Supported backends |
| Examples |
| Model Zoo |
| Contact us |
 |
Contents
Description
The ONNXRT OpenVINO backend is an extension of the ONNXRT backend. This backend is based on Intel's OpenVINO toolkit support available in ONNX Runtime. OpenVINO offers a boost in performance through optimizations for common computer vision and deep learning workloads on Intel's hardware (CPUs, Movidius USB sticks, MyriadX VPUs, and FPGAs).
Installation
The R2Inference ONNXRT backend depends on the C/C++ ONNX Runtime API. The following installation process is based on the ONNX Runtime source code.
x86
The ONNX Runtime C/C++ API can be installed from source code. This section presents instructions to build from the source files based on the official installation guide for OpenVINO execution provider.
The following commands are based on the instruction of the official guide presented before:
- Build and install OpenVINO. Follow the instructions in the OpenVINO documentation, the instructions are detailed and will bring you to complete setup for the OpenVINO toolkit, it is highly recommended to test the sample demos available to make sure everything is set up correctly.
An important aspect of the OpenVINO installation is to make sure that you have run the following script:
source <openvino_install_directory>/bin/setupvars.sh
- Install ONNX Runtime with OpenVINO support
#This step is needed if you require to use ONNX tools included in onnxruntime repo
pip3 install onnx
#Build onnx runtime
git clone --recursive https://github.com/Microsoft/onnxruntime -b v1.4.0
cd onnxruntime
./build.sh --config RelWithDebInfo --build_shared_lib --parallel --use_openvino CPU_FP32 #default hardware execution
cd build/Linux/RelWithDebInfo
sudo make install
sudo cp libonnxruntime.so.1.4.0 /usr/lib/aarch64-linux-gnu/libonnxruntime.so.1.4.0
sudo ln -s /usr/lib/aarch64-linux-gnu/libonnxruntime.so.1.4.0 /usr/lib/aarch64-linux-gnu/libonnxruntime.so
API
Some documentation of the C/C++ ONNX Runtime API can be found in onnxruntime_c_api.h and onnxruntime_cxx_api.h. The R2Inference uses the C++ API which is mostly a wrapper for the C API. R2Inference provides a high-level abstraction for loading the ONNX model, creating the ONNX Runtime session, and executing the inference of the model. We recommend looking at the examples available for this backend in the r2inference repository to get familiar with the R2Inference interface. R2Inference also abstracts many options available in the ONNX Runtime C++ API through the "IParameters class".
The parameters listed below are currently supported:
| Property | C++ API Counterpart | Value | Operation | Description |
|---|---|---|---|---|
| logging-level | OrtLoggingLevel | Integer | R/W and Write before start | Level of log information of the ONNX Runtime session. |
| log-id | N/A | String | R/W and Write before start | String identification of the ONNX Runtime session. |
| intra-num-threads | SessionOptions::SetIntraOpNumThreads() | Integer | R/W and Write before start | Number of threads to parallelize execution within model nodes. |
| graph-optimization-level | GraphOptimizationLevel | Integer | R/W and Write before start | Graph optimization level of the ONNX Runtime session. |
| hardware-id | HardwareId | String | R/W and Write before start | OpenVINO hardware device id |
OpenVINO setup for hardware devices
Neural Compute Stick 2
The following instructions were taken from the official OpenVINO documentation:
#Add the current Linux user to the users group:
sudo usermod -a -G users "$(whoami)"
#Log out and log in for it to take effect.
#To perform inference on Intel® Neural Compute Stick 2, install the USB rules as follows:
sudo cp /opt/intel/openvino/inference_engine/external/97-myriad-usbboot.rules /etc/udev/rules.d/
sudo udevadm control --reload-rules
sudo udevadm trigger
sudo ldconfig
You can try to verify the setup by running Security Barrier Camera Demo with the MYRIAD hardware option.
cd /opt/intel/openvino/deployment_tools/demo
./demo_security_barrier_camera.sh -d MYRIAD
