View source for Panoramic Stitching and WebRTC Streaming on NVIDIA Jetson

<seo title="Panoramic Stitching | Panoramic Stitching and WebRTC Streaming on NVIDIA Jetson | RidgeRun" titlemode="replace" keywords=" GStreamer, Linux SDK, Linux BSP,  Embedded Linux, Device Drivers, NVIDIA, Jetson, TX1, TX2, Xilinx, TI, NXP, Freescale, Embedded Linux driver development, Linux Software development, Embedded Linux SDK, Embedded Linux Application development, GStreamer Multimedia Framework, Jetson, Jetson AGX Xavier, Jeston Xavier NX, Xavier NX, Jetson Xavier, Xavier, Jetson TX1, Jetson TX2, AI, Deep Learning, NVIDIA Jetson Orin, Jetson Orin, Orin, NVIDIA Orin, NVIDIA Jetson AGX Orin, Jetson AGX Orin, Panoramic Stitching, WebRTC Streaming, WebRTC."  description="This wiki serves as a user guide for the Panoramic Stitching and WebRTC Streaming on NVIDIA Jetson reference design."></seo>


This wiki serves as a user guide for the '''Panoramic Stitching and WebRTC Streaming on NVIDIA Jetson''' reference design.

<table>
<tr>
<td><div class="clear; float:right">__TOC__</div></td>
<td>
{{NVIDIA Preferred Partner logo}}
<td> 
<center>
{{ContactUs Button}}
</center>
</tr>
</table>

<br>
<center>
<embedvideo service="youtube">https://youtu.be/XNMZoRdIguc</embedvideo>
</center>
<br>
== Overview ==

The demo makes use of a Jetson AGX Orin devkit to create a 360 panoramic image in real time from 3 different fisheye cameras. The result is then streamed to a remote browser via WebRTC. The following image summarizes the overall functionality.
<br>
<br>
[[File:Panoramic_webrtc_overview.png|frame|center|Overview of the reference design.]]
<br>
'''1.''' The system captures three independent video streams from three different RTSP cameras. Each camera captures an image size of 3840x2160 (4K) at a rate of 30 frames per second. Furthermore, they are equipped with a 180° equisolid fisheye lens. The cameras are arranged so that they each capture perspectives at 120° between them. The following image explains graphically this concept.
<br>
<br>
[[File:panoramic_webrtc_rtsp_cameras.png|frame|center|Physical configurations of the cameras.]]
<br>
As you can see, the cameras, combined, cover the full 360° field of view of the scene. Since each camera is able to capture a 180° field of view, and they facing 120° apart from each other, there is a 60° overlap between them. This overlap is important because it gives the system some safeguard to blend smoothly the two images. Besides this configuration, it is important that the cameras have little to none tangential or radial angles.

'''2.''' The Jetson AGX Orin receives these RTSP streams and decodes them. These images are then used to create a panoramic representation of the full 360° view of the scene. The different HW accelerators in the SoM are carefully configured to achieve processing in real-time. The image below shows an example of the resulting panorama.
<br>
<br>
[[File:panoramic_webrtc_panorama_example.png|800px|thumb|center|Example of the generated panorama.]]
<br>
'''3.''' Finally, the panorama is encoded and streamed through the network to a client browser using WebRTC. Along with the panorama, the original fisheye images are also sent. The client is presented with an interactive web page that allows them to explore each camera stream independently. The following image shows the implemented web page in mid-operation.
<br>
<br>
[[File:Panoramic_webrtc_web_page.png|650px|thumb|center|Screenshot of the WebRTC powered web page.]]
<br>
== System Description ==

While the system should be portable to different platforms of the Jetson family, the results presented in this guide were obtained using a Jetson AGX Orin. More specifically, the configuration used in this demo follows the information detailed in the table below.

<center>
{| class="wikitable"
|-
! Configuration !! Value
|-
| SoM || Jetson AGX Orin NX
|-
| Memory || 64GB
|-
| Carrier || Jetson Orin AGX Devkit
|-
| Camera protocol || RTSP
|-
| Image size || 4K (3840x2160)
|-
| Framerate || 30fps
|-
| Codec || AVC (H.264)
|}
</center>

=== Resource Scheduling ===

To maintain a real-time operation of steady 30fps, this reference design makes use of different hardware modules in SoM. The following image shows graphically how the different processing stages of the system were scheduled to different partitions in the system.
<br>
[[File:Panoramic_webrc_system_description.png|600px|thumb|center|Resource scheduling of the system]]
<br>
To access these units GStreamer is used. In a GStreamer pipeline, each processing accelerator is abstracted in a GStreamer element. The following table describes the different processing stages, the HW accelerators used for them, and the software framework used to access them.

<center>
{| class="wikitable"
|-
! Processing Stage !! Amount !! HW Unit !! Software Framework !! Notes
|-
| Camera || 3 || CPU || GStreamer ([https://gstreamer.freedesktop.org/documentation/rtsp/rtspsrc.html?gi-language=c gstrtspsrc]) ||
|-
| Decoder || 3 || NVENC || GStreamer ([https://docs.nvidia.com/jetson/l4t-multimedia/group__V4L2Dec.html gstnvv4l2decoder]) ||
|-
| Projection || 3 || VIC || [https://docs.nvidia.com/vpi/algo_remap.html VPI (remap)] + [https://www.ridgerun.com/gstvpi GstVPI] || Converts fisheye projections into a partial equirectangular image.
|-
| Image Stitching || 1 || GPU || [https://developer.nvidia.com/cuda-toolkit CUDA] + [https://developer.ridgerun.com/wiki/index.php/GstCUDA GstCUDA] + [https://developer.ridgerun.com/wiki/index.php/Image_Stitching_for_NVIDIA_Jetson CudaStitcher] || Takes the partial equirectangular images and merges them into a representation. Blends seams smoothly.
|-
| Crop and Scaling || 4 || VIC || GStreamer ([https://docs.nvidia.com/jetson/l4t-multimedia/classNvVideoConverter.html gstnvvidconv]) || Crops out black borders resulting from the stitching process. Also scales images to a proper network 1080P transmission size.
|-
| Encoder || 4 || NVENC || GStreamer ([https://docs.nvidia.com/jetson/l4t-multimedia/group__V4L2Enc.html gstnvv4l2h264enc]) ||
|-
| WebRTC || 4 || CPU || GStreamer ([https://gstreamer.freedesktop.org/documentation/webrtc/index.html?gi-language=c webrtcbin]) + [https://developer.ridgerun.com/wiki/index.php/GStreamer_WebRTC_Wrapper GstWebRTCWrapper] || Encrypted network streaming of each camera and the panoramic representation.
|-
|}
</center>

== Running the Demo ==

=== Prepare the camera sources ===

In order to perform panoramic stitching of a set of cameras, a thorough calibration process is needed. The details of the calibration are out of the scope of this document. In order to reproduce the exact same results presented in this guide, we provide 3 pre-recorded videos of the cameras calibrated for the demo. You can use these videos to simulate RTSP cameras from a host PC, for example. Being an emulated video source, synchronization may likely be a little off. Download and store the videos in a known location:
* [https://TODO Left]
* [https://TODO Center]
* [https://TODO Right]

Install VLC, which is the tool we are going to use to simulate the RTSP cameras:
<syntaxhighlight lang=bash>
# In Debian based systems
sudo apt install vlc
</syntaxhighlight>

The following commands are used to loop each file indefinitely:
<syntaxhighlight lang=bash>
# Run each one in a different terminal
cvlc -vvv --loop panoramic_webrtc_demo_left.mp4 --sout="#gather:rtp{sdp=rtsp://127.0.0.1:8554/left}" --sout-keep --sout-all &
cvlc -vvv --loop panoramic_webrtc_demo_center.mp4 --sout="#gather:rtp{sdp=rtsp://127.0.0.1:8554/center}" --sout-keep --sout-all &
cvlc -vvv --loop panoramic_webrtc_demo_right.mp4 --sout="#gather:rtp{sdp=rtsp://127.0.0.1:8554/right}" --sout-keep --sout-all &
</syntaxhighlight>

If everything is working as expected, you should be able to test the camera RTSP source:
<syntaxhighlight lang=bash>
# Run each one in a different terminal
HOST_IP_ADDRESS=127.0.0.1

vlc rtsp://${HOST_IP_ADDRESS}:8554/left
vlc rtsp://${HOST_IP_ADDRESS}:8554/center
vlc rtsp://${HOST_IP_ADDRESS}:8554/right
</syntaxhighlight>
A video player showing the camera should open. The system were the cameras are going to be run should be in the same local network as the Jetson AGX Orin.

=== Run the Docker Image ===

Once you can simulate camera sources, you need to download and run the Docker image that contains the demo binaries. For this, you'll need a Jetson AGX Orin board with the newest L4T version installed. We specifically developed the demo for JetPack 5.0 Developer Preview. The Docker container requires the host (the Orin) to have CUDA and GPU drivers properly installed and running. An install via de SDK Manager should take care of this automatically.

In the Orin terminal, type the following Docker command to pull and run a container from the pre-built image:
<syntaxhighlight lang=bash>
docker run --gpus all -it --rm --net=host --privileged  dockerhub.com/ridgerun/panoramic_stitching_webrtc_demo:latest
</syntaxhighlight>

Due to all the software dependencies, the image is quite big (> 2GB) so it may take a while to download and build the container. In the command above you'll see the following mandatory options:
;--gpus all
:Makes all GPUs available to the container.
;-it
:Makes the container interactive by allocating a terminal so you can interact with it.
;--rm
:Remove the container after usage.
;--net=host
:Connects the container to the host network. Necessary for proper device execution.
;--privileged
:Grants the container access to the host resources.

After the container has been created, you'll be welcomed with a brand new terminal you can interact with. Type ''exit'' when you wish to return to the host system and discard the container.

=== Start the Web Services ===

The web services have two purposes:
# Host the web page
# Perform WebRTC signaler between the Orin and the browser

For convenience, these are provided within the Docker container to be run in the Orin. To run them, execute the following in the Orin terminal:
<syntaxhighlight lang=bash>
cd ~/web_services
HTTPS_KEY=demo.key HTTPS_CERT=demo.crt ./start_services
</syntaxhighlight>
The HTTPS_KEY and HTTPS_CERT point to the key and certificate necessary to perform an HTTPS connection between the browser, Orin, and the services. This is required for proper WebRTC streaming.

If everything went okay, you should be able to open a browser and navigate to https://ORIN_IP_ADDRESS:8443

==== Regenerating Expired Key Pairs ====
The container provides an example key and certificate for demonstration purposes. These, however, have an expiration data and may need to be regenerated. If this is your case, please run the following:
<syntaxhighlight lang=bash>
cd ~/web_services
openssl req -newkey rsa:2048 -nodes -keyout demo_new.key -x509 -days 365 -out demo_new.crt
</syntaxhighlight>

<!--
<pre style=background-color:yellow>
A word of caution, NEVER deploy these demo keys to production or a public server.
</pre>
-->
{{Ambox
|nocat=true
|small=left
| type  = content
| text  = <span style="text-weight:bold;color:red">'''Warning:''' <u></span>'''<pre style=background-color:yellow>A word of caution, NEVER deploy these demo keys to production or a public server.</pre>'''</u> 
|style=width:unset;
}}

==== Running the Services Independently ====

The signaler and the web server are independent processes and may, as such, be run separately. To do so, run the following:
<syntaxhighlight lang=bash>
# The WebRTC signaler
HTTPS_KEY=demo.key HTTPS_CERT=demo.crt ./start_signaler

# The web server
HTTPS_KEY=demo.key HTTPS_CERT=demo.crt ./start_web_server
</syntaxhighlight>

They can even be run from a different machine. If you decide to do so, please make sure you update the IP address in the GStreamer pipelines and from the web page.

=== Running the Demo ===

We are finally ready to run the demo.

'''1.''' In a host computer, open '''Google Chrome''' and browse to https://ORIN_IP_ADDRESS:8554. Note that, this computer must be in the same local network as the Jetson Orin and, in consequence, as the RTSP cameras. <br>
'''2.''' In the web page, click on the '''Connect!''' button. You should see a spinner indicating that the system is ready and waiting for the Jetson Orin to connect.
<br>
<br>
[[File:Panoramic_webrc_system_spinner.png|600px|thumb|center|Spinner indicating that the system is ready for the Orin to connect.]]
<br>
'''3.''' In the Jetson Orin container, start the processing pipeline by running the following:
<syntaxhighlight lang=bash>
cd ~/processing_pipeline
./run_pipeline
</syntaxhighlight>
'''4.''' The webpage should recognize the Orin connection and replace the spinner with a video component. Be aware that, for various reasons, this may take up to a few seconds.
<br>
<br>
[[File:Panoramic_webrtc_web_page.png|600px|thumb|center|The view of the web page with the Orin property connected.]]
<br>
'''5.''' At any point, you should be able to disconnect the orin and the web page will return to the spinner view. By reconnecting the Orin the videos will resume.

==== Troubleshooting ====

First of all, make sure you are using Google Chrome. WebRTC is very picky with the browser and we've found that other browsers may present various different problems.

If the problem persists, please:
'''1.''' Refresh the web page.
'''2.''' Open the browser console: ''Ctrl + Shift + J'' (Or ''Command + Option + J'' on Mac)
'''3.''' Start the processing pipeline with debug activated:
<syntaxhighlight lang=bash>
cd ~/processing_pipeline
GST_DEBUG=3 ./run_pipeline
</syntaxhighlight>
'''6.''' [mailto:support@ridgerun.com '''Email us'''] the problem, what you see, and both the terminal and browser console log.

== Results and Performance ==

The following video shows the demo in action by streaming to the web page. 
<br>
<br>
<center>
<embedvideo service="youtube">https://youtu.be/XNMZoRdIguc</embedvideo>
</center>
<br>
The following figure presents the resource usage of the different HW unit involved in the processing.
<br>
<br>
[[File:Panoramic_webrtc_performance.png|750px|thumb|center|Resource usage of the different HW units involved in the processing]]
<br>
<br>
In a similar way, the table below summarizes the same information but in a tabular fashion.

<center>
{| class="wikitable"
|-
! HW Unit !! colspan=12 | Measurement
|-
| rowspan=2 | CPU (% @ MHz) || 0 || 1 || 2 || 3 || 4 || 5 || 6 || 7 || 8 || 9 || 10 || 11
|-
| 20%@729 || 23%@729 || 27%@729 || 9%@729 || 7%@729 || 11%@729 || 10%@729 || 10%@729 || 12%@2201 || 20%@2201 || 18%@2201 || 12%@2201
|-
| rowspan=2 | Mem || colspan=6 | RAM (MB) || colspan=6 | EMC (% @ MHz)
|-
| colspan=6 | 3937/30623MB || colspan=6 | 21%3199MHz
|-
| rowspan=2 | Codec || colspan=6 | NVENC (MHz) || colspan=6 | NVDEC (MHz)
|-
| colspan=6 | 998MHz || colspan=6 | 806MHz 
|-
| rowspan=2 | Image || colspan=12 | VIC (% @ MHz)
|-
| colspan=12 | 92%729MHz
|-
| rowspan=2 | GPU || colspan=6 | GR3D (% @ MHz) || colspan=6 | GR3D2 (% @ MHz)
|-
| colspan=6 | 65%318 || colspan=6 | 15@318
|}
</center>

== Known Issues ==

At the time being, the design is under development, and are certain issues that need to be addressed:
* Image quality on individual cameras is very poor. This is a work in progress and is expected to be fixed in the near future.
* Delay during initialization.

{{ContactUs}}

[[Category:Jetson]][[Category:JetsonNano]][[Category:JetsonTX2]][[Category:NVIDIA Xavier]][[Category:JetsonXavierNX]][[Category:NVIDIA Jetson Orin]]