Image Stitching for NVIDIA Jetson/User Guide/Additional Details

Image Stitching for NVIDIA®Jetson™
Image Stitching for NVIDIA Jetson Basics
Getting Started
Evaluating the Stitcher Getting the code Building Image Stitching for NVIDIA Jetson
User Guide
Controlling the Stitcher Homography estimation Homography list Homography refinement Blending Additional details
Compiling OpenCV for Image Stitching
Examples
Basic pipelines Using Gstd Other pipelines
360 Video
About 360 video Equirectangular Projection Projector Building Projector Stitcher Calibration Tool
Performance
Xavier Nano
Contact Us

Additional Details

This wiki holds some additional notes and useful details to consider when using the stitcher element.

Video Cropping

The stitcher element applies multiple transformations to the target sources in order to achieve a seamless match with the target. These transformations can in some cases distortion the output resolution and provide an image that is a different size and shape than desired. The image below represents an example of this behavior:

To crop the stitcher's output into the expected resolutions Gstreamer's videocrop element can be used. This element takes 4 main parameters (top, bottom, left, right) specifying the number of pixels to remove from each side respectively.

The basic idea for determining these parameters is to take the original (without cropping) output of the stitcher, and based on its dimensions, subtract from each side accordingly.

Using videocrop after the stitcher on your pipeline can reduce the output to only what is needed. The original image can then turn into:

Integration with the homography estimation tool

The homography estimation tool by default limits its output to a resolution keeps its height the same as the original image and the width is cropped at the addition between input widths; for example, stitching two 1920x1080 images, produces an output with dimensions width=(1920+1920)=3840; height=1080.

If you want the output for the stitcher element to look the same as the homography estimation, you should add a videocrop element at the output and determine its parameters like so:

To get the complete output you can run the homography estimation tool using the --non_crop option, this will you the complete image so from there it is easier to decide what you want to keep and from what direction to crop. Using this option the output you get in result.jpg will have the dimensions of the complete image.

Then you can run the tool again without the --non-crop option; the resulting resolution will be as explained before (same height, widths added).

From these two runs, we have the following information:

• [math]\displaystyle{ width_{full} }[/math]: the full width of the image without cropping; same as the stitcher element.
• [math]\displaystyle{ height_{full} }[/math]: the full height of the image without cropping; same as the stitcher element.
• [math]\displaystyle{ width_{crop} }[/math]: the cropped width of the image, same as the desired output.
• [math]\displaystyle{ height_{crop} }[/math]: the cropped height of the image, same as the desired output.

From there the videocrop parameters can be determined:

• top: [math]\displaystyle{ \frac{(height_{full} - height_{crop})}{2} }[/math]

• bottom: [math]\displaystyle{ \frac{(height_{full} - height_{crop})}{2} }[/math]

• left: [math]\displaystyle{ (width_{full}-width_{crop}) }[/math] if mode is right_fixed; 0 otherwise.

• right: [math]\displaystyle{ (width_{full}-width_{crop}) }[/math] if mode is left_fixed; 0 otherwise.

The formulas for top and bottom parameters assume that the transformation produced deformation symmetrically in the vertical dimension; that is usually the case if cameras are parallel to one another; however depending on your use case; you might need to adjust the proportion between top and bottom accordingly.

That 50/50 assumption is usually a good place to start and adjust from there according to the output.

Known Issues

From working with multiple clients and integrating the stitcher into their environments we have gathered the following information regarding some limitations for the element:

The stitcher is a project under constant development, and this issues are likely to be solved in the future; if you would like to sponsor the development of a new feature or enhancment please get in touch at contactus@ridgerun.com.

Deepstream NVMM integration

The stitcher element works with NVMM buffers for dma acceleration; therefore it can integrate with other elements that handle this video/x-raw(Memory:NVMM) media format. However, there is an issue when working with NVIDIA's deepstream elements; because even though those elements use the same caps (video/x-raw(Memory:NVMM)) as the stitcher (so the negotiation occurs as expected) the underlying format is different and produces a runtime error.