Stitcher element Additional Details
| Image Stitching for NVIDIA®Jetson™ | |
|---|---|
 | |
| Image Stitching for NVIDIA Jetson Basics | |
|
|
|
| Getting Started | |
|
|
|
| User Guide | |
|
|
|
| Compiling OpenCV for Image Stitching | |
|
|
|
| Examples | |
|
|
|
| 360 Video | |
|
|
|
| Performance | |
|
|
|
| Contact Us |
|
Contents
Additional Details
This wiki holds some additional notes and useful details to consider when using the stitcher element.
Video Cropping
The stitcher element applies multiple transformations to the target sources in order to achieve a seamless match with the target. These transformations can in some cases distort the output resolution and provide an image that is a different size and shape than desired. The image below represents an example of this behavior:
To crop the stitcher's output into the expected resolutions Gstreamer's videocrop element can be used. This element takes 4 main parameters (top, bottom, left, right) specifying the number of pixels to remove from each side respectively.
The basic idea for determining these parameters is to take the original (without cropping) output of the stitcher, and based on its dimensions, subtract from each side accordingly.
Using videocrop after the stitcher on your pipeline can reduce the output to only what is needed. The original image can then turn into:
Integration with the homography estimation tool
The homography estimation tool by default limits its output to a resolution that keeps its height the same as the original image and the width is cropped at the addition between input widths; for example, stitching two 1920x1080 images produces an output with dimensions width=(1920+1920)=3840; height=1080.
If you want the output for the stitcher element to look the same as the homography estimation, you should add a videocrop element at the output and determine its parameters like so:
To get the complete output you can run the homography estimation tool using the --non_crop option, this will you the complete image so from there it is easier to decide what you want to keep and from what direction to crop. Using this option the output you get in result.jpg will have the dimensions of the complete image.
Then you can run the tool again without the --non-crop option; the resulting resolution will be as explained before (same height, widths added).
From these two runs, we have the following information:
- [math]\displaystyle{ width_{full} }[/math]: the full width of the image without cropping; same as the stitcher element.
- [math]\displaystyle{ height_{full} }[/math]: the full height of the image without cropping; same as the stitcher element.
- [math]\displaystyle{ width_{crop} }[/math]: the cropped width of the image, the same as the desired output.
- [math]\displaystyle{ height_{crop} }[/math]: the cropped height of the image, the same as the desired output.
From there the videocrop parameters can be determined:
- top: [math]\displaystyle{ \frac{(height_{full} - height_{crop})}{2} }[/math]
- bottom: [math]\displaystyle{ \frac{(height_{full} - height_{crop})}{2} }[/math]
- left: [math]\displaystyle{ (width_{full}-width_{crop}) }[/math] if mode is
right_fixed; 0 otherwise.
- right: [math]\displaystyle{ (width_{full}-width_{crop}) }[/math] if mode is
left_fixed; 0 otherwise.
The formulas for top and bottom parameters assume that the transformation produced deformation symmetrically in the vertical dimension; that is usually the case if cameras are parallel to one another; however, depending on your use case; you might need to adjust the proportion between the top and bottom accordingly.
That 50/50 assumption is usually a good place to start and adjust from there according to the output.
Known Issues
From working with multiple clients and integrating the stitcher into their environments we have gathered the following information regarding some limitations for the element:
The stitcher is a project under constant development, and these issues are likely to be solved in the future; if you would like to sponsor the development of a new feature or enhancement please get in touch at contactus@ridgerun.com.
CUDA interpolation algorithm illegal memory access
Currently, there is an issue inside CUDA that produces illegal memory access for some homographies, this issue is dependent on the homography values as well as the interpolation algorithm being used. You can find more information about this issue here: https://github.com/opencv/opencv_contrib/issues/2361
This produces an error in the element that looks like the following:
illegal memory access was encountered in the function 'waitForCompletion'
If you get this error you can change the interpolation algorithm with the interpolation-algorithm property to nearest. This will affect the quality of the image warping but will avoid the issue; another option is to change the homography slightly and see what value is producing the error.
Deepstream NVMM integration
The stitcher element works with NVMM buffers for DMA acceleration; therefore it can integrate with other elements that handle this video/x-raw(Memory:NVMM) media format. However, there is an issue when working with NVIDIA's deepstream elements; because even though those elements use the same caps (video/x-raw(Memory:NVMM)) as the stitcher (so the negotiation occurs as expected) the underlying format is different and produces a runtime error.
