Stitcher

Cuda Stitcher Element

The RidgeRun’s cuda stitching solution includes a GStreamer element capable of performing video stitching, the element is named cudastitcher. The cudastitcher uses CUDA acceleration to perform the stitching process, so as the projector, it has special memory requirements and uses RidgeRun’s project gst-cuda. Specifically this element inherits from CudaBaseMiso, since it can receive video from any amount of inputs and produces a single output video. The stitcher can accept RGBA and GRAY8 video streams in cuda memory and only RGBA format in NVMM memory.

The stitcher requires an homography configuration to determine the relationship between the input images and which transformations need to be applied. For each of the inputs the element creates a dependency tree and generates the final transformation needed, it applies the transformation to each of the inputs to generate the stitched frame. Indeed the output video frames resolution is determined by the estimation of the stitched frame corners, using the corners of each of the image corners transformed with the corresponding homography.

After the homography transformation the stitcher element performs a blending stage, the objective of this stage is to make a smoother transition between the images and reduce as much as possible the visibility of the seam. Take into account that for large overlap regions and depending on the distance and motion of the objects in the video, the blending process may produce ghosting where you can see the superposition of the object in different locations.

The stitcher element can crop the borders of the individual images to reduce the overlap region to reduce ghosting or remove unwanted borders. Individual crop parameters for each image can be configured on the GstStitcherPad, you must take into account that the crop is applied to the input image before applying the homography, so the crop areas are in pixels from the input image. Following is the list of properties available for the stitcher pad:

bottom: amount of pixels to crop at the bottom of the image.
left: amount of pixels to crop at the left side of the image.
right: amount of pixels to crop at the right side of the image.
top: amount of pixels to crop at the top side of the image.

Properties

async-homography-enable: A boolean property that enables the asynchronous homography estimation.

homography-list: A string property that describes the required homography configuration in a JSON format. You can check the documentation to understand the format of this JSON.

Sample Pipelines

The following pipeline creates a stitched 360 frame from 3 fisheye of 180 degrees pre-recorded videos. The cameras configuration can be seen in the following image:

Each of the videos is decoded and projected to an equirectangular using Ridgerun’s equirectangular projector. The projector is configured according to the characteristics of each fisheye lens and the camera's array, the appropriate values were determined with the calibration tool. Each rreqrprojector is linked to the cudastitcher, so it will have three inputs and the appropriate homography configuration is set to the homography-list property shown below Finally the stitcher video is recorded to file.

File move_homographies.json
{
    "homographies":[
        {
            "images":{
                "target":1,
                "reference":0
            },
            "matrix":{
                "h00": 1, "h01": 0, "h02": 700,
                "h10": 0, "h11": 1, "h12": 0,
                "h20": 0, "h21": 0, "h22": 1
            }
        },
        {
            "images":{
                "target":2,
                "reference":0
            },
            "matrix":{
                "h00": 1, "h01": 0, "h02": -710,
                "h10": 0, "h11": 1, "h12": 0,
                "h20": 0, "h21": 0, "h22": 1
            }
        }
    ]
}

gst-launch-1.0 -e  cudastitcher name=stitcher homography-list="`cat move_homographies.json | tr -d "\n" | tr -d " "`" filesrc location= ~/video0-s0.mp4 ! qtdemux ! queue ! h265parse ! nvv4l2decoder ! queue ! nvvidconv ! rreqrprojector center_x=963 center_y=558 radius=788 rot-y=-3.6 name=proj0 !  queue ! stitcher.sink_0 filesrc location= ~/video0-s1.mp4 ! qtdemux ! queue ! h265parse ! nvv4l2decoder ! queue ! nvvidconv !  rreqrprojector center_x=918 center_y=546 radius=782 rot-y=2 rot-x=0 name=proj1 !  queue ! stitcher.sink_1 filesrc location= ~/video0-s2.mp4 ! qtdemux ! queue ! h265parse ! nvv4l2decoder ! queue ! nvvidconv !  rreqrprojector center_x=933 center_y=557 radius=782 rot-y=1 rot-x=3 name=proj2 !  queue ! stitcher.sink_2  stitcher. ! perf print-arm-load=true  ! queue ! nvvidconv ! nvv4l2h264enc bitrate=30000000 ! h264parse ! queue !  qtmux ! filesink location=360_stitched_video.mp4

The following pipeline stitches 2 fisheye cameras back to back of 190 degrees. This pipeline makes use of the stitcher pads properties to cut out the borders, reduce the overlap region and some artifacts caused by the gray regions in the right side of the image.

{
    "homographies":[
        {
            "images":{
                "target":1,
                "reference":0
            },
            "matrix":{
                "h00": 1, "h01": 0, "h02": 2840,
                "h10": 0, "h11": 1, "h12": 0,
                "h20": 0, "h21": 0, "h22": 1
            }
        }
    ]
}

gst-launch-1.0 -e cudastitcher sink_0::right=2800 sink_1::right=3000 name=stitcher homography-list="`cat homographies.json | tr -d "\n" | tr -d " "`" filesrc location= ~/360_0006\ right.jpg ! nvjpegdec ! nvvidconv ! rreqrprojector radius=1447 center_x=1454 center_y=1446 lens=190 ! queue ! stitcher.sink_0 filesrc location= ~/360_left.JPG ! nvjpegdec ! nvvidconv ! rreqrprojector radius=1447 center_x=1440 center_y=1447 lens=190 rot-y=-1 ! queue ! stitcher.sink_1 stitcher. ! perf print-arm-load=true  ! queue ! videoconvert ! jpegenc ! filesink location=360_stitch.jpeg