Difference between revisions of "Image Stitching for NVIDIA Jetson/Examples/Nano,TX1,TX2,Xavier Pipelines"
Crodriguez (talk | contribs) |
Crodriguez (talk | contribs) (→Stitching from three cameras) |
||
Line 108: | Line 108: | ||
nvarguscamerasrc maxperf=true sensor-id=1 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_1 \ | nvarguscamerasrc maxperf=true sensor-id=1 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_1 \ | ||
nvarguscamerasrc maxperf=true sensor-id=2 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_2 \ | nvarguscamerasrc maxperf=true sensor-id=2 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_2 \ | ||
+ | cuda. ! perf print-arm-load=true ! queue ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=360" ! nvv4l2h264enc bitrate=20000000 ! h264parse ! mp4mux ! filesink location=$OUTVIDEO | ||
+ | </syntaxhighlight> | ||
+ | |||
+ | === Generating a MP4 stitch from 3 GRAY8 cameras === | ||
+ | |||
+ | <syntaxhighlight lang=bash> | ||
+ | RC_HOMOGRAPHY="{\"h00\":2.47795806e-01, \"h01\":1.83125651e-01, \"h02\":1511, \"h10\":-9.58951851e-02, \"h11\":1.00466096e+00, \"h12\":3.43046193e+01, \"h20\":-1.84997102e-04, \"h21\":4.48703017e-05, \"h22\":1}" | ||
+ | LC_HOMOGRAPHY="{\"h00\":1.07033034e+00, \"h01\":-8.91500609e-02, \"h02\":-1504, \"h10\":2.06014232e-02, \"h11\":1.06485384e+00, \"h12\":-2.68451772e+01, \"h20\":3.98503995e-06, \"h21\":4.20860985e-05, \"h22\":1}" | ||
+ | OUTVIDEO=/tmp/stitching_result.mp4 | ||
+ | |||
+ | gst-launch-1.0 -e cudastitcher name=cuda left-center-homography="$LC_HOMOGRAPHY" right-center-homography="$RC_HOMOGRAPHY" \ | ||
+ | nvarguscamerasrc maxperf=true sensor-id=0 ! nvvidconv ! "video/x-raw, width=1920, height=1080,format=GRAY8" ! queue ! cuda.sink_0 \ | ||
+ | nvarguscamerasrc maxperf=true sensor-id=1 ! nvvidconv ! "video/x-raw, width=1920, height=1080,format=GRAY8" ! queue ! cuda.sink_1 \ | ||
+ | nvarguscamerasrc maxperf=true sensor-id=2 ! nvvidconv ! "video/x-raw, width=1920, height=1080,format=GRAY8" ! queue ! cuda.sink_2 \ | ||
cuda. ! perf print-arm-load=true ! queue ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=360" ! nvv4l2h264enc bitrate=20000000 ! h264parse ! mp4mux ! filesink location=$OUTVIDEO | cuda. ! perf print-arm-load=true ! queue ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=360" ! nvv4l2h264enc bitrate=20000000 ! h264parse ! mp4mux ! filesink location=$OUTVIDEO | ||
</syntaxhighlight> | </syntaxhighlight> |
Revision as of 10:53, 24 July 2020
Image Stitching for NVIDIA®Jetson™ | |
---|---|
Image Stitching for NVIDIA Jetson Basics | |
|
|
Getting Started | |
|
|
User Guide | |
|
|
Compiling OpenCV for Image Stitching | |
|
|
Examples | |
|
|
360 Video | |
|
|
Performance | |
|
|
Contact Us |
|
This page gives a usage example of the cudastitcher element.
The perf element can be downloaded from this repository, otherwise the element can be removed from the pipeline without any issues. In case of performance issues, consider executing the /usr/bin/jetson_clocks binary.
Example pipelines stitching cameras
Two Camera Stitching
In the two image stitching the left image is always "static" and the transformation is applied on the right image.
Displaying a stitch
HOMOGRAPHY="{\"h00\":7.38511630e-01, \"h01\":1.04317351e-01, \"h02\":1.43471832e+03, \"h10\":-1.07952893e-01, \"h11\":9.89148056e-01, \"h12\":-9.39168804e+00, \"h20\":-2.34496984e-04, \"h21\":3.32061513e-05, \"h22\":1}"
#TODO
Stitching from two cameras
HOMOGRAPHY="{\"h00\":7.38511630e-01, \"h01\":1.04317351e-01, \"h02\":1.43471832e+03, \"h10\":-1.07952893e-01, \"h11\":9.89148056e-01, \"h12\":-9.39168804e+00, \"h20\":-2.34496984e-04, \"h21\":3.32061513e-05, \"h22\":1}"
OUTVIDEO=/tmp/stitching_result.ts
gst-launch-1.0 -e cudastitcher name=cuda right-center-homography="$HOMOGRAPHY" \
nvarguscamerasrc maxperf=true sensor-id=0 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_0 \
nvarguscamerasrc maxperf=true sensor-id=1 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_1 \
cuda. ! perf print-arm-load=true ! queue ! nvvidconv ! nvv4l2h264enc bitrate=20000000 ! h264parse ! mpegtsmux ! filesink location=$OUTVIDEO
Generating a MP4 stitch from 2 cameras
HOMOGRAPHY="{\"h00\":7.38511630e-01, \"h01\":1.04317351e-01, \"h02\":1.43471832e+03, \"h10\":-1.07952893e-01, \"h11\":9.89148056e-01, \"h12\":-9.39168804e+00, \"h20\":-2.34496984e-04, \"h21\":3.32061513e-05, \"h22\":1}"
OUTVIDEO=/tmp/stitching_result.mp4
gst-launch-1.0 -e cudastitcher name=cuda right-center-homography="$HOMOGRAPHY" \
nvarguscamerasrc maxperf=true sensor-id=0 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_0 \
nvarguscamerasrc maxperf=true sensor-id=1 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_1 \
cuda. ! perf print-arm-load=true ! queue ! nvvidconv ! nvv4l2h264enc bitrate=20000000 ! h264parse ! mp4mux ! filesink location=$OUTVIDEO
Streaming a stitch from two cameras via UDP+RTP
HOMOGRAPHY="{\"h00\":7.38511630e-01, \"h01\":1.04317351e-01, \"h02\":1.43471832e+03, \"h10\":-1.07952893e-01, \"h11\":9.89148056e-01, \"h12\":-9.39168804e+00, \"h20\":-2.34496984e-04, \"h21\":3.32061513e-05, \"h22\":1}"
HOST=127.0.0.1
PORT=12345
# Sender
gst-launch-1.0 -e cudastitcher name=cuda right-center-homography="$HOMOGRAPHY" \
nvarguscamerasrc maxperf=true sensor-id=0 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_0 \
nvarguscamerasrc maxperf=true sensor-id=1 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_1 \
cuda. ! perf print-arm-load=true ! nvvidconv ! nvv4l2h264enc ! rtph264pay config-interval=10 ! queue ! udpsink host=$HOST port=$PORT
# Receiver
gst-launch-1.0 udpsrc port=$PORT ! 'application/x-rtp, media=(string)video, encoding-name=(string)H264' ! queue ! rtph264depay ! avdec_h264 ! videoconvert ! xvimagesink
Stitching from three cameras
Displaying a stitch
RC_HOMOGRAPHY="{\"h00\":2.47795806e-01, \"h01\":1.83125651e-01, \"h02\":1511, \"h10\":-9.58951851e-02, \"h11\":1.00466096e+00, \"h12\":3.43046193e+01, \"h20\":-1.84997102e-04, \"h21\":4.48703017e-05, \"h22\":1}"
1962 LC_HOMOGRAPHY="{\"h00\":1.07033034e+00, \"h01\":-8.91500609e-02, \"h02\":-1504, \"h10\":2.06014232e-02, \"h11\":1.06485384e+00, \"h12\":-2.68451772e+01, \"h20\":3.98503995e-06, \"h21\":4.20860985e-05, \"h22\":1}"
#TODO
Stitching from 3 cameras to a TS file
RC_HOMOGRAPHY="{\"h00\":2.47795806e-01, \"h01\":1.83125651e-01, \"h02\":1511, \"h10\":-9.58951851e-02, \"h11\":1.00466096e+00, \"h12\":3.43046193e+01, \"h20\":-1.84997102e-04, \"h21\":4.48703017e-05, \"h22\":1}"
LC_HOMOGRAPHY="{\"h00\":1.07033034e+00, \"h01\":-8.91500609e-02, \"h02\":-1504, \"h10\":2.06014232e-02, \"h11\":1.06485384e+00, \"h12\":-2.68451772e+01, \"h20\":3.98503995e-06, \"h21\":4.20860985e-05, \"h22\":1}"
OUTVIDEO=/tmp/stitching_result.ts
gst-launch-1.0 -e cudastitcher name=cuda left-center-homography="$LC_HOMOGRAPHY" right-center-homography="$RC_HOMOGRAPHY" \
nvarguscamerasrc maxperf=true sensor-id=0 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_0 \
nvarguscamerasrc maxperf=true sensor-id=1 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_1 \
nvarguscamerasrc maxperf=true sensor-id=2 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_2 \
cuda. ! perf print-arm-load=true ! queue ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=360" ! nvv4l2h264enc bitrate=20000000 ! h264parse ! mpegtsmux ! filesink location=$OUTVIDEO
Generating a MP4 stitch from 3 cameras
RC_HOMOGRAPHY="{\"h00\":2.47795806e-01, \"h01\":1.83125651e-01, \"h02\":1511, \"h10\":-9.58951851e-02, \"h11\":1.00466096e+00, \"h12\":3.43046193e+01, \"h20\":-1.84997102e-04, \"h21\":4.48703017e-05, \"h22\":1}"
LC_HOMOGRAPHY="{\"h00\":1.07033034e+00, \"h01\":-8.91500609e-02, \"h02\":-1504, \"h10\":2.06014232e-02, \"h11\":1.06485384e+00, \"h12\":-2.68451772e+01, \"h20\":3.98503995e-06, \"h21\":4.20860985e-05, \"h22\":1}"
OUTVIDEO=/tmp/stitching_result.mp4
gst-launch-1.0 -e cudastitcher name=cuda left-center-homography="$LC_HOMOGRAPHY" right-center-homography="$RC_HOMOGRAPHY" \
nvarguscamerasrc maxperf=true sensor-id=0 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_0 \
nvarguscamerasrc maxperf=true sensor-id=1 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_1 \
nvarguscamerasrc maxperf=true sensor-id=2 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_2 \
cuda. ! perf print-arm-load=true ! queue ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=360" ! nvv4l2h264enc bitrate=20000000 ! h264parse ! mp4mux ! filesink location=$OUTVIDEO
Generating a MP4 stitch from 3 GRAY8 cameras
RC_HOMOGRAPHY="{\"h00\":2.47795806e-01, \"h01\":1.83125651e-01, \"h02\":1511, \"h10\":-9.58951851e-02, \"h11\":1.00466096e+00, \"h12\":3.43046193e+01, \"h20\":-1.84997102e-04, \"h21\":4.48703017e-05, \"h22\":1}"
LC_HOMOGRAPHY="{\"h00\":1.07033034e+00, \"h01\":-8.91500609e-02, \"h02\":-1504, \"h10\":2.06014232e-02, \"h11\":1.06485384e+00, \"h12\":-2.68451772e+01, \"h20\":3.98503995e-06, \"h21\":4.20860985e-05, \"h22\":1}"
OUTVIDEO=/tmp/stitching_result.mp4
gst-launch-1.0 -e cudastitcher name=cuda left-center-homography="$LC_HOMOGRAPHY" right-center-homography="$RC_HOMOGRAPHY" \
nvarguscamerasrc maxperf=true sensor-id=0 ! nvvidconv ! "video/x-raw, width=1920, height=1080,format=GRAY8" ! queue ! cuda.sink_0 \
nvarguscamerasrc maxperf=true sensor-id=1 ! nvvidconv ! "video/x-raw, width=1920, height=1080,format=GRAY8" ! queue ! cuda.sink_1 \
nvarguscamerasrc maxperf=true sensor-id=2 ! nvvidconv ! "video/x-raw, width=1920, height=1080,format=GRAY8" ! queue ! cuda.sink_2 \
cuda. ! perf print-arm-load=true ! queue ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=360" ! nvv4l2h264enc bitrate=20000000 ! h264parse ! mp4mux ! filesink location=$OUTVIDEO
Streaming a stitch from 3 cameras via UDP+RTP
In the three camera case the stitching should always be giving
RC_HOMOGRAPHY="{\"h00\":2.47795806e-01, \"h01\":1.83125651e-01, \"h02\":1511, \"h10\":-9.58951851e-02, \"h11\":1.00466096e+00, \"h12\":3.43046193e+01, \"h20\":-1.84997102e-04, \"h21\":4.48703017e-05, \"h22\":1}"
LC_HOMOGRAPHY="{\"h00\":1.07033034e+00, \"h01\":-8.91500609e-02, \"h02\":-1504, \"h10\":2.06014232e-02, \"h11\":1.06485384e+00, \"h12\":-2.68451772e+01, \"h20\":3.98503995e-06, \"h21\":4.20860985e-05, \"h22\":1}"
HOST=127.0.0.1
PORT=12345
# Sender
gst-launch-1.0 -e cudastitcher name=cuda left-center-homography="$LC_HOMOGRAPHY" right-center-homography="$RC_HOMOGRAPHY" \
nvarguscamerasrc maxperf=true sensor-id=0 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_0 \
nvarguscamerasrc maxperf=true sensor-id=1 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_1 \
nvarguscamerasrc maxperf=true sensor-id=2 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_2 \
cuda. ! perf print-arm-load=true ! nvvidconv ! nvv4l2h264enc ! rtph264pay config-interval=10 ! queue ! udpsink host=$HOST port=$PORT
# Receiver
gst-launch-1.0 udpsrc port=$PORT ! 'application/x-rtp, media=(string)video, encoding-name=(string)H264' ! queue ! rtph264depay ! avdec_h264 ! videoconvert ! xvimagesink
Example pipeline stitching images
Displaying a stitch form two JPEG images
HOMOGRAPHY="{\"h00\":7.38511630e-01, \"h01\":1.04317351e-01, \"h02\":1.43471832e+03, \"h10\":-1.07952893e-01, \"h11\":9.89148056e-01, \"h12\":-9.39168804e+00, \"h20\":-2.34496984e-04, \"h21\":3.32061513e-05, \"h22\":1}"
LEFTIMG=left.jpg
RIGHTIMG=right.jpg
#TODO
Saving a stitch from two JPEG images
HOMOGRAPHY="{\"h00\":7.38511630e-01, \"h01\":1.04317351e-01, \"h02\":1.43471832e+03, \"h10\":-1.07952893e-01, \"h11\":9.89148056e-01, \"h12\":-9.39168804e+00, \"h20\":-2.34496984e-04, \"h21\":3.32061513e-05, \"h22\":1}"
LEFTIMG=left.jpg
RIGHTIMG=right.jpg
OUTIMG=/tmp/stitching_result.jpg
#TODO
Example pipelines stitching videos
Displaying a stitch from two MP4 videos
HOMOGRAPHY="{\"h00\":7.38511630e-01, \"h01\":1.04317351e-01, \"h02\":1.43471832e+03, \"h10\":-1.07952893e-01, \"h11\":9.89148056e-01, \"h12\":-9.39168804e+00, \"h20\":-2.34496984e-04, \"h21\":3.32061513e-05, \"h22\":1}"
LEFTVID=left.mp4
RIGHTVID=right.mp4
#TODO
Saving a stitch from two MP4 videos
HOMOGRAPHY="{\"h00\":7.38511630e-01, \"h01\":1.04317351e-01, \"h02\":1.43471832e+03, \"h10\":-1.07952893e-01, \"h11\":9.89148056e-01, \"h12\":-9.39168804e+00, \"h20\":-2.34496984e-04, \"h21\":3.32061513e-05, \"h22\":1}"
LEFTVID=left.mp4
RIGHTVID=right.mp4
OUTVID=/tmp/stitching_result.mp4
#TODO
Undistortion Pipelines
Capture to Display
CAMERA_MATRIX="{\"fx\":2.8472876737532920e+03, \"fy\":2.8608529052506838e+03, \"cx\":9.7983673800322515e+02, \"cy\":5.0423299551699932e+02}"
DISTORTION_PARAMETERS="{\"k1\":-6.7260720359999060e-01, \"k2\":2.5160831522455513e+00, \"p1\":5.4007310542765141e-02, \"p2\":-1.1365265232659062e-02, \"k3\":-1.2760075297700798e+01 }"
gst-launch-1.0 nvarguscamerasrc ! cudaundistort distortion-model=brown-conrady camera-matrix=$CAMERA_MATRIX distortion-parameters=$DISTORTION_PARAMETERS ! nvoverlaysink
3 cameras stitching with undistort element
This comprises of a system with both undistortion and stitching.
OUTVIDEO=/tmp/stitching_result.mp4
RC_HOMOGRAPHY="{\"h00\":0.18846164727524536,\"h01\":0.3094433487541917, \"h02\":471.7246223741918, \"h10\":-0.1970199049349592, \"h11\":0.9946393933468606, \"h12\":26.75293460418449, \"h20\":-0.000722663093587326, \"h21\":0.00021396368386953087, \"h22\":1}"
LC_HOMOGRAPHY="{\"h00\":1.1217838548058883,\"h01\":0.07335969882181002, \"h02\":-453.68800318355966, \"h10\":0.023170492179664638, \"h11\":1.2173787645317036, \"h12\":-9.14050414930091, \"h20\":0.0002687772494613081, \"h21\":0.00016446099177990192, \"h22\":1}"
CAMERA_MATRIX="{\"fx\":2.8472876737532920e+03, \"fy\":2.8608529052506838e+03, \"cx\":9.7983673800322515e+02, \"cy\":5.0423299551699932e+02}"
DISTORTION_PARAMETERS="{\"k1\":-6.7260720359999060e-01, \"k2\":2.5160831522455513e+00, \"p1\":5.4007310542765141e-02, \"p2\":-1.1365265232659062e-02, \"k3\":-1.2760075297700798e+01 }"
gst-launch-1.0 -e cudastitcher name=cuda left-center-homography="$LC_HOMOGRAPHY" right-center-homography="$RC_HOMOGRAPHY" \
nvarguscamerasrc maxperf=true sensor-id=0 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! cudaundistort distortion-model=brown-conrady camera-matrix="$CAMERA_MATRIX" distortion-parameters="$DISTORTION_PARAMETERS" ! nvvidconv ! "video/x-raw(memory:NVMM), width=640, height=360" ! queue ! cuda.sink_0 \
nvarguscamerasrc maxperf=true sensor-id=1 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! cudaundistort distortion-model=brown-conrady camera-matrix="$CAMERA_MATRIX" distortion-parameters="$DISTORTION_PARAMETERS" ! nvvidconv ! "video/x-raw(memory:NVMM), width=640, height=360" ! queue ! cuda.sink_1 \
nvarguscamerasrc maxperf=true sensor-id=2 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! cudaundistort distortion-model=brown-conrady camera-matrix="$CAMERA_MATRIX" distortion-parameters="$DISTORTION_PARAMETERS" ! nvvidconv ! "video/x-raw(memory:NVMM), width=640, height=360" ! queue ! cuda.sink_2 \
cuda. ! perf print-arm-load=true ! queue ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=360" ! nvv4l2h264enc bitrate=20000000 ! h264parse ! mpegtsmux ! filesink location=$OUTVIDEO
Example pipeline for debugging
Dumping output to fakesink
HOMOGRAPHY="{\"h00\":7.38511630e-01, \"h01\":1.04317351e-01, \"h02\":1.43471832e+03, \"h10\":-1.07952893e-01, \"h11\":9.89148056e-01, \"h12\":-9.39168804e+00, \"h20\":-2.34496984e-04, \"h21\":3.32061513e-05, \"h22\":1}"
gst-launch-1.0 -e cudastitcher name=cuda right-center-homography="$HOMOGRAPHY" nvarguscamerasrc maxperf=true sensor-id=0 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_0 nvarguscamerasrc maxperf=true sensor-id=1 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_1 cuda. ! perf print-arm-load=true ! fakesink
Generating a raw video frames
HOMOGRAPHY="{\"h00\":7.38511630e-01, \"h01\":1.04317351e-01, \"h02\":1.43471832e+03, \"h10\":-1.07952893e-01, \"h11\":9.89148056e-01, \"h12\":-9.39168804e+00, \"h20\":-2.34496984e-04, \"h21\":3.32061513e-05, \"h22\":1}"
OUTVIDEO=/tmp/stitching_result.raw
gst-launch-1.0 -e cudastitcher name=cuda right-center-homography="$HOMOGRAPHY" nvarguscamerasrc maxperf=true sensor-id=0 num-buffers=30 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_0 nvarguscamerasrc maxperf=true sensor-id=1 num-buffers=30 ! nvvidconv ! "video/x-raw(memory:NVMM), width=1920, height=1080" ! queue ! cuda.sink_1 cuda. ! perf print-arm-load=true ! filesink location=$OUTVIDEO