Difference between revisions of "CUDA ISP for NVIDIA Jetson/Examples/GStreamer usage"
(→GStreamer Elements) |
|||
| Line 13: | Line 13: | ||
* cudaawb | * cudaawb | ||
| − | In the following sections, you can see examples of how to use them. You can see more information about the algorithms these elements apply in [[CUDA_ISP_for_NVIDIA_Jetson/CUDA_ISP_for_NVIDIA_Jetson_Basics|CUDA ISP Basics]]. | + | In the following sections, you can see examples of how to use them. You can see more information about the algorithms these elements apply in [[CUDA_ISP_for_NVIDIA_Jetson/CUDA_ISP_for_NVIDIA_Jetson_Basics|CUDA ISP Basics]]. Consider that a patch must be applied to <code>v4l2src</code> to enable captures in bayer10. You can see how to apply this patch in this link: [[CUDA_ISP_for_NVIDIA_Jetson/Getting_Started/Building_custom_v4l2src_element|Apply patch to v4l2src]]. |
== GStreamer Elements == | == GStreamer Elements == | ||
| Line 22: | Line 22: | ||
* '''Choosing the shift value''' | * '''Choosing the shift value''' | ||
| − | On Jetson platforms, the capture subsystem unit (Video Input) has particular formats to pack the RAW data received from the CSI channels. This modifies the RAW Pixel bits organization on the image captured. When capturing, the MSB can be found on | + | On Jetson platforms, the capture subsystem unit (Video Input) has particular formats to pack the RAW data received from the CSI channels. This modifies the RAW Pixel bits organization on the image captured. When capturing, the MSB can be found on positions 13, 14, 15 (depending on the platform), and the lower bits will be replicated MSBs. The values of the MSB have more impact on the image than LSB. To choose a proper value for shifting, please consider which shift value places the 8 MSB data bits into the lower 8 positions. |
| − | For example, when capturing a RAW10 image | + | For example, when capturing a RAW10 image of a Jetson Xavier, the information will be organized as shown in the next image: |
| Line 32: | Line 32: | ||
<br> | <br> | ||
<br><br> | <br><br> | ||
| − | According to the previous image, we have the data (D9-D0) from bit 15 to bit 6 and have replicated bits from bit 5 to bit 0. In this case the proper shift value would be 8. In practice, the best image quality may be obtained using different shift values than | + | According to the previous image, we have the data (D9-D0) from bit 15 to bit 6 and have replicated bits from bit 5 to bit 0. In this case, the proper shift value would be 8. In practice, the best image quality may be obtained using different shift values than theoretical ones. |
| − | |||
| − | |||
| Line 49: | Line 47: | ||
=== cudadebayer === | === cudadebayer === | ||
| − | |||
| − | |||
This element applies the debayering algorithm. For more information about the element properties and capabilities, check here: [https://ridgerun.github.io/libcudaisp-docs/md__builds_ridgerun_rnd_libcudaisp_docs_cudadebayer.html cudadebayer element] | This element applies the debayering algorithm. For more information about the element properties and capabilities, check here: [https://ridgerun.github.io/libcudaisp-docs/md__builds_ridgerun_rnd_libcudaisp_docs_cudadebayer.html cudadebayer element] | ||
| − | Next, you will see examples | + | Next, you will see examples of how to use the cudadebayer element. |
The following pipeline receives a bayer 10 image with 4K resolution from a camera sensor and outputs an RGB image. | The following pipeline receives a bayer 10 image with 4K resolution from a camera sensor and outputs an RGB image. | ||
| Line 77: | Line 73: | ||
=== cudaawb === | === cudaawb === | ||
| − | This element applies de auto-white balance algorithm. Next, you will see examples | + | This element applies de auto-white balance algorithm. Next, you will see examples of how to use the cudaawb element. |
| − | The following pipeline receives | + | The following pipeline receives an RGB image with 4K resolution, applies the auto-white balance algorithm, and outputs an RGB image. |
<source lang=bash> | <source lang=bash> | ||
| Line 126: | Line 122: | ||
== Use cases == | == Use cases == | ||
| − | + | Previous sections described some examples of the three elements of the CUDA ISP. As seen in those previous sections, the cudadebayer and cudashift elements can apply the shifting algorithm. You might wonder in which cases to use just the cudadebayer or both elements. | |
=== cudashift and cudadebayer === | === cudashift and cudadebayer === | ||
*'''Higher framerate''' | *'''Higher framerate''' | ||
| − | If your goal during processing is to maximize the framerate, you should use both elements. Splitting the elements will lead to a higher framerate but | + | If your goal during processing is to maximize the framerate, you should use both elements. Splitting the elements will lead to a higher framerate but increases the process's latency. |
<br> | <br> | ||
| Line 141: | Line 137: | ||
*'''Reduced latency''' | *'''Reduced latency''' | ||
| − | By just using the cudadebayer element with its own shifting property, the latency of the process is reduced but so does the framerate. | + | By just using the cudadebayer element with its own shifting property, the latency of the process is reduced, but so does the framerate. |
<br> | <br> | ||
| Line 148: | Line 144: | ||
<br> | <br> | ||
<br><br> | <br><br> | ||
| − | |||
| − | |||
Revision as of 09:50, 30 March 2023
| CUDA ISP for NVIDIA Jetson | |
|---|---|
 | |
| CUDA ISP for NVIDIA Jetson Basics | |
|
|
|
| Getting Started | |
|
|
|
| User Manual | |
|
|
|
| GStreamer | |
|
|
|
| Examples | |
|
|
|
| Performance | |
|
|
|
| Contact Us | |
|
|
Contents
General
CUDA ISP provides GStreamer plugins that facilitate the usage of the library with GStreamer pipelines. CUDA ISP provides three different GStreamer elements for image signal processing. The three elements are:
- cudashift
- cudadebayer
- cudaawb
In the following sections, you can see examples of how to use them. You can see more information about the algorithms these elements apply in CUDA ISP Basics. Consider that a patch must be applied to v4l2src to enable captures in bayer10. You can see how to apply this patch in this link: Apply patch to v4l2src.
GStreamer Elements
cudashift
This element applies the shifting algorithm. For more information about the element properties and capabilities, check here: cudashift element.
- Choosing the shift value
On Jetson platforms, the capture subsystem unit (Video Input) has particular formats to pack the RAW data received from the CSI channels. This modifies the RAW Pixel bits organization on the image captured. When capturing, the MSB can be found on positions 13, 14, 15 (depending on the platform), and the lower bits will be replicated MSBs. The values of the MSB have more impact on the image than LSB. To choose a proper value for shifting, please consider which shift value places the 8 MSB data bits into the lower 8 positions.
For example, when capturing a RAW10 image of a Jetson Xavier, the information will be organized as shown in the next image:
According to the previous image, we have the data (D9-D0) from bit 15 to bit 6 and have replicated bits from bit 5 to bit 0. In this case, the proper shift value would be 8. In practice, the best image quality may be obtained using different shift values than theoretical ones.
The following pipeline receives a bayer 10 image with 4k resolution from a camera sensor, applies a shifting with a value of 5 and outputs a bayer 8 image.
gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=10, format=rggb' ! cudashift shift=5 ! fakesink
cudadebayer
This element applies the debayering algorithm. For more information about the element properties and capabilities, check here: cudadebayer element
Next, you will see examples of how to use the cudadebayer element.
The following pipeline receives a bayer 10 image with 4K resolution from a camera sensor and outputs an RGB image.
gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=10, width=3840, height=2160' ! cudadebayer ! fakesink
This next pipeline receives a bayer 10 image with 4K resolution and outputs an I420 image. You can see that source caps were added to choose which output format you desire,in this case it would be I420.
gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=10, width=3840, height=2160' ! cudadebayer ! 'video/x-raw, format=I420' ! fakesink
This next pipeline receives a bayer 10 image with 4K resolution and outputs an RGB image with shifting added. In this case the output format is RGB.
gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=10, width=3840, height=2160' ! cudadebayer shift=5 ! 'video/x-raw, format=RGB' ! fakesink
cudaawb
This element applies de auto-white balance algorithm. Next, you will see examples of how to use the cudaawb element.
The following pipeline receives an RGB image with 4K resolution, applies the auto-white balance algorithm, and outputs an RGB image.
gst-launch-1.0 videotestsrc ! 'video/x-raw, width=3840,height=2160' ! cudaawb ! fakesink
The following pipeline receives a bayer10 image with 4K resolution from a camera sensor and outputs an RGB image after applying the auto-white balance algorithm. This element requires an RGB image input. So, when using a camera sensor, before using the cudaawb element, we should add the cudadebayer element to obtain the RGB image that the AWB element will use as input.
gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=10, width=3840, height=2160' ! cudadebayer ! cudaawb ! fakesink
The following pipeline receives a bayer10 image with 4K resolution from a camera sensor, applies debayering and auto-white balance, and outputs an I420 image.
gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=10, width=3840, height=2160' ! cudadebayer ! cudaawb ! 'video/x-raw, format=I420' ! fakesink
Using the three elements
The following pipeline receives a bayer 10 image with 4K resolution and outputs a RGB image, using the three elements.
gst-launch-1.0 -ve v4l2src io-mode=userptr ! 'video/x-bayer, bpp=10, width=3840, height=2160' ! cudashift shift=5 ! cudadebayer ! cudaawb ! fakesink
Use cases
Previous sections described some examples of the three elements of the CUDA ISP. As seen in those previous sections, the cudadebayer and cudashift elements can apply the shifting algorithm. You might wonder in which cases to use just the cudadebayer or both elements.
cudashift and cudadebayer
- Higher framerate
If your goal during processing is to maximize the framerate, you should use both elements. Splitting the elements will lead to a higher framerate but increases the process's latency.
- Reduced latency
By just using the cudadebayer element with its own shifting property, the latency of the process is reduced, but so does the framerate.
