Xilinx ZYNQ UltraScale+ MPSoC Accelerating Image Processing Elements

Xilinx ZYNQ UltraScale+ MPSoC
Introduction
Overview Family Developer kits
Getting Started
Xilinx Kria ZCU 102-106
Development Flows Examples
Simple Vector Addition with XRT

Xilinx MPSoC's VCU IP is capable of simultaneously encoding and decoding up to 32 streams of H.264 and H.265 codecs, with a maximum bandwidth equivalent to 4K60. 8K encoding/decoding is also supported at a 15 FPS reduced rate mode.

It is important to notice that to make use of the VCU, firmware with the VCU enabled must be loaded into the FPGA. For this wiki, the smartcam example firmware was used during testing. Xilinx seems to have stopped including these apps in the newer versions of Petalinux, for this reason, you may need to add them manually. The firmware can be found in this repository and follow the instructions to add a firmware recipe to the Petalinux build.

Video Encoding

H.264

The H.264 encoder is available through the omxh264enc GStreamer element.

gst-inspect-1.0 omxh264enc

root@xilinx-kv260-starterkit-20222:~# gst-inspect-1.0 omxh264enc Factory Details: Rank primary + 1 (257) Long-name OpenMAX H.264 Video Encoder Klass Codec/Encoder/Video/Hardware Description Encode H.264 video streams Author Sebastian Dr?ge <sebastian.droege@collabora.co.uk> Plugin Details: Name omx Description GStreamer OpenMAX Plug-ins Filename /usr/lib/gstreamer-1.0/libgstomx.so Version 1.18.5 License LGPL Source module gst-omx Source release date 2021-09-08 Binary package GStreamer OpenMAX Plug-ins Origin URL Unknown package origin GObject +----GInitiallyUnowned +----GstObject +----GstElement +----GstVideoEncoder +----GstOMXVideoEnc +----GstOMXH264Enc +----GstOMXH264Enc-omxh264enc Implemented Interfaces: GstPreset Pad Templates: SINK template: 'sink' Availability: Always Capabilities: video/x-raw(memory:XLNXLL) format: { (string)NV16_10LE32, (string)NV12_10LE32, (string)NV16, (string)NV12, (string)GRAY10_LE32, (string)GRAY8, (string)Y444, (string)Y444_10LE32 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] video/x-raw format: { (string)NV16_10LE32, (string)NV12_10LE32, (string)NV16, (string)NV12, (string)GRAY10_LE32, (string)GRAY8, (string)Y444, (string)Y444_10LE32 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] SRC template: 'src' Availability: Always Capabilities: video/x-h264 width: [ 16, 4096 ] height: [ 16, 4096 ] framerate: [ 0/1, 2147483647/1 ] stream-format: byte-stream alignment: { (string)au, (string)nal } Element has no clocking capabilities. Element has no URI handling capabilities. Pads: SINK: 'sink' Pad Template: 'sink' SRC: 'src' Pad Template: 'src' Element Properties: aspect-ratio : Display aspect ratio of the video sequence to be written in SPS/VUI flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncAspectRatio" Default: 0, "auto" (0): auto - 4:3 for SD video,16:9 for HD video,unspecified for unknown format (1): 1-1 - 1:1 aspect ratio (2): 4-3 - 4:3 aspect ratio (3): 16-9 - 16:9 aspect ratio (4): none - Aspect ratio information is not present in the stream b-frames : Number of B-frames between two consecutive P-frames flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state Unsigned Integer. Range: 0 - 4294967295 Default: 0 constrained-intra-prediction: If enabled, prediction only uses residual data and decoded samples from neighbouring coding blocks coded using intra prediction modes flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false control-rate : Bitrate control method flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncControlRate" Default: -1, "default" (0): disable - Disable (1): variable - Variable (2): constant - Constant (3): variable-skip-frames - Variable Skip Frames (4): constant-skip-frames - Constant Skip Frames (2130706434): capped-variable - Capped Variable (2130706433): low-latency - Low Latency (-1): default - Component Default cpb-size : Coded Picture Buffer as specified in the HRD model in msec. Not used when control-rate=disable flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 3000 default-roi-quality : The default quality level to apply to each Region of Interest flags: readable, writable Enum "GstOMXVideoEncRoiQuality" Default: 0, "high" (0): high - Delta QP of -5 (1): medium - Delta QP of 0 (2): low - Delta QP of +5 (3): dont-care - Maximum delta QP value (4): intra - Region all LCU encoded with intra prediction mode dependent-slice : If encoding with multiple slices, specify whether the additional slices are dependent slice segments or regular slices flags: readable, writable Boolean. Default: false entropy-mode : Entropy mode for encoding process flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXH264EncEntropyMode" Default: -1, "default" (0): CAVLC - CAVLC entropy mode (1): CABAC - CABAC entropy mode (-1): default - Component Default filler-data : Enable/Disable Filler Data NAL units for CBR rate control flags: readable, writable, changeable only in NULL or READY state Boolean. Default: true gdr-mode : Gradual Decoder Refresh scheme mode. Only used if gop-mode=low-delay-p flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncGdrMode" Default: 0, "disabled" (0): disabled - No GDR (1): vertical - Gradual refresh using a vertical bar moving from left to right (2): horizontal - Gradual refresh using a horizontal bar moving from top to bottom gop-length : Distance between two consecutive I frames flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state Unsigned Integer. Range: 0 - 1000 Default: 30 gop-mode : Group Of Pictures mode flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncGopMode" Default: 0, "basic" (0): basic - Basic GOP settings (1): basic-b - Basic GOP settings, includes only B-frames (2): pyramidal - Advanced GOP pattern with hierarchical B-frames (3): pyramidal-b - Advanced GOP pattern with hierarchical B-frames, includes only B-frames (5): low-delay-p - Single I-frame followed by P-frames only (6): low-delay-b - Single I-frame followed by B-frames only (4): adaptive - Advanced GOP pattern with adaptive B-frames hlg-sdr-compatible : If enabled and input caps contain HLG colorimetry, insert BT2020 EOTF and ATC SEI instead flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false initial-delay : The initial removal delay as specified in the HRD model in msec. Not used when control-rate=disable flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 1500 input-crop : Cropping parameters for input video buffer ('<left, top, width, height>') flags: writable GstValueArray of GValues of type "gint" Write only interval-intraframes: Interval of coding Intra frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 -h264,alignment=nal ! ...). flags: readable, writable Enum "GstOMXVideoEncLatencyMode" Default: 0, "normal" (0): normal - Normal mode (1): low-latency - Low latency mode (-1): default - Component Default long-term-freq : Periodicity of LongTerm reference picture marking in encoding process Units in frames, distance between two consequtive long-term reference pictures flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 0 long-term-ref : If enabled, encoder accepts dynamically inserting and using long-term reference picture events from upstream elements flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false look-ahead : The number of frames processed ahead of second pass encoding. If smaller than 2, dual pass encoding is disabled flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 0 loop-filter-alpha-c0-offset: Alpha C0 offset for the deblocking filter, used only when loop-filter-mode is enabled flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state Integer. Range: -6 - 6 Default: -1 loop-filter-beta-offset: Beta offset for the deblocking filter, used only when loop-filter-mode is enabled flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state Integer. Range: -6 - 6 Default: -1 loop-filter-mode : Enable or disable the deblocking filter (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXH264EncLoopFilter" Default: -1, "default" (0): enable - Enable deblocking filter (1): disable - Disable deblocking filter (2): disable-slice-boundary - Disables deblocking filter on slice boundary (-1): default - Component Default low-bandwidth : If enabled, decrease the vertical search range used for P-frame motion estimation to reduce the bandwidth flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false max-bitrate : Max bitrate in Kbps, only used if control-rate=variable (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 max-consecutive-skip: Max number of consecutive frames to be skipped. Only applicable if skip-frame is enabled flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 ze will not have any effect flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 0 icture-size-x value. If set it to 0 then max-picture-size-x will not have any effect flags: writable GstValueArray of GValues of type "gint" Write only max-qp : Maximum QP value allowed for the rate control flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 51 Default: 51 le flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 min-force-key-unit-interval: Minimum interval between force-keyunit requests in nanoseconds flags: readable, writable Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 min-qp : Minimum QP value allowed for the rate control flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 51 Default: 10 name : The name of the object flags: readable, writable, 0x2000 String. Default: "omxh264enc-omxh264enc0" egularly as possible. If slice-size is defined as well more slices may be produced to fit the slice-size requirement (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 1 - 4294967295 Default: 4294967295 output-crop : Cropping parameters for output bitstream ('<left, top, width, height>') flags: writable GstValueArray of GValues of type "gint" Write only parent : The parent of the object flags: readable, writable, 0x2000 Object of type "GstObject" periodicity-idr : Periodicity of IDR frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 periodicty-idr : Periodicity of IDR frames (0xffffffff=component default) DEPRECATED - only for backwards compat flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 prefetch-buffer : Enable/Disable L2Cache buffer in encoding process flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false qos : Handle Quality-of-Service events from downstream flags: readable, writable Boolean. Default: false qp-mode : QP control mode used by the VCU encoder flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncQpMode" Default: -1, "default" (0): uniform - Use the same QP for all coding units of the frame (1): roi - Adjust QP according to the regions of interest defined on each frame. Must be set to handle ROI metadata. (2): auto - Let the VCU encoder change the QP for each coding unit according to its content (3): load-qp-absolute - Uses absolute QP values set by user. Must be set to use External QP buffer (4): load-qp-relative - Uses Relative/Delta QP values set by user. Must be set to use External QP buffer (-1): default - Component Default quant-b-frames : Quantization parameter for B-frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 quant-i-frames : Quantization parameter for I-frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 quant-p-frames : Quantization parameter for P-frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 ref-frames : Number of reference frames used for inter-motion search (0=component default) flags: readable, writable, changeable only in NULL or READY state Unknown type 16 "guchar" scaling-list : Scaling list mode flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncScalingList" Default: 1, "default" (1): default - Default scaling list mode (0): flat - Flat scaling list mode Skip. Only used if control-rate=constant or variable and b-frames are less than 2 flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false slice-size : Target slice size (in bytes) that the encoder uses to automatically split the bitstream into approximately equally-sized slices flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 65535 Default: 0 target-bitrate : Target bitrate in Kbps (0xffffffff=component default) flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 uniform-slice-type : Enable/Disable uniform slice type in slice header flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false use-out-port-pool : Use a buffer pool on the output port flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false cture size will be limited to max-picture-size-x value. If set it to 0 then max-picture-size-x will not have any effect flags: writable GstValueArray of GValues of type "gint" Write only y444-to-gray : Enable/Disable YUV format manipulation to Y8 flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false

Performance

Using AR1335 camera through AP1302 ISP at 1080p:

gst-launch-1.0 mediasrcbin media-device=/dev/media0 v4l2src0::io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh264enc ! perf ! fakesink

Board	CPU %	FPS	Element latency (ms)
KV260	2 %	30.000	Not tested

Using AR1335 camera through AP1302 ISP at 4K:

gst-launch-1.0 mediasrcbin media-device=/dev/media0 v4l2src0::io-mode=4 ! video/x-raw, width=3840, height=2160, format=NV12, framerate=30/1 ! omxh264enc ! perf ! fakesink

Board	CPU %	FPS	Element latency (ms)
KV260	3 %	30.000	Not tested

H.265

The H.265 encoder is available through the omxh265enc GStreamer element.

gst-inspect-1.0 omxh265enc

root@xilinx-kv260-starterkit-20222:~# gst-inspect-1.0 omxh265enc Factory Details: Rank primary + 1 (257) Long-name OpenMAX H.265 Video Encoder Klass Codec/Encoder/Video/Hardware Description Encode H.265 video streams Author Sebastian Dr?ge <sebastian.droege@collabora.co.uk> Plugin Details: Name omx Description GStreamer OpenMAX Plug-ins Filename /usr/lib/gstreamer-1.0/libgstomx.so Version 1.18.5 License LGPL Source module gst-omx Source release date 2021-09-08 Binary package GStreamer OpenMAX Plug-ins Origin URL Unknown package origin GObject +----GInitiallyUnowned +----GstObject +----GstElement +----GstVideoEncoder +----GstOMXVideoEnc +----GstOMXH265Enc +----GstOMXH265Enc-omxh265enc Implemented Interfaces: GstPreset Pad Templates: SINK template: 'sink' Availability: Always Capabilities: video/x-raw(format:Interlaced) format: { (string)NV16_10LE32, (string)NV12_10LE32, (string)NV16, (string)NV12, (string)GRAY10_LE32, (string)GRAY8, (string)Y444, (string)Y444_10LE32 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] interlace-mode: alternate video/x-raw(memory:XLNXLL) format: { (string)NV16_10LE32, (string)NV12_10LE32, (string)NV16, (string)NV12, (string)GRAY10_LE32, (string)GRAY8, (string)Y444, (string)Y444_10LE32 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] video/x-raw format: { (string)NV16_10LE32, (string)NV12_10LE32, (string)NV16, (string)NV12, (string)GRAY10_LE32, (string)GRAY8, (string)Y444, (string)Y444_10LE32 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] SRC template: 'src' Availability: Always Capabilities: video/x-h265 width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] stream-format: byte-stream aligmment: { (string)au, (string)nal } Element has no clocking capabilities. Element has no URI handling capabilities. Pads: SINK: 'sink' Pad Template: 'sink' SRC: 'src' Pad Template: 'src' Element Properties: aspect-ratio : Display aspect ratio of the video sequence to be written in SPS/VUI flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncAspectRatio" Default: 0, "auto" (0): auto - 4:3 for SD video,16:9 for HD video,unspecified for unknown format (1): 1-1 - 1:1 aspect ratio (2): 4-3 - 4:3 aspect ratio (3): 16-9 - 16:9 aspect ratio (4): none - Aspect ratio information is not present in the stream b-frames : Number of B-frames between two consecutive P-frames flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state Unsigned Integer. Range: 0 - 4294967295 Default: 0 constrained-intra-prediction: If enabled, prediction only uses residual data and decoded samples from neighbouring coding blocks coded using intra prediction modes flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false control-rate : Bitrate control method flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncControlRate" Default: -1, "default" (0): disable - Disable (1): variable - Variable (2): constant - Constant (3): variable-skip-frames - Variable Skip Frames (4): constant-skip-frames - Constant Skip Frames (2130706434): capped-variable - Capped Variable (2130706433): low-latency - Low Latency (-1): default - Component Default cpb-size : Coded Picture Buffer as specified in the HRD model in msec. Not used when control-rate=disable flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 3000 default-roi-quality : The default quality level to apply to each Region of Interest flags: readable, writable Enum "GstOMXVideoEncRoiQuality" Default: 0, "high" (0): high - Delta QP of -5 (1): medium - Delta QP of 0 (2): low - Delta QP of +5 (3): dont-care - Maximum delta QP value (4): intra - Region all LCU encoded with intra prediction mode dependent-slice : If encoding with multiple slices, specify whether the additional slices are dependent slice segments or regular slices flags: readable, writable Boolean. Default: false filler-data : Enable/Disable Filler Data NAL units for CBR rate control flags: readable, writable, changeable only in NULL or READY state Boolean. Default: true gdr-mode : Gradual Decoder Refresh scheme mode. Only used if gop-mode=low-delay-p flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncGdrMode" Default: 0, "disabled" (0): disabled - No GDR (1): vertical - Gradual refresh using a vertical bar moving from left to right (2): horizontal - Gradual refresh using a horizontal bar moving from top to bottom gop-length : Distance between two consecutive I frames flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state Unsigned Integer. Range: 0 - 1000 Default: 30 gop-mode : Group Of Pictures mode flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncGopMode" Default: 0, "basic" (0): basic - Basic GOP settings (1): basic-b - Basic GOP settings, includes only B-frames (2): pyramidal - Advanced GOP pattern with hierarchical B-frames (3): pyramidal-b - Advanced GOP pattern with hierarchical B-frames, includes only B-frames (5): low-delay-p - Single I-frame followed by P-frames only (6): low-delay-b - Single I-frame followed by B-frames only (4): adaptive - Advanced GOP pattern with adaptive B-frames hlg-sdr-compatible : If enabled and input caps contain HLG colorimetry, insert BT2020 EOTF and ATC SEI instead flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false initial-delay : The initial removal delay as specified in the HRD model in msec. Not used when control-rate=disable flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 1500 input-crop : Cropping parameters for input video buffer ('<left, top, width, height>') flags: writable GstValueArray of GValues of type "gint" Write only interval-intraframes: Interval of coding Intra frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 -h264,alignment=nal ! ...). flags: readable, writable Enum "GstOMXVideoEncLatencyMode" Default: 0, "normal" (0): normal - Normal mode (1): low-latency - Low latency mode (-1): default - Component Default long-term-freq : Periodicity of LongTerm reference picture marking in encoding process Units in frames, distance between two consequtive long-term reference pictures flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 0 long-term-ref : If enabled, encoder accepts dynamically inserting and using long-term reference picture events from upstream elements flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false look-ahead : The number of frames processed ahead of second pass encoding. If smaller than 2, dual pass encoding is disabled flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 0 loop-filter-beta-offset: Beta offset for the deblocking filter, used only when loop-filter-mode is enabled flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state Integer. Range: -6 - 6 Default: -1 loop-filter-mode : Enable or disable the deblocking filter (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXH265EncLoopFilter" Default: -1, "default" (0): enable - Enable deblocking filter (1): disable - Disable deblocking filter (2): disable-cross-slice - Disable deblocking filter on slice boundary (3): disable-cross-tile - Disable deblocking filter on tile boundary (4): disable-slice-and-tile - Disable deblocking filter on slice and tile boundary (-1): default - Component Default loop-filter-tc-offset: TC offset for the deblocking filter, used only when loop-filter-mode is enabled flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state Integer. Range: -6 - 6 Default: -1 low-bandwidth : If enabled, decrease the vertical search range used for P-frame motion estimation to reduce the bandwidth flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false max-bitrate : Max bitrate in Kbps, only used if control-rate=variable (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 max-consecutive-skip: Max number of consecutive frames to be skipped. Only applicable if skip-frame is enabled flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 ze will not have any effect flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 0 icture-size-x value. If set it to 0 then max-picture-size-x will not have any effect flags: writable GstValueArray of GValues of type "gint" Write only max-qp : Maximum QP value allowed for the rate control flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 51 Default: 51 le flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 min-force-key-unit-interval: Minimum interval between force-keyunit requests in nanoseconds flags: readable, writable Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 min-qp : Minimum QP value allowed for the rate control flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 51 Default: 10 name : The name of the object flags: readable, writable, 0x2000 String. Default: "omxh265enc-omxh265enc0" egularly as possible. If slice-size is defined as well more slices may be produced to fit the slice-size requirement (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 1 - 4294967295 Default: 4294967295 output-crop : Cropping parameters for output bitstream ('<left, top, width, height>') flags: writable GstValueArray of GValues of type "gint" Write only parent : The parent of the object flags: readable, writable, 0x2000 Object of type "GstObject" periodicity-idr : Periodicity of IDR frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 prefetch-buffer : Enable/Disable L2Cache buffer in encoding process flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false qos : Handle Quality-of-Service events from downstream flags: readable, writable Boolean. Default: false qp-mode : QP control mode used by the VCU encoder flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncQpMode" Default: -1, "default" (0): uniform - Use the same QP for all coding units of the frame (1): roi - Adjust QP according to the regions of interest defined on each frame. Must be set to handle ROI metadata. (2): auto - Let the VCU encoder change the QP for each coding unit according to its content (3): load-qp-absolute - Uses absolute QP values set by user. Must be set to use External QP buffer (4): load-qp-relative - Uses Relative/Delta QP values set by user. Must be set to use External QP buffer (-1): default - Component Default quant-b-frames : Quantization parameter for B-frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 quant-i-frames : Quantization parameter for I-frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 quant-p-frames : Quantization parameter for P-frames (0xffffffff=component default) flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 scaling-list : Scaling list mode flags: readable, writable, changeable only in NULL or READY state Enum "GstOMXVideoEncScalingList" Default: 1, "default" (1): default - Default scaling list mode (0): flat - Flat scaling list mode Skip. Only used if control-rate=constant or variable and b-frames are less than 2 flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false slice-size : Target slice size (in bytes) that the encoder uses to automatically split the bitstream into approximately equally-sized slices flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 0 - 65535 Default: 0 target-bitrate : Target bitrate in Kbps (0xffffffff=component default) flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 uniform-slice-type : Enable/Disable uniform slice type in slice header flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false use-out-port-pool : Use a buffer pool on the output port flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false cture size will be limited to max-picture-size-x value. If set it to 0 then max-picture-size-x will not have any effect flags: writable GstValueArray of GValues of type "gint" Write only y444-to-gray : Enable/Disable YUV format manipulation to Y8 flags: readable, writable, changeable only in NULL or READY state Boolean. Default: false

Performance

Using AR1335 camera through AP1302 ISP at 1080p:

gst-launch-1.0 mediasrcbin media-device=/dev/media0 v4l2src0::io-mode=4 ! video/x-raw, width=1920, height=1080, format=NV12, framerate=30/1 ! omxh265enc ! perf ! fakesink

Board	CPU %	FPS	Element latency (ms)
KV260	2 %	30.000	Not tested

Using AR1335 camera through AP1302 ISP at 4K:

gst-launch-1.0 mediasrcbin media-device=/dev/media0 v4l2src0::io-mode=4 ! video/x-raw, width=3840, height=2160, format=NV12, framerate=30/1 ! omxh265enc ! perf ! fakesink

Board	CPU %	FPS	Element latency (ms)
KV260	3 %	30.000	Not tested

Video Decoding

H.264

The H.264 decoder is available through the omxh264dec GStreamer element.

gst-inspect-1.0 omxh264dec

root@xilinx-kv260-starterkit-20222:~# gst-inspect-1.0 omxh264dec Factory Details: Rank primary + 1 (257) Long-name OpenMAX H.264 Video Decoder Klass Codec/Decoder/Video/Hardware Description Decode H.264 video streams Author Sebastian Dr?ge <sebastian.droege@collabora.co.uk> Plugin Details: Name omx Description GStreamer OpenMAX Plug-ins Filename /usr/lib/gstreamer-1.0/libgstomx.so Version 1.18.5 License LGPL Source module gst-omx Source release date 2021-09-08 Binary package GStreamer OpenMAX Plug-ins Origin URL Unknown package origin GObject +----GInitiallyUnowned +----GstObject +----GstElement +----GstVideoDecoder +----GstOMXVideoDec +----GstOMXH264Dec +----GstOMXH264Dec-omxh264dec Pad Templates: SINK template: 'sink' Availability: Always Capabilities: video/x-h264 alignment: au stream-format: byte-stream width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] video/x-h264 alignment: nal stream-format: byte-stream width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] SRC template: 'src' Availability: Always Capabilities: video/x-raw(memory:GLMemory) format: RGBA width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] video/x-raw(memory:XLNXLL, format:Interlaced) format: { (string)NV16_10LE32, (string)NV12_10LE32, (string)NV16, (string)NV12, (string)GRAY10_LE32, (string)GRAY8, (string)Y444, (string)Y444_10LE32 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] interlace-mode: alternate video/x-raw(format:Interlaced) format: { (string)NV16_10LE32, (string)NV12_10LE32, (string)NV16, (string)NV12, (string)GRAY10_LE32, (string)GRAY8, (string)Y444, (string)Y444_10LE32 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] interlace-mode: alternate video/x-raw(memory:XLNXLL) format: { (string)NV16_10LE32, (string)NV12_10LE32, (string)NV16, (string)NV12, (string)GRAY10_LE32, (string)GRAY8, (string)Y444, (string)Y444_10LE32 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] video/x-raw format: { (string)NV16_10LE32, (string)NV12_10LE32, (string)NV16, (string)NV12, (string)GRAY10_LE32, (string)GRAY8, (string)Y444, (string)Y444_10LE32 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] Element has no clocking capabilities. Element has no URI handling capabilities. Pads: SINK: 'sink' Pad Template: 'sink' SRC: 'src' Pad Template: 'src' Element Properties: g high bitrate streams. Decreasing it reduces the memory footprint flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 2 - 16 Default: 5 t to 'au' for normal latency and 'nal' for subframe latency (e.g. ... ! video/x-h264,alignment=nal ! omxh264dec low-latency=1 ! ...). flags: readable, writable Enum "GstOMXVideoDecLatencyMode" Default: 0, "normal" (0): normal - Normal mode (1): reduced-latency - Low ref dpb mode(reduced-latency) (2): low-latency - Low latency mode (-1): default - Component Default low-latency : When enabled, buffers will be pushed before they are evicted from the DBP, reducing decoding latency. flags: readable, writable Boolean. Default: false max-errors : Max consecutive decoder errors before returning flow error flags: readable, writable Integer. Range: -1 - 2147483647 Default: 10 name : The name of the object flags: readable, writable, 0x2000 String. Default: "omxh264dec-omxh264dec0" output-position : Position coordinates of start of active pixel data in video buffer ('<position_x, position_y>') flags: writable GstValueArray of GValues of type "gint" Write only parent : The parent of the object flags: readable, writable, 0x2000 Object of type "GstObject" qos : Handle Quality-of-Service events from downstream flags: readable, writable Boolean. Default: true buffer and process them. flags: readable, writable Boolean. Default: false

Performance

1080p

File: 1080p-h264-60fps.mp4

gst-launch-1.0 filesrc location="1080p-h264-60fps.mp4" ! qtdemux ! h264parse ! queue ! omxh264dec ! perf ! queue ! fakesink

Board	CPU %	FPS	Element latency (ms)
KV260	105 %	40.582	Not tested

4K

File:

gst-launch-1.0 filesrc location="" ! qtdemux ! queue ! omxh264dec ! perf ! queue ! fakesink

Board	CPU %	FPS	Element latency (ms)
KV260	Not tested	Not tested	Not tested

H.265

The H.265 decoder is available through the omxh265dec GStreamer element.

gst-inspect-1.0 omxh265dec

root@xilinx-kv260-starterkit-20222:~# gst-inspect-1.0 omxh265dec Rank primary + 1 (257) Long-name OpenMAX H.265 Video Decoder Klass Codec/Decoder/Video/Hardware Description Decode H.265 video streams Author Sebastian Dr?ge <sebastian.droege@collabora.co.uk> Plugin Details: Name omx Description GStreamer OpenMAX Plug-ins Filename /usr/lib/gstreamer-1.0/libgstomx.so Version 1.18.5 License LGPL Source module gst-omx Source release date 2021-09-08 Binary package GStreamer OpenMAX Plug-ins Origin URL Unknown package origin GObject +----GInitiallyUnowned +----GstObject +----GstElement +----GstVideoDecoder +----GstOMXVideoDec +----GstOMXH265Dec +----GstOMXH265Dec-omxh265dec Pad Templates: SINK template: 'sink' Availability: Always Capabilities: video/x-h265 alignment: au stream-format: byte-stream width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] video/x-h265 alignment: nal stream-format: byte-stream width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] SRC template: 'src' Availability: Always Capabilities: video/x-raw(memory:GLMemory) format: RGBA width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] video/x-raw(memory:XLNXLL, format:Interlaced) format: { (string)NV16_10LE32, (string)NV12_10LE32, (string)NV16, (string)NV12, (string)GRAY10_LE32, (string)GRAY8, (string)Y444, (string)Y444_10LE32 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] interlace-mode: alternate video/x-raw(format:Interlaced) format: { (string)NV16_10LE32, (string)NV12_10LE32, (string)NV16, (string)NV12, (string)GRAY10_LE32, (string)GRAY8, (string)Y444, (string)Y444_10LE32 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] interlace-mode: alternate video/x-raw(memory:XLNXLL) format: { (string)NV16_10LE32, (string)NV12_10LE32, (string)NV16, (string)NV12, (string)GRAY10_LE32, (string)GRAY8, (string)Y444, (string)Y444_10LE32 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] video/x-raw format: { (string)NV16_10LE32, (string)NV12_10LE32, (string)NV16, (string)NV12, (string)GRAY10_LE32, (string)GRAY8, (string)Y444, (string)Y444_10LE32 } width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] Element has no clocking capabilities. Element has no URI handling capabilities. Pads: SINK: 'sink' Pad Template: 'sink' SRC: 'src' Pad Template: 'src' Element Properties: g high bitrate streams. Decreasing it reduces the memory footprint flags: readable, writable, changeable only in NULL or READY state Unsigned Integer. Range: 2 - 16 Default: 5 t to 'au' for normal latency and 'nal' for subframe latency (e.g. ... ! video/x-h264,alignment=nal ! omxh264dec low-latency=1 ! ...). flags: readable, writable Enum "GstOMXVideoDecLatencyMode" Default: 0, "normal" (0): normal - Normal mode (1): reduced-latency - Low ref dpb mode(reduced-latency) (2): low-latency - Low latency mode (-1): default - Component Default low-latency : When enabled, buffers will be pushed before they are evicted from the DBP, reducing decoding latency. flags: readable, writable Boolean. Default: false max-errors : Max consecutive decoder errors before returning flow error flags: readable, writable Integer. Range: -1 - 2147483647 Default: 10 name : The name of the object flags: readable, writable, 0x2000 String. Default: "omxh265dec-omxh265dec0" output-position : Position coordinates of start of active pixel data in video buffer ('<position_x, position_y>') flags: writable GstValueArray of GValues of type "gint" Write only parent : The parent of the object flags: readable, writable, 0x2000 Object of type "GstObject" qos : Handle Quality-of-Service events from downstream flags: readable, writable Boolean. Default: true buffer and process them. flags: readable, writable Boolean. Default: false

Performance

1080p

File: 1080p-hevc-30fps.mkv

gst-launch-1.0 filesrc location="1080p-hevc-30fps.mkv" ! matroskademux ! h265parse ! queue ! omxh265dec ! perf ! queue ! fakesink

Board	CPU %	FPS	Element latency (ms)
KV260	108 %	40.109	Not tested

4K

File: 4k-hevc-60fps.mkv

gst-launch-1.0 filesrc location="4k-hevc-60fps.mkv" ! matroskademux ! h265parse ! queue ! omxh265dec ! perf ! queue ! fakesink

Board	CPU %	FPS	Element latency (ms)
KV260	103 %	7.919	Not tested

Simultaneous encoding and decoding

gst-launch-1.0 mediasrcbin media-device=/dev/media0 v4l2src0::io-mode=4 ! video/x-raw, width=3840, height=2160, format=NV12, framerate=30/1 ! omxh265enc ! queue ! omxh265dec ! perf ! queue ! fakesink

Board	CPU %	FPS	Element latency (ms)
KV260	102 %	9.775	Not tested