FPGA Image Signal Processor - GStreamer Pipelines

Overview

The V4L2-FPGA project makes it possible to use the FPGA-ISP modules with GStreamer. Depending on the accelerator, you can adjust the input/output format of the streaming. Also, the accelerators support ranges of sizes, making them versatile for almost any resolution, including non-standard sizes.

With FPGA-ISP, you can accelerate your image processing application with GStreamer without sacrificing your CPU resources and letting the FPGA do the job for you.

Typical applications

Let's suppose that our FPGA application is based on an accelerator scheme. This implies that there will be two pipelines; one for transferring the data to the FPGA and another for receiving the processed data from it.

Sink (sender) GStreamer Pipeline

It makes the most of the v4l2sink element, which receives the frame and transfers it to the FPGA aid by the V4L2-FPGA driver. The typical caps to set in the pipeline are:

width
height
format

Those caps are meant to set the input image on the accelerator. In most of the cases, the accelerator will mirror these properties to its output, given that currently, they do not support image scaling.

The pipeline, therefore, has the following structure:

# Properties
WIDTH=1920
HEIGHT=1080
FORMAT=ARGB

# GStreamer pipeline launcher
gst-launch-1.0 videotestsrc ! video/x-raw,width=$WIDTH,height=$HEIGHT,format=$FORMAT ! v4l2sink device=/dev/video1

Also, it is possible to use the GStreamer caps autonegotiation:

# GStreamer pipeline launcher
gst-launch-1.0 videotestsrc ! v4l2sink device=/dev/video1

Source (receiver) GStreamer pipeline

The source receives the processed data from the FPGA. It uses the v4l2src element to do this task. The process is quite similar to the sink pipeline, given that defining the caps is also required. Let's suppose that we are using the sink specified above, therefore, the corresponding source pipeline is:

# Properties
WIDTH=1920
HEIGHT=1080
FORMAT=ARGB

# GStreamer pipeline launcher
gst-launch-1.0 v4l2src device=/dev/video2 ! video/x-raw,width=$WIDTH,height=$HEIGHT,format=$FORMAT ! videoconvert ! xvimagesink

Execution order

By defining the caps, it is possible to execute the pipelines in any order. However, because of GStreamer negotiation, is recommended to start the sink pipeline first and before of source pipeline.

Each accelerator has its supported properties specified on its page. Please, refer to FPGA ISP Accelerators