OpenGL Accelerated HTML Overlay: Examples - Library Usage

OpenGL Accelerated HTML Overlay
Basics
Getting Started
Evaluating HTML Overlay Getting the Code Building HTML Overlay
Library User Manual
GStreamer
GstHTMLOverlay
Examples
Library Usage GStreamer Usage
Performance
NVIDIA Jetson NXP i.MX 6 NXP i.MX 8
Contact Us

Library Usage

Pre-requisites

NVIDIA Jetson

If you are on an NVIDIA Jetson, you may need to set the following variables

export DISPLAY=:0
export WEBKIT_DISABLE_COMPOSITING_MODE=1

Graphics Backend

The graphics backend class encapsulates the different OpenGL operations, such as the buffer creation or the drawing. It is capable of overlaying one buffer over the other. The graphics backend allows using user-defined parameters to determine the overlay position and size against a background. To create the graphics backend object, it is necessary to specify the background's backend type and width and height. Currently, the supported graphics backend is OpenGL. The width and height should be the size of the whole frame in which both images will be rendered.

auto opengl_backend = rr::IGraphicsBackend::GetGraphicsBackend (rr::Backends::kOpenGLBackend, background_width, background_height);

Content Loader

The content loader class allows us to create and obtain the content overlaid on the background frame. Currently, the supported backend is WebKitGtk. This class provides functionality to update the contents and set the parameters on how the website will behave. Creating the content loader object requires the type of loader to use, in this case, WebkitGtk.

auto content_loader = rr::IContentLoader::GetContentLoader(rr::Loaders::kWebkitGtkLoader);

Set ContentLoader parameters

Here we set the parameters to manipulate the website. These parameters are web URL, refresh rate and a flag to enable javascript execution within the browser.

rr::WebParams params = rr::WebParams();
params.url = "https://www.ridgerun.com/"; //Website url to render
params.refresh_rate = 0; // Refresh rate(Hz) of the content loader, can be 0 to not auto update
params.enable_js = true; // Flag to enable javascript
content_loader->SetParams(params);

Create ROIs and Drawing parameters

Specifies the location and dimension of the ROI on the background buffer and the ROI of the buffer that will be overlaid on the background buffer.

rr::Rectangle overlay_roi (o_x, o_y, o_w, o_h)
rr::Rectangle background_roi (b_x,b_y,b_w,b_h)

Then attach both to the drawing parameters.

rr::DrawingParams drawing_params(overlay_roi,background_roi);

Creating the buffers

HOST buffers

First we create the buffers that will be used to manipulate and load the data corresponding to the video frames.

• Background buffer
  • Uses RGBA color, and corresponds to the content that will be used as background
  • Created by the GraphicsBackend

std::shared_ptr<rr::Buffer> background_buffer_host =
      std::make_shared<rr::Buffer>(nullptr, background_buffer_host_width,background_buffer_host_height, color_channels, background_buffer_host_pitch, rr::Format::RGBA, rr::Memory:HOST);

• Overlay buffer
  • Uses ARGB color, and corresponds to the buffer that will be overlaid on the background.
  • Created by the GraphicsBackend

std::shared_ptr<rr::Buffer> webkit_buffer_host =
      std::make_shared<rr::Buffer>(nullptr,webkit_buffer_host_width,webkit_buffer_host_height, color_channels, webkit_buffer_host_pitch, rr::Format::ARGB, rr::Memory::HOST);

OpenGL buffers

Initializes the buffers that will be used as textures to use on OpenGL graphics Library. For the two of them, the graphics backend is responsible for creating the data field on the buffers.

• Background texture

std::shared_ptr<rr::Buffer> background_buffer_GL =
      std::make_shared<rr::Buffer>(nullptr, background_buffer_host_width,background_buffer_host_height, color_channels, background_buffer_host_pitch, rr::Format::RGBA, rr::Memory::OPENGL)

• Overlay texture
  • Even though the webkit content loader uses ARGB, the graphics backend does the color conversion

std::shared_ptr<rr::Buffer> webkit_buffer_GL =
      std::make_shared<rr::Buffer>(nullptr, webkit_buffer_host_width,
                                   webkit_buffer_host_height, color_channels,
                                   nullptr, webkit_buffer_host_pitch, drawing_params,
                                   rr::Format::RGBA, rr::Memory::OPENGL);

Then, create them to allow the backend to manipulate them.

opengl_backend->CreateBuffer(webkit_buffer_host)
opengl_backend->CreateBuffer(background_buffer_host)
opengl_backend->CreateBuffer(webkit_buffer_GL)
opengl_backend->CreateBuffer(background_buffer_GL)

Attach buffer for website

Attach the buffer that will be used by the content loader to offload the results from the rendered content.

content_loader->AttachBuffer(webkit_buffer_host);

Note: The content loader uses two buffers to help to keep up with higher frame-rates. Two different buffers are needed, with their respective call to AttachBuffer.

Obtaining the content to overlay

To render the content, set a refresh rate using the WebParams that can be set with the SetParams method on the content loader, so it updates the buffer asynchronous, or manually call the Update method, with the force flag as such:

content_loader->Update(true);

The update can be called any time, even with the refresh_rate member set. To toggle between manual or automatic updates, the refresh_rate can be set to 0 to avoid automatic updates or more than 0 to enable them.

Upload

Then, upload the contents on the HOST buffers to the OpenGL buffers. It expects a buffer with HOST memory, with the data to be loaded to the corresponding OpenGL buffer.

opengl_backend->Upload(webkit_buffer_GL,webkit_buffer_host);
opengl_backend->Upload(background_buffer_GL,background_buffer_host);

If using multiple buffers on the content loader:

auto buffer = content_loader->GiveBuffer();
opengl_backend->Upload(webkit_buffer_GL,buffer);
content_loader->LeaveBuffer(buffer->data);
opengl_backend->Upload(background_buffer_GL,background_buffer_host);

Perform overlay

The draw function overlays the ROI from the contents of the web on the ROI on the background buffer. It expects two buffers with OpenGL memory.

opengl_backend->Draw(background_buffer_GL,webkit_buffer_GL);

Download

To save the overlaid buffer, create a CPU buffer to download the contents of the OpenGL framebuffer into it. This buffer should have the same size as the background buffer. Use the Download method from the graphics backend to obtain and use the data.

std::shared_ptr<rr::Buffer> overlayed_buffer_host =
      std::make_shared<rr::Buffer>(background_buffer_host_data,background_buffer_host_width,
                                   background_buffer_host_height,color_channels ,
                                   background_buffer_host_pitch, nullptr, 
rr::Format::RGBA, rr::Memory::HOST);

opengl_backend->CreateBuffer(overlayed_buffer_host);
opengl_backend->Download(overlayed_buffer_host,background_buffer_GL);

Free memory

When finished, free the memory used by the buffers.

opengl_backend->DeleteBuffer(background_buffer_host);
opengl_backend->DeleteBuffer(webkit_buffer_host);
opengl_backend->DeleteBuffer(background_buffer_GL);
opengl_backend->DeleteBuffer(webkit_buffer_GL);
opengl_backend->DeleteBuffer(overlayed_buffer_host);

The graphics backend and the content loader are responsible for freeing any other buffer or context that was used or started.

Complete example

/*
* Copyright (C) 2023 RidgeRun, LLC (http://www.ridgerun.com)
* All Rights Reserved.
* Author: Andres Artavia Lopez <andres.artavia@ridgerun.com>
*
* The contents of this software are proprietary and confidential to RidgeRun,
* LLC. No part of this program may be photocopied, reproduced or translated
* into another programming language without prior written consent of
* RidgeRun, LLC. The user is free to modify the source code after obtaining
* a software license from RidgeRun. All source code changes must be provided
* back to RidgeRun without any encumbrance.
*/

#include "libhtmloverlay/buffer.hpp"
#include "libhtmloverlay/icontentloader.hpp"
#include "libhtmloverlay/igraphicsbackend.hpp"
#include "libhtmloverlay/memory.hpp"
#include "libhtmloverlay/webparams.hpp"
#include <cstring>
#include <iostream>
#include <gtk/gtk.h>
#include <opencv2/opencv.hpp>


/* Easy to use macro to check for errors on libhtmloverlay library calls */
/**
* If @p expr != rr::RuntimeError::EOk returns false
*/
#define TRY(err, expr) \
  err = expr; \
  if (err.GetCode() != rr::RuntimeError::EOk) { \
    std::cout << "Error, code: " << err.GetCode() \
    << ",msj: " << err.GetDescription() << std::endl; \
    gtk_main_quit(); \
    return false; \
}
/* Struct used to share data with the async update method */
struct SharedData {
  std::shared_ptr<rr::IContentLoader> content_loader;
  std::shared_ptr<rr::IGraphicsBackend> graphics_backend;
  std::shared_ptr<rr::Buffer> overlay_texture;
  std::shared_ptr<rr::Buffer> background_buffer;
  std::shared_ptr<rr::Buffer> background_texture;
  std::shared_ptr<rr::Buffer> output_buffer;
};
/* Constants */
static constexpr int kSizeWidth = 500;
static constexpr int kSizeHeight = 500;
static constexpr int kChannels = 4;
static constexpr int kBpp = 4;
static constexpr int kPitch = kSizeWidth * kBpp;
static constexpr int kDataSize = kPitch * kSizeHeight;
static constexpr float kSizeMax = 100.0f;
static constexpr int kWindowBorder = 100;
static constexpr int kRefresh_rate = 0;
static constexpr int kTimeout_rate = 16;
/**
 * Note this loop counter, 
 * the content loader needs some time to properly load and
 * display it, this varies according to the complexity of the wesite.
 */
static int loop = 350;
/* Main method to update the content, and make the library calls to make the overlay */
static gboolean do_update(gpointer data) {
  rr::RuntimeError ret;
  struct SharedData *in_data = static_cast<struct SharedData *>(data);
  in_data->content_loader->Update(true);
  auto buff = in_data->content_loader->TakeBuffer();
  TRY(ret, in_data->graphics_backend->Upload(in_data->overlay_texture, buff));
  TRY(ret, in_data->graphics_backend->Upload(in_data->background_texture,
  in_data->background_buffer));
  in_data->content_loader->UnlockReadBuffer();
  TRY(ret, in_data->graphics_backend->Draw(in_data->background_texture,
  in_data->overlay_texture));
  in_data->graphics_backend->Synchronize();
  TRY(ret, in_data->graphics_backend->Download(in_data->output_buffer,
  in_data->background_texture));
  if (loop == 0) {
    gtk_main_quit();
    return false;
  }
  loop--;
  return true;
}

int main() {
  auto content_loader =
  rr::IContentLoader::GetContentLoader(rr::Loaders::kWebkitGtkLoader);
  auto graphics_backend = rr::IGraphicsBackend::GetGraphicsBackend(
  rr::Backends::kOpenGLBackend, kSizeWidth, kSizeHeight);
  rr::RuntimeError ret;
  rr::WebParams params = rr::WebParams();
  params.url = "https://commons.wikimedia.org/wiki/File:Videotestsrc-720x576-16-15.ogv"; // Url to load
  params.refresh_rate = 60; // Refresh rate of the content loader
  params.enable_js = true; // Flag to enable javascript execution
  content_loader->SetParams(params);
  rr::Rectangle in_roi{0, 0, kSizeWidth-100, kSizeHeight-100}; // Parameters to define the viewport of the overlay
  rr::Rectangle out_roi{50, 50, kSizeWidth, kSizeHeight}; // Parameters to define the position of the overlay in the output
  rr::DrawingParams drawing_params{in_roi, out_roi, false};
  /* Buffers reserved for the content loader */
  std::shared_ptr<rr::Buffer> overlay_buffer_1 =
    std::make_shared<rr::Buffer>(nullptr, kSizeWidth, kSizeHeight, kBpp,
    kPitch, rr::Format::ARGB, rr::Memory::HOST);
  std::shared_ptr<rr::Buffer> overlay_buffer_2 =
    std::make_shared<rr::Buffer>(nullptr, kSizeWidth, kSizeHeight, kBpp,
    kPitch, rr::Format::ARGB, rr::Memory::HOST);
  std::shared_ptr<rr::Buffer> overlay_texture = std::make_shared<rr::Buffer>(
    nullptr, kSizeWidth, kSizeHeight, kBpp, kPitch, rr::Format::RGBA,
    rr::Memory::OPENGL, drawing_params);
  /* Buffer dedicated for the background */
  std::shared_ptr<rr::Buffer> background_buffer =
     std::make_shared<rr::Buffer>(nullptr, kSizeWidth, kSizeHeight, kBpp,
     kPitch, rr::Format::RGBA, rr::Memory::HOST);
  std::shared_ptr<rr::Buffer> background_texture = std::make_shared<rr::Buffer>(
     nullptr, kSizeWidth, kSizeHeight, kBpp, kPitch, rr::Format::RGBA,
     rr::Memory::OPENGL, drawing_params);
  /* Buffer dedicated to contain the result */
  std::shared_ptr<rr::Buffer> output_buffer =
    std::make_shared<rr::Buffer>(nullptr, kSizeWidth, kSizeHeight, kBpp,
    kPitch, rr::Format::RGBA, rr::Memory::HOST);
  /* Call to the graphics backend, to setup each buffer */
  graphics_backend->CreateBuffer(overlay_buffer_1);
  graphics_backend->CreateBuffer(overlay_buffer_2);
  graphics_backend->CreateBuffer(overlay_texture);
  graphics_backend->CreateBuffer(background_buffer);
  graphics_backend->CreateBuffer(background_texture);
  graphics_backend->CreateBuffer(output_buffer);
  /* Attach the 2 buffers that the content loader needs */
  content_loader->AttachBuffer(overlay_buffer_1);
  content_loader->AttachBuffer(overlay_buffer_2);

  std::shared_ptr<struct SharedData> shared_data =
  std::make_shared<struct SharedData>();
  shared_data->content_loader = content_loader;
  shared_data->graphics_backend = graphics_backend;
  shared_data->overlay_texture = overlay_texture;
  shared_data->background_buffer = background_buffer;
  shared_data->background_texture = background_texture;
  shared_data->output_buffer = output_buffer;
  /* Add sample data to the background buffer */
  memset(background_buffer->data, 150, kDataSize);
  /* Set up the loop to update the content and generate the overlay */
  g_timeout_add(kTimeout_rate, do_update, shared_data.get());
  gtk_main();
  /* Output the result as a png image */
  cv::Mat image_result(kSizeHeight, kSizeWidth, CV_8UC4, output_buffer->data);
  cv::Mat image_result_RGBA;
  cv::cvtColor(image_result, image_result_RGBA, cv::COLOR_BGRA2RGBA);
  cv::imwrite("result.png", image_result_RGBA);
  /* Free the used resources */
  graphics_backend->DeleteBuffer(overlay_texture);
  graphics_backend->DeleteBuffer(background_texture);
  graphics_backend->DeleteBuffer(background_buffer);
  graphics_backend->DeleteBuffer(overlay_buffer_1);
  graphics_backend->DeleteBuffer(overlay_buffer_2);
  graphics_backend->DeleteBuffer(output_buffer);
}

This will output a file similar to the next figure:

Error creating thumbnail: Unable to save thumbnail to destination