I.MX8 Deep Learning Reference Designs - Implementing Custom Media

i.MX8 Deep Learning Reference Designs
Getting Started
Environment Setup Evaluating the Project Purchasing a License
Project Architecture
High Level Design
Reference Designs
Restricted Zone Detector
Customizing the Project
Implementing Custom Inference Listener Implementing Custom Policies Implementing Custom Actions Implementing Custom Media Implementing a Custom Application
Contact Us

The Media Module

This element is responsible for encapsulating and managing the information data received through the system's camera. The main functionality of this component is that it allows the Camera Capture subsystem to carry out a transparent and simple manipulation of the data received, through the operations provided by the Media interface. In addition, the flexibility of the system is increased since the Camera Capture module is not aware of the underlying technology used for processing camera data, so one media can be replaced by another without affecting overall functionality. For example, in the case of the Restricted Zone Detector, the Gstd plugin is used to manage media instances. And within the media descriptor, the camera protocol is indicated, which corresponds to RTSP in said system. Other examples of components that could be used for this purpose were mentioned in the High Level Design.

As indicated in other modules of this system, the design of this component is not restricted to a specific implementation, so different technologies, plugins, or your own components can be added within the project structure, however, certain conditions must be respected prior to the incorporation of a custom component. After reading this wiki page, you will know what are the requirements to add your custom Media module to this reference design.

Class Diagram

The following figure shows a class diagram representation of the Media structure. To have a better understanding of the component, the diagram also shows which modules within the system are related to the Media instance.

Media Module Class Diagram

Communication Between Modules

• Camera Capture: As previously explained in other sections of this Wiki, Camera Capture represents the module in charge of managing the reception and processing of the information data coming from the cameras used by the application. To fulfill this purpose, Camera Capture interacts with entities called Medias, which encapsulate within its structure, the type of camera used, the protocol in which the data is received, and provides the necessary operations to manipulate the flow of information. Camera Capture is in charge of creating the respective Media instances. For this, it uses a Media Factory, and a Media Descriptor, so that through the description entered into the module, the factory can be used to build the new Media of the system. Of course, the tasks of Camera Capture also include managing the processes of starting and stopping the flow of information. In this way, it could be said that Camera Capture is the entity in charge of manipulating the instantiation, deletion, and regular operation of the Media entities.

• Media Factory: Each custom implementation of a Media type must have an associated module in charge of its creation. As shown in the class diagram, a Factory Pattern is used for the design of this part of the system, where a Factory Interface is provided, which indicates the minimum operations that this module must include. The Factory will receive a Media descriptor as a parameter, which indicates the information required to instantiate a new Media type. The Camera Capture module will be in charge of using the functionalities of the Factory for the construction of the Media.

Media Interface Operations

As shown in the diagram, any custom Media module must implement the operations that are defined by the interface named "Media". As a user, you can add more elements to the design of the said component, for example, add new methods, modify the constructor of the class, and even carry out your implementations of each of the operations exposed by the interface. The important thing is that these methods are maintained. Next, there will be a brief explanation of the purpose of each of the operations defined by the interface. Remember that the specific implementation is up to your criteria or needs.

• start: As its name indicates, the purpose of this method is to initialize the data transmission process coming from the cameras. The idea is that Camera Capture can manage the right times to start receiving information from each Media.

• stop: Method that represents the counterpart of the start operation. As its name indicates, the idea is that the Camera Capture module can stop the flow of information received by the cameras, through this function. The fact that a Media stops its operation does not mean that it has been eliminated, it is simply in a paused state, which can be reactivated later.

• get_name: This is a simple method that allows you to get the name with which a certain instance of Media is created. Other modules or components of the system may require some type of management with the Media, so when obtaining the identifier they can carry out the respective processing tasks.

• get_triggers: According to the system design, each Media created will have one or more associated triggers, which abstract the business rules of the application and determine the moment necessary to activate a certain action in that specific Media. With this method provided by the interface, it is possible to obtain a list of the triggers associated with each Media.

• register_error_callback: Through this function, the Camera Capture module that manages the Media can register a callback function for error handling. The idea is that this callback is in charge of monitoring the flow of information coming from each Media, and being able to carry out actions when an error occurs in the camera system or in the process of obtaining the information. For example, in the RZD application, the error callback is responsible for restarting the Media instance that reported an error and stopped. What is sought with this type of strategy is that the system can recover automatically in any eventuality, without the need for manual intervention by the user. Of course, the above is just an example of what an implementation of this type of callback could be and the user can implement other types of actions.

Supporting a New Camera

It is well known that any Media application that involves Computer Vision and AI processing tasks may require the use of different types of cameras that capture the input information to the system. That is why the design of this system provides flexibility in the section that involves selecting the camera type. As previously indicated and shown in the class diagram, each instance of Media created by the Custom Factory uses a Media Descriptor as an input parameter. Using this descriptor it is possible to abstract the camera component used. For example, in the Restricted Zone Detector, a URL is used as a central component of the descriptor that refers to a camera that transmits via RTSP, which is adapted to the context of a monitoring system in a determined zone with several cameras. This description is used by the GStreamer Daemon plugin (a custom component used in RZD to create Media instances), which creates a GStreamer pipeline that encapsulates the flow of information received.