DeepStream Reference Designs - Reference Designs - Automatic Parking Lot Vehicle Registration
|
| DeepStream Reference Designs | |
|---|---|
 | |
| Getting Started | |
|
|
|
| Project Architecture | |
|
|
|
| Reference Designs | |
|
|
|
| Customizing the Project | |
|
|
|
| Contact Us | |
|
|
Contents
Introduction
System Components
Next, an explanation of the main components that make up the APLVR application will be provided. As you will notice, the structure of the system is based on the design presented in the High-Level Design section, reusing the components of the main framework and including custom modules to meet the business rules required by the context of this application.
Camera Capture
Class Diagram
Below is the class diagram that represents the design of the Camera Capture subsystem for the APLVR application.
Media Entity
For handling the media, APLVR relies on the use of the methods provided by the Python API of GStreamer Daemon. This library allows us to easily manipulate the GStreamer pipelines that transmit the information received by the system's cameras. In this way, the Media and Media Factory entities are used to abstract the methods that allow creating, starting, stopping, and deleting pipeline instances. In addition, the parking system uses three cameras that transmit video information through the Real-Time Streaming Protocol (RTSP). Each of these cameras represents the entrance, exit, and parking sectors of the parking lot system. The video frames received from the three cameras are processed simultaneously in the DeepStream pipeline, which is in charge of carrying out the inference process. To do this, the media are described using the GstInterpipe, which is a GStreamer plug-in that allows pipeline buffers and events to flow between two or more independent pipelines.
AI Manager
Class Diagram
Below is the class diagram that represents the design of the AI Manager for the APLVR application. As can be seen in the diagram, the AI Manager bases its operation on the entity called engine, which is an extension of the media module used by the Camera Capture subsystem. This extension allows the media entity to handle the inference operations required for the application. Also, the media descriptor for the engine abstracts the components of the DeepStream Pipeline.
DeepStream Models
The following figure shows the general layout of the DeepStream pipeline used by the APLVR application. For simplicity of the diagram, some components are excluded to give emphasis to the most important ones.
As can be seen in the pipeline, the input information comes from the three RTSP cameras used by the APLVR parking system. These data are multiplexed by the plugin called nvstreammux, which is in charge of forming a batch of frames from multiple input sources. Then will come the inference process, where APLVR relies on the use of three models to achieve its goal. The models that are included are part of version 3.0 of the TAO Toolkit, which is a tool that allows you to combine pre-trained NVIDIA models with your own data, to create custom AI models that can be used in computer vision applications. The models shown in the pipeline are:
- Car Detection Model: It corresponds to the first inference made on the data, to make the first object detection, although for APLVR purposes we focus on the detected vehicles. This model is based on the TrafficCamNet neural network.
- License Plate Detection: Once the vehicles have been detected in the input frames, a vehicle license plate detection is carried out. For this, the LPDNet network is used. In the diagram, you can see that at the output of this model, the information obtained is passed through a tracker. This allows us to assign a unique identifier to each of the detected license plate instances, and then we can perform specific tasks within the context of the problem that APLVR seeks to solve.
- License Plate Recognization: Finally, once the license plate detection process is finished, this model is used to provide us with textual information on the characters presented by each of the detected license plates. This model is based on the LPRNet network, and it is also combined with a custom parser, which performs the text extraction based on the character dictionary that is included along with the configuration files.
Note: The diagram shows the PGIE and SGIE notations for each of the models. These notations stand for "Primary GPU Inference Engine" and "Secondary GPU Inference Engine". It is also important to note that for the operation of these models, we want to enter a series of hyperparameters that the neural network will use in its configuration. All these configuration parameters are provided in files in txt format and will be explained later in the Config Files section.
Once the inference process has been carried out, the information obtained can be displayed graphically, and in the case of APLVR, the metadata is sent to the nvmsgconv and nvmsgbroker plugins, which allow us to manipulate the information generated. It is out of the scope of this Wiki to provide a more detailed explanation of each of the components shown in the APLVR DeepStream pipeline, however, if you are interested, you can review the Plugin Guide provided by the DeepStream SDK in its documentation.
Inference Listener
In the case of the inference listener, the APLVR application uses a DeepStream plugin called Gst-nvmsgbroker. Basically, when inserted in the DeepStream pipeline, this element will be in charge of sending the payload messages (after the inference process) to the server in charge of receiving them, for which a specified communication protocol is used, in this case, the AMQP protocol. On the server-side, APLVR locally deploys a rabbitmq broker, which can be installed via the rabbitmq-server package. On the client-side, there is an application developed in Python, which represents the APLVR custom inference listener, and is supported by the methods provided by the Pika library, to consume the broker's messages asynchronously. To establish communication, it is necessary that, both the client and the server, use the same credentials and configuration parameters of the broker, which are established in the configuration files used by the application. If you want to know more details about these configuration files, we invite you to read the APLVR Config Files section.
Inference Parser
This custom module is responsible for parsing the information received by the inference listener. The format of the messages transmitted by the used message broker is defined by the DeepStream plugin called Gst-nvmsgconv, which is responsible for converting metadata into message payloads based on schema. From DeepStream 5.0 release, there is support for two variations of this schema, full and minimal. APLVR uses the format provided by the minimal schema since it includes the most relevant information needed by the application to apply its respective logic. Below is an example of the format provided by the payload minimal schema of this plugin:
{
"version": "4.0",
"id": 60,
"@timestamp": "2022-03-28T23:49:46.377Z",
"sensorId": "CAMERA_ID",
"objects": [
"18446744073709551615|357.758|285.072|411.014|317.846|lpd|#|90MZ205|0.972624",
"0|579.805|133.696|639.437|244.258|Car",
"1|212.615|102.074|513.669|400.402|Car"
]
}
The inference parser receives this format as a parameter, and after processing it, it returns a dictionary that is stored in an object of type Inference Info, which can be interpreted by any component of the system. Note the difference in the context and format of the stored information, which shows the importance of this module, since it facilitates the interpretation of the inferences received:
{
"sensorId": "Exit",
"date": "2022-05-25",
"time": "22:44:25.019",
"instances": [
{
"id": "10",
"category": "lpd",
"bbox": {
"topleftx": 280.683,
"toplefty": 155.192,
"bottomrightx": 379.966,
"bottomrighty": 206.743
},
"extra": [
"#",
"P370651",
"0.979938"
]
},
{
"id": "13",
"category": "car",
"bbox": {
"topleftx": 137.766,
"toplefty": 0.0,
"bottomrightx": 494.419,
"bottomrighty": 309.661
},
"extra": []
}
]
}
Actions
The following actions are available to the user of the APLVR application. Although the actions are specific to the parking lot context, they could be adapted for use in other designs, so these ideas can serve as a basis for different projects.
- Stdout Log: This action represents printing relevant information on the processed inferences to the system's terminal, using the standard output. This information can serve as a kind of runtime log about what is happening in the managed parking lot. An example of the information format shown in the standard output is as follows:
A vehicle with license plate: XYZ123 is in sector "Zone A". Status: Parked. 2022-06-02, 20:56:04.143- Dashboard: The dashboard action is very similar to the Stdout Log in terms of the content and format of the information it handles. The difference is that in this case, the information is not printed on the terminal using the standard output but is sent to a web dashboard, using an HTTP request to the API that controls the operations where the web dashboard is located. The APLVR application provides an API and a basic design that represents the web dashboard that an administrator of this parking system would use. Later, in the Additional Features section of this wiki, you can find more information about the implementation of this web dashboard.
- Snapshot: What this action seeks is to create a snapshot, of the flow of information that the media pipeline in question is transmitting. Like the other ones, this action is linked to the camera on which the business rules (policies) will be applied, and eventually, they will activate the execution of the corresponding task, depending on certain conditions. The snapshot will be stored in an image file with the format ... and with the generic name of ... The path where the image will be stored can be entered as a configuration parameter of the aplvr.yaml file.
 | Fabricio: It remains to add an example of the snapshot demonstration. (please remove this box when addressed) |
| Fabricio: Explaining the recording action is pending (please remove this box when addressed) |
Important Note: Remember that if you want a more detailed explanation of what are actions in the DeepStream Reference Designs project, including visualization of their class diagrams with code examples, we recommend reading the sections of High-Level Design and Implementing Custom Actions.
Policies
The following policies are available to the user of the APLVR application. Given the nature of policies, the examples shown depend on the context of a parking system. APLVR uses a monitoring system with three cameras, which represent the entry, exit, and parking area sectors. So the policies explained below are based on the inferences received through each of these sector cameras, to apply the required business rules.
- Parking Entrance/Exit Policy: This policy is responsible for determining the exact moment a vehicle enters the parking lot and leaves it. Once the logic of the policy has established that one of these two conditions is met, one of the previously explained actions can be executed, where the user can see information about the license plate of the vehicle that entered or left the parking lot, with its respective date and hour.
- Parking Sector Policy: As its name indicates, the idea of this policy is to verify the status of the vehicles that are captured by the camera in the parking area, whether they are parked or have left said area and are about to leave the parking lot. The logic of this policy bases its operation on the use of timers and filters. The filters are used to validate the accuracy of the information received, while the timers allow determining the moment to make a change of state, after not receiving any type of inference in the parking sector, which is an indicator that a vehicle left that area.
Important Note: Remember that if you want a more detailed explanation of what are the policies in the DeepStream Reference Designs project, including visualization of their class diagrams with code examples, we recommend reading the sections of High-Level Design and Implementing Custom Policies.
Config Files
APLVR YAML
cfg_mqtt
Support Platforms -> Deepstream Models configuration
Main Module
Additional Features
Web Dashboard
Automatic Script
Demonstration
| Fabricio: Missing: add images or animated gifs of the application demo (please remove this box when addressed) |
