NVIDIA Jetson TX2 - Argus vs V4L2 Latency Analysis
Contents
Introduction to NVIDIA Jetson TX2 - Argus vs V4L2 Latency Analysis
In this wiki, a practical example of the measurement of latency when using libargus and V4L2 is shown. This wiki assumes that the reader is familiar with the Jetson TX2 VI concepts outlined in NVIDIA Jetson TX2 - Video Input Timing Concepts and the proposed measurement techniques in NVIDIA Jetson TX2 - VI Latency Measurement Techniques.
Jetson TX2 latency measurement
Description of the experiment
The experiment consisted of an OV5693 capturing at 1280x720@120 fps. Each captured frame was labeled as metadata_timestamp-system_timestamp.jpg, so that metadata_timestamp corresponds to the timestamp extracted with getSensorTimestamp and system timestamp is the timestamp (clock monotonic) captured after the sensor timestamp right after the frame metadata is retrieved.
During camera capture, a kernel module captures system timestamp (clock monotonic) and then turns on a GPIO (initially off) that has a LED connected within the field of view of the camera. The timestamp is printed after the GPIO has been turned on to avoid extra delays between the timestamp extraction and the LED turn on.
The board was power-cycled per every iteration of the experiment.
It is possible to associate the first frame where the led was turned on, whose name includes the metadata_timestamp (t1) and the system timestamp (t2) with the timestamp printed by the kernel module (t0), as shown in figure 1. In summary:
- t0 is the system timestamps acquired at the kernel module when turning on the LED.
- t1 is the metadata_timestamps, that is, the timestamps obtained with getSensorTimestamps() and coming out of Argus (from either the ISP or the RTCPU).
- t2 is the system timestamps obtained within the Argus application just right at the moment we extract t1.
Before performing the test, the following assumptions were made:
- t1 is a timestamp set during ISP processing, after image capture. It shall be greater than the RTCPU timestamps (frame acquisition time).
- t1 shall be greater than t0 since it includes post-capture processing.
- t0 - t1 shall be always negative.
- t2 - t0 shall be always positive and provide a measurement of the total time since the sensor integration until its acquisition at the user space level.
A total of six experiments were performed to measure the deltas. The summarized in average (rounded) results are shown in the table below:
| delta 0 | 4.6 ms |
| delta 1 | 29.4 ms |
| delta 2 | 34 ms |
Some of the initial assumptions had to be reviewed since there were non-consistent results in the time stamping of the frames, meaning that there might be different clock sources for the frame's time stamps as it passes through the different modules of the capture interface (starting at the VI and ending at the user space application).
Analysis
This experiment consisted of enabling tracing, use RTCPU reference for led timestamp (t0) and frame timestamp (t1) (without system clock offset adjustment), so that tracing VI information could be associated with the frame of interest (in which the led turns on). For extracting the frame ID, it was found that getCaptureId() function was misleading, instead, the value was extracted of status->frame directly in the vi_notify.c file.
Some additional VI's concepts to keep in mind:
- According to the Parker Technical Reference Manual, ISP is physically separated from VI, and works in a different clock domain, often running faster, then VI must buffer data for ISP. Also, it mentions that there is a FIFO in the ISP that stores a small number of VI packets.
- According to L4T multimedia API reference, getSensorTimestamp() (t1) returns the time that the first data from the capture arrives from the sensor.
- According to NVIDIA's, getSensorTimestamp() (t1) function call returns the software shutter event time.
- VI has a notification engine that logs events timestamped by RTCPU clock, and can be accessed in CPU by enabling tracing. There are four events of interest for this experiment:
- ISPBUF_FS: Occurs when the Frame Start event that was detected in CHANSEL reaches the ISPBUF.
- CHANSEL_PXL_SOF: First pixel of frame.
- CHANSEL_PXL_EOF: Last pixel of frame.
- ISPBUF_FE: Occurs when the Frame End event that was detected in CHANSEL reaches the ISPBUF.
Results
The results obtained are shown in the table below.
We assume that t1 is taken when the first row of pixel data arrives at the VI, this is, as soon as the first sensor row exposure ends. Also, we know that VI has to buffer data for ISP, so there must be a delay between t1 and ISP first event, that is ISP_FS. This delay is shown in the table below and seems to be pretty precise of 150us. With this result, we can answer the question of the origin of the timestamp, knowing that it is pre-ISP.
Consider ISPBUF delta result, this was computed as ISPBUF_FS - ISPBUF_FE, and seems to be pretty precise 8ms. We don't know why this delta is so big, Could this be equal to the sensor integration time? This would be confirmed if we had enough data of the EOF timestamp. This timestamp is usually zero as per vi_notify extraction, however, in these tests there were a couple of cases in which it was not zero, then allowing us to calculate the delta SOF-EOF, that resulted in the value -7881952 for both cases, then confirming an integration time of approx. 8ms.
Finally, consider delta 0. These values are not so precise and we still see a couple of cases where delta0 is positive. For this experiment, in contrast with the previous, we ensured that the led covert fully the vertical FoV of the camera. We found that in tests 3 and 10, there was a rolling shutter artifact, the led light was observed only in the lower half of the image. We think that the led turned on in the middle of the sensor integration, and given that t1 is captured when the first row of pixel arrives to VI, we assume that this probably happens as soon as it's ready, then causing delta0 to be positive, since it actually occurred after the first row of data arrived to VI.
Consider delta0 for test 4, it is the smallest of the experiment, and it could be interpreted that the led turned on 0.5 ms before the led was turned on.
Positive delta0 actual explanation
Consider the picture below. We know that the frame is timestamped when the first pixel packet arrives at the CHANSEL module. Given that the sensor has rolling shutter, then it could be possible that the led turns on in the middle of the sensor exposure, then causing that some rows capture the light while others not. If the sensor sends pixels as soon as their row exposure time finishes, then the top rows not exposed to led light would trigger the frame timestamp. Then the bottom rows in the same frame would capture led light, causing the positive delta 0. For this week experiment, we put the led much closer to the camera so that we could capture these artifacts, and found that in fact the cases where the delta0 was positive, some of the top rows had little light in comparison with bottom rows.
V4l2 path vs ISP path timing comparison
We will define an additional delta, called delta 3 that will be the time elapsed since the ATOMP_FE(for the v4l2 case) or ISPBUF_FE (for argus case) event to the moment when the frame data is available in user space. With this delta, we may be able to estimate the additional delay of the ISP path with respect to the v4l2 path.
Also we will denote with delta4 the time elapsed since the PXL_SOF event to the ATOMP_FE (for the v4l2 case) or ISPBUF_FE (for argus case) event to have an idea of the time elapsed between the moment when the first pixel packet arrives to VI to when the frame data has finished arriving to ISP or ATOMP in the v4l2 case.
We performed ten tests for each case, with a resolution of 2592x1458@30fps and averaged the results shown in the summaries below.
Summary of argus results
| Delta | Value (ms) |
|---|---|
| delta1 | 40.9 |
| delta2 | 44.2 |
| delta3 | 16.4 |
| delta4 | 24.5 |
Summary of v4l2 results
| Delta | Value (ms) |
|---|---|
| delta1 | 76.6 |
| delta2 | 90.4 |
| delta3 | 52 |
| delta4 | 24.5 |
Given these results, we looked into the code to find out why the deltas for v4l2 and in the channel.c file function tegra_channel_ring_buffer, we found the following comment: "release buffer N at N+2 frame start event". Based on this, we extracted the ATOMP_FS event for the N+2 frame for each measurement, so that we could estimate appropriately the value of the deltas. Assuming that the frame N data is ready when the ATOMP_FE occurs for frame N, but it is released to user space until ATOMP_FS event occurs for the frame N+2, then this mechanism would be adding ATOMP_FS(N+2) - ATOMP_FE(N) of delay to all deltas that use the user space timestamp (delta1, delta2 and delta3). In average the additional delay due to the queuing mechanism is 42.1ms. The adjusted deltas are shown below.
| Delta | Value (ms) |
|---|---|
| delta1 | 34.4 |
| delta2 | 48.2 |
| delta3 | 9.9 |
| delta4 | 24.5 |
Conclusion
The difference between the delta3 of both paths is approximately 6.5ms, so we think that this is the actual ISP latency. The latency added previous to the ISP processing is of 24.5ms.
Related links
- NVIDIA Jetson TX2 - Video Input Timing Concepts
- NVIDIA Jetson TX2 - VI Latency Measurement Techniques
|
RidgeRun Resources | |||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Visit our Main Website for the RidgeRun Products and Online Store. RidgeRun Engineering informations are available in RidgeRun Professional Services, RidgeRun Subscription Model and Client Engagement Process wiki pages. Please email to support@ridgerun.com for technical questions and contactus@ridgerun.com for other queries. Contact details for sponsoring the RidgeRun GStreamer projects are available in Sponsor Projects page. |  
|
