Jetson Camera Drivers - Fixed Sensor ID for Argus Capture with GStreamer
Contents
- 1 Overview
- 2 Introduction
- 3 How to Fix
- 3.1 1. Capture using Directly V4L2 using a Stable Camera System
- 3.2 2. Capture using Directly V4L2 using an Unstable Camera System
- 3.3 3. Capture using nvarguscamerasrc with no V4L2 Controls using an Unstable Camera System
- 3.4 4. Capture using nvarguscamerasrc with V4L2 Controls using an Unstable Camera System
Overview
This wiki intends to show how to index the CSI camera devices used by LibArgus (nvarguscamerasrc) and V4L2 (v4l2src) for ensuring that a given index will always refer the same physical camera.
Introduction
First of all, we need to know that V4L2 enumeration/indexing is different than the one from LibArgus. By default V4L2 enumeration is done according to the driver loading order, whereas LibArgus enumeration is based on the modules definition order in the tegra-camera-platform section of the device-tree.
Issues
If both indexing mechanisms were the same, the ideal case would be the following:
Cam # -> V4L2 Domain | Argus Domain
------------------------------------
Cam 0 -> /dev/video0 | sensor-id=0
Cam 1 -> /dev/video1 | sensor-id=1
Cam 2 -> /dev/video2 | sensor-id=2
Cam 3 -> /dev/video3 | sensor-id=3
Where the indexing matches for both domains and the enumeration will be persistent across reboots or camera disconnections.
However, when using systems with multiple cameras we have identified the following issues:
1. Different V4L2 Enumeration for the same Camera between Reboots
This usually happens when there are multiple camera drivers trying to load at the same time (not serialized). The behavior is the following:
Cam # -> V4L2 Domain | Argus Domain
------------------------------------
Cam 0 -> /dev/video2 | sensor-id=0
Cam 1 -> /dev/video3 | sensor-id=1
Cam 2 -> /dev/video1 | sensor-id=2
Cam 3 -> /dev/video0 | sensor-id=3
---- Reboot ----
Cam # -> V4L2 Domain | Argus Domain
------------------------------------
Cam 0 -> /dev/video1 | sensor-id=0
Cam 1 -> /dev/video2 | sensor-id=1
Cam 2 -> /dev/video0 | sensor-id=2
Cam 3 -> /dev/video3 | sensor-id=0
This kind of issue affects V4L2 capture (v4l2src) and any operation that relies on a V4L2 device such as camera controls. However, it doesn't affect the Argus domain because all the cameras are still connected and the drivers loaded correctly (even if not in the appropriate order), so the sensor-id enumeration is not affected.
2. V4L2 /dev/videoX and LibArgus Enumeration Shifting when a Camera is Missing
This happens when a camera is physically disconnected or the driver failed to load for any reason. The behavior is the following:
Cam # -> V4L2 Domain | Argus Domain
------------------------------------
Cam 0 -> /dev/video0 | sensor-id=0
Cam 1 -> /dev/video1 | sensor-id=1
Cam 2 -> /dev/video2 | sensor-id=2
Cam 3 -> /dev/video3 | sensor-id=3
---- Reboot and disconnect Cam 0 ----
Cam # -> V4L2 Domain | Argus Domain
------------------------------------
Cam 0 -> ----------- | ----------
Cam 1 -> /dev/video0 | sensor-id=0
Cam 2 -> /dev/video1 | sensor-id=1
Cam 3 -> /dev/video2 | sensor-id=2
This kind of issue affects V4L2 capture (v4l2src) and any operation that relies on a V4L2 device such as camera controls. It also affects the Argus domain because not all the drivers loaded correctly.
3. V4L2 Fixed Devices but LibArgus Shifting when a Camera is Missing
This issue seems to be similar than the one before but we treat it as a different one because it demonstrates an extra step required for getting to the ideal behavior. Let's say we already fixated the V4L2 enumeration but we still disconnect a camera, the behavior would be the following:
Cam # -> V4L2 Domain | Argus Domain
------------------------------------
Cam 0 -> /dev/video0 | sensor-id=0
Cam 1 -> /dev/video1 | sensor-id=1
Cam 2 -> /dev/video2 | sensor-id=2
Cam 3 -> /dev/video3 | sensor-id=3
---- Reboot and disconnect Cam 1 ----
Cam # -> V4L2 Domain | Argus Domain
------------------------------------
Cam 0 -> /dev/video0 | sensor-id=0
Cam 1 -> ----------- | -----------
Cam 2 -> /dev/video2 | sensor-id=1
Cam 3 -> /dev/video3 | sensor-id=2
As we can see the V4L2 domain seems to be good, but the Argus domain still shifted, and this is because as we mentioned before, their indexing works different for each of them, so fixing one won't necessarily fix the other. It depends on the customer use case if they need both domains to maintain always the relationship between an index and a given camera, so in the next section you will see how to fix each case.
How to Fix
Here are some use cases where you might need to apply the fix for the previous issues if consistency between indices and cameras is required.
1. Capture using Directly V4L2 using a Stable Camera System
Here we call a stable camera system a system where cameras are not expected to disconnect, but still their drivers might still load in different order so it is vulnerable to Issue 1.
For this, you need to patch the kernel by adapting these instructions to your Jetpack version. They have already been tested in Jetpack 4.5.1 and 4.6.
2. Capture using Directly V4L2 using an Unstable Camera System
Similarly, an unstable camera system is a system where cameras are expected to disconnect due to hardware motion or human manipulation, therefore, they are vulnerable to Issue 2.
For this, you can also use the previous solution which is to patch the kernel by adapting these instructions to the corresponding Jetpack version. If Issue 1 was also affecting, that solution would get rid of both of them.
3. Capture using nvarguscamerasrc with no V4L2 Controls using an Unstable Camera System
In this case we are not using V4L2 directly so the enumeration it has doesn't matter, but we know that since cameras might disconnect we are still vulnerable to Issue 2.
In order to fix this we need to change the behavior of the sensor-id property of nvarguscamerasrc element. The current behavior uses that property to index an array of camera devices that is made of all the available cameras, which leads to different behavior for the same number if one of them is not available at boot time.
The proposed change makes use of the GUID (Global Unique Identifier) of the camera, available through the LibArgus API. The GUID is assigned based on the tegra-camera-platform section of the device tree, so it won't change across reboots. For example, let's look at the following extract of a device tree:
tegra-camera-platform {
compatible = "nvidia, tegra-camera-platform";
/*...*/
modules {
/* GUID = 0 */
module0 {
status = "okay";
badge = "nvidia_0_imx219";
position = "0"
/*...*/
drivernode0 {
status = "okay";
pcl_id = "v4l2_sensor";
devname = "imx219 9-0060";
proc-device-tree = "/proc/device-tree/cam_i2cmux/i2c@0/ds90ub960@30/link@0/ub953@40/imx219@60";
};
};
/* GUID = 1 */
module1 {
status = "okay";
badge = "nvidia_1_imx219";
position = "1"
/*...*/
drivernode0 {
status = "okay";
pcl_id = "v4l2_sensor";
devname = "imx219 9-0061";
proc-device-tree = "/proc/device-tree/cam_i2cmux/i2c@0/ds90ub960@30/link@1/ub953@41/imx219@61";
};
};
/* GUID = 2 */
module2 {
status = "okay";
badge = "nvidia_2_imx219";
position = "2"
/*...*/
drivernode0 {
status = "okay";
pcl_id = "v4l2_sensor";
devname = "imx219 9-0062";
proc-device-tree = "/proc/device-tree/cam_i2cmux/i2c@0/ds90ub960@30/link@2/ub953@42/imx219@62";
};
};
/* GUID = 3 */
module3 {
status = "okay";
badge = "nvidia_3_imx219";
position = "3"
/*...*/
drivernode0 {
status = "okay";
pcl_id = "v4l2_sensor";
devname = "imx219 9-0063";
proc-device-tree = "/proc/device-tree/cam_i2cmux/i2c@0/ds90ub960@30/link@3/ub953@43/imx219@63";
};
};
};
Each moduleX entry will have its own GUID that we can use to identify from the others. So with this approach the sensor-id property will look among all the available devices if one of them has the same GUID as the sensor-id, in which case it will be considered a match. Otherwise it will throw an error to let the user know that the camera that is requesting is not connected or did not load correctly.
Note: The GUID assignment from the device-tree might change if the position field is described with strings instead of numbers. For example, we've seen the following GUID values for these strings:
| String | GUID |
|---|---|
| bottomleft | 0 |
| centerleft | 1 |
| centerright | 2 |
| topleft | 3 |
| bottomright | 4 |
| topright | 5 |
In order to apply this solution you might need to adapt this patch to the gstnvarguscamerasrc code that is available with the L4T public sources and then recompile and install the modified version:
diff --git a/gstnvarguscamerasrc.cpp b/gstnvarguscamerasrc.cpp
index 31d5c4b..e53a227 100644
--- a/gstnvarguscamerasrc.cpp
+++ b/gstnvarguscamerasrc.cpp
@@ -636,10 +636,15 @@ bool StreamConsumer::threadShutdown(GstNvArgusCameraSrc* src) { return true; }
static bool execute(int32_t cameraIndex, int32_t cameraMode, const Size2D<uint32_t>& streamSize, int32_t secToRun,
GstNvArgusCameraSrc* src) {
gfloat frameRate = 0, duration = 0;
+ uint64_t guid = 0;
+ uint64_t val = 0;
uint32_t index = 0;
+ uint32_t i = 0, j = 0;
uint32_t err = 0;
gint found = 0;
gint best_match = -1;
+ gint uuidNodeNumBytes = 6;
+ gint argusIndex = -1;
Range<float> sensorModeAnalogGainRange;
Range<float> ispDigitalGainRange;
Range<uint64_t> limitExposureTimeRange;
@@ -650,6 +655,7 @@ static bool execute(int32_t cameraIndex, int32_t cameraMode, const Size2D<uint32
ISensorMode* iSensorMode_ptr = NULL;
IAutoControlSettings* iAutoControlSettings = NULL;
ISourceSettings* requestSourceSettings = NULL;
+ UUID uuid = { 0 };
// Domain for errors to bus
static GQuark domain = g_quark_from_static_string("NvArgusCameraSrc");
@@ -687,17 +693,46 @@ static bool execute(int32_t cameraIndex, int32_t cameraMode, const Size2D<uint32
ORIGINATE_ERROR("No cameras available");
}
- if (static_cast<uint32_t>(cameraIndex) >= cameraDevices.size()) {
+ /* Look for the camera device that has the given sensor-id as GUID.
+
+ The GUID is assigned based on the tegra-camera-platform modules definition
+ order and it's not affected if a camera is disconnected, which was the
+ case for the previous sensor-id implementation. */
+
+ for (i = 0; i < cameraDevices.size(); i++) {
+ camProps = interface_cast<ICameraProperties>(cameraDevices[i]);
+ if (!camProps) {
+ ORIGINATE_ERROR("Failed to get camera properties for camera %d", i);
+ }
+
+ uuid = camProps->getUUID();
+ /* In UUID, clock_seq contains low 16 bits of GUID and node[6] contains
+ the high 48 bits of GUID. */
+ guid = 0;
+ /* Fill the first 48 high bits */
+ for (j = 0; j < uuidNodeNumBytes; j++) {
+ val = uuid.node[j];
+ guid = guid | (val << (j * 8 + 16));
+ }
+ /* Fill the last 16 bits */
+ guid = guid | uuid.clock_seq;
+
+ if (guid == cameraIndex) {
+ argusIndex = i;
+ break;
+ }
+ }
+
+ if (argusIndex == -1) {
error = g_error_new(domain, ERROR_INVALID_CAMERA_INDEX, "Invalid camera index");
GstMessage* message = gst_message_new_error(GST_OBJECT(src), error, "Argus Error Status");
gst_element_post_message(GST_ELEMENT_CAST(src), message);
- ORIGINATE_ERROR("Invalid camera device specified %d specified, %d max index", cameraIndex,
- static_cast<int32_t>(cameraDevices.size()) - 1);
+ ORIGINATE_ERROR("No camera device found for index %d.", cameraIndex);
}
// Create the capture session using the specified device.
UniqueObj<CaptureSession> captureSession = UniqueObj<CaptureSession>(
- iCameraProvider->createCaptureSession(cameraDevices[cameraIndex]));
+ iCameraProvider->createCaptureSession(cameraDevices[argusIndex]));
ICaptureSession* iCaptureSession = interface_cast<ICaptureSession>(captureSession);
if (!iCaptureSession) {
error = g_error_new(domain, ERROR_CAPTURE_SESSION, "Failed to create CaptureSession");
@@ -762,7 +797,7 @@ static bool execute(int32_t cameraIndex, int32_t cameraMode, const Size2D<uint32
if (!iAutoControlSettings) ORIGINATE_ERROR("Failed to get AutoControlSettings");
src->iAutoControlSettings_ptr = iAutoControlSettings;
- camProps = interface_cast<ICameraProperties>(cameraDevices[cameraIndex]);
+ camProps = interface_cast<ICameraProperties>(cameraDevices[argusIndex]);
if (!camProps) ORIGINATE_ERROR("Failed to create camera properties");
camProps->getAllSensorModes(&modes);
@@ -842,11 +877,12 @@ static bool execute(int32_t cameraIndex, int32_t cameraMode, const Size2D<uint32
GST_ARGUS_PRINT(
"Running with following settings:\n"
" Camera index = %d \n"
+ " Argus Camera index = %d \n"
" Camera mode = %d \n"
" Output Stream W = %d H = %d \n"
" seconds to Run = %d \n"
" Frame Rate = %f \n",
- cameraIndex, cameraMode, iSensorMode_ptr->getResolution().width(), iSensorMode_ptr->getResolution().height(),
+ cameraIndex, argusIndex, cameraMode, iSensorMode_ptr->getResolution().width(), iSensorMode_ptr->getResolution().height(),
secToRun, (1e9 / (iSensorMode_ptr->getFrameDurationRange().min())));
IDenoiseSettings* denoiseSettings = interface_cast<IDenoiseSettings>(src->request);
@@ -1815,7 +1851,8 @@ static void gst_nv_argus_camera_src_class_init(GstNvArgusCameraSrcClass* klass)
g_object_class_install_property(
gobject_class, PROP_SENSOR_ID,
- g_param_spec_int("sensor-id", "Sensor ID", "Set the id of camera sensor to use. Default 0.", 0, G_MAXUINT8,
+ g_param_spec_int("sensor-id", "Sensor ID", "Set the id of camera sensor to use according to the "
+ "tegra-camera-platform modules definition order in the device tree. Default 0.", 0, G_MAXUINT8,
NVARGUSCAM_DEFAULT_SENSOR_ID, (GParamFlags)(G_PARAM_READWRITE | G_PARAM_STATIC_STRINGS)));
g_object_class_install_property(
4. Capture using nvarguscamerasrc with V4L2 Controls using an Unstable Camera System
For this case we care about both LibArgus and V4L2 enumeration because we want to make sure we are setting the control (V4L2) for the camera we are capturing (LibArgus) from.
This means we need to apply the solutions for both case 2 and case 3.
Note that these 2 solutions are independent and they don't attach V4L2 to LibArgus. The consistency between both solutions will depend on your implementation of both of them in the device tree. For example, this is an extract of a device tree that would ensure the ideal case depicted in the introduction. Notice, how the order of declaration is consistent with the devnode names.
vi@15c10000 {
compatible = "nvidia,tegra194-vi";
/* ... */
ports {
/* ... */
port@0 {
/* ... */
endpoint {
devnode = "video0";
/* ... */
};
};
port@1 {
/* ... */
endpoint {
devnode = "video1";
/* ... */
};
};
port@2 {
/* ... */
endpoint {
devnode = "video2";
/* ... */
};
};
port@3 {
/* ... */
endpoint {
devnode = "video3";
/* ... */
};
};
};
};
/* ... */
i2c@0 {
/* ... */
ds90ub960@30 {
/* ... */
link@0 {
/* ... */
ub953@40 {
/* ... */
imx219@60 {
compatible = "nvidia,imx219";
reg = <0x60>;
devnode = "video0";
/* ... */
ports {
/* ... */
port@0 {
/* ... */
endpoint {
/* ... */
};
};
};
};
};
};
/* ... */
link@1 {
/* ... */
ub953@41 {
/* ... */
imx219@61 {
compatible = "nvidia,imx219";
reg = <0x61>;
devnode = "video1";
/* ... */
ports {
/* ... */
port@0 {
/* ... */
endpoint {
/* ... */
};
};
};
};
};
};
/* ... */
link@2 {
/* ... */
ub953@42 {
/* ... */
imx219@62 {
compatible = "nvidia,imx219";
reg = <0x62>;
devnode = "video2";
/* ... */
ports {
/* ... */
port@0 {
/* ... */
endpoint {
/* ... */
};
};
};
};
};
};
/* ... */
link@3 {
/* ... */
ub953@43 {
/* ... */
imx219@63 {
compatible = "nvidia,imx219";
reg = <0x63>;
devnode = "video3";
/* ... */
ports {
/* ... */
port@0 {
/* ... */
endpoint {
/* ... */
};
};
};
};
};
};
};
/*...*/
tegra-camera-platform {
compatible = "nvidia, tegra-camera-platform";
/*...*/
modules {
/* GUID = 0 */
module0 {
status = "okay";
badge = "nvidia_0_imx219";
/*...*/
drivernode0 {
status = "okay";
pcl_id = "v4l2_sensor";
devname = "imx219 9-0060";
proc-device-tree = "/proc/device-tree/cam_i2cmux/i2c@0/ds90ub960@30/link@0/ub953@40/imx219@60";
};
};
/* GUID = 1 */
module1 {
status = "okay";
badge = "nvidia_1_imx219";
/*...*/
drivernode0 {
status = "okay";
pcl_id = "v4l2_sensor";
devname = "imx219 9-0061";
proc-device-tree = "/proc/device-tree/cam_i2cmux/i2c@0/ds90ub960@30/link@1/ub953@41/imx219@61";
};
};
/* GUID = 2 */
module2 {
status = "okay";
badge = "nvidia_2_imx219";
/*...*/
drivernode0 {
status = "okay";
pcl_id = "v4l2_sensor";
devname = "imx219 9-0062";
proc-device-tree = "/proc/device-tree/cam_i2cmux/i2c@0/ds90ub960@30/link@2/ub953@42/imx219@62";
};
};
/* GUID = 3 */
module3 {
status = "okay";
badge = "nvidia_3_imx219";
/*...*/
drivernode0 {
status = "okay";
pcl_id = "v4l2_sensor";
devname = "imx219 9-0063";
proc-device-tree = "/proc/device-tree/cam_i2cmux/i2c@0/ds90ub960@30/link@3/ub953@43/imx219@63";
};
};
};
