View source for NVIDIA Jetson Orin/Video Capture and Display/Cameras/IMX477 J20

<noinclude>
{{NVIDIA Jetson Orin/Head|previous=Video Capture and Display|next=GStreamer Pipelines|metakeywords=imx477, j20, mipi camera, Auvidea, Auvidea J20 board, Raspberry Pi HQ camera, IMX477 Linux driver for NVIDIA, IMX477 Linux driver}}
</noinclude>

{{DISPLAYTITLE:NVIDIA Jetson Orin - IMX477 Auvidea J20 board Camera Sensor Driver Support|noerror}}

<table>
<tr>
<td><div class="clear; float:right">__TOC__</div></td>
<td>
{{GStreamer debug}}
<td> 
<center>
{{ContactUs Button}}
</center>
</tr>
</table>

==Introduction to Sony IMX477 Linux driver==

[https://www.ridgerun.com/ RidgeRun] is actively working to enable customers with great software solutions on top of powerful hardware. For this reason, RidgeRun has worked on enabling a triple capture for the V4L2 driver for the Raspberry Pi HQ camera using an Auvidea J20 board. This wiki contains a brief introduction to the sensor and hardware that will be used to use the driver. Also please keep in mind that the driver can be ported to other Jetson platforms if needed. Please [https://www.ridgerun.com/contact contact us] if you require more information.

==Raspberry Pi HQ camera==
<br>
[[Image:rpi-hq-imx477.png|thumb|center|500px]]

===Camera Features===

* 12.3-Megapixel high-resolution Sony IMX477 sensor.
* 1.55μm × 1.55μm Pixel Size – double the pixel area of IMX219 for improved low-light performance.
* Back-illuminated sensor architecture for improved sensitivity.
* Support for different C- and CS-mount lenses.
* Integrated back-focus adjustment ring and tripod mount.

[[Image:rpi-hq-imx477-sensor.jpg|thumb|center|500px]]

===Sony IMX477 Basic Drive Mode===

The Raspberry Pi HQ camera uses the IMX477 CMOS sensor of Sony to capture. Such a sensor provides the modes included in the table below. At this point, RidgeRun is working to support one mode only but others can be added per request.

{| class="wikitable" style="margin-left: auto; margin-right: auto; border: none;
|-
! '''Drive Mode''' !! '''Number of active pixels''' !! '''Maximum frame rate<br>[frame/s]''' !! '''Output Interface''' !! '''ADC<br>[bit]'''
|-
| rowspan="2" | '''Full (4:3) (Normal)''' || rowspan="2" | 4056 x 3040<br>approx. 12.33 M pixels
| 60 || CSI-2 || 10
|-
| 40 || CSI-2 || 12
|-
| '''Full (4:3) (DOL-HDR)''' || 4056 x 3040<br>approx. 12.33 M pixels || DOL 2 frame:30<br>DOL 3 frame:15 || CSI-2 || 10
|-
| '''Full (16:9) 4K2K (Normal)''' || 4056 x 2288<br>approx. 9.28 M pixels || 79 || CSI-2 || 10
|-
| '''Full (16:9) 4K2K (DOL-HDR)''' || 4056 x 2288<br>approx. 9.28 M pixels || DOL 2 frame:39<br>DOL 3 frame:19 || CSI-2 || 10
|-
| '''Full (4:3) Binning (Normal)''' || 2028 x 1520<br>approx. 3.08 M pixels || 179 || CSI-2 || 10
|-
| '''Full (16:9) Binning 1080P (Normal)''' || 2028 x 1128<br>approx. 2.29 M pixels || 240 || CSI-2 || 10
|-
| '''Full (16:9) Binning 720P (Normal)''' || 1348 x 750<br>approx. 1.01 M pixels || 240 || CSI-2 || 10
|-
| '''Full (16:9) Scaling 1080P (Normal)''' || 2024 x 1142<br>approx. 2.31 M pixels || 79 || CSI-2 || 10
|-
| '''Full (16:9) Scalling 720P (Normal)''' || 1348 x 762<br>approx. 1.03 M pixels || 79 || CSI-2 || 10
|-
|}

===Compatibility with NVIDIA<sup>®</sup>Jetson™ Platforms===

=== R8 resistor ===

The Raspberry Pi HQ camera module requires a hardware modification in order to work with Jetson Platforms.
Jetson Platforms provide 1.8V for reset GPIO in the camera interface, but the camera module requires 3.3V.

To fix this issue a resistor labeled as R8 must be removed from the camera module. If you decide to apply this fix it is under your own risk. RidgeRun is not responsible for any damage caused to your board. The following picture shows the location of the R8 in the module.
<br>
<br>
[[Image:R8ToBeRemoved.png|thumb|center|500px]] 
<div style="text-align: center;">'''Figure 1'''. R8 resistor on RPi HQ camera module.</div>

=== Auvidea J20 adapter ===

The NVIDIA<sup>®</sup>Jetson Orin™ makes use of a Samtec connector, this driver makes use of the J20 from Auvidea
More information recopilated from RidgeRun can be found [[Getting_started_guide_for_Auvidea_J20_board|here]].

The driver will work with IMX477 connected to the J1_1, J1_2, and J1_3 ports of the J20.

==IMX477 Linux driver for NVIDIA<sup>®</sup>Jetson Orin™==

RidgeRun Engineering has developed the driver with NVIDIA Corporation and Leopard Imaging Inc. and Auvidea. as a collaborative initiative. The driver works with three cameras on the J1 ports from the J20. 
In this section, you will find the instructions to patch the JetPack sources enabling the IMX477 sensor driver for the Jetson Orin. These  instructions have been validated in the following JetPack versions:
* Jetpack 5.0.2

===Download JetPack===

The current version of the driver is supported in Jetpack 5.0.2. Porting the driver to other versions and platforms is possible.

1. Download and install the [https://developer.nvidia.com/nvidia-sdk-manager NVIDIA SDK Manager] (membership is required).

2. Log in with your NVIDIA membership credentials in the SDK Manager.

3. Select JetPack version 5.0.2 DP and Jetson Orin as shown below and press ''Continue'':
[[Image:Sdk1.png|thumb|center|650px]] 
<div style="text-align: center;">'''Figure 1'''. Required SDK Manager options.</div>

4. Accept the terms and conditions and press ''Continue''. The SDK Manager will download and install the selected components:
[[Image:imx477_sdkmanager2.png|thumb|center|650px]] 
<div style="text-align: center;">'''Figure 2'''. SDK Manager download and install software.</div>
5. Select '''Manual Setup'''. Create a  Make sure the board is in ''Recovery Mode'' and then press '''Flash''': 
[[Image:imx477_sdkmanager3.png|thumb|center|650px]] 
<div style="text-align: center;">'''Figure 3'''. Manual setup to start the flashing process.</div>

====Recovery Mode for Jetson Orin AGX====

{{Ambox
|type=notice
|small=left
|issue=To set the Orin AGX developer kit in '''Recovery Mode''' the following steps are required:
|style=width:unset;
}}

# Make sure the device is connected to the power adapter but powered off.
# Connect the host computer to the front USB Type-C connector on the device.
# Power the device 
# Press and hold the middle (Force Recovery) button.
# Press and hold the left (Power) button.
# Release both buttons.

6. When the OS has been installed successfully in the Jetson Orin AGX, you have multiple options to use the board:

# Use the username and password previously selected to connect over SSH to the corresponding IP (normally 192.168.55.1 over the USB connection). 
# Connect over the UART connection opened by attaching the micro-USB, for example, minicom: <code>sudo minicom -D /dev/ttyUSB0 -b 115200</code> and use the same username and password
# Connect over display.

<br>
{{Ambox
|type=notice
|small=left
|issue='''IMPORTANT:'''
*From now on we will refer as <code>$JETPACK_DIR</code> to the directory where the SDK Manager placed the <code>JetPack_5.0.2_DP_Linux_JETSON_AGX_ORIN_TARGETS<br>
      This directory is usually in <code>$HOME/nvidia/nvidia_sdk<code>
|style=width:unset;
}}
<br>

== Installing the Driver - JetPack sources patch==

===Download the JetPack sources===

You can obtain the JetPack 5.0.2 DP sources by doing the following:

1. Go to https://developer.nvidia.com/embedded/jetson-linux-archive and download the source files for your release.

2. Extract the .tbz2 file:
<pre>
tar -xjf public_sources.tbz2
</pre>
3. Extract the kernel source file:
<pre>
cd Linux_for_Tegra/source/public
JETSON_SOURCES=$PWD
tar –xjf kernel_src.tbz2
</pre>
This extracts the kernel source to the kernel/ subdirectory.

===Patch instructions===

====Download the patches====

RidgeRun has created a [https://github.com/RidgeRun/NVIDIA-Jetson-IMX477-RPIV3/tree/jetpack-5.0.2 repository] from where you can download the patch files.

You need to clone it and use the branch for the jetpack. 
<pre>
cd $HOME
git clone https://github.com/RidgeRun/NVIDIA-Jetson-IMX477-RPIV3.git -b jetpack-5.0.2
</pre>

Once you have downloaded the repository, copy the files to the <code>$JETSON_SOURCES</code> directory:

<pre>
KERNEL_PATCH=$(pwd)/NVIDIA-Jetson-IMX477-RPIV3/
cp $KERNEL_PATCH/patches_orin/5.0.2_agx_imx477_v0.1.patch $JETSON_SOURCES/
</pre>

====Apply the patches====
<pre>
cd $JETSON_SOURCES
git apply 5.0.2_agx_imx477_v0.1.patch
</pre>

===Kernel build instructions===

Once the sources have been patched, perform the following steps in order to build and install the driver.

====Install the Toolchain====
First, download the [https://developer.nvidia.com/embedded/jetson-linux Bootlin toolchain binaries] from the NVIDIA Jetson Linux home page. Go to the bottom of the page and click on the Bootlin Toolchain gcc option as shown in the image below:
<br>
[[File:Toolchain.png|thumb|center|650px]]

After that do:
<syntaxhighlight lang=bash>
mkdir $HOME/l4t-gcc
cd $HOME/l4t-gcc
tar xf <path_to_toolchain_archive>
</syntaxhighlight>

====Build the kernel====

* Establish the building paths and create directories:
<pre>
KERNEL_OUT=$JETSON_SOURCES/kernel_out
MODULES_OUT=$JETSON_SOURCES/modules_out
mkdir -p $KERNEL_OUT
mkdir -p $MODULES_OUT
</pre>

* Use the following command to install dependencies on Ubuntu:
<pre>
sudo apt install libncurses-dev
</pre>

* Export environment variables:
<pre>
export CROSS_COMPILE_AARCH64_PATH=$HOME/l4t-gcc
export CROSS_COMPILE_AARCH64=$HOME/l4t-gcc/bin/aarch64-buildroot-linux-gnu-
</pre>

* Compile kernel:
<pre>
cd $JETSON_SOURCES
./nvbuild.sh -o $KERNEL_OUT
</pre>

*Install the modules
<pre>
make -C $JETSON_SOURCES/kernel/kernel-5.10/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra INSTALL_MOD_PATH=$MODULES_OUT modules_install
</pre>

===Flash the Jetson===

We present two methods to install the customized kernel, one requires a complete flash, and the other is for quick updating while developing. 

==== Complete flash ====

'''Update the Linux_for_Tegra directory'''

<syntaxhighlight lang=bash>
# Copy the new compiled kernel.
cp -r $KERNEL_OUT/arch/arm64/boot/Image $JETPACK_DIR/Linux_for_Tegra/kernel/

# Copy all new compiled DTBs and DTB overlays. 
cp -r $KERNEL_OUT/arch/arm64/boot/dts/nvidia/* $JETPACK_DIR/Linux_for_Tegra/kernel/dtb/

# Update rootfs with compiled modules, which include the driver.
rsync -azPu $MODULES_OUT/lib/modules/5.10.104-tegra $JETPACK_DIR/Linux_for_Tegra/rootfs/lib/modules/

# Update the rootfs with all the binaries (to make sure the /boot/ folder is updated too).
cd $JETPACK_DIR/Linux_for_Tegra
sudo ./apply_binaries.sh

# Copy the module to the correct location.
sudo cp $KERNEL_OUT/drivers/media/i2c/nv_imx477.ko $JETPACK_DIR/Linux_for_Tegra/rootfs/lib/modules/5.10.104-tegra/kernel/drivers/media/i2c/
</syntaxhighlight>

'''Flash'''

Finally, flash the driver
<syntaxhighlight lang=bash>
sudo ./flash.sh jetson-agx-orin-devkit mmcblk0p1
</syntaxhighlight>

==== Manual installation ====

Update the kernel and DTB files used by /boot/extlinux/extlinux.conf when there is an already running board. 
This assumes that the Jetson can be reach by SSH using:

<pre>
ssh jetson
</pre>

'''Updating the kernel'''

Updating the kernel is as simple as copying into the board:

<syntaxhighlight lang=bash>
# Update Image on Linux_for_Tegra folder just to sync the flash folders
sudo cp -rfv $KERNEL_OUT/arch/arm64/boot/Image $JETPACK_DIR/Linux_for_Tegra/kernel/

# Copy the Image to the tmp folder on the board
scp $JETPACK_DIR/Linux_for_Tegra/kernel/Image jetson:/tmp/

# Copy the Image to the boot directory on the board
sudo mv /tmp/Image /boot/Image
</syntaxhighlight>
'''Updating the DTB'''

The new JetPack uses the FDT property of the /boot/extlinux/extlinux.conf file, and it sets it to a device tree located in /boot/dtb/ with a prefix of kernel_tegra. We can update this DTB with the following:
<syntaxhighlight lang=bash>
# Update on Linux_for_Tegra folder with the new DTBs and overlays
sudo cp -rfv $KERNEL_OUT/arch/arm64/boot/dts/nvidia/* $JETPACK_DIR/Linux_for_Tegra/kernel/dtb/

# Generate required prefixed DTB using flash script with no flash option
cd  $JETPACK_DIR/Linux_for_Tegra
sudo ./flash.sh -r --no-flash jetson-agx-orin-devkit mmcblk0p1

# Copy the device tree to the board
scp $JETPACK_DIR/Linux_for_Tegra/bootloader/kernel_tegra234-p3701-0000-p3737-0000.dtb jetson:/tmp/

# Update the device tree on the board
sudo mv /tmp/kernel_tegra234-p3701-0000-p3737-0000.dtb /boot/dtb/
</syntaxhighlight>

==== Updating modules ====

Inside the board run the following to update the modules folder, copying them from the host:

<syntaxhighlight lang=bash>
HOST_NAME=host
HOST_IP=192.168.55.100
cd /lib/modules/
sudo rsync -azPu $HOST_NAME@$HOST_IP:$MODULES_OUT/lib/modules/5.10.104-tegra .
</syntaxhighlight>

==ISP camera overrides installation==

Please, visit the [[JetsonTX2/HowTo%27s/NVIDIA_Jetson_ISP_Control#Custom_ISP_Configuration | Custom ISP Configuration page]] in order to learn how to use the <code>camera_overrides.isp</code> file to configure the ISP.

{{Ambox
|type=notice
|small=left
|issue='''NOTE:''' This is causing some timeout issues currently.
|style=width:unset;
}}

==Supported Features==

===Resolutions and framerates===

* 1920x1080 @ 60fps
* 3840x2160 @ 30fps

===Controls===

* Gain
* Exposure
* Framerate
* Group Hold

==Example Pipelines==

Find some example pipelines to use the IMX477 on Jetson Xavier NX below:

===Display=== 

====1920x1080====
<syntaxhighlight lang=bash>
SENSOR_ID=0 # 0 to 2 depending on cam port. 
FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode
gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvvidconv ! nvoverlaysink
</syntaxhighlight>

====3840x2160====

<syntaxhighlight lang=bash>
SENSOR_ID=0 # 0 to 2 depending on cam port. 
FRAMERATE=30 # Framerate can go from 2 to 30 for 3840x2160 mode
gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=3840,height=2160,framerate=$FRAMERATE/1" ! nvvidconv ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvoverlaysink
</syntaxhighlight>

===MP4 Recording===

====1920x1080====
<syntaxhighlight lang=bash>
SENSOR_ID=0 # 0 to 2 depending on cam port. 
FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode
gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4
</syntaxhighlight>

====3840x2160====

<syntaxhighlight lang=bash>
SENSOR_ID=0 # 0 to 2 depending on cam port. 
FRAMERATE=30 # Framerate can go from 2 to 30 for 3840x2160 mode
gst-launch-1.0 -e nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=3840,height=2160,framerate=$FRAMERATE/1" ! nvv4l2h264enc ! h264parse ! mp4mux ! filesink location=rpi_v3_imx477_cam$SENSOR_ID.mp4
</syntaxhighlight>

===JPEG snapshots===

====1920x1080====
<syntaxhighlight lang=bash>
SENSOR_ID=0 # 0 to 2 depending on cam port. 
FRAMERATE=60 # Framerate can go from 2 to 60 for 1920x1080 mode
NUMBER_OF_SNAPSHOTS=20
gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=1920,height=1080,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg
</syntaxhighlight>

====3840x2160====

<syntaxhighlight lang=bash>
SENSOR_ID=0 # 0 to 2 depending on cam port. 
FRAMERATE=30 # Framerate can go from 2 to 30 for 3840x2160 mode
NUMBER_OF_SNAPSHOTS=20
gst-launch-1.0 -e nvarguscamerasrc num-buffers=$NUMBER_OF_SNAPSHOTS sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=3840,height=2160,framerate=$FRAMERATE/1" ! nvjpegenc ! multifilesink location=%03d_rpi_v3_imx477_cam$SENSOR_ID.jpeg
</syntaxhighlight>

{{ContactUs}}

[[Category:GStreamer]][[Category:Jetson]][[Category:JetsonNano]][[Category:JetsonTX2]][[Category:NVIDIA Xavier]][[Category:JetsonXavierNX]][[Category:Sony]][[Category:Jetson V4L2 Drivers]][[Category:NVIDIA Jetson Orin]]



<noinclude>
{{NVIDIA Jetson Orin/Foot|Video Capture and Display|GStreamer Pipelines}}
</noinclude>