Raspberry Pi Camera Module 3 IMX708 Linux driver for Jetson

From RidgeRun Developer Connection
Jump to: navigation, search

Nvidia-preferred-partner-badge-rgb-for-screen.png

Error something wrong.jpg Problems running the pipelines shown on this page?
Please see our GStreamer Debugging guide for help.

RR Contact Us.png

Introduction to Sony IMX708 Linux driver

RidgeRun is actively working to enable customers with great software solutions on top of powerful hardware. For this reason, RidgeRun created the V4L2 driver for the Raspberry Pi Camera Module 3. This wiki contains a brief introduction to the sensor and hardware that will be used to develop the driver.

The instructions in this wiki are useful for Jetson Nano (A02 Model, which is the single camera model) and Jetson Orin Nano. Keep in mind that the driver can be ported to other Jetson platforms if needed. Please Contact Us if you require more information.

Raspberry Pi Camera Module 3


Rpi-hq-imx708.png
Figure 1. Raspberry Pi Camera Module 3 IMX708.

Camera Features

  • 11.9-Megapixel Sony IMX708 sensor.
  • 1.4μm × 1.4μm Pixel Size.
  • Binning support.
  • HDR Support.
  • Autofocus.
  • 5 cm minimum focus distance.

Sony IMX708 Basic Drive Mode

The Raspberry Pi Camera Module 3 uses the IMX708 CMOS sensor from Sony to capture. Such a sensor provides the modes included in the table below. At this point, RidgeRun developed one mode only, but many others can be added per request.

Officially the camera on the Raspberry Pi supports these resolutions:

  • 4608 x 2592 @ 14fps
  • 2304 × 1296 @ 56fps.
  • 2304 × 1296 @ 30fps HDR.
  • 1536 × 864 @ 120 fps.

The driver supports only one mode, if you want you can contact us to start the development of your specific mode need. We offer for the free driver, this resolution:

  • 4608 x 2592 @ 14fps (only 1 camera attached, dual streaming is not supported in the free driver version)

Compatibility with NVIDIA®Jetson™ Platforms

The Raspberry Pi Camera Module 3 is now compatible with Jetson Orin Nano and Jetson Nano thanks to RidgeRun's commitment to helping the developer community to create their projects and achieve their objectives. As you saw above, RidgeRun has developed only one mode accessible for free, but we are ready to develop many more, depending on your needs.

IMX708 Linux driver for NVIDIA®Jetson™

In this section, you will find the instructions to patch the JetPack sources enabling the IMX708 sensor driver for the Jetson Orin Nano and Jetson Nano. These instructions have been validated in the following JetPack versions:

  • Jetpack 5.1.1 (only tested in kernel version 5.10.104-tegra) for Jetson Orin Nano
  • Jetpack 4.6.4 (only tested in kernel version 4.9.253-tegra) for Jetson Nano


Download JetPack

The current version of the driver is supported in Jetpack 5.1.1 (in case you are in Jetson Orin Nano), 4.6.2 (in case you are in Jetson Nano). Porting the driver to other versions and platforms is possible.


1. Download and install the NVIDIA SDK Manager (membership is required).

2. Put your NVIDIA membership credentials in the SDK Manager main window:

SDKMCredentials.png
Figure 2. Required SDK Manager credentials.

3. Select JetPack version and Jetson [Jetson Orin Nano | Jetson Nano] as shown below and press Continue:

SDKMOptions2-orin.png
Figure 3. Required SDK Manager options.

4. Accept the terms and conditions and press Continue. The SDK Manager will download and install the selected components:

SDKMInstall.png
Figure 4. SDK Manager download and install software.

5. Be careful with the platform you will be flashing, the image shown is for Jetson Xavier, in your case Jetson Orin Nano or Nano must be shown. Select Manual Setup. Make sure the board is in Recovery Mode and then press Flash:

ManualOption.png
Figure 5. Manual setup to start the flashing process.

Recovery Mode for Jetson Orin Nano

  • To set the Jetson Orin Nano developer kit in Recovery Mode the following steps are required:
  1. Ensure the device is powered off and the power adapter is disconnected.
  2. Place a jumper across the Force Recovery Mode pins (9 and 10) on the button header [J14]. Please, see Figure 6 below.
  3. Connect your host computer to the developer kit's USB Micro-B connector.
  4. Connect the power supply to the power jack [J16]. The developer kit automatically powers on in Force Recovery Mode.
  5. Remove the jumper from the Force Recovery Mode pins.
  6. Use the lsusb command to check if the NVidia corp message is shown.


RecmodeNX.png
Figure 6. Jumper between pins 9 and 10 to get the board on Recovery Mode (top view).

Recovery Mode for Jetson Nano

  • To set the Jetson Nano in Recovery Mode, follow these steps:
  1. Ensure the device is powered off and the power adapter is disconnected.
  2. Verify that a microSD card is inserted in the Jetson Nano SD card slot.
  3. Place a jumper across the Force Recovery pins.
  4. Enable the use of the Power Jack [J15] by placing a jumper across the Power Select Header [J48] pins.
  5. Connect your host computer to the device's Micro USB connector.
  6. Connect the power supply to the power jack [J25]. The developer kit automatically powers on in Force Recovery Mode.
  7. Remove the jumper from the Force Recovery Mode pins.


6. When the OS has been installed successfully in the Orin Nano, you will be asked to enter the board IP, a username and a password, as shown below in Figure7:

OSIPOption.png
Figure 7. Required username and password for your profile in the board.

7. Don't close the window of Figure 7 and don't modify its content yet. You need to connect a screen to the board via HDMI, a keyboard/mouse via USB port, and a network cable via Ethernet port. You can remove the jumper that keeps the board in Recovery Mode and then reboot the Jetson board at this point if necessary.

8. After reboot, you will be asked to accept the terms of the license' on the board's screen. Select the checkbox and press Continue.

9. Select your language in the next screen and press Continue:

Error creating thumbnail: Unable to save thumbnail to destination
Figure 8. Language selection.

10. Select your keyboard layout and press Continue:

KeyboardNX.png
Figure 9. Keyboard layout selection.

11. Select your location and press Continue:

LocationNX.png
Figure 10. Location selection.

12. Enter a username and a password. Then, press Continue:

UsernameNX.png
Figure 11. User credentials for Ubuntu.

13. Wait for the system to apply the changes. The board will restart automatically.

14. Now you are able to access Ubuntu as a new user in the [Orin Nano | Nano]. You can find the board IP by using ifconfig from a terminal.

15. Get back to the window presented in Figure 7 (in your host PC) and enter the IP, username, and password that the [Orin Nano | Nano] has been configured with. Then press Install.

PreinstallNX.png
Figure 12. User credentials to start the software installation process.

16. The software installation process should finish successfully, as shown in Figure 13:

FinishNX.png
Figure 13. Finishing software install.



Installing the Driver - Option A: Debian Packages (Recommended)

This is the simplest way to install the IMX477 driver on Jetson platforms.

Downloading the debian packages

You can download the Debian packages according to your platform from the following link:

Kernel Debian Packages



Installing the debian packages

Jetson Orin Nano

  • Copy the Debian package to your board filesystem:
scp orin_nano.deb [jetson-username]@[jetson-ip]:/home/nvidia/
  • Install the Debian package in your board:
sudo dpkg -i --force-overwrite ./orin_nano.deb
  • Enable the dtb changes by modifying the /boot/extlinux/extlinux.conf file:
TIMEOUT 30
DEFAULT Develop

MENU TITLE L4T boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      FDT /boot/dtb/kernel_tegra234-p3767-0003-p3768-0000-a0.dtb
      INITRD /boot/initrd
      APPEND ${cbootargs} root=PARTUUID=77f45bd2-b473-4f54-8ff7-4ba0edc48af8 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 console=ttyAMA0,115200 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0 nv-auto-config
  • Reboot your board:
sudo reboot

Jetson Nano

  • Copy the Debian package to your board filesystem:
scp nano.deb [jetson-username]@[jetson-ip]:/home/nvidia/
  • Install the Debian package in your board:
sudo dpkg -i --force-overwrite ./nano.deb
  • Enable the dtb changes by modifying the /boot/extlinux/extlinux.conf file:
TIMEOUT 30
DEFAULT Develop

MENU TITLE L4T boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      FDT /boot/dtb/kernel_tegra210-p3448-0003-p3542-0000.dtb
      INITRD /boot/initrd
      APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 nv-auto-config
  • Reboot your board:
sudo reboot
  • After these steps, the driver should be working and you can run some capture tests described below.

Installing the Driver - Option B: JetPack sources patch or SCP files copy

Install dependencies

sudo apt update
sudo apt install git \
wget \
quilt \
build-essential \
bc \
libncurses5-dev libncursesw5-dev \
rsync

Get the source code from NVIDIA oficial repository

For Jetson Orin Nano:

cd
wget https://developer.nvidia.com/downloads/embedded/l4t/r35_release_v3.1/sources/public_sources.tbz2/ -O public_sources.tbz2
tar -xjf public_sources.tbz2 Linux_for_Tegra/source/public/kernel_src.tbz2 --strip-components 3

For Jetson Nano:

cd
wget https://developer.nvidia.com/embedded/l4t/r32_release_v7.2/sources/t210/public_sources.tbz2 -O public_sources.tbz2
tar -xjf public_sources.tbz2 Linux_for_Tegra/source/public/kernel_src.tbz2 --strip-components 3

Export the Jetpack directory as the environment variable DEVDIR:

(In our case the nvidia directory is in the HOME directory, which in our case is /home/nvidia/ Remember that this directory is created by the sdk manager when you downloaded the sources and flashed your Jetson board).

For Jetson Orin Nano:

export DEVDIR=/home/nvidia/nvidia_sdk/JetPack_5.1.1_Linux_JETSON_ORIN_NANO_TARGETS/Linux_for_Tegra

For Jetson Nano:

export DEVDIR=/home/nvidia/nvidia_sdk/JetPack_4.6.4_Linux_JETSON_NANO_TARGETS/Linux_for_Tegra

And now create the sources directory and extract the kernel sources:

mkdir $DEVDIR/sources
tar -xjf kernel_src.tbz2 -C $DEVDIR/sources

Build and install the driver

Jetson Orin Nano

Get the driver patches

Download contents provided in RidgeRun GitHub by running the command:

git clone git@github.com:RidgeRun/NVIDIA-Jetson-IMX708-RPIV3.git
cd NVIDIA-Jetson-IMX708-RPIV3/patches_orin_nano

Then you will need the patch in the directory $DEVDIR/sources by running:

cp -r patches/ $DEVDIR/sources

Next, you can then apply the patch and go back to the Linux_for_Tegra directory.

cd $DEVDIR/sources
quilt push -a
cd ..
Set up the toolchain

You will need to download the toolchain to compile the sources:

mkdir $HOME/l4t-gcc
cd $HOME/l4t-gcc
wget -O toolchain.tar.xz https://developer.nvidia.com/embedded/jetson-linux/bootlin-toolchain-gcc-93
sudo tar -xf toolchain.tar.xz
Build

Create output directories:

KERNEL_OUT=$DEVDIR/sources/kernel_out
MODULES_OUT=$DEVDIR/sources/modules_out
mkdir -p $KERNEL_OUT
mkdir -p $MODULES_OUT
export CROSS_COMPILE_AARCH64=$HOME/l4t-gcc/bin/aarch64-buildroot-linux-gnu-

Compile the Kernel:

cd $DEVDIR/sources
make -C kernel/kernel-5.10/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra CROSS_COMPILE=${CROSS_COMPILE_AARCH64} tegra_defconfig

make -C kernel/kernel-5.10/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra CROSS_COMPILE=${CROSS_COMPILE_AARCH64} menuconfig
  • Be sure that IMX708 is selected and deselect any other IMX driver:
Device Drivers  --->
  <*> Multimedia support  --->
      Media ancillary drivers  --->
          NVIDIA overlay Encoders, decoders, sensors and other helper chips  --->
              <*> IMX708 camera sensor support
make -C kernel/kernel-5.10/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra CROSS_COMPILE=${CROSS_COMPILE_AARCH64} Image

make -C kernel/kernel-5.10/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra CROSS_COMPILE=${CROSS_COMPILE_AARCH64} dtbs

make -C kernel/kernel-5.10/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra CROSS_COMPILE=${CROSS_COMPILE_AARCH64} modules

make -C kernel/kernel-5.10/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra INSTALL_MOD_PATH=$MODULES_OUT modules_install
Installation options
Install using scp command

Get the Jetson IP address going to your board, connect it by ethernet, run the command shown below and copy the ip address showed in eth0 (inet):

ifconfig

In your computer, copy the files via SSH by running

scp $KERNEL_OUT/arch/arm64/boot/Image nvidia@orin_ip_address:/tmp
scp $KERNEL_OUT/arch/arm64/boot/dts/nvidia/tegra234-p3767-0003-p3768-0000-a0.dtb nvidia@orin_ip_address:/tmp

Open a terminal on your Jetson and run:

sudo cp /tmp/Image /boot/
sudo cp /tmp/tegra234-p3767-0003-p3768-0000-a0.dtb /boot/dtb/kernel_tegra234-p3767-0003-p3768-0000-a0.dtb

Modify the extlinux.conf to accept the modified dtb by copying this code in the extlinux.conf file:

sudo nano /boot/extlinux/extlinux.conf
TIMEOUT 30
DEFAULT primary

MENU TITLE L4T boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      FDT /boot/dtb/kernel_tegra234-p3767-0003-p3768-0000-a0.dtb
      INITRD /boot/initrd
      APPEND ${cbootargs} root=PARTUUID=77f45bd2-b473-4f54-8ff7-4ba0edc48af8 rw rootwait rootfstype=ext4 mminit_loglevel=4 console=ttyTCU0,115200 console=ttyAMA0,115200 firmware_class.path=/etc/firmware fbcon=map:0 net.ifnames=0 nv-auto-config 

# When testing a custom kernel, it is recommended that you create a backup of
# the original kernel and add a new entry to this file so that the device can
# fallback to the original kernel. To do this:
#
# 1, Make a backup of the original kernel
#      sudo cp /boot/Image /boot/Image.backup
#
# 2, Copy your custom kernel into /boot/Image
#
# 3, Uncomment below menu setting lines for the original kernel
#
# 4, Reboot

# LABEL backup
#    MENU LABEL backup kernel
#    LINUX /boot/Image.backup
#    FDT /boot/dtb/kernel_tegra234-p3767-0003-p3768-0000-a0.dtb
#    INITRD /boot/initrd
#    APPEND ${cbootargs}

And now you can reboot

Install using Jetpack (not working yet)

Copy the built files to the locations required for flashing:

# Copy the new compiled kernel.
cp -r $KERNEL_OUT/arch/arm64/boot/Image $DEVDIR/kernel/

# Copy all new compiled DTBs and DTB overlays. 
cp -r $KERNEL_OUT/arch/arm64/boot/dts/* $DEVDIR/kernel/dtb/
# Update rootfs with compiled modules, which include the driver.
sudo rsync -azPu $MODULES_OUT/lib/modules/5.10.104-tegra $DEVDIR/rootfs/lib/modules/
# Update the rootfs with all the binaries (to make sure the /boot/ folder is updated too).
cd $DEVDIR
sudo ./apply_binaries.sh

Make sure to put the board in recovery mode and plug it into your computer. Then run the command that you usually use to flash your hardware, for instance:

sudo ./flash.sh jetson-orin-nano-devkit mmcblk0p1


When this command finishes, you can connect a display and keyboard to set up the user/password and continue the configuration process.

Jetson Nano

Get the driver patches

Download contents provided in RidgeRun GitHub by running the command:

git clone git@github.com:RidgeRun/NVIDIA-Jetson-IMX708-RPIV3.git
cd NVIDIA-Jetson-IMX708-RPIV3/patches_nano

Then you will need the patch in the directory $DEVDIR/sources by running:

cp -r patches/ $DEVDIR/sources

Next, you can then apply the patch and go back to the Linux_for_Tegra directory.

cd $DEVDIR/sources
quilt push -a
cd ..
Set up the toolchain

You will need to download the toolchain to compile the sources:

mkdir $HOME/l4t-gcc
cd $HOME/l4t-gcc
wget -O toolchain.tar.xz https://developer.nvidia.com/embedded/dlc/l4t-gcc-7-3-1-toolchain-64-bit
sudo tar -xf toolchain.tar.xz
Build

Create output directories:

KERNEL_OUT=$DEVDIR/sources/kernel_out
MODULES_OUT=$DEVDIR/sources/modules_out
mkdir -p $KERNEL_OUT
mkdir -p $MODULES_OUT

Export environment variables for Jetson Nano

export CROSS_COMPILE_AARCH64=$HOME/l4t-gcc/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-

Compile the kernel:

cd $DEVDIR/sources
make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra CROSS_COMPILE=${CROSS_COMPILE_AARCH64} tegra_defconfig

make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra CROSS_COMPILE=${CROSS_COMPILE_AARCH64} menuconfig
  • Be sure that IMX708 is selected and deselect any other IMX driver:
Device Drivers  --->
  <*> Multimedia support  --->
      NVIDIA overlay Encoders, decoders, sensors and other helper chips  --->
          <*> IMX708 camera sensor support

make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra CROSS_COMPILE=${CROSS_COMPILE_AARCH64} Image

make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra CROSS_COMPILE=${CROSS_COMPILE_AARCH64} dtbs

make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra CROSS_COMPILE=${CROSS_COMPILE_AARCH64} modules

make -C kernel/kernel-4.9/ ARCH=arm64 O=$KERNEL_OUT LOCALVERSION=-tegra INSTALL_MOD_PATH=$MODULES_OUT modules_install
Installation options
Install using scp command

Get the Jetson IP address going to your board, connect it by ethernet, run the command shown below and copy the ip address showed in eth0 (inet):

ifconfig

In your computer, copy the files via SSH by running

scp $KERNEL_OUT/arch/arm64/boot/Image nvidia@nano_ip_address:/tmp
scp $KERNEL_OUT/arch/arm64/boot/dts/tegra210-p3448-0003-p3542-0000.dtb nvidia@nano_ip_address:/tmp

Open a terminal on your Jetson and run:

sudo cp /tmp/Image /boot/
sudo cp /tmp/tegra210-p3448-0003-p3542-0000.dtb /boot/dtb/kernel_tegra210-p3448-0003-p3542-0000.dtb

Modify the extlinux.conf to accept the modified dtb by copying this code in the extlinux.conf file:

sudo nano /boot/extlinux/extlinux.conf
TIMEOUT 30
DEFAULT Develop

MENU TITLE L4T boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image
      FDT /boot/dtb/kernel_tegra210-p3448-0003-p3542-0000.dtb
      INITRD /boot/initrd
      APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyS0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 nv-auto-config

# When testing a custom kernel, it is recommended that you create a backup of
# the original kernel and add a new entry to this file so that the device can
# fallback to the original kernel. To do this:
#
# 1, Make a backup of the original kernel
#      sudo cp /boot/Image /boot/Image.backup
#
# 2, Copy your custom kernel into /boot/Image
#
# 3, Uncomment below menu setting lines for the original kernel
#
# 4, Reboot

# LABEL backup
#    MENU LABEL backup kernel
#    LINUX /boot/Image.backup
#    INITRD /boot/initrd
#    APPEND ${cbootargs}

And now you can reboot

Install using Jetpack

Copy the built files to the locations required for flashing:

# Copy the new compiled kernel.
cp -r $KERNEL_OUT/arch/arm64/boot/Image $DEVDIR/kernel/

# Copy all new compiled DTBs and DTB overlays. 
cp -r $KERNEL_OUT/arch/arm64/boot/dts/* $DEVDIR/kernel/dtb/
# Update rootfs with compiled modules, which include the driver.
sudo rsync -azPu $MODULES_OUT/lib/modules/4.9.253-tegra $DEVDIR/rootfs/lib/modules/
# Update the rootfs with all the binaries (to make sure the /boot/ folder is updated too).
cd $DEVDIR
sudo ./apply_binaries.sh

Make sure to put the board in recovery mode and plug it into your computer. Then run the command that you usually use to flash your hardware, for instance:

sudo ./flash.sh jetson-nano-devkit mmcblk0p1

When this command finishes, you can connect a display and keyboard to set up the user/password and continue the configuration process.

Supported Features

Resolutions and framerates

  • 4608x2592 @ 14fps

Controls

  • Gain
  • Exposure
  • Framerate
  • Group Hold

Example Pipelines

Find some example pipelines to use the IMX708 on Jetson Orin Nano and Jetson Nano below:

Display

4608x2592

SENSOR_ID=0 
FRAMERATE=14 
gst-launch-1.0 nvarguscamerasrc sensor-id=$SENSOR_ID ! "video/x-raw(memory:NVMM),width=4032,height=3040,framerate=$FRAMERATE/1" ! queue ! nv3dsink

MP4 Recording

4608x2592

SENSOR_ID=0 
FRAMERATE=14
gst-launch-1.0 nvarguscamerasrc sensor_id=$SENSOR_ID -e ! 'video/x-raw(memory:NVMM),width=4608,height=2592,framerate=$FRAMERATE/1, format=NV12' ! nvvidconv ! "video/x-raw,width=1920,height=1080" ! x264enc ! qtmux ! filesink location=RidgeRun_out.mp4

JPEG snapshots

4608x2592

SENSOR_ID=0 
FRAMERATE=14
gst-launch-1.0 nvarguscamerasrc num-buffers=1 sensor_id=0 ! 'video/x-raw(memory:NVMM), width=4608, height=2592, framerate=14/1, format=NV12' ! nvjpegenc ! filesink location=RidgeRun_test.jpg


RidgeRun Resources

Quick Start Client Engagement Process RidgeRun Blog Homepage
Technical and Sales Support RidgeRun Online Store RidgeRun Videos Contact Us

OOjs UI icon message-progressive.svg Contact Us

Visit our Main Website for the RidgeRun Products and Online Store. RidgeRun Engineering informations are available in RidgeRun Professional Services, RidgeRun Subscription Model and Client Engagement Process wiki pages. Please email to support@ridgerun.com for technical questions and contactus@ridgerun.com for other queries. Contact details for sponsoring the RidgeRun GStreamer projects are available in Sponsor Projects page. Ridgerun-logo.svg
RR Contact Us.png