Qualcomm Robotics RB5/RB6 - Building your own camera sensor driver

Qualcomm Robotics RB5/RB6
Introduction
SoM Overview Carrier Boards Qualcomm Robotics RB5 developer kit Qualcomm Robotics RB6 platform
Software Development Support
Boot Devices Thundercomm SDK Downloading SDK Installing SDK Generating Image and flashing Linaro Getting Started Flashing Bootloader Flashing Boot Image and Rootfs Special / Debug Modes Setting up board in EDL Mode Setting up board in Fastboot Mode
Development in the Board
Getting into the Board Using ADB Using Serial Console Using SSH Device Setup WiFi setup HDMI setup Update Ubuntu Packages
Build from source
Base Image Customize Image
Capture Subsystem
Hardware Capture Components Software Capture Components Building your own camera sensor driver
Image Processing and Acceleration Software
GPU Profiling Fast CV Installing Hexagon SDK Installing Fast CV Building your own Fast CV application
GStreamer Pipelines
Capture and Display Video Encoding Video Decoding
AI Acceleration
Neural Processing SDK Downloading Requirements Setup SDK Environment Install qtimlesnpe Example Project TensorFlow Example pipeline
Contact Us

The Qualcomm Robotics RB5 Developer Kit include two cameras ready to use for your applications, however, in some cases, you want to use your own camera sensor. In this section you will learn about the files and the building process needed to create a camera sensor driver for the Qualcomm Robotics RB5/RB6 platform. This procedure involves building several images with yocto in a Host PC, that will later be flashed in to the board.

Note

The building process of the images in yocto is very resource intensive. For this we used a host PC with a 14-core 12th Gen Intel(R) Core(TM) i7-12700H processor.

Downloading the LinuxEmbedded source code

Note

To be able to get the repository you will need to have a Thundercomm account, if you don't already have an account click the following link to create one. Note that Thundercomm will take some time to validate your account.

Note

These instructions are based on the document [tc-ble-2412_TurboX%20C5165%20DK_LE%20Software%20Release%20Notes_(for%20Pre-FC.r001001)_20210415_V1.2.pdf Thundercomm Turbox™ C5165 Development Kit Linux Enablement Software Release Notes].

The notes are from 2021 and in words of a Thundercomm engineer:

The latest version of LE was released on January 16, 2023. Currently, the release notes of RB5 series has not been released on the official website of Thundercomm, so you need to contact your sales to obtain it separately.

Environment setup

The image is built using a Ubuntu 18.04 container based on an Image provided by Thundercomm.

For this build at least 200GB of disk space is required.

1. Make sure you have docker installed.

2. Download Thundercomm Ubuntu 18.04 image from: Ubuntu-18.04

3. Load the downloaded image with:

docker load < ubuntu-18.04.tar

4. Make sure to authenticate with:

Note

with <User Email> and <password> being your Thundercomm login information.

echo "machine partner.thundercomm.com login <User Email> password <password>" >> ~/.netrc

5. Install repo

sudo apt-get install repo

Get SDK

Once your system is fully configured you can proceed with your build.

1. Create a folder for your SDK:

mkdir SDK_SRC_ROOT
cd SDK_SRC_ROOT

2. Initialize repo

Note

If you file to init the repository make sure your credentials are correct.

repo init -u https://partner.thundercomm.com/sdkgit/a/manifest -b SDK.turbox-rb5-le1.0-dev -m turbox-rb5-le1.0-dev.Pre-FC.r001001.xml

3. Sync repository

repo sync -c -j4 --no-tags

4. Create the docker container for the build.

Note

SDK_SRC_ROOT is the full path in your host where the SDK lives (the folder created in step 1).

Note

ubuntu:18.04 is the docker image you loaded in the environment setup. Be careful if you already had an Ubuntu 18.04 docker image installed when you load Thundercomm's version as there will be a conflict with the names.

sudo docker run -v <SDK_SRC_ROOT>:/home/user/host/ -d --name thundercomm-sdk -p36000:22 ubuntu:18.04

After this step, you will end up with a docker container called thundercomm-sdk with /home/user/host mapped to the host SDK_SRC_ROOT directory.

Start the container

Once you have your docker container ready:

1. Start the container

docker start thundercomm-sdk

2. Log into the container with:

Note

The password of the container is 123456

ssh -p 36000 user@127.0.0.1

Once you are in the container you can enter the host directory where the source code is:

cd host

Note

From this point forward, all the paths specified will be relative to this root directory of the repository, which is /home/user/host.

Camera sensor driver files

To create a camera driver binary, which will allow access to the camera sensor frames, the CamX framework contains a layer called CHI-CDK with the necessary files and scripts. A camera driver is built upon a series of XML files, shared libraries, and Kernel device tree files, such as the following:

Sensor Software configuration XML

This XML file contains sensor specific information such as power settings, resolution, initialization settings, color filter arrangements and exposure settings. A file with a template showing the complete set of features to specify is found at apps_proc/src/vendor/qcom/proprietary/chi-cdk/api/sensor/camxsensordriver.xsd.
The XML file you create for your sensor must be located at apps_proc/src/vendor/qcom/proprietary/chi-cdk/oem/qcom/sensor/<sensor_name>/ <sensor_name>_sensor.xml, where <sensor_name> is replaced by the name of your sensor, for example, imx577.

Module configuration XML

This XML contains camera module-specific information, including the lens information, mount angles, actuator, OIS, and flash-related information. A file with a template showing the complete set of features to specify is found at apps_proc/src/vendor/qcom/proprietary/chi-cdk/api/sensor/camxmoduleconfig.xsd.
The XML file you create for your sensor module must be located at apps_proc/src/vendor/qcom/proprietary/chi-cdk/oem/qcom/module/<module_name>_module.xml, where <module_name> is replaced by the name of your sensor module, for example, cmk_imx577.

Sub-Module configuration XMLs

The camera you are implementing may have additional components, referred to as sub-modules, such as an actuator, EEPROM, Flash and OIS. For these, there are also XML files to describe them. Templates showing the complete set of features to specify for each of them are found at apps_proc/src/vendor/qcom/proprietary/chi-cdk/api/sensor/camx<sub_module>driver.xsd, where <sub_module> corresponds to one of the submodules, actuator, eeprom, flash or ois.
The XML file you create for your sensor submodule must be located at apps_proc/src/vendor/qcom/proprietary/chi-cdk/oem/qcom/<sub_module>/<sub_module_name>_<sub_module>.xml, where <sub_module_name> is replaced by the name of your specific sub-module, for example, ak7371, so an actuator XML file would look like ak7371_actuator.xml.

Sensor gain/exposure customization

The sensor infrastructure provides an alternative for developers and OEMs to create their own gain/exposure functions. The make use of this feature, you will need to create a C/C++ source file that utilizes the C API camxsensordriverAPI.h, which is found at apps_proc/src/vendor/qcom/proprietary/chi-cdk/api/sensor. You will need to generate it as a shared library with the following name com.<vendor_name>.sensor.<sensor_name>.so, for example, com.qti.sensor.imx318.so.
You can find an example at apps_proc/src/vendor/qcom/proprietary/chi-cdk/oem/qcom/sensor/imx318. Inside the previous directory, the folder build/linuxembedded contains a CMakeLists.txt where the shared library creation is specified and the path where it should be installed.

Camera sensor binary build setup

For the system to build the camera driver binary file you need to modify the following files:
1. Include in the device vendor makefile located at apps_proc/src/vendor/qcom/proprietary/chi-cdk/configs/product.mk the name of the camera driver binary. Open the file and add the following line:

Note

Replace <module_name> with the same name you specified for the sensor module in the Module configuration XML section, for example cmk_imx577

PRODUCT_PACKAGES += com.qti.sensormodule.<module_name>.bin

If you created custom gain/exposure functions as in the Sensor gain/exposure customization section. You need to add the name of the library to the product.mk file, as well, for example:

Note

Replace <sensor_name> with the same name you specified for the sensor in the Sensor Software configuration XML section, for example imx577

PRODUCT_PACKAGES += com.qti.sensor.<sensor_name>.bin

2. The infrastructure uses a script to generate the binary file of the camera driver, to locate the paths of the XML files describing the camera, the script reads a yaml file, in which you will need to add the paths for your specific XMLs, open the file apps_proc/src/vendor/qcom/proprietary/chi-cdk/tools/buildbins/buildbins_kona.yaml and add the following lines:

Note

Make sure to replace all the placeholders <> with the corresponding names for your sensor. Also note that the sub-modules such as the EEPROM, actuator and flash are not necessary, so if you did not implement them, their respective lines should be removed from the file.

- com.qti.sensormodule.<module_name>:
    - sensor/<sensor_name>/<sensor_name>_sensor.xml
    - module/<module_name>_module.xml
    - eeprom/<sub_module_name>_eeprom.xml
    - actuator/<sub_module_name>_actuator.xml
    - flash/<sub_module_name>_flash.xml

Sensor Device tree configuration

The sensor must be specified in the device tree of your board's kernel by including it in the DTSI files, before the binary can work. The dtsi files corresponding to the RB5/RB6 board is located at apps_proc/src/kernel/msm-4.19/arch/arm64/boot/dts/vendor/qcom/camera/. For examples on how to customize the DTSI files, see the files located at apps_proc/src/kernel/msm-4.19/arch/arm64/boot/dts/vendor/qcom/camera/bindings.

Building the yocto images and flashing them to the board

Once you are ready with your driver related and device tree changes, you can build the yocto images with the following steps: 1. Build the yocto images:
First, set the right owner on the host folder to avoid permission issues:

chown -R $USER:$USER /home/user/host

Change directory to the host directory and run the building script:

Note

Take into account that with the host PC mentioned at the beginning, the first time you run the script, the process takes around 3.5 hours. In the consequent times that you execute the script after you made a change, for example in the device tree, the process takes around 30 minutes.

cd /home/user/host
./turbox_build.sh

When the images are built you should see a message like the following:

[09:48:14] - update_meta.py: Validating File paths in contents.xml
[09:48:14] - =====================================================
[09:48:14] - update_meta.py: Total number of file paths found in contents.xml = 121
[09:48:14] - update_meta.py: file path validation passed.
[09:48:14] - ============== CHECK COMPLETE ===============
[09:48:14] - update_common_info.py:============ UPDATE COMMON INFO
COMPLETE====================
user@93ed2f966144:~/host$

2. Prepare images for flashing:

./turbox_build.sh --zip_flat_build

This will generate the images for flat installation.

The generated zip file should be under SDK_SRC_ROOT/turbox/output/ in your host PC, with the following name:

FlatBuild_Turbox-RB5_xx.xx_LE1.0.l.userdebug.Pre-CS.r001001.zip

Flash the image

Once the images are built and packaged you can proceed to the installation:

1. Make sure you have fastboot and adb installed:

sudo apt-get install android-tools-adb fastboot

2. Configure host USB

sudo touch /etc/udev/rules.d/99-android.rules
sudo chmod 777 /etc/udev/rules.d/99-android.rule

Edit /etc/udev/rules.d/99-android.rule and add the following content:

SUBSYSTEM=="usb", ATTR{idVendor}=="18d1", MODE="0777", GROUP="adm"
SUBSYSTEM=="usb", ATTR{idVendor}=="05c6", MODE="0777", GROUP="adm"

and restart udev:

sudo chmod a+r /etc/udev/rules.d/99-android.rules
sudo service udev restart

3. Unzip the flat build images

unzip FlatBuild_Turbox-RB5_xx.xx_LE1.0.l.userdebug.Pre-CS.r001001.zip

4. Set the board into EDL mode, to learn how to do it see the following section EDL_MODE.

5. Once the board is in EDL mode, enter the directory where the flash script is located and flash the image:

cd FlatBuild_Turbox-RB5_xx.xx_LE1.0.l.userdebug.Pre-CS.r001001/ufs/
sudo ./turbox_flash_flat.sh

If the Flash succeeded you will see something like the following:

10:45:54: INFO:      _             (done)
10:45:54: INFO:     | |                 
10:45:54: INFO:   __| | ___  _ __   ___ 
10:45:54: INFO:  / _` |/ _ \| '_ \ / _ \
10:45:54: INFO: | (_| | (_) | | | |  __/
10:45:54: INFO:  \__,_|\___/|_| |_|\___|
10:45:54: INFO: {All Finished Successfully}

10:45:54: INFO: Overall to target 84.943 seconds (42.93 MBps)
10:45:54: INFO: {percent files transferred 100.00%}

Writing log to '/home/user/FlatBuild_Turbox-RB5_xx.xx_LE1.0.l.userdebug.Pre-CS.r001001/ufs/port_trace.txt', might take a minute


Log is '/home/user/FlatBuild_Turbox-RB5_xx.xx_LE1.0.l.userdebug.Pre-CS.r001001/ufs/port_trace.txt'

6. Reboot the board by pressing the ON/OFF button for 20 seconds. It should boot into your new image.

Rebuilding camera driver binaries

In the process of testing the camera driver you may notice a configuration in an XML file must be modified to meet your requirements or to make the driver functional. In that case, after making the necessary changes in the XMLs, you can rebuild the binary in the source code and copy it back to the /usr/lib/camera directory in your board, which simplifies the bring-up process. The steps are the following:
1. To generate the binary for your camera driver go to the directory where the generation script called buildbins3.py is located:

cd apps_proc/src/vendor/qcom/proprietary/chi-cdk/tools/buildbins

2. Run the following command to create the binary:

Note

Take into account this script only generates the binary com.qti.sensormodule.<module_name>.bin.

python3 buildbins3.py --yaml-file-name buildbins_kona.yaml

The binary file will be generated at apps_proc/src/vendor/qcom/proprietary/chi-cdk/oem/qcom/bin directory.
3. To copy the binary file to the board you will need to connect to it via ADB. Click on ADB_GUIDE to learn how to connect via ADB, please skip Step 4 in the mentioned guide.
4. Once the host PC detected the board with adb, run the following commands to give permissions on the board:

adb root
adb disable-verity
adb root
adb remount

5. Change to the directory where the binary file was generated:

cd apps_proc/src/vendor/qcom/proprietary/chi-cdk/oem/qcom/bin

6. Push the binary to the board:

adb push com.qti.sensormodule.<module_name>.bin /usr/lib/camera/

7. Restart the board:

adb reboot