Mender.io - Software updates for embedded Linux devices

Mender is a project that allows the OTA (over-the-air) updates on embedded Linux devices in a robust a secure way. Mender has support for multiple platforms; this tutorial will be a focus on the use of Mender in NVIDIA Jetson platforms.

Overview

Mender architecture

Error creating thumbnail: Unable to save thumbnail to destination

Figure 1. Mender architecture

Mender implements a client-server architecture, wherein on the server-side the different updates will be stored to be deployed to the corresponding devices, and the client is installed in all those devices to be able to install the updates.

In general, for every update, mender implements the following logic:

1. The update is stored in a .menderfile called artifact.
2. These artifacts are stored in the Mender server.
3. The devices ask for new updates to the server.
4. The artifact is downloaded on different devices.
5. The artifact is installed on each device.
6. The board is rebooted if the artifact requires it, and perform the commit in case of making the update persistent.

The server could be the official Mender server implementation or a custom local server. In case of using the official Mender server, the client installed on the boards will install the updates in a managed mode, and in case of installing the updates using a custom local server or from a USB device, the client will install the update in a standalone mode.

Mender allows its integration in a custom build system, like in the case of Yocto. This wiki page will be explained how to test and then integrate Mender into a Yocto environment.

Basic concepts

Error creating thumbnail: Unable to save thumbnail to destination

Figure 2. State machine pattern implemented to Mender's update module.

The following are important concepts needed to have an adequate understanding of the Mender update system:

A/B redundancy systems

In an A/B redundant system, the partition layout is configured to have an active filesystem A and a redundant filesystem B. During a RootFS update, the update is written to the inactive filesystem B, and when the update is completed, the bootloader is configured to boot at the redundant partition. In case the update has some problem and the board is not able to boot from the redundant filesystem partition, Mender has a recovery mechanism to roll back the upgrade .

Artifacts

The artifacts are the files that Mender uses to package the update. These files have the suffix .mender and basically consist of a tar file with all the needed files needed to perform the update on the board, like some configuration files with information like the type of devices that are compatible with the update, and a payload with the updates files that need to be applied .

Update modules

By default, Mender implements the mechanism for the dual rootfs update but also is flexible enough to allow the implementation of custom logic that allows the use of other types of updates . To do this, Mender has the update modules that follow a state machine design pattern as shown in figure 2.

State scripts

The state scripts are used by Mender to be executed in state transitions to overcome a specific use case. The mender client executes this type of script either before or after some state of the update . The state script has the following naming convention:

<STATE_NAME>_<ACTION>_<ORDERING_NUMBER>_<OPTIONAL_DESCRIPTION>

For example, ArtifactInstall_Enter_00_remount_to_rw_filesystem will be executed before the ArtifactInstall state. These state scripts can be included into the Mender installation on the board, or added directly to some update artifact.

Typical partition layout

The expected partition layout to have the correct functioning of Mender is one with A/B partitions for the filesystem updates and a /data partition used by mender to store configuration files that will keep persistent among different updates (See figure 3).

Error creating thumbnail: Unable to save thumbnail to destination

Figure 3. Mender typical partition layout.

Is also possible to add extra partition, making use of Mender variables for Yocto like MENDER_EXTRA_PARTS, MENDER_EXTRA_PARTS_FSTAB or MENDER_EXTRA_PARTS_SIZES_MB.

Types of updates

RootFS updates

This is the default type of update implemented by Mender to perform a dual rootfs update. A typical flow for this type of update is the following:

1. Create the Mender artifact: In a Yocto integration, when the build process completes, a .mender file is created automatically with the rootfs update.
2. Upload the artifact to the corresponding server.
3. On the board, use the Mender client to download and install the artifact in the inactive filesystem partition.
4. For this type of update a reboot is required.
5. After performing the update, to make the update persistent is needed to commit the changes using the Mender client.
6. In case of revert the update changes, the Mender client can perform a rollback.

Application updates

For uses cases different from the filesystem update, Mender implemented the applications updates. These updates have custom implementations of update-modules to perform actions like single file updates, directory updates, Deb packages updates, Rpm packages updates, docker updates, among others .

By default, the update modules come disabled. To enable the mender update-modules, create mender-client_%.bbappend file in your custom meta layer and add the following:

PACKAGECONFIG_append = " modules"

Then, to include some custom update module, add the following to the same .bbappend file:

FILESEXTRAPATHS_prepend := "${THISDIR}/<DIRECTORY-WITH-UPDATE-MODULE>:"
SRC_URI_append = " file://custom-update-module"

do_install_append() {
    install -d ${D}/${datadir}/mender/modules/v3
    install -m 755 ${WORKDIR}/custom-update-module ${D}/${datadir}/mender/modules/v3/custom-update-module
}

Updates deployments

Managed deployments

The managed deployment method is the one that makes use of the official Mender server. The server can be the one hosted by Mender, with a paid stipend, or hosted in your own infrastructure. From the client-side, when configured as managed mode, the client will run a daemon that constantly will be asking for updates to the server. Mender recommends the managed deployment mode.

Standalone deployments

With the standalone mode the update will be performed manually on the board by executing the following command:

mender -install <URL>

Where the URL can be the address of the artifact in a local web server or in a USB device connected to the board. In some cases, is required to perform a commit of the update changes to make them persistent by running the command:

mender -commit

Security features

You can use the mender-artifact application in your host machine to sign the different artifacts files.

First is needed to generate a private key, used to sign the artifact, and a public key to distribute into the devices:

openssl genpkey -algorithm RSA -out private.key -pkeyopt rsa_keygen_bits:3072
openssl rsa -in private.key -out private.key

Then, you need to integrate the public key to the Yocto image by adding the following to the mender-client_%.bbappend file:

FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI_append = " file://artifact-verify-key.pem"

Please note that you need to rename your public key to artifact-verify-key.pem. After that, with the following command you can sign an existing artifact:

mender-artifact validate artifact-signed.mender -k public.key

Now the mender client installed in the boards won't allow the installation of not signed artifacts.

Recovery features

Mender provides the functionality to recover the system from a failed filesystem update or to perform a rollback to some particular update recently installed. For the case of the filesystem update, after the update is installed the system must be rebooted. If during boot time the system is not able the start from the redundant partition, Mender implements a mechanism to apply a rollback from the bootloader. But even if the update works correctly after the reboot was done, the user can perform a rollback manually, running the following command on the board:

mender -rollback

Some application updates also implement the rollback feature.

Paid features

In general, Mender is an open-source project, but there are some features and services that are not free. The following are some paid features:

• Production server hosted by mender.
• Robust delta updates.
• Secure hosted server.

For the complete list of features, consult .

Mender demo using NVIDIA Jetson platforms

The Mender community provides support for different NVIDIA Jetson platforms. To test the Mender functionalities, there is also a demo project for the following platforms:

• Jetson Nano .
• Jetson TX2 .
• Jetson Xavier NX .
• Tegra AGX Xavier .

Prerequisites

• 1. Clone the demo repository:

git clone https://github.com/OE4T/tegra-demo-distro.git

• 2. Switch to the appropriate branch .

• 3. Initialize the git submodules:

git submodule update --init

• 4. Source the environment:

source ./setup-env --distro tegrademo-mender --machine <MACH>

Replace MACH with the corresponding machine as shown in the following table:

Device	<MACH>
TX2	jetson-tx2*	jetson-tx2i*	jetson-tx2-4gb*
XAVIER	jetson-xavier*	jetson-xavier-8gb*
NANO	jetson-nano-devkit*	jetson-nano-emmc
XAVIER NX	jetson-xavier-nx-devkit*	jetson-xavier-nx-devkit-emmc*
Note: the machines with * have NOT been tested by RidgeRun yet

Setup build environment

Initialize build environment:

source ./setup-env --distro tegrademo-mender --machine <MACH>

Build process

For this Yocto environment, you have available different images options to build:

Image	Description
demo-image-base	Basic image with no graphics
demo-image-egl	Base image with DRM/EGL graphics, no window manager
demo-image-sato	X11 image with Sato UI
demo-image-weston	Wayland with Weston compositor
demo-image-full	Sato image plus nvidia-docker, openCV, multimedia API samples

To start the build run:

bitbake demo-image-base

Testing

When the build completes, in <YOCTO_DEVDIR>/build/tmp/deploy/images/<MACH> you will find a .mender file that corresponds to the artifact to do a dual rootfs update. You can store that artifact in a USB device or a local web server, and using the Mender client on the board, manually install the update (see Standalone deployment section).

Yocto integration

Mender allows its integration to Yocto. For this guide, we will be using the dunfell version of Yocto.

Dependencies

1. Go to your Yocto devdir:

YOCTO_DIR=/path/to/yocto-devdir
cd $YOCTO_DIR

2. Select the branch:

export BRANCH="dunfell"

3. Download the mender layers:

git clone -b ${BRANCH} https://github.com/mendersoftware/meta-mender.git
git clone -b ${BRANCH} https://github.com/mendersoftware/meta-mender-community.git
git clone -b ${BRANCH} https://github.com/OE4T/meta-tegra-community.git

4. Add the Mender meta layers to the Yocto bblayers configuration files located in YOCTO_DIR/build/conf:

# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"

BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
  ...
  YOCTO_DIR/meta-mender/meta-mender-core \
  YOCTO_DIR/meta-mender-community/meta-mender-tegra \
  YOCTO_DIR/meta-tegra-community \
  "

Distro setup

Add the following settings values to your custom Yocto distro:

### Additions for mender ###

INHERIT += "mender-full"
INHERIT_append_tegra = " tegra-mender-setup"

# Disabling unneeded mender features
MENDER_FEATURES_DISABLE_append = " mender-systemd"

# Setup mender artifacts
MENDER_ARTIFACT_NAME = "my-release"
ARTIFACTIMG_NAME = "${IMAGE_BASENAME}-boot"

IMAGE_FSTYPES_tegra = "tegraflash mender dataimg"
IMAGE_FSTYPES_pn-tegra-minimal-initramfs_tegra = "${INITRAMFS_FSTYPES}"

# Mender partition settings
MENDER_STORAGE_DEVICE_BASE = "/dev/mmcblk0p"
MENDER_DATA_PART = "${MENDER_STORAGE_DEVICE_BASE}24"
MENDER_ROOTFS_PART_A = "${MENDER_STORAGE_DEVICE_BASE}1"
MENDER_ROOTFS_PART_B = "${MENDER_STORAGE_DEVICE_BASE}18"

# Mender machines need to run checks after an update
SYSTEMD_DEFAULT_TARGET = "finished-booting.target"

In this setup is intended the use of the standalone mode, which means the Mender SystemD service is not needed. That is why the mender-systemd is disabled. Also, verify your partition layout very carefully to set the correct block devices for the rootfs A, rootfs B, and the data partition.

Now you can trigger the build process of your Yocto image again, and this time you should get a .mender file with the rootfs update generated in YOCTO_DIR/build/tmp/deploy/images/<MACH>. Finally, is recommended to follow the Mender checklist to verify this integration work .

Artifact creation

To create the artifacts, you need to download and install the mender-artifact tool in your host machine .

RootFS artifacts

As mentioned in previous sections, when Mender is integrated with Yocto, each time you complete a build is generated a .mender file with the rootfs artifact. But is also possible to create it manually running the following command:

mender-artifact write rootfs-image \
   -t jetson-nano-emmc \
   -n release-1 \
   --software-version rootfs-v1 \
   -f rootfs.ext4 \
   -o artifact.mender

Where rootfs.ext4 corresponds to the filesystem image, and:

• -t: specifies the compatible device. In this case, we are creating the artifact for the jetson-nano-emmc.
• -n: this is the name of the artifact.
• --software-version: corresponds to the string software version.
• -o: path of the output file.

Application artifacts

There are different types of application artifacts. In this case, we are going to present an example to create an artifact to perform a single file update.

To create the single file artifact, first download the single-file-artifact-gen tool:

wget https://raw.githubusercontent.com/mendersoftware/mender/master/support/modules-artifact-gen/single-file-artifact-gen
chmod +x single-file-artifact-gen

Then, create the sample file to deploy:

echo "File created by Mender single-file Update Module!" > my_update_file.txt

Then, create the artifact with the following command:

ARTIFACT_NAME="my-update-1.0"
DEVICE_TYPE="jetson-nano-emmc"
OUTPUT_PATH="my-update-1.0.mender"
DEST_DIR="/opt/installed-by-single-file/"
FILE="my_update_file.txt"
./single-file-artifact-gen -n ${ARTIFACT_NAME} -t ${DEVICE_TYPE} -d ${DEST_DIR} -o ${OUTPUT_PATH} ${FILE}

Where:

• ARTIFACT_NAME - The name of the Mender Artifact
• DEVICE_TYPE - The compatible device type of this Mender Artifact
• OUTPUT_PATH - The path where to place the output Mender Artifact. This should always have a .mender suffix
• DEST_DIR - The path on target device where FILE will be installed.
• FILE - The path to the file to be sent to the device in the update.

References

RidgeRun Resources

Quick Start Client Engagement Process RidgeRun Blog Homepage Technical and Sales Support RidgeRun Online Store RidgeRun Videos Contact Us

Contact Us Visit our Main Website for the RidgeRun Products and Online Store. RidgeRun Engineering informations are available in RidgeRun Professional Services, RidgeRun Subscription Model and Client Engagement Process wiki pages. Please email to support@ridgerun.com for technical questions and contactus@ridgerun.com for other queries. Contact details for sponsoring the RidgeRun GStreamer projects are available in Sponsor Projects page.