Yocto - Tools

Yocto
What is Yocto
Advantages Disadvantages Components Resources Tasks
Tools
Repo Devtool Devtool for Kernel Development Devshell Devshell Yocto Extensible SDK QEMU
How to
Create Custom Distro Create Custom Image Preferred Recipe Version
Contact Us

Tools

In here you can find some tools and how to use them related to Yocto.

Repo

What is Repo?

From repo documentation: repo is a tool built on top of Git. Repo helps manage many Git repositories, does the uploads to revision control systems, and automates parts of the development workflow. Repo is not meant to replace Git, only to make it easier to work with Git. The repo command is an executable Python script that you can put anywhere in your path.

Install repo

You can install it using apt:

sudo apt-get install repo

Or downloading it directly

1. Add repo script to bin directory in home.

mkdir -p ~/bin
curl http://commondatastorage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
chmod a+x ~/bin/repo

2. Append the directory if it's not already in PATH

PATH=$PATH:~/bin

How to use repo with Yocto

Repo uses an XML manifest in order to enumerate multiple remotes, repositories, branches and revisions, then it fetches them. For example, using a Boundary Devices repository with a manifest we can use:

cd <ANY_DIR>
repo init -u https://github.com/boundarydevices/boundary-bsp-platform -b dunfell

With these options being the most important:

-u URL, --manifest-url=URL : manifest repository location
-b REVISION, --manifest-branch=REVISION : manifest branch or revision (use HEAD for default)
-m NAME.xml, --manifest-name=NAME.xml: initial manifest file (defaults to default.xml), you can also use this to specify a completely different manifest in your system and it will be used. 

After doing repo init, you can sync it using:

repo sync

This will generate a directory structure with the given repositories defined in the XML manifest, normally in Yocto it will create a sources directory with each layer, and a setup script to create a build.

Analyzing the repo manifest

Take as an example the manifest located in: https://github.com/boundarydevices/boundary-bsp-platform/blob/gatesgarth/default.xml

Defaults

The default settings are set as:

<default sync-j="4" revision="gatesgarth"/>

Remotes

Remotes are defined as below, they are named to be used later for each project:

<remote fetch="https://git.yoctoproject.org/git" name="yocto"/>
<remote fetch="https://github.com/Freescale" name="freescale"/>
<remote fetch="https://github.com/openembedded" name="oe"/>
<remote fetch="https://github.com/boundarydevices" name="boundary"/>

Projects

Project are basically the repositories to fetch, they will use the remote defined above, the name on the hosting server, and then clone them into a path. You can also use the revision which is the commit to checkout, and upstream which is the branch to use, this is not necessarily the same for all projects obviously.

<project remote="yocto"     revision="6bd890d9e011014cf323e61267f8b256949d44aa" upstream="pyro" name="poky" path="sources/poky"/>
<project remote="yocto"     revision="83b779a7ef69006564fb8d27e65c31917bc3b6f0" upstream="pyro" name="meta-freescale" path="sources/meta-freescale"/>
<project remote="oe"        revision="5e82995148a2844c6f483ae5ddd1438d87ea9fb7" upstream="pyro" name="meta-openembedded" path="sources/meta-openembedded"/>
<project remote="freescale" revision="a73a14247afbbcf840bab7b76f953004fe109253" upstream="pyro" name="meta-freescale-3rdparty" path="sources/meta-freescale-3rdparty"/>
<project remote="freescale" revision="cd5c7a2539f40004f74126e9fdf08254fd9a6390" upstream="pyro" name="meta-freescale-distro" path="sources/meta-freescale-distro"/>

You can also perform operations like copy files from the project directly after syncing:

<project remote="boundary" name="boundary-bsp-base" path="sources/base">
  <copyfile dest="README" src="README"/>
  <copyfile dest="setup-environment" src="setup-environment"/>
</project>

Creating a manifest

You can create a manifest of your current build by using:

repo manifest -r -o my_manifest.xml

Using tools inside a docker container

A docker image with the environment almost ready is available to download, here are the steps to use it for the training

Install docker

First you will need to install Docker from this guide: https://docs.docker.com/engine/install/ubuntu/

You can also check this guide in order to have permissions to run docker without sudo: https://docs.docker.com/engine/install/linux-postinstall/

Run a container with the image

docker run -it ridgerun/yocto-tips-training-prebuilt

You need to clean gst-perf after running the container:

cd $DEVDIR
source ./sources/poky/oe-init-build-env training-build
bitbake -c cleanall gst-perf
bitbake gst-perf

This DEVDIR variable is already set and it contains the sources.

Devtool

Let's assume we are building for qemuarm64 and you have set the `$DEVDIR` variable.

First use devtool to set a new workspace for the recipe:

devtool modify gst-perf

Source code now will be located in the new created folder:

ls $BUILDDIR/workspace/sources/gst-perf

When you compile and install this recipe, these sources will be used, you can check it by running:

bitbake -c install gst-perf # Compilation and installation
tree $BUILDDIR/tmp/work/aarch64-poky-linux/gst-perf/1.0-r0/image # Files installed

Devtool deploy

deploy method is agnostic to the platform and any paths, it takes into account everything and just needs the name of the recipe to update it in the target by using SSH

Setting up

cd $BUILDDIR
runqemu qemuarm64 core-image-training nographic

Let's get the IP from the QEMU instance, inside the instance:

ifconfig eth0 | grep "inet " | awk -F'[: ]+' '{ print $4 }'

Save this IP, we will use it later.

Testing

Modify the source code, for example the Description of the plugin, and then build again:

bitbake -c install gst-perf

Then you can deploy the changes using devtool:

devtool deploy-target gst-perf root@$QEMU_IP

You can see the new version by running in the QEMU:

ldconfig
gst-inspect-1.0 perf | sed -r "s/[[:cntrl:]]\[[0-9]{1,3}m//g" | grep Description -m 1

You can also undeploy the changes

undeploy-target gst-perf root@$QEMU_IP

Saving changes

To save the changes

devtool update-recipe --append=$DEVDIR/sources/meta-training/ gst-perf
devtool reset gst-perf

Equivalent to:

devtool finish gst-pref $DEVDIR/sources/meta-training/

Both commands will just save the changes made in the new commits made for the branch inside the $BUILDDIR/workspace/sources/ folder

NFS

NFS method let's you mount the filesystem in your host and QEMU will load it, allowing changes to it in real time. Reference: https://docs.yoctoproject.org/dev-manual/qemu.html#running-under-a-network-file-system-nfs-server

Setting up

# Set variables
cd $BUILDDIR
export NFS_ROOT=`pwd`/test-nfs

# Install dependencies for NFS bitbake and QEMU support
sudo apt-get install rpcbind
bitbake meta-ide-support

# Set up NFS and start QEMU instance
runqemu-extract-sdk ./tmp/deploy/images/qemuarm64/core-image-training-qemuarm64.tar.bz2 $NFS_ROOT
runqemu qemuarm64 $NFS_ROOT nographic

Now, every change to the filesystem inside `$NFS_ROOT` will be seen in real time on the QEMU instance running

Testing

Modify the source code, and then:

cd $BUILDDIR
bitbake -c install gst-perf
cp -r ./tmp/work/aarch64-poky-linux/gst-perf/1.0-r0/image/* $NFS_ROOT/

Stopping

To stop the NFS sharing use:

runqemu-export-rootfs stop $NFS_ROOT

Then remove the folders

rm -rf $NFS_ROOT
rm -rf $NFS_ROOT.pseudo_state

Devtool for Kernel Development

Modify

Setup the kernel sources:

devtool modify virtual/linux

Modify $BUILDDIR/workspace/sources/linux-yocto/init/calibrate.c by adding these printk to the calibrate_delay function.

     void calibrate_delay(void)
     {
         unsigned long lpj;
         static bool printed;
         int this_cpu = smp_processor_id();

         printk("*************************************\n");
         printk("*                                   *\n");
         printk("*        HELLO YOCTO KERNEL         *\n");
         printk("*                                   *\n");
         printk("*************************************\n");

     	if (per_cpu(cpu_loops_per_jiffy, this_cpu)) {

Deploy

We need to copy the image /boot/

scp tmp/work/qemuarm64-poky-linux/linux-yocto/5.4.153+git999-r0/image/boot/Image root@<QEMU_IP>:/boot/

Then exit Qemu, and enter again. You should see the message from above while running:

dmesg | less

---

---

Devshell

Setting up

Remember the $DEVDIR folder? Get into there and get into the build directory if you haven't done so.

cd $DEVDIR
source ./sources/poky/oe-init-build-env training-build/

We are going to modify the perf element. So get into the board and check the current Description for this element.

gst-inspect-1.0 perf | sed -r "s/[[:cntrl:]]\[[0-9]{1,3}m//g" | grep Description -m 1

Usage

If you added gst-perf to the workspace using devtool, first remove it: devtool reset gst-perf.

Let's start using devshell, execute:

bitbake gst-perf -c devshell

This will open up the sources on the $S variable, normally $WORKDIR/git if using git.

Updating the code

0. Check if the source code is using git or not, if not and you are planning to make changes that you are planning to capture you will need a new git local repo or to use with quilt, in any case you need to either start the repo or create a new quilt patch before modifying the files. More information in the Saving changes section below.

1. Modify the code, for example update the Description for the element:

Use vim or your editor, and search for gst_element_class_set_static_metadata function. Modify it.

Compiling

1. Run by hand:

./autogen.sh
./configure --prefix `pwd`/../image/usr/ --libdir `pwd`/../image/usr/lib --build=x86_64-linux --host=aarch64-poky-linux
make && make install

Then all the will be installed at first level inside

tree ../image/
../image/
└── usr
    └── lib
        └── gstreamer-1.0
            ├── libgstperf.la
            └── libgstperf.so

Testing

You can test by updating the binaries and files in the board or in the QEMU instance or

scp -r ../image/* root@192.168.7.2:/

Look that the copies need to be done by hand, different to devtool.

Then run on the board:

ldconfig
gst-inspect-1.0 perf | sed -r "s/[[:cntrl:]]\[[0-9]{1,3}m//g" | grep Description -m 1

Saving changes

Create the patch

git

Create patch, if using git for example:

git add plugins/gstperf.c
git commit -m "update"
git format-patch HEAD~1

quilt

quilt new 0001-update.patch
quilt add plugins/gstperf.c
edit plugins/gstperf.c

Modify file and then

quilt refresh

Now you can copy the quilt generated file.

Update recipe with patch

Save the path of the patch file, using pwd and the name of it:

echo `pwd`/0001-update.patch

Update recipe with patch:

cd $DEVDIR/sources/meta-training/recipes-multimedia/gstreamer
mkdir files
cp $PATCH_PATH files/

And append it to the SRC_URI for the recipe manually.

cd $DEVDIR/sources/meta-training/recipes-multimedia/gstreamer/

And then add to the gst-perf_%.bbapped:

FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI += "file://0001-update.patch \
           "

Now to test you may copy the values from the image folder in the workdir, first get this value doing a grep:

bitbake -e gst-perf | grep ^D=

Then just the same copy and test commands.

NOTE: Take a look at how many manual steps are needed in order to achieve the same that the devtool is doing for us.

run.do_* scripts

Setting up

Remember the $DEVDIR folder again? Get into there and get into the build directory if you haven't done so.

cd $DEVDIR
source ./sources/poky/oe-init-build-env training-build/

We are going to modify the perf element again. So get into the board and check the current Description for this element.

gst-inspect-1.0 perf | sed -r "s/[[:cntrl:]]\[[0-9]{1,3}m//g" | grep Description -m 1

Usage

To start, we need to get into the code of the package, for this check where it is located by using:

bitbake -e gst-perf | grep ^S=

And move to the directory it is pointing.

Updating the code

0. Check if the source code is using git or not, if not and you are planning to make changes that you are planning to capture you will need a new git local repo or to use with quilt, in any case you need to either start the repo or create a new quilt patch before modifying the files. More information in the Saving changes section of devshell, same applies here.

1. Modify the code, for example update the Description for the element:

Use vim or your editor, and search for gst_element_class_set_static_metadata function. Modify it.

Compiling

This is the easy part, you can just run the compilation and install scripts autogenerated by bitbake:

../temp/run.do_compile
../temp/run.do_install

If these scripts don't exist, execute the install task for the desired recipe.

Testing

You can test by updating the binaries and files in the board or in the QEMU instance or

scp -r ../image/* root@192.168.7.2:/

Look that the copies need to be done by hand, different to devtool.

Then run on the board:

ldconfig
gst-inspect-1.0 perf | sed -r "s/[[:cntrl:]]\[[0-9]{1,3}m//g" | grep Description -m 1

Saving changes

To save these changes refer to: Saving changes section of devshell, it is the same process for this.

Yocto Extensible SDK

Building Yocto can take time. There's an alternative for bigger teams working on a Yocto image, so that only one person builds Yocto, and the rest of the team uses the resulting build to develop their applications. This is the so-called Yocto SDK.

This way, Yocto SDK provides a simpler environment to application engineers, to avoid all the complexity and time spent to build the whole Yocto image.

Yocto provides two SDK types: Standard and Extensible. We will build the extensible SDK, as it contains more useful features as can be seen in the following table:

Feature	Standard SDK	Extensible SDK
Toolchain	Yes	Yes
Debugger	Yes	Yes
Size	100+MB	1+GB
devtool	No	Yes
Build Images	No	Yes
Updateable	No	Yes
Managed Sysroot	No	Yes
Installed Packages	No	Yes
Construction	Packages	Shared State

Build the SDK

To build the Extensible SDK:

bitbake core-image-training -c populate_sdk_ext

Install the SDK

The SDK can be found inside the $BUILDDIR/tmp/deploy/sdk directory.

To install the SDK:

cd $BUILDDIR
./tmp/deploy/sdk/poky-glibc-x86_64-core-image-training-aarch64-qemuarm64-toolchain-ext-3.1.11.sh

You will be asked for the directory where the SDK will be installed. From now on, we are going to call this directory $SDK_DIR

Initialize the SDK

Every time you want to use the SDK, you have to source the environment script:

cd $SDK_DIR
source environment-setup-aarch64-poky-linux

Check that you get the following text after the source command:

SDK environment now set up; additionally you may now run devtool to perform development tasks.
Run devtool --help for further details.

Extended SDK with Devtool

First we need to create the image with

devtool build-image core-image-training

To add the gst-perf sources to the workspace, run:

devtool modify gst-perf

Now you can modify it at:

workspace/sources/gst-perf/

Then, to build the changes:

devtool deploy-target gst-perf root@$QEMU_IP

devtool knows which files need to be updated and where they need to be installed, so you don't have to worry.