Yocto - What is Yocto

Yocto
What is Yocto
Advantages Disadvantages Components Resources Tasks
Tools
Repo Devtool Devtool for Kernel Development Devshell Devshell Yocto Extensible SDK QEMU
How to
Create Custom Distro Create Custom Image Preferred Recipe Version
Contact Us

What is Yocto?

You can watch this funny video or continue reading.

The purpose of the Yocto Project is to help you build a custom Linux-based system for embedded computing devices. Contains a set of tools that help you develop and test your system based on your needs.

Advantages

• Automates build process
• Easy to customize (add, remove, modify) Linux packages
• Most popular and used build system in the semiconductor industry.
• Easy to maintain package layers and package building recipes.
• Fully functional and tested tool-chain across a variety of architectures and platforms.
• Has a somehow good enough documentation
• Customizable tool-chain if needed.

Disadvantages

• Learning curve can be steep.
• Not very good forums for troubleshooting
• Due to a large amount of packages, sometimes compatibility is broken between versions
• Usually takes too much space
• Generally, consumes too many resources from your computer, such as CPU and RAM

Components

Yocto Project is a collection of build framework (build system), tools, and metadata. its major components are openEmbedded, Poky, and the Bitbake, being the base for any BSP created for a new board or system. Yocto Project supports many different architectures (PowerPC, ARM, x86/64, etc) and hardware platforms.

Poky

It is the reference distribution of the Yocto Project and it provides an OpenEmbedded build system (Bitbake and OpenEmbedded Core) as well as a set of metadata that helps to create a custom distro. Poky is basically bitbake + meta data It supports as a reference for the following platforms: Beaglebone, QEMU (Emulator), and generic x86-64. Every six-month Yocto project team releases a new version of poky, these can be seen from here: https://wiki.yoctoproject.org/wiki/Releases

OpenEmbedded

OpenEmbedded is a build framework that consists of OE-core metadata and Bitbake build engine. OE-core is the base layer of the Yocto project and contains a Bitbake layer, recipe, classes, and tools.

Bitbake

It is the build engine in the Yocto project, it executes the task according to provided recipes, classes, configuration files, and metadata. Bitbake is a python program which takes recipe files (.bb and .bbappend files ) as an input and executes multiple tasks.

Resources

There are some more important concepts we need to understand to develop projects using Yocto.

Recipes

A recipe has information to interpret and build a piece of software. This information usually is:

• The location of the source code.
• Patches to apply (if needed)
• Any special configuration to be applied.
• Instructions about how to compile the source.
• And how to package the compiled output.

Layers

Yocto uses what is called a "Layer Model". A layer is just a collection of related recipes. This model makes it easy to collaborate, share and customize a specific image. So one example of why using separate layers is useful is when using specific hardware: all the hardware-specific configuration can be abstracted into a single layer, maintaining other layers that could be hardware-agnostic in separate layers. This way, these last layers are independent and can be used for other hardware projects.

Images

Yocto produces compressed forms of the filesystem ready to boot on the device. The images are located at: tmp/deploy/images/$MACHINE/

Vendor BSPs

BSP stands for Board Support Package. It collects information to support particular hardware devices. It contains specific information about hardware features, kennel configuration and required hardware drivers. It also contains information about any software needed for essential or optional features.

Usually, hardware vendors create a BSP layer so that customers can start the developing process easily.

Examples:

• NVIDIA Jetson: https://github.com/OE4T/meta-tegra
• Boundary Devices: https://github.com/boundarydevices/boundary-bsp-platform
• Variscite: https://github.com/varigit/variscite-bsp-platform
• Qualcomm: https://git.yoctoproject.org/cgit/cgit.cgi/meta-qcom

Tasks

Most software packages will generally need some set of steps or tasks in order to build them and install their software, that's why bitbake has a set of common tasks that get executed in order to build a package, some other recipes will have more tasks, or other will have less.

Normal Recipes

Here is a small layout of the BitBake general tasks for a single recipe:

1. fetch: The fetch task is responsible for fetching any source code that is required. This means things such as downloading files and checking out from source control repositories such as git or svn.
2. unpack: The unpack task is responsible for extracting files from archives, such as .tar.gz, or from a repository, into the working area and copying any additional files, such as init scripts, into the working area. This will unpack the source code into a working directory pointed to by ${WORKDIR}, which is the path specified in as ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}. The S variable also plays a role in where unpacked source files ultimately reside, normally for git is S=${WORKDIR}/git, for a tarball, it is normally S=${WORKDIR}/<tarball_name>
   1. WORKDIR elements are:
      1. TMPDIR: The top-level build output directory
      2. MULTIMACH_TARGET_SYS: The target system identifier
      3. PN: The recipe name
      4. EXTENDPE: The epoch - (if PE is not specified, which is usually the case for most recipes, then EXTENDPE is blank)
      5. PV: The recipe version
      6. PR: The recipe revision
3. patch: The patch task is responsible for locating and applying any patches to the unpacked source code. Normally these are .patch or .diff, and are contained inside the files directory of the recipe. All these files get copied directly into ${WORKDIR}.
4. configure: The configure task takes care of the configuration of the package. Running the autotools configure script ("./configure ") is probably the form of configuration that is most recognized but it's not the only configuration system that exists. The task runs with the current working directory set to ${B}, which normally points to ${S}, even thought it can be separated.
5. compile: The compile task actually compiles the software. This task runs with the current working directory set to ${B}. The default behavior of this task is to run the oe_runmake function if a makefile (Makefile, makefile, or GNUmakefile) is found. If no such file is found, the do_compile task does nothing.
6. install The install task is responsible for installing files that are to be packaged into the holding area ${D} which is normally ${WORKDIR}/image. This directory won't actually be a part of the final package though. In other words if you install something into ${D}/bin then it will end up in the /bin directory in the package and therefore on the target.
7. package : The package task takes the installed files and splits them into separate directories under the ${WORKDIR}/install directory, one per package. It moves the files for the destination directory, ${D}, that they were installed in into the appropriate packages subdirectory. Usually there will be a main package, a separate documentation (-doc), development (-dev) and debugging packages (-dbg) for example. It makes usage of the ${PACKAGES} and ${FILES} variables, the ${PACKAGES} by default is ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}, during packaging, the do_package task goes through PACKAGES and uses the FILES variable corresponding to each package to assign files to the package. If a file matches the FILES variable for more than one package in PACKAGES, it will be assigned to the earliest (leftmost) package. The FILES variable are used inside the .bb file, like this: FILES_${PN} += "${bindir}/mydir1 ${bindir}/mydir2/myfile", here we have two considerations:
   1. When specifying files or paths, you can pattern match using Python’s glob syntax.
   2. When specifying paths as part of the FILES variable, it is good practice to use appropriate path variables. For example, use ${sysconfdir} rather than /etc, or ${bindir} rather than /usr/bin. You can find a list of these variables at the top of the meta/conf/bitbake.conf file in the Source Directory at here: https://github.com/openembedded/openembedded-core/blob/master/meta/conf/bitbake.conf. You will also find the default values of the various FILES_* variables in this file.
8. package_write The package_write task is responsible for taking each packages subdirectory and creating any actual installation package, such as .ipk, .deb or .rpm.
9. populate_sysroot: The do_populate_sysroot recipe stages (copies) a subset of the files installed by the do_install task into the appropriate sysroot. When another recipe depends on one, this is the task necessary to be completed before starting of that another recipe.

An example taken from $BUILDDIR/tmp/log/cooker/ logs:

Click expand to show the full log:

Sstate summary: Wanted 7 Found 0 Missed 7 Current 630 (0% match, 98% complete)
NOTE: Executing Tasks
NOTE: Setscene tasks completed
NOTE: Running task 1108 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_fetch)
NOTE: recipe busybox-1.31.1-r0: task do_fetch: Started
NOTE: recipe busybox-1.31.1-r0: task do_fetch: Succeeded
NOTE: Running task 1993 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_unpack)
NOTE: Running task 1994 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_prepare_recipe_sysroot)
NOTE: recipe busybox-1.31.1-r0: task do_unpack: Started
NOTE: recipe busybox-1.31.1-r0: task do_prepare_recipe_sysroot: Started
NOTE: recipe busybox-1.31.1-r0: task do_prepare_recipe_sysroot: Succeeded
NOTE: recipe busybox-1.31.1-r0: task do_unpack: Succeeded
NOTE: Running task 1995 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_patch)
NOTE: recipe busybox-1.31.1-r0: task do_patch: Started
NOTE: recipe busybox-1.31.1-r0: task do_patch: Succeeded
NOTE: Running task 1996 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_deploy_source_date_epoch)
NOTE: Running task 1997 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_populate_lic)
NOTE: recipe busybox-1.31.1-r0: task do_deploy_source_date_epoch: Started
NOTE: recipe busybox-1.31.1-r0: task do_populate_lic: Started
NOTE: recipe busybox-1.31.1-r0: task do_deploy_source_date_epoch: Succeeded
NOTE: Task /home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_deploy_source_date_epoch unihash changed to c037bd0f4613f48a16241d390c2ddfc5ae8ac520f284cb7f2475e872e8b09bdf
NOTE: Running task 1998 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_configure)
NOTE: recipe busybox-1.31.1-r0: task do_populate_lic: Succeeded
NOTE: Task /home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_populate_lic unihash changed to dcba114cfb0c983a53ed1a2fdd53a8c2cb378b236d92897f4c6126be52dc8f30
NOTE: Setscene tasks completed
NOTE: recipe busybox-1.31.1-r0: task do_configure: Started
NOTE: recipe busybox-1.31.1-r0: task do_configure: Succeeded
NOTE: Running task 1999 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_configure_ptest_base)
NOTE: recipe busybox-1.31.1-r0: task do_configure_ptest_base: Started
NOTE: recipe busybox-1.31.1-r0: task do_configure_ptest_base: Succeeded
NOTE: Running task 2000 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_compile)
NOTE: recipe busybox-1.31.1-r0: task do_compile: Started
NOTE: recipe busybox-1.31.1-r0: task do_compile: Succeeded
NOTE: Running task 2001 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_compile_ptest_base)
NOTE: recipe busybox-1.31.1-r0: task do_compile_ptest_base: Started
NOTE: recipe busybox-1.31.1-r0: task do_compile_ptest_base: Succeeded
NOTE: Running task 2002 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_install)
NOTE: recipe busybox-1.31.1-r0: task do_install: Started
NOTE: recipe busybox-1.31.1-r0: task do_install: Succeeded
NOTE: Running task 2003 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_install_ptest_base)
NOTE: recipe busybox-1.31.1-r0: task do_install_ptest_base: Started
NOTE: recipe busybox-1.31.1-r0: task do_install_ptest_base: Succeeded
NOTE: Running task 2004 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_package)
NOTE: Running task 2005 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_populate_sysroot)
NOTE: recipe busybox-1.31.1-r0: task do_populate_sysroot: Started
NOTE: recipe busybox-1.31.1-r0: task do_package: Started
NOTE: recipe busybox-1.31.1-r0: task do_populate_sysroot: Succeeded
NOTE: Task /home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_populate_sysroot unihash changed to 987a52c097b8226babd585e0f7680436b0101ef5e5a4bb0198dcf074587c84c5
NOTE: recipe busybox-1.31.1-r0: task do_package: Succeeded
NOTE: Running task 2006 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_packagedata)
NOTE: recipe busybox-1.31.1-r0: task do_packagedata: Started
NOTE: recipe busybox-1.31.1-r0: task do_packagedata: Succeeded
NOTE: Task /home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_packagedata unihash changed to fd29a1a7c2b98597690c4c578aa37a8e634e3c1c0f159768542608ea6e495fe8
NOTE: Setscene tasks completed
NOTE: Running task 2007 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_package_write_rpm)
NOTE: Running task 2008 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_package_qa)
NOTE: recipe busybox-1.31.1-r0: task do_package_write_rpm: Started
NOTE: recipe busybox-1.31.1-r0: task do_package_qa: Started
NOTE: recipe busybox-1.31.1-r0: task do_package_qa: Succeeded
NOTE: Task /home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_package_qa unihash changed to fc79955fe62c4f7532b9b7f94456a1494a1824ddd72fbdf616a7e15b53a2dabf
NOTE: recipe busybox-1.31.1-r0: task do_package_write_rpm: Succeeded
NOTE: Task /home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_package_write_rpm unihash changed to 6863e0dce6e818dc4b184dd54da73f4a17e892857221fefd1ec13e6b6941ac4f
NOTE: Running noexec task 2009 of 2009 (/home/rgutierrez/yocto-training/poky/meta/recipes-core/busybox/busybox_1.31.1.bb:do_build)
NOTE: Tasks Summary: Attempted 2009 tasks of which 1991 didn't need to be rerun and all succeeded.

Image Recipes

There are specific tasks to build images (e.g. core-image-minimal)

1. do_bootimg: This task creates a bootable image.
2. do_bundle_initramfs: This task creates a single image from a ramdisk image and a kernel.
3. do_rootfs: Creates the image's root filesystem.
4. do_testimage: First boots the image and then performs tests on it.