Device firmware update and hardware diagnostics using change root

Doctor Jekyll and Mister Hyde

Everytime I dig into Linux to learn about a some aspect I have heard about, but never used, I am always impressed by the level of flexibility and power the operating system designers enabled. In the case of change root, normally called chroot, Wikipedia indicates the feature was added to Unix in 1982 by none other than Bill Joy. The reason for adding the feature was to simplify testing new operating system installations.

I came across chroot in an effort to solve two different customer requirements:

• Use the exact same code image to update devices in the field that is used on the manufacturing line for new devices.
• Provide a way to test devices in the field using software functionality not normally in the customer's device.

The first requirement was to minimize the number of different software deployments that needed support. Basically if someone had trouble with a device, the first support step is to get them on the latest released codebase, then address the issue.

The second requiement came about by the desire to limit what software was on the device, since it ships in an easily hackable configuration - just telnet in. The goal was to allow advanced users to enhance what the product could do, while minimizing the number of ways they could do it wrong.

In both cases chroot turns out to be a clever way of meeting these two requirements. The rest of this documentation focuses of device firmware upgrade using chroot. You can see how the same approach can be used to have a diagnostics file system that is enabled via chroot.

Change root overview

From the man page: chroot() changes the root directory of the calling process to that specified in path. This directory will be used for pathnames beginning with /. The root directory is inherited by all children of the calling process. chroot() does not close open file descriptors, and such file descriptors may allow access to files outside the chroot tree.

Only files and directories that are open at the time the chroot() call is made can access the old root file system. After chroot() all new file system accesses can only access the file system passed as a parameter to the chroot system call. For this reason, the term chroot jail is used to indicate processes don't have the freedom to access the old root file system.

In addition to passing in the path to the new root file system, chroot() also takes the name of an application to run in the new root file system. When getting a chroot environment in place, I often pass in /bin/sh so I have a shell to look around the new file system to make sure everything I need is in place.

Change root and device firmware update

If the device hardware is setup to boot from SD card, and is using either an eMMC chip, internal (non-user removable) micro SD card, or even a user removable SD card, then a simple way to update the firmware is to completely rewrite the SD card. However, you can't be using the SD card when you are rewriting the SD card. We use change root so we can the unmount the file systems residing on the SD card, which allows us to then be able to rewrite the contents of the entire SD.

If a device is booting from NAND, this same approach should work as well, but wasn't tested at the time this document was written.

Making a new root file system

Before calling chroot, you need to have a new root file system in place. You can use a tmpfs based file system if you have sufficient RAM. The easist way to get the new file system in place is to grab what is needed from the current file system before doing the chroot. Here is an example shell script to get the applications and libraries needed for doing a device update in place in a tmpfs based file system. The new root file system is located in the old root file system directory at /tmp/chrootdir.

	mkdir -p /tmp/chrootdir/proc /tmp/chrootdir/sys /tmp/chrootdir/tmp /tmp/chrootdir/usr/local 
	mkdir -p /tmp/chrootdir/var  /tmp/chrootdir/home/root
	for d in bin sbin usr/bin usr/sbin usr/local/bin usr/local/sbin lib dev etc ; do 
		cp -R /$d /tmp/chrootdir/$d
	done

Freeing up current file system

Before doing the chroot, all applications that have open files on the root file system need to be terminated. Also we can unmount the other file systems so we can remount them after the chroot.

	killall lighttpd inetd dropbear # whatever apps that are running

	umount /dev/pts
	umount /var
	umount /sys
	umount /proc

Performing change root

At this point, we are ready to switch to the new root file system we have put in place in tmpfs. Assuming we have an application called /usr/local/bin/do-update in the new root file system, we can switch from using the current file system to the new file system and start the do-update application:

chroot /tmp/chrootdir /usr/local/bin/do-update

Remonting after change root

Next remount the file systems so they are attached to the new file system

	mount /proc
	mount /sys
	mount /var
	mkdir -p /dev/pts
	mount -t devpts -ogid=5,mode=620 devpts /dev/pts

Rewrite the SD card or NAND chip

The logic in the do-update application can now overwrite the SD card. A simple example way using busybox dd command is:

tar -xOf $UPDATE_FILE sdcard.img.xz | xzcat | dd bs=67174400 of=/dev/mmcblk0

where $UPDATE_FILE is a tarball that contains a xz compressed SD card image file and the SD card shows up in the chroot file system at /dev/mmcblk0.