Built-in Optimization

Since the on-demand generation of block hash indexes for the slots will (depending on CPU and IO performance) add some time to the installation, they are cached in the RAUC data directory. Only if there is no cache or the cached information does not match the actual data, the hash index will be re-generated from the block device.

Especially since images often do not fill the entire space in a slot, one of the very frequently used hashes will be those for the zero-chunk. RAUC has a zero chunk in memory that it will use before any lookup mechanism takes place.

Adaptive Update Method tree-rsync-checksum

While the previously introduced adaptive method is applicable to block images only, the underlying approach can be used for directory-tree-based updates (as RAUC already supports in the form of tar archives), too.

Instead of generating a hash index for a block device on a chunk basis, for directory trees one can simply generate a hash index by calculating and storing one hash for each file. By doing this for the directory tree that comes with the update 'images' as well as for the directory trees of the currently active and inactive slots one again has a set of three hash index files.

When installing the update, we can again download the hash list describing the update directory tree and iterate over it while searching for a file with the same hash in the local slots first. Only files whose hash RAUC cannot find locally have to be downloaded from the remote bundle.

With rsync, there is already a tool available that can handle the checksum-based synchronization of files with using multiple sources as reference.

The only missing piece of the puzzle is the ability to store/cache the checksums so that they do not have to be re-calculated on each update.

There are two possible solutions for this out in the field as patches:

• saving and storing checksums in .rsyncsums files:

  https://github.com/WayneD/rsync-patches/blob/e8526f7ffa33e7b3a55f592133070b43d82b04d6/checksum-reading.diff

• saving and storing checksums in xattrs:

  https://github.com/WayneD/rsync-patches/blob/86a72ad7dcf34ca0ffb52fb36e291ef253894da4/checksum-xattr.diff

The actual copying will the be as similar as running

rsync --delete --copy-dest=<active-slot> <bundle>/rootfs.tree <inactive-slot>

Adaptive Update Method delta-image

Remember? In the beginning of this post we've argued against the use of conventional delta updates. Now, having adaptive updates introduced, we can rehabilitate them a bit.

The idea behind the adaptive method delta-image is quite simple: We create conventional delta images between old bundle versions and a new bundle version using some binary diff tool.

But, instead of using only one delta image, we can add several as additional artifacts to the bundle so that we gain benefit from both: If there is an optimized delta artifact matching a currently available bundle version, use it. If there is none, just perform a normal image update.

Measuring & Debugging

To check if adaptive updates are working, you should inspect the rauc service log, e.g. with:

journalctl -u rauc

when using systemd.

Look for lines noting 'hash index', like this one:

using existing hash index for target_slot from /data/slot.rootfs.1/hash-d60b4e16226f704249fc6bf4d99c9aae0be41706ac7dcee0e4aa1d53792c9577/block-hash-index
hash index (35223 chunks) does not cover complete data range (38400 chunks), ignoring the rest
building new hash index for active_slot with 38400 chunks
using existing hash index for source_image from /mnt/rauc/bundle/core-image-minimal-qemux86-64.ext4.block-hash-index

To see if adaptive updates are working and for evaluating how it performs on your system, you should have a look at the access stats that are printed:

access stats for zero chunk: count=8235 sum=8235.000 min=1.000 max=1.000 avg=1.000 recent-avg=1.000
access stats for target_slot_written (reusing source_image): count=25567 sum=204.000 min=0.000 max=1.000 avg=0.008 recent-avg=0.000
access stats for target_slot: count=25363 sum=1.000 min=0.000 max=1.000 avg=0.000 recent-avg=0.000
access stats for active_slot: count=25362 sum=25304.000 min=0.000 max=1.000 avg=0.998 recent-avg=0.969
access stats for source_image: count=58 sum=58.000 min=1.000 max=1.000 avg=1.000 recent-avg=1.000

You can ignore the min, max, avg, and recent-avg values since they are only emitted by the generic RAUC stats framework.

You see the four internally used block hash indexes being printed in the order they are accessed by RAUC as well as an additional zero chunk entry that represents the built-in access to the zero chunk hash optimization noted above.

The count refers to the number of times the corresponding index was accessed while the sum is the number of found chunks in this index.

In the example above 58 chunks were downloaded from the bundle, while 25304 were used from the active slot and 204+1 chunks were (re)used from the target slot.

Optimization Approaches

If, even for small rootfs changes, the resulting difference, i.e. the number of fetched chunks is quite high, these are some things one should have a look at:

• Ensure using a build system that has reproducible builds enabled.

• When using ext4, ensure your block size is 4096. You can check this with:

  dumpe2fs rootfs.ext4 | grep "Block size"
  

  To enforce block size for mkfs.ext4, add -b 4096 argument.

• Even if file system contents are (nearly) identical, the generated images may have significant differences. Have a look at https://reproducible-builds.org/docs/system-images/ for possible solutions to this.

• The bundle image chunks are accessed through the bundle SquashFS, thus with a large SquashFS block size, some unnecessary data may have to be downloaded, as each compressed SquashFS block needs to be downloaded completely.

  You can e.g. reduce the block size to 32kiB by adding --mksquashfs-args="-b 32k" to the rauc bundle call.