design and implement a proper artifact format

[koehlma]

As of now, we ship updates in the form of system images that can also be directly flashed to a storage medium. While it is convenient to have only a single artifact, these images waste space and are also very inflexible when it comes to shipping updates. In particular, if we aim to eventually support delta updates, we need to develop a proper and more flexible artifact format. This issue tracks the progress of the design and implementation of an artifact format.

Design requirements

• R1: The format must support streaming updates.
• R2: The format should allow for random accesses.

Prior Art

• RAUC uses Squashfs images which are mounted by the update client via a network-backed block device using HTTP range queries: https://rauc.readthedocs.io/en/latest/reference.html#sec-ref-formats
• Mender uses tar archives.

While RAUC's approach supports random accesses, it does not allow for primitive streaming of artifacts without seeking. While Mender's approach allows streaming, it does not allow random accesses.

Design Proposal

We will be using Casync's catar file format as a basis as catar files support both streaming and random accesses. This will allow the update client to accept a stream while also giving us the flexibility to later use HTTP range queries or similar mechanisms to only retrieve certain parts of an artifact. In particular, this would also allow us to build artifacts containing multiple updates or updates in different formats, leaving it to the client which updates to install.

As with RAUC and Mender, a Rugpi update artifact contains a manifest describing the update. Ideally, this manifest would be the first file in the archive stream, followed by system images, Casync indices for the filesystem, or delta streams.

• Decide on the exact details of the artifact format.