Skip to content

Commit

Permalink
Add toml-patch crate for improved inheritance (#1477)
Browse files Browse the repository at this point in the history
Right now, Hubris has a hard-coded inheritance system that allows for one thing:
adding new features to existing tasks.

The current system is not flexible enough for our current set of boards, and
we've ended up with a huge amount of duplication.  For Gimlet alone, we have
`rev-{b,c,d}{'',-dev,-lab,}`, i.e. 9 variations, of which 6 are mostly
copy-pasta.

This PR adds a more general form of TOML inheritance, implemented in a new
`toml-patch` crate.  For those familiar with the Yocto project, it's similar to
[`.bbappend`
files](https://docs.yoctoproject.org/1.6/dev-manual/dev-manual.html#using-bbappend-files):

- TOML files can declare that they inherit from one or more other files, and
  also contain local data
- Inheritance is processed in order (with local data being applied last)
- Basic data types (e.g. integers, strings) are replaced with new values
- Sequence data types (lists, tables, inline tables; including the root table)
  have new values appended to the end

Before this PR, we shipped both `app.toml` and `patches.toml` in the Hubris
archive, and Humility had special code to apply patches itself.  After this PR,
we flatten everything into a single canonical `app.toml` and only include that
in the archive.  This means that Humility doesn't have to know about the new
patching system at all; it just sees an `app.toml`.

The hard part about this was an unexpected issue with `toml_edit`: that crate
has a _global order_ for TOML tables, which controls the order in which they're
printed.  This means that to insert new tables at the end position, we have to
- Modify the original document to add gaps
- Modify the incoming document, shifting tables into the correct position
- _Then_, splice them together

It gets even tricker when lists of tables (which may contain sub-tables of their
own) are involved!

Anyways, that's all implemented in the `toml-patch` crate, along with some unit
tests.

Note that this PR does not actually deduplicate all of our existing manifests!
That's coming in a later PR, which will require careful testing to make sure it
doesn't break anything.  It does, however, update the style for `-lab` images.
  • Loading branch information
mkeeter committed Aug 7, 2023
1 parent 926418c commit 3c236dc
Show file tree
Hide file tree
Showing 12 changed files with 604 additions and 100 deletions.
17 changes: 17 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ hkdf = { version = "0.12", default-features = false }
hmac = { version = "0.12.1", default-features = false }
hubpack = { version = "0.1.2", default-features = false }
indexmap = { version = "1.4.0", default-features = false, features = ["serde-1"] }
indoc = { version = "2.0.3", default-features = false }
itertools = { version = "0.10.5", default-features = false }
lpc55-pac = { version = "0.4", default-features = false }
memchr = { version = "2.4", default-features = false }
Expand Down Expand Up @@ -106,6 +107,7 @@ stm32g0 = { version = "0.15.1", default-features = false }
strsim = { version = "0.10.0", default-features = false }
syn = { version = "1", default-features = false, features = ["derive", "parsing", "proc-macro"] }
toml = { version = "0.7", default-features = false, features = ["parse", "display"] }
toml_edit = { version = "0.19", default-features = false }
vcell = { version = "0.1.2", default-features = false }
walkdir = { version = "2.0.0", default-features = false }
zerocopy = { version = "0.6.1", default-features = false }
Expand Down
5 changes: 2 additions & 3 deletions app/gimlet/rev-b-lab.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
inherit = "rev-b-dev.toml"

[patches]
name = "gimlet-b-lab"
features.gimlet_seq = ["stay-in-a2"]
features.packrat = ["boot-kmdb"]
tasks.gimlet_seq.features = ["stay-in-a2"]
tasks.packrat.features = ["boot-kmdb"]
5 changes: 2 additions & 3 deletions app/gimlet/rev-c-lab.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
inherit = "rev-c-dev.toml"

[patches]
name = "gimlet-c-lab"
features.gimlet_seq = ["stay-in-a2"]
features.packrat = ["boot-kmdb"]
tasks.gimlet_seq.features = ["stay-in-a2"]
tasks.packrat.features = ["boot-kmdb"]
5 changes: 2 additions & 3 deletions app/gimlet/rev-d-lab.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
inherit = "rev-d-dev.toml"

[patches]
name = "gimlet-d-lab"
features.gimlet_seq = ["stay-in-a2"]
features.packrat = ["boot-kmdb"]
tasks.gimlet_seq.features = ["stay-in-a2"]
tasks.packrat.features = ["boot-kmdb"]
5 changes: 2 additions & 3 deletions app/sidecar/rev-b-lab.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
inherit = "rev-b-dev.toml"

[patches]
name = "sidecar-b-lab"
features.sequencer = ["stay-in-a2"]
features.ignition = ["always-transmit"]
tasks.sequencer.features = ["stay-in-a2"]
tasks.ignition.features = ["always-transmit"]
5 changes: 2 additions & 3 deletions app/sidecar/rev-c-lab.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
inherit = "rev-c-dev.toml"

[patches]
name = "sidecar-c-lab"
features.sequencer = ["stay-in-a2"]
features.ignition = ["always-transmit"]
tasks.sequencer.features = ["stay-in-a2"]
tasks.ignition.features = ["always-transmit"]
11 changes: 11 additions & 0 deletions build/toml-patch/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[package]
name = "toml-patch"
version = "0.1.0"
edition = "2021"

[dependencies]
anyhow.workspace = true
toml_edit.workspace = true

[dev-dependencies]
indoc.workspace = true
Loading

0 comments on commit 3c236dc

Please sign in to comment.