Add pubsys: make Bottlerocket repos using cargo make repo

Simplify repo creation and tie it into the existing `cargo make` build system
using a new `repo` target.  This is intended to replace the common pattern of
calls to tuftool and updata - either creating a repo or extending an existing
one, adding the latest built artifacts, then updating the manifest to match.

**Usage:**
`cargo make repo` depends on the `build` target, so if your goal is building an
update for a repo you don't have to separately build it first.  It uses the
same `BUILDSYS_VARIANT` and `BUILDSYS_ARCH` variables to determine the metadata
for the update added to the repo.

**Requirements:**
* An Infra.toml file, based on Infra.toml.example, listing paths to keys,
  existing repos, etc.
* Release.toml updated with a new version

**Optional further configuration:**
* Repo expiration policy - by default, uses a policy file with 2 week target
  and snapshot expiration and 1 week timestamp expiration.
* Wave policy - same policy files you give to updata today; defaults to
  "default" wave policy.
* Release start time - when waves start and when expiration starts counting
  down; defaults to now.
* Can select which named repo and signing key to use from Infra.toml.

**Design decisions:**
* Built repo metadata is written to a directory like
  /build/repos/bottlerocket-0.4.1-5880e5d/aws-k8s-1.15/x86_64 so that you can
  prepare repos for multiple releases in parallel.  Targets are written to
  a shared directory like /build/repos/bottlerocket-0.4.1-5880e5d/targets -
  they're unique across variants and arches so there's no conflict.  The
  directory structure as a whole can be synced to your final repo
  location; it's the structure expected by Bottlerocket and updog.
* buildsys uses environment variables set by cargo-make; we opted instead for
  more standard arg parsing.  It seems more likely that someone would use
  pubsys separately from cargo-make, and pubsys has more input information, so
  arg parsing was clearer.
* cargo-make environment variable expansion is done in phases, and you can't
  refer to a variable defined in the same section if you intend to let the user
  override the earlier variable on the command line.  If you do, the variable
  won't expand, as seen in
  bottlerocket-os/bottlerocket#963.  Because of this,
  until we figure out a better strategy, a couple of variables can't be
  overridden - the path to Release.toml (which we made a variable in this
  change) and the repo output directory.

Co-authored-by: Zac Mrowicki <mrowicki@amazon.com>
Co-authored-by: Tom Kirchner <tjk@amazon.com>

sources/updater/update_metadata/Cargo.toml

@@ -18,6 +18,7 @@ serde = { version = "1.0.100", features = ["derive"] }
 serde_json = "1.0.40"
 serde_plain = "0.3.0"
 snafu = "0.6.0"
+toml = "0.5"
 tough = "0.7.1"
 
 [lib]

sources/updater/update_metadata/src/error.rs

@@ -78,8 +78,15 @@ pub enum Error {
         backtrace: Backtrace,
     },
 
-    #[snafu(display("Failed to write manifest file {}: {}", path.display(), source))]
-    ManifestWrite {
+    #[snafu(display("Failed to read '{}': {}", path.display(), source))]
+    FileRead {
+        path: PathBuf,
+        source: std::io::Error,
+        backtrace: Backtrace,
+    },
+
+    #[snafu(display("Failed to write '{}': {}", path.display(), source))]
+    FileWrite {
         path: PathBuf,
         source: std::io::Error,
         backtrace: Backtrace,
@@ -91,6 +98,12 @@ pub enum Error {
         backtrace: Backtrace,
     },
 
+    #[snafu(display("Invalid TOML in '{}': {}", path.display(), source))]
+    InvalidToml {
+        path: PathBuf,
+        source: toml::de::Error,
+    },
+
     #[snafu(display(
         "Migration {} given for {} but name implies it is for {}",
         name,

sources/updater/update_metadata/src/lib.rs

@@ -58,6 +58,18 @@ pub struct UpdateWaves {
     pub waves: Vec<UpdateWave>,
 }
 
+impl UpdateWaves {
+    /// Deserializes an UpdateWaves from a given path
+    pub fn from_path<P>(path: P) -> Result<Self>
+    where
+        P: AsRef<Path>,
+    {
+        let path = path.as_ref();
+        let wave_data = fs::read_to_string(path).context(error::FileRead { path })?;
+        toml::from_str(&wave_data).context(error::InvalidToml { path })
+    }
+}
+
 #[derive(Debug, Serialize, Deserialize)]
 pub struct UpdateWave {
     pub start_after: String,
@@ -101,14 +113,26 @@ pub struct Release {
     pub migrations: BTreeMap<(Version, Version), Vec<String>>,
 }
 
+impl Release {
+    /// Deserializes a Release from a given path
+    pub fn from_path<P>(path: P) -> Result<Self>
+    where
+        P: AsRef<Path>,
+    {
+        let path = path.as_ref();
+        let release_data = fs::read_to_string(path).context(error::FileRead { path })?;
+        toml::from_str(&release_data).context(error::InvalidToml { path })
+    }
+}
+
 pub fn load_file(path: &Path) -> Result<Manifest> {
-    let file = File::open(path).context(error::ManifestRead { path })?;
+    let file = File::open(path).context(error::FileRead { path })?;
     serde_json::from_reader(file).context(error::ManifestParse)
 }
 
 pub fn write_file(path: &Path, manifest: &Manifest) -> Result<()> {
     let manifest = serde_json::to_string_pretty(&manifest).context(error::UpdateSerialize)?;
-    fs::write(path, &manifest).context(error::ManifestWrite { path })?;
+    fs::write(path, &manifest).context(error::FileWrite { path })?;
     Ok(())
 }