makefile.toml #30
Merged
makefile.toml #30
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Ben Cressey <bcressey@amazon.com>
We expect the lock file to be current and all crates to be fetched, so changes are unexpected and could trigger unnecessary builds. Signed-off-by: Ben Cressey <bcressey@amazon.com>
Signed-off-by: Ben Cressey <bcressey@amazon.com>
We want to clean up any existing build artifacts, and that now includes the crates built as a side effect of creating packages and images. Signed-off-by: Ben Cressey <bcressey@amazon.com>
Signed-off-by: Ben Cressey <bcressey@amazon.com>
Adds host-ctr (pending name), a go program that interacts with containerd runtime to pull images and start container tasks. It currently takes the image URI and container specifications from the command-line. It also able to gracefully terminate by cleaning up container snapshots and tasks.
All `go` invokations are now containerized This removes the go build dependency during the `fetch` step.
Signed-off-by: iliana destroyer of worlds <iweller@amazon.com>
This adds a host-containers service that asks the API for the defined host-containers and tells systemd to start/stop them to match their 'enabled' setting. The settings are written to an EnvironmentFile that's read by the new host-container@ service, a systemd templated service used for all host containers by way of a suffix like "host-container@admin". The existing container-specific thar-be-settings templated systemd unit files were removed in favor of this more general approach. Existing metadata for these was also removed, replaced with a single metadata entry that invokes the new tool. The existing host-containers package, which just contains the Go host-ctr source, was renamed to host-ctr.
Signed-off-by: iliana destroyer of worlds <iweller@amazon.com>
On build success create symlinks to the Thar output images, with a arch-flavor-version format. The version is relative to the latest git tag if available or the current date as a fallback. Old symlinks are cleaned up to avoid confusion. Signed-off-by: Samuel Mendoza-Jonas <samjonas@amazon.com>
This adds a `fetch-external-files` task allowing for explicitly performing a "whole-project fetch" on demand and as a dependency: ``` $ cargo make fetch $ cargo make fetch-external-files ``` buildsys is now supporting the front-loaded effort of pulling external sources so as to not fail further into builds. This takes advantage of the fact that the build.rs script is invoked for build with certain cargo make env variables in scope in order to appropriately short circuit. Signed-off-by: Jacob Vallejo <jakeev@amazon.com>
This adds flags to prevent issues on hosts without configured iptables or otherwise problematic networking.
Signed-off-by: iliana destroyer of worlds <iweller@amazon.com>
Add script to export HTML docs
This reverts commit 4f63ff62944c8bfe391e73b8a5d99288bd241c21, reversing changes made to 5c3aee62d5de6eaa714d104c8a33f733fccde485.
…al-files"" This reverts commit 745a4c80465569d39c496b3efe6d71333aa92fe3.
Create versioned symlinks to output images
Fix incremental with fetch
Newer versions of Docker have BuildKit integrated, and we already depend on Docker to provide some of our build tools. BuildKit's ability to copy files directly out of a build image is convenient, but we can work around that using Docker functionality, and eliminate the need to run another daemon for builds. Signed-off-by: Ben Cressey <bcressey@amazon.com>
The build step is causing Cargo to invalidate some portion of the context that is used to facilitate incremental builds. Until we can determine why and how to fix this issue the removal of the task from the dependency graph should allow for incremental builds to work properly. Signed-off-by: Jacob Vallejo <jakeev@amazon.com>
Signed-off-by: Ben Cressey <bcressey@amazon.com>
This continues to trigger a low rate of spurious rebuilds. Signed-off-by: Ben Cressey <bcressey@amazon.com>
build: remove direct support for fetching external files
Adds buildspec for running the test stage codebuild project that runs rust workspaces unit tests and go unit tests
Previously, doing a full build that respected the dependency graph would cause almost every package to be rebuilt when the kernel was updated, which happened at regular intervals. The cascade feature was added to allow a developer to opt out of the full dependency tracking in the interests of speeding up iterative builds. However, making use of it required two full builds to get cargo in sync, with no reliable way to predict whether either build would be quick. This limited its usefulness in practice. Now that the kernel is a leaf package, we will only rebuild the world when glibc changes, which we expect to happen less often. Signed-off-by: Ben Cressey <bcressey@amazon.com>
Add a `check-migrations` task that runs as part of `check`. It attempts to check for common problems with migrations. It might be nice to have it run during build tests. While it does not actually run migrations or check their logic, it can spot the kind of inconsistencies that are easily lost in reviews. Signed-off-by: Markus Boehme <markubo@amazon.com>
Signed-off-by: John McBride <jpmmcb@amazon.com>
Makefile: add check-migrations task to check for common problems
Signed-off-by: Ben Cressey <bcressey@amazon.com>
Ensure that at least one Secure Boot profile is always available, for cases where the variant has Secure Boot enabled. This uses the "local" version of the script, since locally generated keys cost nothing and are guaranteed to be available. This is similar to the locally generated keys used by default for TUF repositories, in that neither is suitable for long-term production use. The individual Secure Boot profile and the directory where profiles are stored can both be overridden, so that profiles can be stored in a different location, such as another Git repository. Profiles are checked for completeness, to allow the expected files to evolve over time, for example to add support for a new platform that expects EFI variables in a different format. Signed-off-by: Ben Cressey <bcressey@amazon.com>
`buildsys` needs to pass through files from the Secure Boot keys profile in order to sign artifacts during the variant build step. Those files might include an `aws-kms-pkcs11` configuration that uses KMS for signing, which requires network access to be enabled for variant builds. On an EC2 instance, credentials from IMDS will be used automatically by `aws-kms-pkcs11`, but otherwise they need to come from environment variables or an AWS CLI configuration file. Accordingly, `buildsys` now passes the most important of these AWS environment variables as additional secrets. Signed-off-by: Ben Cressey <bcressey@amazon.com>
If Secure Boot is enabled for the variant, AMIs should be registered with the UEFI boot mode along with the relevant EFI variable payload. Signed-off-by: Ben Cressey <bcressey@amazon.com>
Twoliter will be calling this script from within a container that has the host's docker socket mounted. It is necessary to match internal and external paths for any state that we wish to preserve on the host. Because the data we want to cache is at $GOPATH/pkg/go, it was necessary to create this as a subdirectory under .gomodcache and mount that so that we could point GOPATH to .gomodcache.
|
Do you mind describing your process here? Since the commits themselves aren't relevant. |
Oh yeah, I meant to do that. Updated. |
cargo make build-variant and build-package typically run several RPM builds concurrently, and those allow for their own levels of parallelism. Leaving this unrestricted would very easily lead to context switching overhead. On the other hand, the rest of the build commands are mostly focused on building a directory of rust sources. It shouldn't be a problem to let cargo use as many cores as it needs for those.
cbgbt
approved these changes
Aug 14, 2023
There was a problem hiding this comment.
History-preserving process looks fine and contents match the current develop branch in the mono repo. The embeded directory has a typo in its name; would be good to get this fixed before merging lest it becomes too entrenched to change. :-)
As for further development, is the expectation to keep making changes to Makefile.toml in both repositories for now?
For a short period of time, between the merging of this and bottlerocket-os/bottlerocket#3342, changes would need to be made in both places, or a patch would need to be applied here to match bottlerocket develop. After that, The Makefile will exist only here, and the one left behind in Bottlerocket will be a thin facade that will diverge over time. |
Thank you for catching this! Fixed https://github.com/bottlerocket-os/twoliter/compare/5e54155dde8defe169839d70e99ee9b4dda326a0..29844f2cb8f13733fe1873206df301f5ce4ae8fb |
markusboehme
approved these changes
Aug 15, 2023
bcressey
approved these changes
Aug 16, 2023
Signed-off-by: Matthew James Briggs <matthew.james.briggs@gmail.com>
|
Unnecessary force push to see if anything else has landed in Bottlerocket (it hasn't). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Related to #14
Description of changes:
Bring in the history of
Makefile.tomlProcedure:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.