Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

import tools history #37

Merged
merged 552 commits into from
Sep 14, 2023
Merged

import tools history #37

merged 552 commits into from
Sep 14, 2023

Conversation

webern
Copy link
Contributor

@webern webern commented Aug 19, 2023

Note: As long as build and test work, then I may merge this without the clippy fixes and get those minor fixes a into follow-up PR.

Issue #, if available:

Part of #14
Replaced #31

Description of changes:

Move the tools history from the Bottlerocket monorepo to this repo like so:

#!/usr/bin/env bash

#where=$REPOS/tools-import && $REPOS/notes/ootb/import-tools.sh $where && cd $where/twoliter
#git push -u deleteme-tools-import:tools-$(date '+%Y%m%d_%H%M%S')
set -e

cd /tmp
DIR="${1:=/tmp/bottlerocket-tools}"
BR=$DIR/bottlerocket
TL=$DIR/twoliter

BOTTLEROCKET_BRANCH=deleteme-tools-export
TWOLITER_IMPORT_BRANCH=deleteme-tools-import
TWOLITER_FINAL_BRANCH=tools

rm -rf $DIR

git clone git@github.com:bottlerocket-os/bottlerocket.git $BR
# TODO change remotes

git clone git@github.com:bottlerocket-os/twoliter.git $TL
cd $TL
git remote remove origin
git remote add origin git@github.com:webern/twoliter.git
git remote add BECAREFUL_UPSTREAM git@github.com:bottlerocket-os/twoliter.git

cd $BR
git checkout -b $BOTTLEROCKET_BRANCH
git fast-export $BOTTLEROCKET_BRANCH -- \
  sources/generate-readme \
  sources/bottlerocket-variant \
  sources/parse-datetime \
  sources/updater/update_metadata \
	tools/buildsys \
	tools/infrasys \
	tools/pubsys \
	tools/pubsys-config \
	tools/pubsys-setup \
	tools/testsys \
	tools/testsys-config \
	tools/Cargo.lock \
	tools/Cargo.toml \
	tools/deny.toml \
	tools/docker-go \
	tools/partyplanner \
	tools/rpm2img \
	tools/rpm2kmodkit \
	tools/rpm2migrations > $DIR/export.fi

cd $TL
git switch --orphan $BOTTLEROCKET_BRANCH
git fast-import < $DIR/export.fi
git checkout develop
git checkout -b $TWOLITER_IMPORT_BRANCH
git merge --signoff --allow-unrelated-histories -m'merge tools history' $BOTTLEROCKET_BRANCH
mkdir -p twoliter/embedded

mv sources/bottlerocket-variant tools/
mv sources/generate-readme tools/
mv sources/parse-datetime tools/
mv sources/updater/update_metadata tools/update-metadata
mv tools/Cargo.lock .
mv tools/deny.toml .
mv tools/docker-go twoliter/embedded/docker-go
mv tools/partyplanner ./twoliter/embedded/partyplanner
mv tools/rpm2img ./twoliter/embedded/rpm2img
mv tools/rpm2kmodkit ./twoliter/embedded/rpm2kmodkit
mv tools/rpm2migrations  ./twoliter/embedded/rpm2migrations

git add --all
git commit -m'move imported tools into desired directories'
rm -rf sources/

rm tools/Cargo.toml
git add --all
git commit -m'remove old tools workspace Cargo.toml'

cat <<- "EOF" > Cargo.toml
[workspace]
resolver = "2"

members = [
    "tools/bottlerocket-variant",
    "tools/buildsys",
    "tools/generate-readme",
    "tools/infrasys",
    "tools/parse-datetime",
    "tools/pubsys",
    "tools/pubsys-config",
    "tools/pubsys-setup",
    "tools/testsys",
    "tools/testsys-config",
    "tools/update-metadata",
    "twoliter",
]
EOF

sed -i 's|../../sources/bottlerocket-variant|../bottlerocket-variant|' tools/buildsys/Cargo.toml
sed -i 's|../../sources/parse-datetime|../parse-datetime|' tools/pubsys-config/Cargo.toml
sed -i 's|../../sources/parse-datetime|../parse-datetime|' tools/pubsys/Cargo.toml

sed -i 's|../../sources/updater/update_metadata|../update-metadata|' tools/pubsys/Cargo.toml
sed -i 's|update_metadata|update-metadata|' tools/pubsys/Cargo.toml

sed -i 's|../../sources/bottlerocket-variant|../bottlerocket-variant|' tools/testsys-config/Cargo.toml
sed -i 's|../../sources/bottlerocket-variant|../bottlerocket-variant|' tools/testsys/Cargo.toml
sed -i 's|name = "update_metadata"|name = "update-metadata"|' tools/update-metadata/Cargo.toml
sed -i 's|../../parse-datetime|../parse-datetime|' tools/update-metadata/Cargo.toml

# chop off the unnecessary [lib] section
sed -i '22,$d' tools/update-metadata/Cargo.toml

cargo test

git add --all
git commit -m'add tools to twoliter workspace'

git push origin -u "$TWOLITER_IMPORT_BRANCH":"tools-$(date '+%Y%m%d_%H%M%S')"

sed -i 's|let mut image = images.get_mut|let image = images.get_mut|' tools/pubsys/src/aws/publish_ami/mod.rs
cargo clippy -- -D warnings
git add --all
git commit -m'pubsys: clippy fix'

git checkout develop
git checkout -b $TWOLITER_FINAL_BRANCH
git merge --signoff --allow-unrelated-histories $TWOLITER_IMPORT_BRANCH
git push -fu BECAREFUL_UPSTREAM $TWOLITER_FINAL_BRANCH

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

cbgbt and others added 30 commits August 12, 2022 16:47
fix clippy warnings for tools and run lints for pull requests
Clippy warnings about a redundant closure. The closure just makes a call
to a function that would take the passed argument anyway, so we should
remove the closure and just provide the function directly.

Signed-off-by: Sean McGinnis <stmcg@amazon.com>
Parse the variant tuple into its components with the standard module
for that purpose, and pass each component into the build environment,
to avoid the need for ad-hoc parsing in spec files.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
Adds support for ecs variants testing. Everything works the same way as aws-k8s variants except ECS clusters are created instead of EKS, and there is not supported conformance testing for ecs variants.
Bumps [iana-time-zone](https://github.com/strawlab/iana-time-zone) from 0.1.44 to 0.1.46.
- [Release notes](https://github.com/strawlab/iana-time-zone/releases)
- [Changelog](https://github.com/strawlab/iana-time-zone/blob/main/CHANGELOG.md)
- [Commits](strawlab/iana-time-zone@0.1.44...v0.1.46)

---
updated-dependencies:
- dependency-name: iana-time-zone
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Using the `bundle-*` keys on
`packag.metadata.build-package.external-files`, buildsys can vendor Go
dependencies into a new archive (leaving the prestine upstream archive
intact). Both archives may then be used to build the package in an
isolated environment.

toosl/docker-go now also retrieves the `GOSUMDB` environment variable
from the host.

Co-authored-by: John McBride <jpmmcb@amazon.com>
Co-authored-by: Samuel Karp <skarp@amazon.com>
Signed-off-by: John McBride <jpmmcb@amazon.com>
buildsys: extend `external-files` to vendor go modules
The previous definition of variant sensitivity treated any change to
the variant as requiring a rebuild.

However, packages might depend on only one component of the variant
tuple. For example, the kernel package might conditionally enable
some modules for the "aws" platform, and a different set for the
"metal" platform.

To prevent unnecessary rebuilds, packages are now allowed to specify
which component, or group of components, they depend upon.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
This allows package specs to reference particular components of the
variant, without duplicating the parsing logic.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
This allows RPM spec files to conditionalize parts of the build in an
idiomatic way, using `with` tests rather than string comparisons.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
The `bundled-output-path` setting for external files had a small error
when processing the output bundle for Go project dependencies where it
would always use the default value, even when a different value was
provided. This fixes the name of the variable used so it uses the
provided name instead. This already handles setting this to the default
value if the package maintainer has not provided an overridden value.

Signed-off-by: Sean McGinnis <stmcg@amazon.com>
buildsys: Use bundle name setting for output
We've rotated signing keys for TUF and have deployed a new root.json for interacting with Bottlerocket's TUF repositories.
This updates all the checksums for the new root.json in the lookaside cache.
docs: update checksum for new 4.root.json
…iana-time-zone-0.1.46

build(deps): bump iana-time-zone from 0.1.44 to 0.1.46 in /tools
Signed-off-by: Ben Cressey <bcressey@amazon.com>
Otherwise `cargo` will detect that the file is newer than the output
log, and always rebuild.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
With the 1.63.0 versions of rustc/cargo, there is a clippy error due to
`return` being used when it is not needed. This removes the return
statement to make the linter happy.

Signed-off-by: Sean McGinnis <stmcg@amazon.com>
Fedora 36 automatically sets certain flags during RPM builds:
  https://fedoraproject.org/wiki/Changes/SetBuildFlagsBuildCheck

Since the `set_cross_build_flags` macro will use these flags if they
are specified in the environment, this causes the expected flags for
cross-compilation to be ignored, leading to build failures.

It's necessary to undefine the macro to suppress the new behavior.
Unfortunately, that can't be done through the usual macro inheritance
approach, because macros cannot be undefined, only reset. Therefore
the `rpmbuild` invocation is modified to remove the definition.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
Fedora 36 moved the RPM database to /usr:
  https://fedoraproject.org/wiki/Changes/RelocateRPMToUsr

For Bottlerocket, /usr should be a symlink to the sysroot directory.
If the RPM database is stored in /usr, then it will be created prior
to installing any Bottlerocket packages, and create a conflict with
the intended symlink.

Ordinarily this would be changed in the shared macros, but doing that
could confuse `rpm` during the package build step, since the SDK's
rpm database is still in `/usr` and package builds require a mix of
host and target packages to be installed.

Instead the definition is overridden just before image creation, when
host packages are no longer needed and the target packages have not
yet been installed.

Signed-off-by: Ben Cressey <bcressey@amazon.com>
Ran 'cargo update' in sources workspace and updated license check tomls.
dependabot bot and others added 8 commits August 30, 2023 19:58
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.11.19 to 0.11.20.
- [Release notes](https://github.com/seanmonstar/reqwest/releases)
- [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md)
- [Commits](seanmonstar/reqwest@v0.11.19...v0.11.20)

---
updated-dependencies:
- dependency-name: reqwest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [serde_yaml](https://github.com/dtolnay/serde-yaml) from 0.8.26 to 0.9.21.
- [Release notes](https://github.com/dtolnay/serde-yaml/releases)
- [Commits](dtolnay/serde-yaml@0.8.26...0.9.21)

---
updated-dependencies:
- dependency-name: serde_yaml
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
This gets rid of a warning that comes with rust 1.72.0. This does not
impact any of our workspaces since we've always been defaulted to using
resolver version 1. We can't use resolver v2 yet since that breaks the
'cargo make build-package' task.

See rust-lang/cargo#10112 for more details.
…serde_yaml-0.9.21

build(deps): bump serde_yaml from 0.8.26 to 0.9.21 in /tools
…reqwest-0.11.20

build(deps): bump reqwest from 0.11.19 to 0.11.20 in /tools
…assert-json-diff-2.0.2

build(deps): bump assert-json-diff from 1.1.0 to 2.0.2 in /tools
Fix Clippy warnings in the tools workspace.
dependabot bot and others added 4 commits September 11, 2023 18:47
Bumps [webpki](https://github.com/briansmith/webpki) from 0.22.0 to 0.22.1.
- [Commits](https://github.com/briansmith/webpki/commits)

---
updated-dependencies:
- dependency-name: webpki
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
XFS reads information from the device on mkfs. This results in an
optimized filesystem for the particular situation. Boot time is slightly
better as well since the IO from mkfs is less than growfs.

Having a prepared XFS partition with no data can result in similar
problems as ac0cacc where data read in from an encrypted EBS volume
returns random garbage which makes the filesystem appear corrupted.
Having no filesystem sidesteps this problem.

Signed-off-by: Matthew Yeazel <yeazelm@amazon.com>
rpm2img: create XFS partition on boot
Previously all migration testing required a bottlerocket crd to be
present, but since EKS Anywhere is used to provision machines, the
bottlerocket crd is not created for metal. This means the migration
agent needs to use the bottlerocket crd for instance ids if it's
available and if not, rely on the cluster crd to get the instance ids.
ecpullen and others added 8 commits September 13, 2023 16:49
This adds the plumbing necessary to override the EKS service endpoint
for the EKS cluster agent when it queries cluster metadata for
populating information necessary to launch nodes into the cluster.
It's more likely for a test cluster to at least have one or more public
subnets than it is for it to have at least one or more private subnet.
Signed-off-by: Matthew James Briggs <matthew.james.briggs@gmail.com>
@webern
Copy link
Contributor Author

webern commented Sep 14, 2023

@webern webern merged commit 7da74bd into develop Sep 14, 2023
1 check passed
@webern webern deleted the tools branch September 14, 2023 00:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.