feat: introduce use of kapp-ctrl/packaging #87
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cirocosta
commented
Sep 9, 2021
cirocosta
commented
Sep 9, 2021
cirocosta
force-pushed
the
packaging
branch
from
aa785a6 to
d0689ba
Compare
cirocosta
force-pushed
the
packaging
branch
8 times, most recently
from
ff602d6 to
1d07921
Compare
|
at this point, just missing updating the commit message with the latest details |
cirocosta
force-pushed
the
packaging
branch
2 times, most recently
from
936242f to
c7613f7
Compare
|
todo:
|
cirocosta
force-pushed
the
packaging
branch
from
c7613f7 to
6d62d9e
Compare
|
marking as draft once again to account for the installation based on the tarball |
cirocosta
force-pushed
the
packaging
branch
from
b1b19e1 to
4691c02
Compare
emmjohnson
reviewed
Sep 22, 2021
emmjohnson
approved these changes
Sep 22, 2021
added 23 commits
September 22, 2021 15:45
we used to have the generation of release notes (i.e., what'd become the body of the GitHub release) generated in a way that was suited only for CI - this commit switches to an approach where we need not to rely on anything CI specific as it can figure out the tag it's currently in via plain use of `git` commands, even though the versions to use for the notes generation _can_ be manually specified. example: generate release notes for the assets under `./release` (default) in a diff between the current tag and the previous tag (default), writing them to `./release/CHANGELOG.md` (default) ./hack/release-notes.sh example: generates release notes for assets under `/foo` in a diff between tag v0.0.4 and v0.0.1, writing the result to `/bar` ASSETS_DIR=/foo \ DESTINATION=/bar \ CURRENT_VERSION=v0.0.4 \ PREVIOUS_VERSION=v0.0.1 \ ./hack/release-notes.sh Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
this script allows us to publish a github release, be on CI or not. note that this does not generate any of the files related to the release - it assumes that the files are already there, and so it just makes use of github's cli[1] to submitting the release through its api. see `hack/release.sh` and `hack/release-notes.sh` for release assets generaiton. example: use the current tag as the release version (default), the `release/` directory as the place where `yaml` files to be released can be found (default), and `release/CHANGELOG.md` as the release notes body (default): ./hack/github-release.sh see `readonly` vars in the script for customization. [1]: https://github.com/cli/cli Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
this commit brings back the functionality that we had for the previous `hack/e2e.sh` script, but: - broken down into separate commands - by default, making use of an authenticated registry rather than not - removing all resource reservations from our dependencies - leveraging secretgen[1] for passing registry credential secrets to all namespaces - using `kapp-ctrl/Package*` as the way of installing cartographer with this setup we end up then replicating the setups of the real-world much more closely, letting us be aware of the tweaks necessary for working with private registries. by having the funcionality of e2e.sh broken down into different commands, it also becomes easier to bring a cluster locally and iterate on an example, of install _just_ a subset of the dependencies you want. for instance: 1. bring up a local cluster with a registry and base dependencies we'd expect to be there (kapp-ctrl,cert-manager,secretgen): ./hack/setup.sh cluster 2. install cartographer using local code, pushing that to the local registry ./hack/setup.sh cartographer 3. setup the extra dependencies our examples use (like kpack, knative, etc) ./hack/setup.sh example-dependencies 4. run the example end to end ./hack/setup.sh example [1]: https://github.com/vmware-tanzu/carvel-secretgen-controller Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
with the introduction of `hack/setup.sh`, where everything is local by default, there's no need for the `local-dev` scripts anymore. Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
with the aim at reducing the set of "installation sprawl", removing these targets and focusing on the single installation location (hack/setup.sh) we can instead just always use the commands from there. Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
- make use of `install-binaries` to install the binary dependencies - move to `setup.sh` as the mechanism for bringing the cluster etc up Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
- to make sure that we're really confident about what we shipped, run it through the same e2e tests that we run for commits/prs but with the release candidate that we generate before finializing the release Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
as the stamper is now capable of stamping objects out making use of `ytt`, we need to make it available inside the final container image. leveraging `kodata`, we introduce an extra layer that includes the binary as we want: ``` ... drwxrwxrwx 0:0 0 B ├── tmp drwxr-xr-x 0:0 1.6 MB ├─⊕ usr dr-xr-xr-x 0:0 13 MB └── var drwxr-xr-x 0:0 0 B ├── backups ... dr-xr-xr-x 0:0 13 MB ├── run dr-xr-xr-x 0:0 13 MB │ └── ko -r-xr-xr-x 0:0 5 B │ ├── .gitignore -r-xr-xr-x 0:0 13 MB │ └── ytt-linux-amd64 (!) drwxr-xr-x 0:0 0 B ├── spool drwxrwxrwx 0:0 0 B └── tmp ``` Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
given that release generation is now done throguh `./hack/release.sh` (either used directly or through `hack/setup.sh`), there's no need to have that under Makefile anymore Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
it's great that we can point at specific versions, but _nothing_ prevents those versions from being updated. with checksum verification we ensure that we're really getting what we expect (sure sure, the paranoid in me would want to pgp verify those from authors we trust ... but what we have here is certainly good enough). Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
kubebuilder didn't follow the same pattern of having all of the binaries landing on /usr/local/bin, which by default is open to our non-root user on ubuntu. by switching kubebuilder bins to that direcory, we have to let envtest know that that's where it can find those binaries (apparently, it doesn't search $PATH by default). Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
this way we can make sure that 1. we're getting what we expect 2. if we bump the version, we certainly include the right version Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
given that we don't yet have a public image registry that we can push the bundle to, by generating a bundle tarball that goes to `/release` we can not only preserve the ability for one to still install cartographer, but also incentivize people to always relocate[^1] [^1]: sure, once OSS, it's certainly useful to also have the 0-attrition `kubectl apply -f <release_url>` Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
given that we're now not pushing to dockerhub anymore (as we're still private) and we will be publishing the tarball of the whole bundle, we can make the release workflow _much_ closer Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
we were referring to a version from feb 11 2021, no reason to not use a newer one Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
this way we can more easily check out which versions we're installing by referring to that set of variables at the very beginning of the file. here I also made `example-dependencies` be more explicit about all the dependencies it's installing so that just by looking at `main` one can tell more precisely what goes on with each command. Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
given that we don't have the ability to just push our container images to dockerhub anymore, we have to adapt the flow to be "github"-friendly (aiming at OSS) while still preserving the properties of playing well with carvel-packaging, which is usually aimed at a more enterprise-ready workflow that involves relocation. Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
- previously, a release version would default to `0.0.0-dev` regardless of wheteher we were under a tag or not. - previously, the changeset computation would not produce the correct results if we were working out of a release-candidate (it should diff all the way down to the last non-rc). Signed-off-by: Ciro S. Costa <ciroscosta@vmware.com>
cirocosta
force-pushed
the
packaging
branch
from
9e0a50c to
d880c9e
Compare
|
force-pushed to resolve a conflict after rebasing w/ main |
tbr11
reviewed
Sep 22, 2021
| uses: actions/setup-go@v2 | ||
| with: | ||
| go-version: '~1.17.1' | ||
| with: {go-version: '~1.17.1'} |
There was a problem hiding this comment.
Should we switch to caret version matching? Go has committed to 1.x being compatible. go-version: '^1.17.1' ought to keep us on the latest minor and patch version.
Comment on lines
+20
to
+27
| readonly KO_VERSION=0.8.1 | ||
| readonly KO_CHECKSUM=9f004fa1c2b55ac765ec0c287ad0311a517a86299b7a633bc542f2fbbb3a4ea4 | ||
| readonly KUBERNETES_VERSION=1.19.2 | ||
| readonly KUBERNETES_CHECKSUM=fb13a93a800389029b06fcc74ab6a3b969ff74178252709a040e4756251739d2 | ||
| readonly KUTTL_VERSION=0.11.1 | ||
| readonly KUTTL_CHECKSUM=0fb13f8fbb6109803a06847a8ad3fae4fedc8cd159e2b0fd6c1a1d8737191e5f | ||
| readonly GH_VERSION=2.0.0 | ||
| readonly GH_CHECKSUM=20c2d1b1915a0ff154df453576d9e97aab709ad4b236ce8313435b8b96d31e5c |
There was a problem hiding this comment.
Can these be passed in and defaulted? I foresee pain if we have to repro a build from a particular commit and a dependency version got pulled.
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
the commits tell most of the story, but here's the tl;dr on this pr:
adds a
packagingdirectory where we place templates and extra yaml objects(overlays & kapp configurations) that allow us to ship Cartographer as a
carvel Package
refactors ./hack/ci/e2e.sh and other
cirelated scripts which were not justmeant to be run in CI to be more friendly with running locally, leaving to
cijust those that assume a CI environment (like,install-binaries).envrionment once and then
refactors the relase github actions workflow to be simply an automation on
top of what we could do locally, i.e., removes any assumptions of being run
from a CI environment.
at the end, we have a release directory (
gitignored) like so:with release notes such as
i'm not sure how useful
cartographer.yamlthere is now that we're just bundling though ... might actually get rid of it and going solely for bundle / from source for those that want to try it out usingkoby themselves ..todo:
bundle.tar