From d16f1a969a6bf173917e38a8ee70e32907448a5a Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 30 Jun 2021 17:34:30 -0400 Subject: [PATCH 01/19] Add RFC: Remove Stacks & Mixins Signed-off-by: Stephen Levine --- text/0000-remove-stacks-mixins.md | 83 +++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100644 text/0000-remove-stacks-mixins.md diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md new file mode 100644 index 000000000..38b2726d7 --- /dev/null +++ b/text/0000-remove-stacks-mixins.md @@ -0,0 +1,83 @@ +# Meta +[meta]: #meta +- Name: Remove Stacks & Mixins +- Start Date: 2021-06-30 +- Author(s): sclevine +- RFC Pull Request: (leave blank) +- CNB Pull Request: (leave blank) +- CNB Issue: (leave blank) +- Supersedes: [RFC0069](https://github.com/buildpacks/rfcs/blob/main/text/0069-stack-buildpacks.md), [RFC#167](https://github.com/buildpacks/rfcs/pull/167), many others + +# Summary +[summary]: #summary + +*NOTE: This proposal is part of a larger initiative to reduce complexity originally outlined in https://github.com/buildpacks/rfcs/pull/167* + +This RFC proposes that we remove the "stack" and "mixin" concepts from the project and replace them with existing constructs in the container image ecosystem such as base images, Dockerfiles, and OS packages. + +# Motivation +[motivation]: #motivation + +The "stack" and "mixin" concepts add unnecessary complexity to the project and make it difficult for new users and contributors to understand how buildpacks work. Compatibility guarantees that are strongly enforced by the stack contract could be replaced with metadata validations and warnings. + +# What it is +[what-it-is]: #what-it-is + +Summary of changes: +- Replace mixins with a [CycloneDX](https://cyclonedx.org)-formatted list of packages. +- Replace stack metadata (including stack IDs) with canonical OS metadata. + +# How it Works +[how-it-works]: #how-it-works + +## Base Image Metadata + +Instead of a stack ID, runtime and build-time base images are labeled with the following canonicalized metadata: +- OS (e.g., "linux", `$GOOS`) +- Architecture (e.g., "x86_64", `$GOARCH`) +- Distribution (optional) (e.g., "ubuntu", `$ID`) +- Version (optional) (e.g., "18.04", `$VERSION_ID`) + +For Linux-based images, each field should be canonicalized against values specified in `/etc/os-release` (`$ID` and `$VERSION_ID`). + +The `stacks` list in `buildpack.toml` is replaced by a `platforms` list, where each entry corresponds to a different buildpack image that is exported into a [manifest index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). Each entry may contain multiple valid values for Distribution and/or Version, but only a single OS and Architecture. Each entry may also contain a list of package names (as PURL URLs without versions or qualifiers) that specify detect-time and build-time (but not runtime) OS package dependencies. Buildpacks may express runtime OS package dependencies during detection (see "Runtime Base Image Requirements" below). + +App image builds fail if the build image and selected run image have mismatched metadata. We may consider introducing a flag to skip this validation. + +When an app image is rebased, `pack rebase` will fail if the new run image and previous run image have mismatched metadata. This check may be skipped for Distribution and Version by passing a new `--force` flag to `pack rebase`. + +## Mixins + +The mixins label on each base image is replaced by a layer in each base image containing a single file consisting of a [CycloneDX](https://cyclonedx.org)-formatted list of packages. Each package entry has a [PURL](https://github.com/package-url/purl-spec)-formatted ID that uniquely identifies the package. + +### Validations + +Buildpack base image metadata and packages specified in `buildpack.toml`'s `platforms` list are validated against the runtime and build-time base images. + +Runtime and build-time base image packages are no longer validated against each other. + +When an app image is rebased, `pack rebase` will fail if packages are removed from the new runtime base image. This check may be skipped by passing a new `--force` flag to `pack rebase`. + +## Runtime Base Image Requirements + +Buildpacks may specify a list of package names (as PURL URLs without versions or qualifiers) in a `packages` table in the build plan. The build will fail if these packages are not present in the runtime base image. + +# Drawbacks +[drawbacks]: #drawbacks + +- Involves breaking changes. + +# Alternatives +[alternatives]: #alternatives + +- Keep stacks. + +# Unresolved Questions +[unresolved-questions]: #unresolved-questions + +- Should packages be determined during the detect or build phase? Opinion: detect phase, so that (in a later RFC), a runtime base image's app-specified Dockerfiles may by applied in parallel to the buildpack build process. + +# Spec. Changes (OPTIONAL) +[spec-changes]: #spec-changes + +This RFC requires extensive changes to all specifications. From 1b55b449039a01bbb13862813a1cd4863accde46 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 30 Jun 2021 22:44:07 -0400 Subject: [PATCH 02/19] RFC: Remove Stack & Mixins: Add platform table example Signed-off-by: Stephen Levine --- text/0000-remove-stacks-mixins.md | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index 38b2726d7..1534a36da 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -40,12 +40,39 @@ Instead of a stack ID, runtime and build-time base images are labeled with the f For Linux-based images, each field should be canonicalized against values specified in `/etc/os-release` (`$ID` and `$VERSION_ID`). -The `stacks` list in `buildpack.toml` is replaced by a `platforms` list, where each entry corresponds to a different buildpack image that is exported into a [manifest index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). Each entry may contain multiple valid values for Distribution and/or Version, but only a single OS and Architecture. Each entry may also contain a list of package names (as PURL URLs without versions or qualifiers) that specify detect-time and build-time (but not runtime) OS package dependencies. Buildpacks may express runtime OS package dependencies during detection (see "Runtime Base Image Requirements" below). +The `stacks` list in `buildpack.toml` is replaced by a `platforms` list, where each entry corresponds to a different buildpack image that is exported into a [manifest index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). Each entry may contain multiple valid values for Distribution and/or Version, but only a single OS and Architecture. Each entry may also contain a list of package names (as PURL URLs without versions or qualifiers) that specify detect-time and build-time (but not runtime) OS package dependencies. Packages may be specified for all Versions of a distribution, or for a specific Version. Buildpacks may express **runtime** OS package dependencies during detection (see "Runtime Base Image Requirements" below). -App image builds fail if the build image and selected run image have mismatched metadata. We may consider introducing a flag to skip this validation. +App image builds fail if the build image and selected run image have mismatched metadata. We may introduce flags or additional labels to skip this validation (e.g., for cross-compilation or minimal runtime base images). An image without a specified Distribution is compatible with images specifying any Distribution. An image specifying a Distribution without a Version is compatible with images specifying any Versions of that Distribution. When an app image is rebased, `pack rebase` will fail if the new run image and previous run image have mismatched metadata. This check may be skipped for Distribution and Version by passing a new `--force` flag to `pack rebase`. +#### Example: buildpack.toml `platforms` table + +```toml +[[platforms]] +os = "linux" +arch = "x86_64" +[[platforms.distros]] +name = "ubuntu" +[[platforms.distros.versions]] +version = "18.04" +packages = ["pkg:deb/ubuntu/curl"] +[[platforms.distros.versions]] +version = "20.04" +packages = ["pkg:deb/ubuntu/curl2"] + +[[platforms]] +os = "linux" +arch = "x86_64" +[[platforms.distros]] +name = "ubuntu" +packages = ["pkg:deb/ubuntu/curl"] +[[platforms.distros.versions]] +version = "14.04" +[[platforms.distros.versions]] +version = "16.04" +``` + ## Mixins The mixins label on each base image is replaced by a layer in each base image containing a single file consisting of a [CycloneDX](https://cyclonedx.org)-formatted list of packages. Each package entry has a [PURL](https://github.com/package-url/purl-spec)-formatted ID that uniquely identifies the package. From 5e7d9ff5175c69ea1d08d212709e52d0a139ed1c Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Thu, 1 Jul 2021 12:23:49 -0400 Subject: [PATCH 03/19] Add alternatives Signed-off-by: Stephen Levine Co-authored-by: Emily Casey --- text/0000-remove-stacks-mixins.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index 1534a36da..93fb1025f 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -98,6 +98,8 @@ Buildpacks may specify a list of package names (as PURL URLs without versions or [alternatives]: #alternatives - Keep stacks. +- Make the above changes but keep some existing terminology: stacks, build-image, run-image. +- Continue to allow for static definition of runtime package requirements in `buildpack.toml`. # Unresolved Questions [unresolved-questions]: #unresolved-questions From df7d7150e74b660f7f384f6029e5fe18b6c9f672 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Thu, 1 Jul 2021 12:24:39 -0400 Subject: [PATCH 04/19] Add unanswered question Signed-off-by: Stephen Levine Co-authored-by: Emily Casey --- text/0000-remove-stacks-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index 93fb1025f..897041401 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -105,7 +105,7 @@ Buildpacks may specify a list of package names (as PURL URLs without versions or [unresolved-questions]: #unresolved-questions - Should packages be determined during the detect or build phase? Opinion: detect phase, so that (in a later RFC), a runtime base image's app-specified Dockerfiles may by applied in parallel to the buildpack build process. - +- How will migration work? Can we make new base images compatible with older buildpacks? Can we make newer buildpacks compatible with older stacks? # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From 02cb22a796e2124a1b6d3a0ac4910cf6e2d67704 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 14 Jul 2021 22:38:46 -0400 Subject: [PATCH 05/19] RFC: Remove stacks & mixins: respond to feedback Signed-off-by: Stephen Levine --- text/0000-remove-stacks-mixins.md | 53 +++++++++++++++++++++++-------- 1 file changed, 40 insertions(+), 13 deletions(-) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index 897041401..f57d8a436 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -38,38 +38,39 @@ Instead of a stack ID, runtime and build-time base images are labeled with the f - Distribution (optional) (e.g., "ubuntu", `$ID`) - Version (optional) (e.g., "18.04", `$VERSION_ID`) +OS and Architecture must be valid identifiers as defined [here](https://golang.org/doc/install/source#environment). For Linux-based images, each field should be canonicalized against values specified in `/etc/os-release` (`$ID` and `$VERSION_ID`). -The `stacks` list in `buildpack.toml` is replaced by a `platforms` list, where each entry corresponds to a different buildpack image that is exported into a [manifest index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). Each entry may contain multiple valid values for Distribution and/or Version, but only a single OS and Architecture. Each entry may also contain a list of package names (as PURL URLs without versions or qualifiers) that specify detect-time and build-time (but not runtime) OS package dependencies. Packages may be specified for all Versions of a distribution, or for a specific Version. Buildpacks may express **runtime** OS package dependencies during detection (see "Runtime Base Image Requirements" below). +The `stacks` list in `buildpack.toml` is replaced by a `targets` list, where each entry corresponds to a different buildpack image that is exported into a [manifest index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). Each entry may contain multiple valid values for Distribution and/or Version, but only a single OS and Architecture. Each entry may also contain a list of package names (as PURL URLs without versions or qualifiers) that specify detect-time and build-time (but not runtime) OS package dependencies. Packages may be specified for all Versions of a distribution, or for a specific Version. Buildpacks may express **runtime** OS package dependencies during detection (see "Runtime Base Image Requirements" below). App image builds fail if the build image and selected run image have mismatched metadata. We may introduce flags or additional labels to skip this validation (e.g., for cross-compilation or minimal runtime base images). An image without a specified Distribution is compatible with images specifying any Distribution. An image specifying a Distribution without a Version is compatible with images specifying any Versions of that Distribution. When an app image is rebased, `pack rebase` will fail if the new run image and previous run image have mismatched metadata. This check may be skipped for Distribution and Version by passing a new `--force` flag to `pack rebase`. -#### Example: buildpack.toml `platforms` table +#### Example: buildpack.toml `targets` table ```toml -[[platforms]] +[[targets]] os = "linux" arch = "x86_64" -[[platforms.distros]] +[[targets.distros]] name = "ubuntu" -[[platforms.distros.versions]] +[[targets.distros.versions]] version = "18.04" packages = ["pkg:deb/ubuntu/curl"] -[[platforms.distros.versions]] +[[targets.distros.versions]] version = "20.04" packages = ["pkg:deb/ubuntu/curl2"] -[[platforms]] +[[targets]] os = "linux" arch = "x86_64" -[[platforms.distros]] +[[targets.distros]] name = "ubuntu" packages = ["pkg:deb/ubuntu/curl"] -[[platforms.distros.versions]] +[[targets.distros.versions]] version = "14.04" -[[platforms.distros.versions]] +[[targets.distros.versions]] version = "16.04" ``` @@ -77,9 +78,33 @@ version = "16.04" The mixins label on each base image is replaced by a layer in each base image containing a single file consisting of a [CycloneDX](https://cyclonedx.org)-formatted list of packages. Each package entry has a [PURL](https://github.com/package-url/purl-spec)-formatted ID that uniquely identifies the package. +### Example: CycloneDX package list + +```json +{ + "bomFormat": "CycloneDX", + "specVersion": "1.3", + "version": 1, + "components": [ + { + "type": "library", + "name": "curl", + "version": "1.2.3", + "purl": "pkg:deb/ubuntu/curl@1.2.3" + }, + { + "type": "library", + "name": "libcurl", + "version": "4.5.6", + "purl": "pkg:deb/ubuntu/libcurl@4.5.6" + } + ] +} +``` + ### Validations -Buildpack base image metadata and packages specified in `buildpack.toml`'s `platforms` list are validated against the runtime and build-time base images. +Buildpack base image metadata and packages specified in `buildpack.toml`'s `targets` list are validated against the runtime and build-time base images. Runtime and build-time base image packages are no longer validated against each other. @@ -99,13 +124,15 @@ Buildpacks may specify a list of package names (as PURL URLs without versions or - Keep stacks. - Make the above changes but keep some existing terminology: stacks, build-image, run-image. -- Continue to allow for static definition of runtime package requirements in `buildpack.toml`. +- Continue to allow static definition of runtime package requirements in `buildpack.toml`. # Unresolved Questions [unresolved-questions]: #unresolved-questions -- Should packages be determined during the detect or build phase? Opinion: detect phase, so that (in a later RFC), a runtime base image's app-specified Dockerfiles may by applied in parallel to the buildpack build process. +- Should runtime packages be determined during the detect or build phase? Opinion: detect phase, so that (in a later RFC), a runtime base image's app-specified Dockerfiles may by applied in parallel to the buildpack build process. - How will migration work? Can we make new base images compatible with older buildpacks? Can we make newer buildpacks compatible with older stacks? +- What should builder.toml (and similar stack-dependent config files) look like? What should assets look like? Note: these could be decided in subsequent subteam RFCs / spec PRs. + # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From f888612caccd636e5564915c22809c4fa4f044d7 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 21 Jul 2021 22:42:42 -0400 Subject: [PATCH 06/19] RFC: Remove Stacks & Mixins: remove packages Signed-off-by: Stephen Levine --- text/0000-remove-stacks-mixins.md | 37 +++++++++++-------------------- 1 file changed, 13 insertions(+), 24 deletions(-) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index f57d8a436..63ff20b72 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -24,7 +24,7 @@ The "stack" and "mixin" concepts add unnecessary complexity to the project and m [what-it-is]: #what-it-is Summary of changes: -- Replace mixins with a [CycloneDX](https://cyclonedx.org)-formatted list of packages. +- Remove mixins - Replace stack metadata (including stack IDs) with canonical OS metadata. # How it Works @@ -41,7 +41,7 @@ Instead of a stack ID, runtime and build-time base images are labeled with the f OS and Architecture must be valid identifiers as defined [here](https://golang.org/doc/install/source#environment). For Linux-based images, each field should be canonicalized against values specified in `/etc/os-release` (`$ID` and `$VERSION_ID`). -The `stacks` list in `buildpack.toml` is replaced by a `targets` list, where each entry corresponds to a different buildpack image that is exported into a [manifest index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). Each entry may contain multiple valid values for Distribution and/or Version, but only a single OS and Architecture. Each entry may also contain a list of package names (as PURL URLs without versions or qualifiers) that specify detect-time and build-time (but not runtime) OS package dependencies. Packages may be specified for all Versions of a distribution, or for a specific Version. Buildpacks may express **runtime** OS package dependencies during detection (see "Runtime Base Image Requirements" below). +The `stacks` list in `buildpack.toml` is replaced by a `targets` list, where each entry corresponds to a different buildpack image that is exported into a [manifest index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). Each entry may contain multiple valid values for Distribution and/or Version, but only a single OS and Architecture. App image builds fail if the build image and selected run image have mismatched metadata. We may introduce flags or additional labels to skip this validation (e.g., for cross-compilation or minimal runtime base images). An image without a specified Distribution is compatible with images specifying any Distribution. An image specifying a Distribution without a Version is compatible with images specifying any Versions of that Distribution. @@ -55,30 +55,21 @@ os = "linux" arch = "x86_64" [[targets.distros]] name = "ubuntu" -[[targets.distros.versions]] -version = "18.04" -packages = ["pkg:deb/ubuntu/curl"] -[[targets.distros.versions]] -version = "20.04" -packages = ["pkg:deb/ubuntu/curl2"] +versions = ["18.04", "20.04"] [[targets]] os = "linux" arch = "x86_64" [[targets.distros]] name = "ubuntu" -packages = ["pkg:deb/ubuntu/curl"] -[[targets.distros.versions]] -version = "14.04" -[[targets.distros.versions]] -version = "16.04" +versions = ["14.04", "16.04"] ``` ## Mixins -The mixins label on each base image is replaced by a layer in each base image containing a single file consisting of a [CycloneDX](https://cyclonedx.org)-formatted list of packages. Each package entry has a [PURL](https://github.com/package-url/purl-spec)-formatted ID that uniquely identifies the package. +Mixins are no longer used. If an SBoM is available, platforms may warn when, e.g., a rebase operation would change the available packages. -### Example: CycloneDX package list +### Example: CycloneDX SBoM ```json { @@ -97,22 +88,21 @@ The mixins label on each base image is replaced by a layer in each base image co "name": "libcurl", "version": "4.5.6", "purl": "pkg:deb/ubuntu/libcurl@4.5.6" - } + }, + ... ] } ``` ### Validations -Buildpack base image metadata and packages specified in `buildpack.toml`'s `targets` list are validated against the runtime and build-time base images. +Buildpack base image metadata specified in `buildpack.toml`'s `targets` list are validated against the runtime and build-time base images. Runtime and build-time base image packages are no longer validated against each other. -When an app image is rebased, `pack rebase` will fail if packages are removed from the new runtime base image. This check may be skipped by passing a new `--force` flag to `pack rebase`. - -## Runtime Base Image Requirements - -Buildpacks may specify a list of package names (as PURL URLs without versions or qualifiers) in a `packages` table in the build plan. The build will fail if these packages are not present in the runtime base image. +If an SBoM is available, `pack rebase` will fail if packages are removed from the new runtime base image. +This check may be skipped by passing a new `--force` flag to `pack rebase`. +However, this validation is not enforced by the specification. # Drawbacks [drawbacks]: #drawbacks @@ -124,12 +114,11 @@ Buildpacks may specify a list of package names (as PURL URLs without versions or - Keep stacks. - Make the above changes but keep some existing terminology: stacks, build-image, run-image. -- Continue to allow static definition of runtime package requirements in `buildpack.toml`. +- Continue to allow buildpacks to specify package requirements (e.g., by PURL instead of mixins) # Unresolved Questions [unresolved-questions]: #unresolved-questions -- Should runtime packages be determined during the detect or build phase? Opinion: detect phase, so that (in a later RFC), a runtime base image's app-specified Dockerfiles may by applied in parallel to the buildpack build process. - How will migration work? Can we make new base images compatible with older buildpacks? Can we make newer buildpacks compatible with older stacks? - What should builder.toml (and similar stack-dependent config files) look like? What should assets look like? Note: these could be decided in subsequent subteam RFCs / spec PRs. From ce16a0490f5dabffcd4f6ae421a69b2ff8d3ddd6 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Tue, 10 Aug 2021 14:14:03 -0400 Subject: [PATCH 07/19] RFC: Remove Stacks & Mixins: address outstanding feedback Signed-off-by: Stephen Levine --- text/0000-remove-stacks-mixins.md | 50 ++++++++++++++++++++++++------- 1 file changed, 39 insertions(+), 11 deletions(-) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index 63ff20b72..2a89f8da4 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -32,16 +32,18 @@ Summary of changes: ## Base Image Metadata -Instead of a stack ID, runtime and build-time base images are labeled with the following canonicalized metadata: -- OS (e.g., "linux", `$GOOS`) -- Architecture (e.g., "x86_64", `$GOARCH`) -- Distribution (optional) (e.g., "ubuntu", `$ID`) -- Version (optional) (e.g., "18.04", `$VERSION_ID`) - -OS and Architecture must be valid identifiers as defined [here](https://golang.org/doc/install/source#environment). +Instead of a stack ID, runtime and build-time base images must contain the following canonicalized metadata: +- OS (e.g., "linux", `$GOOS`), specified as `os` in the base image `config` +- Architecture (e.g., "arm", `$GOARCH`), specified as `architecture` in the base image `config` +- Architecture Variant (optional) (e.g., "v6", `$GOARM`), specified as `variant` in the base image `config` +- Distribution (optional) (e.g., "ubuntu", `$ID`), specified as a label `io.buildpacks.distribution.name` +- Version (optional) (e.g., "18.04", `$VERSION_ID`), specified as a label `io.buildpacks.distribution.version` + +OS, Architecture, and Architecture Variant must be valid identifiers as defined in the [OCI Image specification](https://github.com/opencontainers/image-spec/blob/main/config.md). For Linux-based images, each field should be canonicalized against values specified in `/etc/os-release` (`$ID` and `$VERSION_ID`). +The `os.version` field in an base image `config` may contain combined distribution and version information, but it is not used by the lifecycle. -The `stacks` list in `buildpack.toml` is replaced by a `targets` list, where each entry corresponds to a different buildpack image that is exported into a [manifest index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). Each entry may contain multiple valid values for Distribution and/or Version, but only a single OS and Architecture. +The `stacks` list in `buildpack.toml` is replaced by a `targets` list, where each entry corresponds to a different buildpack image that is exported into a [manifest index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). Each entry may contain multiple valid values for Distribution and/or Version, but only a single OS, Architecture, and Variant. App image builds fail if the build image and selected run image have mismatched metadata. We may introduce flags or additional labels to skip this validation (e.g., for cross-compilation or minimal runtime base images). An image without a specified Distribution is compatible with images specifying any Distribution. An image specifying a Distribution without a Version is compatible with images specifying any Versions of that Distribution. @@ -53,18 +55,37 @@ When an app image is rebased, `pack rebase` will fail if the new run image and p [[targets]] os = "linux" arch = "x86_64" -[[targets.distros]] +[[targets.distributions]] name = "ubuntu" versions = ["18.04", "20.04"] [[targets]] os = "linux" arch = "x86_64" -[[targets.distros]] +[[targets.distributions]] +name = "ubuntu" +versions = ["14.04", "16.04"] + +[[targets]] +os = "linux" +arch = "arm" +variant = "v6" +[[targets.distributions]] name = "ubuntu" versions = ["14.04", "16.04"] ``` +## Runtime Metadata + +To allow different runtime base images to be used, and to support cross-compilation in the future, buildpacks may need access to the runtime base image's target metadata. +The following environment variables will be available to buildpacks directly in the build-time environment (not via `/platform/env`): +- `CNB_TARGET_OS` +- `CNB_TARGET_ARCH` +- `CNB_TARGET_VARIANT` +- `CNB_TARGET_DISTRO_NAME` +- `CNB_TARGET_DISTRO_VERSION` +- `CNB_TARGET_ID` (optional ID, if present on runtime base image `io.buildpacks.id` label) + ## Mixins Mixins are no longer used. If an SBoM is available, platforms may warn when, e.g., a rebase operation would change the available packages. @@ -104,10 +125,17 @@ If an SBoM is available, `pack rebase` will fail if packages are removed from th This check may be skipped by passing a new `--force` flag to `pack rebase`. However, this validation is not enforced by the specification. +### Migration + +All existing labels and environment variables for stacks and mixins may be preserved on a base image until all users have migrated to the new format. +These labels will be deprecated (but allowed) for the forseeable future. +If the newly-specified field values are missing, the lifecycle and pack may used existing, standardized stack IDs (i.e., `io.buildpacks.stacks.*`) to determine the values of the missing fields, as long as the lifecycle and pack provide a warning for the user. + # Drawbacks [drawbacks]: #drawbacks -- Involves breaking changes. +- Involves breaking changes when existing metadata is removed. +- `CNB_TARGET_*` env vars assume a single target -- may require more breaking changes to support parallel, single-container cross-compilation in the future. # Alternatives [alternatives]: #alternatives From b876676e3913e8548681007289d197b51ef39b9b Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Sun, 15 Aug 2021 20:52:38 -0400 Subject: [PATCH 08/19] RFC: Remove Stacks & Mixins: Add default value for target list Signed-off-by: Stephen Levine --- text/0000-remove-stacks-mixins.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index 2a89f8da4..9cee32793 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -43,9 +43,13 @@ OS, Architecture, and Architecture Variant must be valid identifiers as defined For Linux-based images, each field should be canonicalized against values specified in `/etc/os-release` (`$ID` and `$VERSION_ID`). The `os.version` field in an base image `config` may contain combined distribution and version information, but it is not used by the lifecycle. -The `stacks` list in `buildpack.toml` is replaced by a `targets` list, where each entry corresponds to a different buildpack image that is exported into a [manifest index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). Each entry may contain multiple valid values for Distribution and/or Version, but only a single OS, Architecture, and Variant. +The `stacks` list in `buildpack.toml` is replaced by a `targets` list, where each entry corresponds to a different buildpack image that is exported into a [manifest index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). +Each entry may contain multiple valid values for Distribution and/or Version, but only a single OS, Architecture, and Variant. +If the `targets` list is empty, a single target with `os = "linux"` and `arch = "x86_64"` is assumed by tools reading `buildpack.toml`. -App image builds fail if the build image and selected run image have mismatched metadata. We may introduce flags or additional labels to skip this validation (e.g., for cross-compilation or minimal runtime base images). An image without a specified Distribution is compatible with images specifying any Distribution. An image specifying a Distribution without a Version is compatible with images specifying any Versions of that Distribution. +App image builds fail if the build image and selected run image have mismatched metadata. We may introduce flags or additional labels to skip this validation (e.g., for cross-compilation or minimal runtime base images). +An image without a specified Distribution is compatible with images specifying any Distribution. +An image specifying a Distribution without a Version is compatible with images specifying any Versions of that Distribution. When an app image is rebased, `pack rebase` will fail if the new run image and previous run image have mismatched metadata. This check may be skipped for Distribution and Version by passing a new `--force` flag to `pack rebase`. From 891104e9728195e3d6e1a8389e1e7d251ee4a501 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Sun, 15 Aug 2021 21:00:57 -0400 Subject: [PATCH 09/19] RFC: Remove Stacks & Mixins: determine OS for default target Signed-off-by: Stephen Levine --- text/0000-remove-stacks-mixins.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index 9cee32793..1c43a0c0b 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -45,7 +45,9 @@ The `os.version` field in an base image `config` may contain combined distributi The `stacks` list in `buildpack.toml` is replaced by a `targets` list, where each entry corresponds to a different buildpack image that is exported into a [manifest index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). Each entry may contain multiple valid values for Distribution and/or Version, but only a single OS, Architecture, and Variant. -If the `targets` list is empty, a single target with `os = "linux"` and `arch = "x86_64"` is assumed by tools reading `buildpack.toml`. +If the `targets` list is empty and `/bin/build` is present, a target with `os = "linux"` and `arch = "x86_64"` is assumed by tools reading `buildpack.toml`. +If the `targets` list is empty and `/bin/build.bat` or `/bin/build.exe` is present, a target with `os = "windows"` and `arch = "x86_64"` is assumed by tools reading `buildpack.toml`. + App image builds fail if the build image and selected run image have mismatched metadata. We may introduce flags or additional labels to skip this validation (e.g., for cross-compilation or minimal runtime base images). An image without a specified Distribution is compatible with images specifying any Distribution. From e2587fbaaca35b349f699ba491219c74fd2ed523 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 25 Aug 2021 14:22:28 -0400 Subject: [PATCH 10/19] RFC: Stacks & Mixins: rebaser Signed-off-by: Stephen Levine Co-authored-by: Emily Casey --- text/0000-remove-stacks-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index 1c43a0c0b..bb5c36085 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -53,7 +53,7 @@ App image builds fail if the build image and selected run image have mismatched An image without a specified Distribution is compatible with images specifying any Distribution. An image specifying a Distribution without a Version is compatible with images specifying any Versions of that Distribution. -When an app image is rebased, `pack rebase` will fail if the new run image and previous run image have mismatched metadata. This check may be skipped for Distribution and Version by passing a new `--force` flag to `pack rebase`. +When an app image is rebased, `rebaser` must fail if the new run image and previous run image have mismatched metadata. This check can be skipped for Distribution and Version by passing a new `--force` flag to `rebaser`. #### Example: buildpack.toml `targets` table From 0dd5899dc65c3d94509e83d02641f089f31627bf Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 25 Aug 2021 14:23:11 -0400 Subject: [PATCH 11/19] RFC: Stacks & Mixins: soften SBoM wording Signed-off-by: Stephen Levine Co-authored-by: Emily Casey --- text/0000-remove-stacks-mixins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index bb5c36085..1645b1381 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -127,7 +127,7 @@ Buildpack base image metadata specified in `buildpack.toml`'s `targets` list are Runtime and build-time base image packages are no longer validated against each other. -If an SBoM is available, `pack rebase` will fail if packages are removed from the new runtime base image. +Platforms may choose to add additional validation. For example, if an SBoM is available, `pack rebase` may fail if packages are removed from the new runtime base image. This check may be skipped by passing a new `--force` flag to `pack rebase`. However, this validation is not enforced by the specification. From b1f606cf7aaa33d06912c8b580c1a1d33f024792 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 8 Sep 2021 15:14:57 -0400 Subject: [PATCH 12/19] Reword intro Signed-off-by: Stephen Levine Co-authored-by: Emily Casey --- text/0000-remove-stacks-mixins.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index 1645b1381..956a07bc4 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -13,7 +13,11 @@ *NOTE: This proposal is part of a larger initiative to reduce complexity originally outlined in https://github.com/buildpacks/rfcs/pull/167* -This RFC proposes that we remove the "stack" and "mixin" concepts from the project and replace them with existing constructs in the container image ecosystem such as base images, Dockerfiles, and OS packages. +This RFC proposes that we remove the stack ID concept from the project and replace them with existing constructs in the container image ecosystem such as operating system name, operating system version and architecture. + +This RFC also proposes that we remove the "mixin" concept from the project entirely, eliminating strict validation of all facets of base image, buildpack compatibility, in favor of a a more optimistic approach. + +This proposal prioritizes simplicity and flexibility over the avoidance of all failure scenarios, by allowing more potentially successful build to execute. # Motivation [motivation]: #motivation From 4299b4bc03c9c0f73a5675d74d13292b1e061583 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 8 Sep 2021 15:17:10 -0400 Subject: [PATCH 13/19] Fix typos in intro Signed-off-by: Stephen Levine --- text/0000-remove-stacks-mixins.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index 956a07bc4..8f7987ace 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -13,11 +13,11 @@ *NOTE: This proposal is part of a larger initiative to reduce complexity originally outlined in https://github.com/buildpacks/rfcs/pull/167* -This RFC proposes that we remove the stack ID concept from the project and replace them with existing constructs in the container image ecosystem such as operating system name, operating system version and architecture. +This RFC proposes that we remove the stack ID concept from the project and replace it with existing constructs in the container image ecosystem such as operating system name, operating system version, and architecture. -This RFC also proposes that we remove the "mixin" concept from the project entirely, eliminating strict validation of all facets of base image, buildpack compatibility, in favor of a a more optimistic approach. +This RFC also proposes that we remove the "mixin" concept from the project entirely, eliminating strict validation of all facets of base-image-to-buildpack compatibility in favor of a more optimistic approach. -This proposal prioritizes simplicity and flexibility over the avoidance of all failure scenarios, by allowing more potentially successful build to execute. +This proposal prioritizes simplicity and flexibility over the avoidance of all failure scenarios by allowing more potentially successful builds to execute. # Motivation [motivation]: #motivation From ef6d5313989f4ae880a7dc511a793997806e733f Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 8 Sep 2021 15:23:41 -0400 Subject: [PATCH 14/19] Clarify values for windows Signed-off-by: Stephen Levine --- text/0000-remove-stacks-mixins.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index 8f7987ace..e348bc254 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -44,9 +44,12 @@ Instead of a stack ID, runtime and build-time base images must contain the follo - Version (optional) (e.g., "18.04", `$VERSION_ID`), specified as a label `io.buildpacks.distribution.version` OS, Architecture, and Architecture Variant must be valid identifiers as defined in the [OCI Image specification](https://github.com/opencontainers/image-spec/blob/main/config.md). + For Linux-based images, each field should be canonicalized against values specified in `/etc/os-release` (`$ID` and `$VERSION_ID`). The `os.version` field in an base image `config` may contain combined distribution and version information, but it is not used by the lifecycle. +For Windows-based images, Distribution should be empty. Version should be the [suggested value of `os.version`](https://github.com/opencontainers/image-spec/blob/main/config.md#properties) in the OCI spec (e.g., `10.0.14393.1066`). + The `stacks` list in `buildpack.toml` is replaced by a `targets` list, where each entry corresponds to a different buildpack image that is exported into a [manifest index](https://github.com/opencontainers/image-spec/blob/master/image-index.md). Each entry may contain multiple valid values for Distribution and/or Version, but only a single OS, Architecture, and Variant. If the `targets` list is empty and `/bin/build` is present, a target with `os = "linux"` and `arch = "x86_64"` is assumed by tools reading `buildpack.toml`. From ff4c94a01292ccc6beaf01778ca5e24398b5f82e Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 8 Sep 2021 15:34:47 -0400 Subject: [PATCH 15/19] Explain `CNB_TARGET_ID` Signed-off-by: Stephen Levine --- text/0000-remove-stacks-mixins.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index e348bc254..097157076 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -43,8 +43,14 @@ Instead of a stack ID, runtime and build-time base images must contain the follo - Distribution (optional) (e.g., "ubuntu", `$ID`), specified as a label `io.buildpacks.distribution.name` - Version (optional) (e.g., "18.04", `$VERSION_ID`), specified as a label `io.buildpacks.distribution.version` +Additionally, the runtime base may contain the following metadata: +- Target ID (optional) (e.g., "minimal"), specified as a label `io.buildpacks.id` + OS, Architecture, and Architecture Variant must be valid identifiers as defined in the [OCI Image specification](https://github.com/opencontainers/image-spec/blob/main/config.md). +Target ID is an identifier specified on the runtime base image that must be provided to buildpacks as `CNB_TARGET_ID` during the build process. +This allows buildpacks to change their behavior if a run image is selected (e.g., distroless) that has special properties outside of OS, architecture, etc. + For Linux-based images, each field should be canonicalized against values specified in `/etc/os-release` (`$ID` and `$VERSION_ID`). The `os.version` field in an base image `config` may contain combined distribution and version information, but it is not used by the lifecycle. @@ -55,7 +61,6 @@ Each entry may contain multiple valid values for Distribution and/or Version, bu If the `targets` list is empty and `/bin/build` is present, a target with `os = "linux"` and `arch = "x86_64"` is assumed by tools reading `buildpack.toml`. If the `targets` list is empty and `/bin/build.bat` or `/bin/build.exe` is present, a target with `os = "windows"` and `arch = "x86_64"` is assumed by tools reading `buildpack.toml`. - App image builds fail if the build image and selected run image have mismatched metadata. We may introduce flags or additional labels to skip this validation (e.g., for cross-compilation or minimal runtime base images). An image without a specified Distribution is compatible with images specifying any Distribution. An image specifying a Distribution without a Version is compatible with images specifying any Versions of that Distribution. From 3fe2b89c2d832ebf032d56d879573a4aeaa02a5d Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 8 Sep 2021 15:36:56 -0400 Subject: [PATCH 16/19] Add unresolved question about packages Signed-off-by: Stephen Levine Co-authored-by: Emily Casey --- text/0000-remove-stacks-mixins.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index 097157076..e4b7fd034 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -167,6 +167,7 @@ If the newly-specified field values are missing, the lifecycle and pack may used - How will migration work? Can we make new base images compatible with older buildpacks? Can we make newer buildpacks compatible with older stacks? - What should builder.toml (and similar stack-dependent config files) look like? What should assets look like? Note: these could be decided in subsequent subteam RFCs / spec PRs. +- How will this affect the UX for pack buidpack package? For example package.toml contains a conflicting [platform table](https://buildpacks.io/docs/reference/config/package-config/#platform-_optional_). This should be resolved in a separate subteam RFC. # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes From e24bcad59061b74000db56bbc88424d01dc5d3f5 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 8 Sep 2021 15:37:32 -0400 Subject: [PATCH 17/19] Add examples Signed-off-by: Stephen Levine Co-authored-by: Terence Lee --- text/0000-remove-stacks-mixins.md | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index e4b7fd034..edec1fc7f 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -147,7 +147,26 @@ However, this validation is not enforced by the specification. All existing labels and environment variables for stacks and mixins may be preserved on a base image until all users have migrated to the new format. These labels will be deprecated (but allowed) for the forseeable future. -If the newly-specified field values are missing, the lifecycle and pack may used existing, standardized stack IDs (i.e., `io.buildpacks.stacks.*`) to determine the values of the missing fields, as long as the lifecycle and pack provide a warning for the user. +If the newly-specified field values are missing, the lifecycle and pack may used existing, standardized stack IDs (i.e., `io.buildpacks.stacks.*`) to determine the values of the missing fields, as long as the lifecycle and pack provide a warning for the user. More concretely, when using a stack with `io.buildpacks.stacks.bionic` lifecycle will assume the following metadata is set if these fields are empty: + +config.os = Linux +config.architecture = x86_64 +io.buildpacks.distribution.name = "ubuntu" +io.buildpacks.distribution.version = "18.04" + +Moving forward it's encouraged for buildpack authors to support both `[[stacks]]` and `[[targets]]` sections in `buildpack.toml` for maximum compatibility. In order to ease this process for those using the `io.buildpacks.stacks.bionic`, lifecycle will translate any section that sets this as on of the `stacks`: + +[[stacks]] +id = "io.buildpacks.stacks.bionic + +to + +[[targets]] +os = "linux" +arch = "x86_64" +[[targets.distributions]] +name = "ubuntu" +versions = ["18.04"] # Drawbacks [drawbacks]: #drawbacks From 85545b5ca0e660a512594bfe89d030ca8c636af0 Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Wed, 8 Sep 2021 15:39:50 -0400 Subject: [PATCH 18/19] Fix examples Signed-off-by: Stephen Levine --- text/0000-remove-stacks-mixins.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index edec1fc7f..0aa40eeab 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -149,24 +149,30 @@ All existing labels and environment variables for stacks and mixins may be prese These labels will be deprecated (but allowed) for the forseeable future. If the newly-specified field values are missing, the lifecycle and pack may used existing, standardized stack IDs (i.e., `io.buildpacks.stacks.*`) to determine the values of the missing fields, as long as the lifecycle and pack provide a warning for the user. More concretely, when using a stack with `io.buildpacks.stacks.bionic` lifecycle will assume the following metadata is set if these fields are empty: -config.os = Linux -config.architecture = x86_64 +``` +config.os = "linux" +config.architecture = "x86_64" io.buildpacks.distribution.name = "ubuntu" io.buildpacks.distribution.version = "18.04" +``` Moving forward it's encouraged for buildpack authors to support both `[[stacks]]` and `[[targets]]` sections in `buildpack.toml` for maximum compatibility. In order to ease this process for those using the `io.buildpacks.stacks.bionic`, lifecycle will translate any section that sets this as on of the `stacks`: +```toml [[stacks]] id = "io.buildpacks.stacks.bionic +``` to +```toml [[targets]] os = "linux" arch = "x86_64" [[targets.distributions]] name = "ubuntu" versions = ["18.04"] +``` # Drawbacks [drawbacks]: #drawbacks From ad051c4b833f122a186ca0756bdf2645159ef78f Mon Sep 17 00:00:00 2001 From: Stephen Levine Date: Tue, 28 Sep 2021 22:52:04 -0400 Subject: [PATCH 19/19] RFC: Remove Stacks & Mixins: Add question re: targets Signed-off-by: Stephen Levine --- text/0000-remove-stacks-mixins.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0000-remove-stacks-mixins.md b/text/0000-remove-stacks-mixins.md index 0aa40eeab..4423fcd51 100644 --- a/text/0000-remove-stacks-mixins.md +++ b/text/0000-remove-stacks-mixins.md @@ -193,6 +193,8 @@ versions = ["18.04"] - How will migration work? Can we make new base images compatible with older buildpacks? Can we make newer buildpacks compatible with older stacks? - What should builder.toml (and similar stack-dependent config files) look like? What should assets look like? Note: these could be decided in subsequent subteam RFCs / spec PRs. - How will this affect the UX for pack buidpack package? For example package.toml contains a conflicting [platform table](https://buildpacks.io/docs/reference/config/package-config/#platform-_optional_). This should be resolved in a separate subteam RFC. +- Are we planning to export an image for each target? Or does `pack buildpack package` require a target? + Suggestion: All targets will live in an immutable OCI Manifest Index, so all targets should be exported at the same time. # Spec. Changes (OPTIONAL) [spec-changes]: #spec-changes