RFC: Remove Stacks & Mixins

[sclevine]

Readable.

This proposal is part of a larger initiative to reduce complexity originally outlined in #167

[imjasonh]

Off-topic: Is there any plan/idea for buildpacks to support building multi-platform images, such that a single image manifest can describe an image for linux/arm64 and linux/arm64 and windows/amd64, etc.? Or is the intention that someone wanting to build a multi-platform image should use buildpacks to produce multiple platform-specific images, then stitch them together with something like docker manifest create ?

[sclevine]

No current RFC, but I definitely agree that we should eventually support exporting multi-platform manifest indexes.

The BuildKit frontend might help us with this, via buildx: buildpacks/pack#768

For non-local builds, we could extend the current exporter to accept a set of layers for each OS/ARCH combination.

In either case, we'd probably want to pass standardized env vars (e.g., $CNB_TARGET_ARCH) to the builder and buildpacks to support cross-compilation.

[ekcasey]

Is there any plan/idea for buildpacks to support building multi-platform images

This is hard to do well given that (some exceptions wrt docker-for-mac notwithstanding) the build takes place in a docker container and a given daemon only supports a single os/arch.

You could theoretically have a buildpacks execute on one target while producing launch layers that are valid for a different target but this puts a lot of responsibility on the buildpack author and comes with a lot of gotchas. e.g

• a single layer is unlikely to be valid as both a build and launch layers
• ecosystem tools that buildpacks delegate to will make assumptions based on the build environment

imho this is unlikely to end well

[sclevine]

You could theoretically have a buildpacks execute on one target while producing launch layers that are valid for a different target

I'm not opposed to this. Cross-compilation seems like a reasonable feature request to me. We would probably want to pass "target" metadata (e.g., from a run image label) to the buildpacks. When multiple run images are provided, buildpacks could produce multiple versions of the same layer for different architectures.

[sclevine]

@ekcasey as requested. Looks very noisy, but I think it meets the requirements.

[dmikusa]

Could you explain the sample, please? If I have this example in my buildpack's buildpack.toml, what does that mean for the buildpack & compatibility with images? What sort of compatibility guarantees do I get?

[cmoulliard]

As the goal of buildpacks is to build an image from an app's project where several build/detect/ (1 to many) steps are done using a Build image and next launched using a Runtime or stack or runtimeStack image, then I think that it is important to continue to use a wording that everyone will understand.

This is the reason why I would like to suggest to keep the stack (or maybe to rename it to: runStack, runtimeStack, ...) word instead of using platform.

Why: to improve the readability of a builder definition with a wording which is more inline with what finally we would like to describe (or what we want to do during a build or when we will launch the image built)

Example of such new definition could be

  # buildPacks could be renamed to buildSteps ;-)
  buildPacks:
  # List of build steps executed by the Lifecycle creator during: detect, build, ... steps
    - id: dev.snowdrop.jdk
      packUri: 'docker://quay.io/snowdrop/pack-jdk:11'
      buildPackImage: 'quay.io/snowdrop/buildpack-jdk:1.0'
    - id: dev.snowdrop.maven
      packUri: 'docker://quay.io/snowdrop/pack-maven:3.5'
      buildPackImage: 'quay.io/snowdrop/buildpack-maven:1.0'
    - id: dev.snowdrop.gradle
      packUri: 'docker://quay.io/snowdrop/pack-gradle:7.1'
      buildImage: 'quay.io/snowdrop/buildpack-gradle:1.0'
      optional: yes
      packages:
        - pkg:rpm/rhel/gradle@7.1
  # List of run stacks image supporting the build image      
  runStacks:
    - id: dev.snowdrop.stack.jdk16
      default: yes
      os: rhel
      version: 8
      arch: x86_64
      runtimeImage: 'quay.io/snowdrop/stack-run:jdk16'
      packages:
        - pkg:rpm/rhel/openjdk11
    - id: dev.snowdrop.stack.jdk11
      os: ubuntu
      version: 
      arch: x86_64
      runtimeImage: 'quay.io/snowdrop/stack-run:jdk11'

See gist created: https://gist.github.com/cmoulliard/04999929df0b758b5edbdf19dfc85f45

[sclevine]

I'm not sure I understand the proposed example. Some questions:

• Where does the build-time base image come from for the JDK and Maven buildpacks?
• Is a different build-time base image selected when the runtime base image OS changes?
• Are the buildpack IDs optional?
• How does the packages list work?
• Are you proposing that we re-define stack to only refer to the runtime base image?

Would be great to dig into this during the weekly WG meeting.

I'm not attached to platforms for the table name, but I would like to remove unnecessary, project-specific terms to make the project more accessible.

[ekcasey]

App image builds fail if the build image and selected run image have mismatched metadata

Haven't thought this through all the way but would it make sense to this in the build plan as well. E.g. a buildpack could specify in buildpack.toml that it needs to run on a particular distribution but could express at detect time that the resulting layers could run on any-linux (distroless images).

[dmikusa]

An image without a specified Distribution is compatible with images specifying any Distribution. An image specifying a Distribution without a Version is compatible with images specifying any Versions of that Distribution.

Is this what would replace wildcard stack support, i.e. stack="*".

[sclevine]

E.g. a buildpack could specify in buildpack.toml that it needs to run on a particular distribution but could express at detect time that the resulting layers could run on any-linux (distroless images).

I'm not sure this needs to be dynamic. I wonder if this could be part of the multi-platform interface? See: #172 (comment)

Is this what would replace wildcard stack support, i.e. stack="*".

The requirement is just describing the contract between the build and run images. Wildcard stack support is achieved by leaving off Distribution and Version entirely. (There is no wildcard support for OS/Architecture.)

[joe-kimmel-vmw]

just updating this thread in case someone comes along in two more years, but anyway it turned out while implementing we did make an explicit extension for "*" OS/Architecture.

[ekcasey]

Example please :)

[jabrown85]

Are these strings able to be anything? Could I, if I wanted to, create a JesseOS OS string to tightly control the buildpacks that have explicitly updated to work on my build/run images?

[sclevine]

OS and Architecture would be restricted to GOOS and GOARCH values. Distribution and Version could technically be anything, but should match /etc/os-release for Linux distros.

[sclevine]

Idea I'd like feedback on: what if we dropped the packages list from both buildpack.toml and detect, at least for now? Builder authors could be responsible for reading buildpack documentation and installing any extra packages. We could re-introduce this functionality, if necessary, as part of #174.

[sclevine]

@hone added, ready to merge

[hone]

I think the bot may be broken. :( @buildpacks/implementation-maintainers @buildpacks/platform-maintainers are there any issues that need to be opened before I merge this in?

[jromero]

I think the bot may be broken. :( @buildpacks/implementation-maintainers @buildpacks/platform-maintainers are there any issues that need to be opened before I merge this in?

It's not broken. The issues only get created when this gets merged via merge-rfc.sh.

EDIT: You can see the list of queued issues here: #172 (comment)

[jromero]

This part is a little unclear to me. This appears to be talking about a set of environment variables that are given to the buildpacks during build-time.

1. It's not talking about these environment variable being set on the run image right?
2. What determines what these environment variables should be set to?
3. What does happen if the platform provides these environment variables via /platform/env as a way to configure the desired outcome? Is it explicitly something we don't want to allow?

[sclevine]

1. Correct: "directly in the build-time environment"
2. "runtime base image's target metadata" -- same metadata as above, but for the runtime base image
3. All CNB_ env vars are set directly so that they are unaffected by "strict" mode that forces buildpacks to read the env vars from /platform/env

[edmorley]

The current buildpack API spec includes CNB_STACK_ID being set in the environment of buildpack detect/build:
https://github.com/buildpacks/spec/blob/buildpack/v0.8/buildpack.md#provided-by-the-platform

That env var is commonly used by buildpacks when determining which binary to download/install for those that are compiled for specific OSes/OS versions.

This RFC doesn't mention what happens to that env var. Is it now considered deprecated too? Or are only the buildpack.toml stacks parts being deprecated?

If CNB_STACK_ID is being deprecated, could the RFC be updated to clarify this + explain what buildpacks should now use instead? (I would presume they now need to construct the stack name/version/... from the new CNB_TARGET_* env vars?)