Push pack functionality into small packages/binaries

[jabrown85]

As we discussed in the sub-team sync recently, it might be useful to have pack broken up into smaller binaries/packages. Like ggcr.

Starting off with some ideas based off of actual use cases I've wanted recently.

• pack remains the same
• projectoml-parser new binary/package to read project.toml and produce inputs for lifecycle (order.toml, /platform/env files, etc.)
• buildpack-fetcher given a list of buildpack references (or maybe an order.toml?) - it would resolve the buildpack references, download to a specified location if they do not exist on the builder, and validate the buildpacks are valid for the builder. The reason this would be useful in a project like pack is because pack end-users will expect non-pack platforms to resolve downloading of buildpacks in the same way as it does for them locally. This is especially true when buildpacks are provided in the source code via project.toml. This being in it's own image would also make tekton more powerful as a pre-step to download additional buildpacks not provided in the builder image.

Without at least having some of this exposed as go packages, platform authors will need to try to keep up with any pack changes and the platforms may diverge in a way that is undesirable for the buildpack ecosystem.

[dfreilich]

This would also be great for resolving buildpacks/pack#814, while we decide what public interface we'd like in imgutil

[jromero]

A brain dump of something I've been thinking about recently. Credit goes to the GGCR strategy. 🎤💧

Note that I omitted your projecttoml-parser @jabrown85 based on the conversation in WG today that it's more of a lifecycle concern.

.
├── cmd/
│   ├── pack/ → current pack cli
│   └── fetcher/ → cli to download buildpacks
├── pkg/
│   ├── archive/ → lib for converting multiple sources into OCI layers
│   ├── client/ → current pack client library
│   ├── descriptor/ → lib for processing project descriptor
│   ├── inspect/ → lib for inspecting various CNB resources (app image, buildpack, builder, stack, ...)
│   ├── internal/ → all non-exported go code
│   └── resolve/ → lib for resolving buildpacks via Buildpacks URIs
└── go.mod

[jabrown85]

Yeah, exposing a archive.Filter(WithPackDefaults()) or similar so a platform can optionally filter the way pack does is enough for me. I can imagine a pack build filtering out .git folder and some buildpack using that to grab the current commit and it producing different builds local vs cloud with the same builder/buildpacks. Obviously, the source is technically different in that case...but it feels weird to not have a way to best match local for repeatable build reasons.

[jromero]

FWIW, I'm pretty sure pack doesn't filter out .git directory. The only filtering I recall is in regards to the project.toml include/exclude. I would think that something like the .git directory would be deleted by a buildpack. 🤔

[jabrown85]

ah, I must be thinking of something else. I know the project.toml uses .gitignore syntax.

[dfreilich]

I missed this filtering discussion. Would you like me to add an issue to add that feature?

[jromero]

I believe @dwillist is interested in creating a formal RFC for the platform maintainers from this with some of his ideas incorporated.

[dfreilich]

If we do this, that would be a great time to systematically take care of buildpacks/pack#664

[dfreilich]

Relevant issue: buildpacks/pack#761