RFC Flatten builders

[dlion]

Readable

[buildpack-bot]

Maintainers,

As you review this RFC please queue up issues to be created using the following commands:

/queue-issue <repo> "<title>" [labels]...
/unqueue-issue <uid>

Issues

(none)

[hone]

Thanks for writing this up! For a small nit can you rename your file to text/XXX-flatten-feature.md.

As mentioned in WG today:

• Is there a need for flattening buildpacks or just builders? My concern about flattening buildpacks is if it's distributed this way, then it removes the flexibility for the builder author (if they don't have the layer limit issue for their composition).
• Can you provide an example for exclude and include being used together?
• I'd like to see any of the user research you all are doing being added as context into this RFC!

[tomkennedy513]

In the case of flattening a builder, how would you determine which layers to flatten? If it is just by buildpack include/exclude flags, what would a user expect to happen when a buildpack is shared across multiple meta buildpacks. Also wondering how the flattening of builders would come into play in a platform like kpack since it doesn't use pack to build builders.

[dlion]

Thanks for writing this up! For a small nit can you rename your file to text/XXX-flatten-feature.md

The file is already called https://github.com/dlion/rfcs/blob/rfc-flatten-feature/text/XXX-flatten-feature.md did I miss anything? 🤔

Is there a need for flattening buildpacks or just builders? My concern about flattening buildpacks is if it's distributed this way, then it removes the flexibility for the builder author (if they don't have the layer limit issue for their composition).

Uhm, I don't know about the specific need around one or another choice since I wasn't there (I remember that @jjbustamante told me that he has been asked for this implementation) , maybe @natalieparellano has some context about it.

Can you provide an example for exclude and include being used together?

Uhm, let’s say we have something like:
BP 1, BP 2, BP 3, BP 4, BP 5, BP 6

flowchart TD
    A[CB1]
    A --> BP1[BP1]
    A --> BP2[BP2]
    A  --> BP3[BP3]
    A --> BP4[BP4]
    A --> BP5[BP5]
    A --> BP6[BP6]

Loading

and I want just to specifically merge BP 1 and BP2, but exclude BP 5 and leave the rest flatten to obtain something like: BP1-BP2 BP5 BP3-BP4-BP6

flowchart TD
A[CB1]
A --> BP1BP2[BP1-BP2]
A --> BP3BP4[BP3-BP4-BP6]
A --> BP5

Loading

Does it make sense?

I'd like to see any of the user research you all are doing being added as context into this RFC!
We are planning to have the session this Thursday, I'll keep it updated @hone 😄

[hone]

@dlion

The file is already called https://github.com/dlion/rfcs/blob/rfc-flatten-feature/text/XXX-flatten-feature.md did I miss anything? thinking

Sorry, i meant it should be rename from that with 0000 instead of XXX as talked about in the README.

[dlion]

Sorry, i meant it should be rename from that with 0000 instead of XXX as talked about in the README.

Ops my bad, I misinterpreted the sentence in the readme where 'my-feature' is descriptive. don't assign an RFC number yet as Don't even put 000.
I'll fix it asap, thanks @hone

[dlion]

I've just updated this RFC with the feedback we've got from some of our stakeholders:

• Remove depth since it's not used
• Remove --flatten --flatten-exclude since it's too complicate and it requires some mental gymnastic to be understood and used
• Rename --flatten --flatten-include into just --flatten, it will make the feature simpler and the flatten all it's not the default mode anymore
• It doesn't really matter if this feature is behind the experimental flag.

Here a partial image of the MIRO board we used to ran the exercise with our users:

Thank you

[dlion]

Hi @hone, is this good to be voted?
I'd like to start the voting window, let's say for a period of 1 week, does it sound reasonable?
/cc @jjbustamante @natalieparellano

[jkutner]

It also inhibits the de-dupping of buildpack layers after they are flattened. If we distributed flattened buildpacks, the child-buildpacks can't be de-dupped. This is less of a problem for builders since we don't distribute them the same way.

[jkutner]

Could we limit this RFC to pack builder create with flattened layers?

It seems like that's really the long pole, since that's where most people would accumulate more than 127 layers (are there really examples of individual buildpacks that have more 127 child buildpacks?)

[jjbustamante]

Could we limit this RFC to pack builder create with flattened layers?

It seems like that's really the long pole, since that's where most people would accumulate more than 127 layers (are there really examples of individual buildpacks that have more 127 child buildpacks?)

I think it makes sense, just want to confirm @ForestEckhardt Paketo is flattening only the builders, right?

As an example, I took the Java buildpack from Paketo and count the number of layers there:

> crane manifest gcr.io/paketo-buildpacks/java | jq '.layers | length'
27

I got 27, so I think ti is fine.

[dlion]

Could we limit this RFC to pack builder create with flattened layers?

I think it's a great idea, as I said during our sync would be better to just focus on the builder part to be sure we focus on deliverying value to our users.
From that we can start collecting feedback and if this functionality it is needed for buildpacks as well we can considerate it right away. 🚀

I'll modify the RFC accordingly and start creating the first issue about it. Thanks for the feedback everyone!

[jjbustamante]

From our WG meeting today:

• Let's remove from the RFC the idea of flatening a buildpack package, not because it is not possible or valuable, because it requires another conversation for the impact of doing that.
• We want to move forward with this RFC, so once the change are done, @jkutner will take a look and if it is ok, then he can moves it to next state in the RFC process.
• We can open another RFC for flattening a buildpack package and keep working in that without blocking flattening a builder.

[dlion]

Hi all, I updated the RFC removing any reference that might not be related to a Builder.
Let me know if it is fine 😄
In the meantime I also created the issue related to this RFC: buildpacks/pack#1880 🚀

[hone]

This use case, only happens for platforms that create ephemeral builders (like pack and kpack) right? I was looking at how we consume builders at Heroku and I think this change would "just work".

[hone]

Scoping this to builders minimizes the impact, which I think is a good first step. How many layers is the paketo builder at this point?

[hone]

I'm moving this to voting with a close of Sept 13th. @jkutner I'll be out for half of next week, so can you handle finalizing this?

[dlion]

Hey @jkutner, is there anything else I need to do to have this PR merged? 😄