Ephemeral builder tars are missing base image layers

[matejvasek]

When creating ephemeral builder the pack binary uses hack optimization when creating tar stream. The tar pushed to repo is missing some layers, the layers are however already present as they are downloaded in advance so it's still somehow working. But that is not a valid format. This causes issues for instance for podman which has more strict validation.

[matejvasek]

For more info containers/podman#8132.

[matejvasek]

This will block #564.

[jromero]

Hi @matejvasek, thanks for the report. If it's what I'm thinking about, yes we are using an optimization hack specific to the docker daemon implementation. @ekcasey was the mastermind IIRC.

The "simple fix" as mentioned by @mtrmac is something we've recently contemplated but will take a bit longer to implement in the way we've previously discussed it. We can investigate a little further and determine smaller changes to resolve the issue in a shorter timeframe.

[ekcasey]

Yes this is a hack that we added to improve performance against the docker daemon. It is crucial that we keep this hack in place when using the docker daemon because the performance implications are enormous.

We could likely make the hack configurable in imgutil and toggle it on/off for docker/podman, but performance may suffer for pack+podman users.

[dfreilich]

@ekcasey You had run an experiment somewhere to see what those performance implications were, right? Any chance you could repost those?

[ekcasey]

@dfreilich We did this a looong time ago (sometime in 2018) so suffice it to say I don't have original comparison data handy, but we can do some back of the envelope math. The time saved is the time it takes to docker save and then docker load the base image.

For example, if I am creating an ephemeral builder with a single additional buildpack the "base image" is the original builder. It just took me 29.646s to docker save -o builder.tar paketobuildpacks/builder:full and 26.103s to docker load -i builder.tar. So for this (admittedly larger) builder the hack is saving users close to one minute for the command pack build --builder paketobuildpacks/builder:full --buildpack <buildpack>. And that is just for the ephemeral builder part. If we got rid of the hack, the same problem would arise during export adding another 30-60s most likely.

[jromero]

@matejvasek,

@ekcasey started working on this but found a separate issue, reported by @abitrolly, that looks like might stem from the podman vs docker implementation. Are you able to look at the possible cause for podman not providing platform information as detailed in #966?

[matejvasek]

@jromero
I'll look at it. It might be caused by missing implementation for /containers/{id}/archive endpoint in podman. I recently implemented that containers/podman#8126, but it's probably not released yet.

[matejvasek]

Also 2.1 doesn't contain this fix of mine: containers/podman#8109. Could you please try build from master?

[matejvasek]

After pack update I can reproduce the issue #966 it on master. I just used older version of pack before.
Edit: actually I cannot reproduce it on latest master as it's been fixed. I just forgot to fetch correct remote 🤦‍♂️ .
Edit2: or maybe I fetched correct remote, but the fix has been merged like 20min ago, so I might miss it in first fetch 😄 .

[matejvasek]

Hmm the OS is empty string in response JSON. I'll try to fix that.

[matejvasek]

@jromero After some digging I found out it's been fixed here containers/podman#8494. So podman master should work.

[jromero]

Hi @matejvasek, just trying to button up everything here. Given that buildpacks/lifecycle#485 was released in lifecycle 0.10.0, this should be addresses when using a builder with a lifecycle version >= 0.10.0. I'm assuming it's all tied to #966.

Please reopen if I'm mistaken.

EDIT: I misassociated where the fix would be coming from. In this particular case, it's not the lifecycle but instead the buildpacks/imgutil library. This issue was addressed in commit buildpacks/imgutil@339e186 (Dec 9, 2020), pack updated the library (1b6730e) in release v0.16.0 to buildpacks/imgutil@8581300 (Dec 11, 2020).

[vlk-charles]

I don't know if it's still relevant to this issue but the hack can be made to work with Podman too. You just have to reference something in manifest.json instead of leaving the paths empty for layers. See containers/podman#8132 (comment). This shouldn't break Docker compatibility.