decouple composefs and zstd:chunked

[cgwalters]

Today,

[storage.options.overlay]
use_composefs = "true"

Actually also requires:

[storage.options]
pull_options = {convert_images = "true"} 

And it's especially the convert_images = true that adds huge latency to pulls of images that aren't already in zstd:chunked format.

I am probably willing to put some time in code behind decoupling these two things, as I do not believe at a technical level they need to be coupled.

If someone feels really strongly, it's OK to close this issue though.

In the meantime at a practical level, this issue will just be a reference point for what is likely to be a FAQ of "why does enabling composefs slow down my image pulls"?

[rhatdan]

@giuseppe PTAL

[rhatdan]

@alexlarsson PTAL

[giuseppe]

we have convert_images so we can reuse all the existing infra to deduplicate files across layers.

We could generate a composefs blob directly from the extracted tarball content, but we'd lose the other advantages we have with pkg/chunked.

I think we still need to create a TOC (even in a simpler format without offsets to the payloads in the tarball stream) but we can avoid the temporary conversion to zstd

[mtrmac]

we have convert_images so we can reuse all the existing infra to deduplicate files across layers.

So… the deduplication / lookup code can, in principle, be called from drivers/overlay outside of the zstd call stack. And convert_images should probably not even trigger the PutBlobPartial code path in c/image, it should be “ordinary application of a non-chunked layer”. I do think that would be a better design; but the change would be invasive.

[mtrmac]

[… and I have a self-imposed moratorium on reviewing any changes that add options or complexity to drivers/overlay because I just can’t keep up.]

[cgwalters]

I do think that would be a better design; but the change would be invasive.

Yeah...I didn't look more than superficially at this but I am sure you're right.

[… and I have a self-imposed moratorium on reviewing any changes that add options or complexity to drivers/overlay because I just can’t keep up.]

I guess as far as options, #2017 is actually higher priority from my PoV and obviously way simpler (and I'm curious if you agree with that option).

But just to back up on this and help level set: right now I maintain a whole separate stack in bootc (ostree-rs-ext) that is today mapping non-zstd:chunked images ultimately into composefs (indirectly via ostree). My interest is in reducing this duplication, and that's partly where this issue came from. People are today booting via bootc into composefs from OS containers that are gzip, and I don't want to break that.