background fetching can cause data corruption

[vadimberezniker]

I have been experimenting with container image streaming using stargz-store + podman, but have run into some odd behavior which I have narrowed down to the "background fetch" feature. When background fetching is enabled, the store can return wrong contents for files.

After a few days of debugging, I think the following is happening:
When a layer is first resolved, a new memory.reader is created. memory.NewReader assigns an ID to each file.
It's important to note that the ID assignment is not deterministic because ForEachChild internally iterates over a map which does not provide a stable ordering. The assigned ID is used to compute the key used for storing data in the cache.

Background fetching calls verifiableReader.Cache(reader.WithReader(...)) which calls Clone on metadata.Reader. In the memory.reader case, the IDs are not preserved which means the newly assigned IDs may not match the original assignments. Background fetching can then populate the cache with chunks for the "wrong" IDs.

I think a simple fix is to copy over the ID assignments when Clone is called, what do you think?

[ktock]

@vadimberezniker Thank you for the detailed report.

I think a simple fix is to copy over the ID assignments when Clone is called, what do you think?

Yes, that solution sounds good to me. Would you be willing to create a patch to fix it?

[vadimberezniker]

Yes, I can work on a patch tomorrow.

[ktock]

@vadimberezniker Thanks!