Persist /var/lib/docker across jobs

[fw42]

Hello,

I know this is a bit of a FAQ but I'm trying to figure out how to make it so that /var/lib/docker can persist across jobs. I don't mean that multiple pods should access the same /var/lib/docker concurrently, I know that's discouraged, but I want to either

(a) Have one /var/lib/docker (on some kind of fast storage) per node, limit one runner pod per node, and make that node's /var/lib/docker available to that one pod. When the next pod gets scheduled on the same node, it should see the data that the previous pod wrote into it.

or

(b) Have a couple of different /var/lib/docker volumes that a new job can pick one from, for example if I have 2 nodes and allow 2 pods per node, I would "pre-provision" 2x2=4 volumes. Cache hit ratio would be worse, but still better than nothing.

Unfortunately I can't get either of those to work. The only approach that I've been able to get to work is to use a persistentVolumeClaim in the pod spec and then the next pod will in fact see the same data, but this only works if there's only exactly one pod at a time (a second pod won't be able to claim the volume since it's already in use).

All the documentation, questions, discussion, previous issues, write ups, etc. that I can find online appear to use older/legacy versions of ARC and the code snippets I see there don't appear to work or are leaving out critical details.

Can someone point me in the right direction? What is my Helm values.yaml file supposed to look like? What resources do I need to create manually on top of that? etc.