MacOSX builds fail with lock file exists

[jmid]

I'm trying to publish opam packages from multicoretests ocaml/opam-repository#22620
and I'm seeing failures on MacOSX:
https://opam.ci.ocaml.org/github/ocaml/opam-repository/commit/ad48d0ee5145447acb3e445ee8331c2e79c0aaa1

This is a release of 3 opam packages all from the same repo - and with the same git commit hash.
To me the MacOSX amd64 seem mutually exclusive - if the 5.0 version succeeds the 4.14 version fails (and vice versa) with:

Building on i7-worker-04
fatal: Unable to create '/private/var/lib/ocluster-worker/git/opam-repository.git-b322f7b18d276db68487afc3dfc5a5befdd45818/.git/index.lock': File exists.

Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
git-reset failed with exit-code 128
2022-12-07 21:11.39: Job failed: Failed: Build failed

In all 3 failure cases the offending file is the same:

fatal: Unable to create '/private/var/lib/ocluster-worker/git/opam-repository.git-b322f7b18d276db68487afc3dfc5a5befdd45818/.git/index.lock': File exists.

This makes me suspect a problem around separate packages that are released from the same git commit hash.

[jmid]

I'm unsure if I misfiled this issue. Let me know if it should be moved to https://github.com/ocurrent/ocaml-ci instead.

[kit-ty-kate]

I moved this to ocluster (not sure where the actual issue is though). However I'm pretty sure the error got fixed by removing those lock files on those servers. (but I'm not sure how the error appeared in the first place)

[tmcgilchrist]

This should be fixed in obuilder with changes to how snapshots are pruned and the introduction of ZFS snapshots rather than rsync. Please re-open if you encounter this issue again.