[opam 2.1~beta2] Sandboxed opam var fail when jobs > 1(?)

[kit-ty-kate]

Using opam 2.1.0~beta2, I sometimes get failures when compiling ocaml-freestanding and/or mirage-crypto which both use opam var during their compilation.
For instance if I compile mirage-crypto alone, it compiles fine:

$ opam install mirage-crypto
The following actions will be performed:
  ∗ install mirage-crypto 0.8.5
Do you want to continue? [Y/n] y

<><> Processing actions ><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved mirage-crypto.0.8.5  (cached)
∗ installed mirage-crypto.0.8.5
Done.

But if I compile both ocaml-freestanding and mirage-crypto at the same time, opam fails during the opam var call:

$ opam reinstall mirage-crypto ocaml-freestanding
The following actions will be performed:
  ↻ recompile ocaml-freestanding 0.6.2
  ↻ recompile mirage-crypto      0.8.5
  ↻ recompile hacl_x25519        0.2.0 [uses ocaml-freestanding]
  ↻ recompile fiat-p256          0.2.1 [uses ocaml-freestanding]
===== ↻ 4 =====
Do you want to continue? [Y/n] y

<><> Processing actions ><><><><><><><><><><><><><><><><><><><><><><><><><><><>
⬇ retrieved fiat-p256.0.2.1  (cached)
⬇ retrieved hacl_x25519.0.2.0  (cached)
⬇ retrieved ocaml-freestanding.0.6.2  (cached)
⬇ retrieved mirage-crypto.0.8.5  (cached)
⊘ removed   fiat-p256.0.2.1
⊘ removed   hacl_x25519.0.2.0
⊘ removed   mirage-crypto.0.8.5
⊘ removed   ocaml-freestanding.0.6.2
∗ installed ocaml-freestanding.0.6.2
∗ installed fiat-p256.0.2.1
∗ installed hacl_x25519.0.2.0
[ERROR] The compilation of mirage-crypto.0.8.5 failed at "dune build -p mirage-crypto -j 4".

#=== ERROR while compiling mirage-crypto.0.8.5 ================================#
# context     2.1.0~beta2 | linux/x86_64 | ocaml-base-compiler.4.11.1 | https://opam.ocaml.org/#706bf1d6
# path        ~/.opam/4.11/.opam-switch/build/mirage-crypto.0.8.5
# command     ~/.opam/opam-init/hooks/sandbox.sh build dune build -p mirage-crypto -j 4
# exit-code   1
# env-file    ~/.opam/log/mirage-crypto-554126-ad6566.env
# output-file ~/.opam/log/mirage-crypto-554126-ad6566.out
### output ###
#       cflags freestanding/cflags-freestanding.sexp (exit 1)
# (cd _build/default/freestanding && ./cflags.sh) > _build/default/freestanding/cflags-freestanding.sexp
# Fatal error:
# /usr/local/bin/opam: "open" failed on /home/kit_ty_kate/.opam/4.11/.opam-switch/packages/cache: Read-only file system



<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
┌─ The following actions failed
│ λ build mirage-crypto 0.8.5
└─ 
┌─ The following changes have been performed
│ ⊘ remove    mirage-crypto      0.8.5
│ ↻ recompile fiat-p256          0.2.1
│ ↻ recompile hacl_x25519        0.2.0
│ ↻ recompile ocaml-freestanding 0.6.2
└─ 

[AltGr]

Ouch, I see what happens here. This file is a new cache that can get written to after loading the installed opam packages files.
When a package gets installed or removed, opam wipes that cache to invalidate it, and then the next call to opam var will try to rewrite it. The package builds should really be calling opam var --readonly, though, in general any opam command that is possibly called from within opam should have --readonly.

Do you think we should push for more use of --readonly ? Solving this would otherwise be as simple as ignoring the error...

[AltGr]

Same issue happens with opam install alt-ergo-lib.2.3.3
Cf. OCamlPro/alt-ergo#364

We should probably just silence the error for now.

[kit-ty-kate]

I'm still unsure why opam var isn't readonly by default though. Is there a reason behind this? I think the mirage team (cc @hannesm) already asked this question somewhere but just in case

[AltGr]

Well, opam keeps some internal caches in sync, and opam var might trigger an update of those in some cases. There would be no technical difficulty in making it read-only, but that might increase the cost of repeated calls in a normal (non reentrant) setting.

[kit-ty-kate]

There would be no technical difficulty in making it read-only, but that might increase the cost of repeated calls in a normal (non reentrant) setting.

I don't think this is an issue. I don't think anyone use opam var outside of scripts to be used within opam itself.

[hannesm]

I don't know much about these bits in opam, but I'm curious why this is only an issue in 2.1...

I understand "opam keeps some internal caches in sync, and opam var might trigger an update of those in some cases" - but I do not know the full background story: why are there caches that may be updated? Do they improve the execution time of sth tremendously?

From a user perspective, I'd have never guessed that opam var may lead to a disk write anywhere. Since opam 2.0 did not lead to the observation of the behaviour described above, I'd be in favor to revert to the 2.0 behaviour.

[dra27]

Curiously, this should have been addressed #4405?

[kit-ty-kate]

As noted in #4429, the fix doesn't fully work. Could this issue be reopened?

[AltGr]

@hannesm well, computing the environment is quite involved, since it requires listing all installed packages, getting their definitions, etc. so caching this means a much welcome speedup (esp. when used in the shell prompt, emacs startup script, etc.)

Now when you are in the middle of changing the state of that switch, the cache needs to be invalidated; the default in case of a missing cache is for opam var to update it after computing its value, unless --readonly was specified. The proposed fix is to just give up on the cache update if you happen not to be able to write at that point.

[AltGr]

@kit-ty-kate can't manage to reproduce on master ; could you give a backtrace or at least the precise error you are getting ?

[hannesm]

@AltGr thanks for explanation -- I'm now convinced that --readonly should be added to the invocations when reading a variable -- since when is --readonly available in opam, was it already in 2.0.0 (or even 1.x)?

[kit-ty-kate]

@hannesm --readonly is available since 2.0.0, if you wish to be 1.x-compatible you might want to use --safe instead.

[kit-ty-kate]

I haven't seen these errors for a while. Closing. I'll reopen the issue or recreate it if I see the issue again.