[Cache] Somehow it doesn't seem to hit the cache

[smorimoto]

What I want to do is speed up the opam install process, but somehow it doesn't seem to hit the cache properly. This has been primarily tested on the GitHub Actions VM, but the opam install process doesn't seem to be as fast as expected on my local machine. However, the cache files seem to be generated properly, and it's likely that the cache was not hit or used properly. Is there any way to easily check if it's hit?

Reproduction

1. If the cache exists on cache storage, restore it to the cache directory

2. Put the following dune configuration to ~/.config/dune/config

(lang dune 2.0)
(cache enabled)
(cache-transport direct)

or

(lang dune 2.0)
(cache enabled)

3. Start the cache daemon

4. Install and build dependencies and source

5. Store cache directory contents to cache storage

Specifications

• Version of dune: 2.8.2
• Version of ocaml: 4.11.1
• Operating system (distribution and version):
  • Ubuntu 18.04
  • macOS 10.15

[bobot]

I believe many things conspires against this:

• opam uses sandboxing with bwrap which block Network connection, but for virtual machine it could be deactivated
• dune seems to not use cache when given -p (--for-release-of-packages); and I don't know why

In order to check the use of the cache you can start the cache server before with dune cache start --foreground, I hope it will help you.

[bobot]

The -p activate --release which activate --no-config. So no activation of caching.

[smorimoto]

So is it impossible to speed up the opam install process with the cache?

[rgrinberg]

@mefyl is there an another way to activate the cache somehow?

[mefyl]

You have the counterparts on the command line and in the environment: --cache=enabled --cache-transport=direct

DUNE_CACHE=enabled
DUNE_CACHE_TRANSPORT=direct

[smorimoto]

@mefyl Is there any difference between setting environment variables and adding a config file?

[mefyl]

There shouldn't be any, no.

[smorimoto]

Do you have any idea why the cache is saved but it doesn't seem to be used?

The first one does not use the cache with either opam install or dune build, but the second one seems to be used only with dune build:
https://github.com/smorimoto/ocaml-cohttp/runs/1780796904?check_suite_focus=true
https://github.com/smorimoto/ocaml-cohttp/runs/1781369028?check_suite_focus=true

[mefyl]

@bobot Regarding OPAM sandboxing, it should not be an issue with recent versions of OPAM since the dune cache is now mounted just like ccache. You can check for this snippet in .opam/opam-init/hooks/sandbox.sh:

add_dune_cache_mount() {
  DUNE_CACHE=${XDG_CACHE_HOME:-$HOME/.cache}/dune
  mkdir -p ${DUNE_CACHE}
  add_mounts rw $DUNE_CACHE
 }

@smorimoto I know nothing about github actions, what are the differences between the two builds ? Can you take a look inside the dune log file (_build/log), there should be some mentions of cache hits and misses.

[smorimoto]

The first does not add the dune global configuration file, nor does the daemon start, but the second does. Let me check the build logs.

[smorimoto]

The logs for opam were not logged for some reason, so I don't know, but at least the build of the software itself has hit the cache.

[bobot]

@mefyl @smorimoto great that dune cache work! Just for my understanding, why is the --unshare-net not problematic for the connection to the cache server?

[smorimoto]

(It still seems that the dune command call through opam does not hit the cache.)

[bobot]

Result of my experiments:

• --unshare-net should not be a problem with DUNE_CACHE_TRANSPORT=direct, it could be a problem with DUNE_CACHE_TRANSPORT=daemon.
• Without DUNE_CACHE_TRANSPORT=daemon the daemon is not used so nothing is connecting to it.
• With DUNE_CACHE_TRANSPORT=daemon and bwrap without --unshare-net, it still don't connect
• The environment variable indeed take precedence on -p
• With DUNE_CACHE_TRANSPORT=direct and bwrap (even with --unshare-net), it works. But it is hard to know because we can't give the --verbose through environment variable. I tested by adding the option inside sandbox.sh.

To sum-up it works with @mefyl environement options, but I don't know a simple way to show it.

[smorimoto]

It seems that the direct mode and sandboxing cannot be used at the same time:
https://github.com/smorimoto/ocaml-cohttp/actions/runs/524822187

[smorimoto]

I found that by disabling sandboxing and making the cache as large as possible, it was twice as fast:
https://github.com/smorimoto/ocaml-cohttp/runs/1806548690?check_suite_focus=true

[bobot]

It seems that the direct mode and sandboxing cannot be used at the same time:
smorimoto/ocaml-cohttp/actions/runs/524822187

Dune tells the problem, the script sandbox_exec.sh (sandbox.sh for macos) set DUNE_CACHE to a path instead of enabled or disabled.

[smorimoto]

​What do you mean by "set DUNE_CACHE to a path"? I don't think this is invalid, am I missing something?

[bobot]

OPAM resets(only on macos) the variable DUNE_CACHE (I don't know why), and so we have this error:

### output ###
# dune: environment variable `DUNE_CACHE': invalid value
#       `/Users/runner/.cache/dune', expected either `disabled' or `enabled'
# Usage: dune build [OPTION]... [TARGET]...
# Try `dune build --help' or `dune --help' for more information.

[smorimoto]

Hmm, weird... The same error also seems to be occurring on Ubuntu btw.

[bobot]

Can you give me a link to this error on ubuntu? Because I don't see in the code (bwrap.sh) where it would be done.

[smorimoto]

These 2 errors look exactly the same:
https://github.com/smorimoto/ocaml-cohttp/runs/1806221225?check_suite_focus=true
https://github.com/smorimoto/ocaml-cohttp/runs/1806221305?check_suite_focus=true

[bobot]

Indeed it has been fixed in ubuntu in ocaml/opam@201c304 . It should land in opam 2.1.0.

[smorimoto]

IIUC, that fix is also included in opam 2.0.8, so let's wait for it to be released: ocaml/opam#4425. When the release is published, setup-ocaml will automatically follow it, so we can just re-try the build and see if it works as we expect.

[smorimoto]

Hmm, ocaml/opam@201c304 does not seem to fix this.

[smorimoto]

Does it conflict with this one?
https://github.com/ocaml/opam/blob/cfeb651e565277545985eaaa8e7c2e8fba2bc0d3/src/state/shellscripts/sandbox_exec.sh#L51-L55
But it's weird that only macOS has this problem:
https://github.com/smorimoto/ocaml-cohttp/actions/runs/548452485

Error:  The compilation of stdlib-shims failed at "/Users/runner/.opam/opam-init/hooks/sandbox.sh build dune build -p stdlib-shims -j 5".

#=== ERROR while compiling stdlib-shims.0.3.0 =================================#
# context              2.0.8 | macos/x86_64 | ocaml-system.4.11.1 | git+https://github.com/ocaml/opam-repository.git
# path                 ~/.opam/ocaml-system.4.11.1/.opam-switch/build/stdlib-shims.0.3.0
# command              ~/.opam/opam-init/hooks/sandbox.sh build dune build -p stdlib-shims -j 5
# exit-code            1
# env-file             ~/.opam/log/stdlib-shims-5126-489943.env
# output-file          ~/.opam/log/stdlib-shims-5126-489943.out
### output ###
# dune: environment variable `DUNE_CACHE': invalid value
#       `/Users/runner/.cache/dune', expected either `disabled' or `enabled'
# Usage: dune build [OPTION]... [TARGET]...
# Try `dune build --help' or `dune --help' for more information.

[smorimoto]

@rjbou Do you have any idea why this is happening?

[rjbou]

There is different sandbox scripts, MacOS uses sandbox-exec. As you noticed, it this one, there is the definition of DUNE_CACHE variable in the script, and no link resolution as in bwrap one. It was added in ocaml/opam#4019 and in 2.0.6, i'm surprised that it wasn't reported yet.

[smorimoto]

i'm surprised that it wasn't reported yet.

That is probably due to the fact that very few people use this...
It seems that this problem can be fixed by simply changing the variable name in opam? If so, I can create a quick patch and then raise a PR for release 2.1.0.

[smorimoto]

This has been fixed by ocaml/opam#4535. Thanks to everyone for helping me investigate this issue!