Interleave download actions with build/install actions

[Armael]

Caveats:

• Change of behavior: in the current implementation, if a "remove" needs a "fetch" and the fetch fails, opam will print a warning but do the remove nonetheless (by just removing the files). In this PR, if the fetch fails, the remove will automatically get cancelled. In practice, this can happen only for a few packages that do not use light-uninstall, but it would be annoying nonetheless. It would probably be nice to give the user a way around this, using --force or something.
• Currently, the jobs pool is shared between build and fetch actions. This can be improved, by implementing two separate pools in the scheduler, which should improve parallelism further.
  This can be done in a separate PR.

[Armael]

I just added a commit that restores the existing behavior for "remove" after a failing "fetch": a warning gets printed, and the remove still occurs.

[avsm]

Currently, the jobs pool is shared between build and fetch actions

Weirdly enough, this speeds up some builds for me. The reason is that jobs in opam is propagated to parallel dune builds with the same value, which in turn overcommit the CPU since they all spread to the full load of parallel jobs. With this, some of the opam jobs are fetching, while the dune jobs spread to all CPUs :-)

[Armael]

Well, the point of the PR is to speedup the build! But it's interesting to know that it has a positive effect even without a separate queue for downloads (which would definitely make sure that downloads cannot starve builds).

[AltGr]

removed

[Armael]

I don't understand what you mean here.

[AltGr]

Just one small typo :)
Thanks a lot!

[AltGr]

After testing, one request could be to unify the output of finished downloads with that of actions:

[atdgen.2.0.0] found in cache
[atdgen-runtime.2.0.0] found in cache
[datakit-ci.0.12.3] found in cache
[atdgen-runtime.2.0.0] found in cache
[atdgen-runtime.2.0.0] found in cache
[github.4.0.0] found in cache
[github-unix.4.0.0] found in cache
[merlin.3.2.2] found in cache
[mirage-ci.~dev] no changes from git+https://github.com/avsm/mirage-ci
[yojson.1.7.0] found in cache
⊘ removed   merlin.3.2.2
⊘ removed   mirage-ci.~dev
⊘ removed   datakit-ci.0.12.3
⊘ removed   github-unix.4.0.0
⊘ removed   github.4.0.0
⊘ removed   atdgen.2.0.0
⊘ removed   atdgen-runtime.2.0.0
⊘ removed   yojson.1.7.0
∗ installed yojson.1.7.0
∗ installed atdgen-runtime.2.0.0
∗ installed atdgen.2.0.0
∗ installed merlin.3.2.2
∗ installed github.4.0.0
∗ installed github-unix.4.0.0
∗ installed datakit-ci.0.12.3
∗ installed mirage-ci.~dev

here they don't interleave, but it would be nice for the downloads to be displayed like removals or installs, e.g. :

⇓ fetched github.4.0.0 (cached)