Support Nix depexts with opam env

[RyanGibb]

Nix doesn't install packages in the traditional sense -- instead it puts them in a store and makes them available through environment variables. I.e., nix-shell -p gcc will drop us into a shell with gcc in the $PATH.

We can set appropriate environment variables with Opam to make system dependencies (depexts) available with Nix.
Similar to how nix-shell works under the hood, we create a Nix derivation such as

{ pkgs ? import <nixpkgs> {} }:
with pkgs;
let
  packages = [ gmp ];
  inputs = with buildPackages; packages ++ [ pkg-config ];
in
stdenv.mkDerivation {
  name = "opam-nix-env";
  nativeBuildInputs = inputs;

  phases = [ "buildPhase" ];

  buildPhase = ''
vars=("NIX_CC" "NIX_CC_FLAGS" "NIX_CFLAGS_COMPILE" "NIX_CC_WRAPPER_TARGET_HOST_x86_64_unknown_linux_gnu" "NIX_LDFLAGS" "PKG_CONFIG_PATH")
for var in "''${vars[@]}"; do
  escaped="$(echo "''${!var}" | sed -e 's/^$/@/' -e 's/ /\\ /g')"
  echo "$var    =       $escaped        Nix" >> $out
done
echo "PATH      =       $PATH   Nix"
  '';

  preferLocalBuild = true;
}

Which we can build to output a file with environment variables that make depexts available, in Opam's environment variable format. This file is a Nix store root, so it's dependencies won't be garbage collected by Nix until the file is removed.

This approach came from conversations with @dra27 and is distinct from previous approaches in that it supports providing development environment, which imperative installations with Nix don't #5332 (comment); and doesn't require the user to manage the environment outside of Opam, which would lead to a different workflow #5942.

Initial experiments were done using a package to set such environment variables https://github.com/RyanGibb/nix.opam.
However, in order to work with generic depexts (as opposed to just conf packages), to avoid cyclic dependencies (if conf packages depend on nix.opam but nix.opam depends on conf packages), and due to opam package sandboxing (nix derivation fetching would require opam sandboxing to be disabled or the nix store to be 'primed'), it's better implemented as a depext mechanism.

This has been tested and successfully creates an environment to provide packages conf-gmp and conf-libseccomp, as well as depexts directly.

[RyanGibb]

Rebased on master

[RyanGibb]

Looks like I'm running into NixOS/nix#10587 with the Docker image

[RyanGibb]

We might want to make these environment variables configurable in order to support new dependencies from Nix, which might require additional environment variables, without cutting a new release of Opam.

[RyanGibb]

Rebased on master -- and could do with another depext CI run if it needs re-approval!

[RyanGibb]

Okay! The CI is failing only due to ocaml/opam-repository#26261 (I've got it working locally with my branch of opam-repository)

[kit-ty-kate]

could you explain why all these Option.get are needed?

[RyanGibb]

It's because I don't have access to the switch state here edd9030#diff-6426799f367266b2a65325c870931b51df35852e12ddbe8580e74f4eeeb58177R1606 afaik because it hasn't been created at this point. We need to pass it in to OpamSysInteract.install* in order for Nix to write to files in the switch, but we also need to support installing system packages before this switch has been created in the existing OpamClient logic (which will do nothing for the Nix backend).

[kit-ty-kate]

i think it might be worth using a dedicated record instead of a tuple at this point. Otherwise it could become confusing very quickly with all those similar types around

[RyanGibb]

We have opamSysPkg.status?

[RyanGibb]

Though, we're also passing around the couple of available and required a lot, which perhaps warrants a separate record.

[RyanGibb]

It would be good if we could trigger a re-build of any packages that depend on depexts if the drv path changes, as if they link against a previous derivation it could be GC'd. An alternative might be to keep the old nix.env files around as Nix GC roots.

[RyanGibb]

Anyone wanting to play around with this on a NixOS system can try:

$ nix shell github:RyanGibb/nixos/#legacyPackages.x86_64-linux.nixpkgs.opam
$ opam --version
2.3.0

Or add it to your own overlay with:

{
  nixpkgs.overlays = [
    (final: prev: {
      opam = prev.opam.overrideAttrs (_: {
       src = final.fetchurl {
          url = "http://ryan.freumh.org/software/opam-full-2.3.0-nixos-depexts.tar.gz";
          sha256 = "sha256-mRxxZtWFgQ8v1szVq5g5+qVqa+OffoG1aHzGUiMMvT0=";
        };
        version = "2.3.0";
      });
    })
  ];
}

This URL is just the https://github.com/RyanGibb/opam/tree/nixos-depexts-2.3.0 branch with vendored dependencies since I couldn't get them working in Nix,