node-gyp requires access to the nodejs sources during build phase

[andir]

Describe the bug
Many projects (those with C bindings) use node-gyp as part of their dependencies to generate bindings to C libraries. node-gyp requires the nodejs sources to be available (either supplied via commandline or it will try to download it from the internet). We do not have control over the command line flags that are used to invoke node-gyp as that might be done by any of the dependencies. We also can't allow network access as that would defeat the whole idea of not being a fixed output derivation.

node-gyp is looking into $HOME/.npmrc for the disturl= setting which is supposed to point to a mirror of the distributed source archives.

I tried to "trick" them into downloading from the store by creating a kinda franken-dist-directory with the following expression:

  frankenDist = nodejs: pkgs.runCommand "frankenstein-nodejs-sources" {} ''
    mkdir -p $out/v${nodejs.version}
    ln -s ${nodejs.src} $out/v${nodejs.version}/node-v${nodejs.version}-headers.tar.gz
  '';

It creates the correct folder structure and the file exists on the right location in the store. The problem arises when node-gyp uses requests to download that file. Unlike NPM it doesn't seem to support file:// urls. My current idea is to patch node-gyp to support that but that would still require all of our packages to update to a version of node-gyp that supports file:// URLs.

To Reproduce
Try building a project that requires node-gyp e.g. the tweag website (unfortunately not public).

Expected behavior
node-gyp should just use the source tarball that nodejs.src points at.

[andir]

While not great I managed to workaround that specific issue by spawning a webserver within the build environment. That seemed like a reasonable workaround for the initial PoC. Ideally all of these node packages would provide escape hatches to provide files locally without network I/O. The entire ecosystem doesn't really seem to be there just yet.

The following is my rough shell expression that kinda works (until you hit the next problem):

let
  pkgs = import ./nix { };
  npmlock2nix = import ./default.nix { inherit pkgs; };


  frankenDist = nodejs: pkgs.runCommand "frankenstein-nodejs-sources" {}  (let
    # we must recompress the file as node seems to dislike the .xz extension :/
    recompress = src: pkgs.runCommand "recompress.tar.gz" {
      inherit src;
    } ''
      TMPDIR=$(mktemp -d)
      cd $TMPDIR
      tar xf $src
      tar cf $out *
    '';
  in ''
    mkdir -p $out/v${nodejs.version}
    cd $out/v${nodejs.version}
    ln -s ${recompress nodejs.src} node-v${nodejs.version}-headers.tar.gz
    sha256sum *.tar.gz > SHASUMS256.txt
  '');

in
npmlock2nix.shell {
  src = <tweag-www>;
  npmCommands = [ "npm run build" ];
  nativeBuildInputs = [
    pkgs.python3
    pkgs.pkgconfig
    pkgs.vips
    pkgs.glib
  ];
  installPhase = "cp dist $out";
  preBuild = ''
    #echo "disturl=file://${frankenSteinDist pkgs.nodejs}" > $HOME/.npmrc
    (cd ${frankenDist pkgs.nodejs} && python -m http.server) & # FIXME: stop webserver in postBuild again
    echo "disturl=http://localhost:8000" > $HOME/.npmrc
  '';
}

[m1cr0man]

I have encountered similar issues before trying to build a project using node-sass. The solution there was to yarn2nix' ability to override the preInstall script for that package specifically to do a custom node-gyp build. You can see it here.

Another option I tried previously was pre-warming the headers cache that node-gyp downloads to, as detailed here, but this didn't seem to do the trick. Now that I think of it, it might have been because of the path that $HOME was set to. This solution might work in place of your http server solution.

[andir]

This has been fixed in the add-build branch by simply passing --nodedir to the npm install line.

…

On Thu, Aug 6, 2020, 00:46 Lucas Savva ***@***.***> wrote: I have encountered similar issues before trying to build a project using node-sass. The solution there was to yarn2nix' ability to override the preInstall script for that package specifically to do a custom node-gyp build. You can see it here <https://github.com/m1cr0man/nix-configs/blob/master/packages/chronograf/ui.nix#L30> . Another option I tried previously was pre-warming the headers cache that node-gyp downloads to, as detailed here <nodejs/node-gyp#1154 (comment)>, but this didn't seem to do the trick. Now that I think of it, it might have been because of the path that $HOME was set to. This solution might work in place of your http server solution. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#23 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAE365FS5FIBN6BYU7KJEALR7HOMLANCNFSM4PTJNVNQ> .