Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buildNpmPackage: init #189539

Merged
merged 1 commit into from
Nov 9, 2022
Merged

buildNpmPackage: init #189539

merged 1 commit into from
Nov 9, 2022

Conversation

winterqt
Copy link
Member

@winterqt winterqt commented Sep 3, 2022

Description of changes

This PR introduces buildNpmPackage, a new builder for npm-based projects. Alongside it is fetchNpmDeps, a new fetcher for npm dependencies inspired by fetchYarnDeps in that it constructs a cache for npm to install from using a small purpose-built fetcher.

I've tested this with both npm 8 (shipped in Node 18, the latest version) as well as npm 6 (shipped in Node 14). The cache format has been stable for 5 years, which makes me confident in using it as an output for a FOD.

I believe this is an improvement in terms of reproducibility over existing solutions to this issue, most notably #128749. This is notably more reproducible for a few reasons, but the main one is that it's guaranteed to have the same output across systems. This is because npm allows installing dependencies only on a specific platform, which obviously causes irreproducibility.

npmInstallHook depends on npm/cli#5430 when using packages with {pre,post}pack scripts. This will be released in npm 9, but I'll need to backport it to Node versions that don't include it.

Things done
  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 22.11 Release Notes (or backporting 22.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
    • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
  • Fits CONTRIBUTING.md.

# TODO: how do we expose these like Rust? `npmHooks`?
inherit (callPackage ./hooks { inherit nodejs; }) npmConfigHook npmBuildHook npmInstallHook;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really sure what the best name for exposing the hooks would be, which is why I didn't do it here.

pkgs/build-support/node/build-npm-package/default.nix Outdated Show resolved Hide resolved

outputHashMode = "recursive";
outputHash = hash;
} // args); # TODO: removeAttrs? what attrs?
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw this in fetchCargoTarball -- what's the benefit of removing the args it does? Should we remove any here?

@@ -62,6 +34,5 @@ stdenv.mkDerivation rec {
homepage = "https://jellyfin.org/";
license = licenses.gpl2Plus;
maintainers = with maintainers; [ nyanloutre minijackson purcell jojosch ];
platforms = nodejs.meta.platforms;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to self: set this by default in buildNpmPackage.

@@ -4112,7 +4112,9 @@ with pkgs;

jellyfin-mpv-shim = python3Packages.callPackage ../applications/video/jellyfin-mpv-shim { };

jellyfin-web = callPackage ../servers/jellyfin/web.nix { };
jellyfin-web = callPackage ../servers/jellyfin/web.nix {
buildNpmPackage = buildNpmPackage.override { nodejs = nodejs-14_x; };
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should probably comment why this is done (Jellyfin's web client uses lockfile v1 at this tag, which doesn't work well on newer versions with our setup).

Maybe I can also add a hint for this somewhere, based on the version? Or maybe just in docs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it'd be better to have an attribute along the lines of nodeJs which defines the version used.

That would make it more clear from the child derivation which version is used, and make changing it (upon a package update or something) more intuative.

Copy link
Member Author

@winterqt winterqt Sep 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused -- what do you mean by this, where would that attribute be located? Can you provide an example of what that API would look like?

Ah, I see what you mean -- define the Node copy used in the arguments to the builder. My only issue with that is that we don't do it for any other language, see rustPlatform and buildGo117Module. We should probably strive for consistency with other languages, in this case. How about at least defining aliases like buildNpmPackageNode14 (I can't come up with any clear naming, but you get what I mean), so overrides for common cases don't have to be done manually?

Copy link
Member

@IvarWithoutBones IvarWithoutBones Sep 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right, I forgot other builders do it like that. Creating buildNpmPackage14 etc sounds like a good idea 👍

Maybe buildNpm14Package would be better naming? I'm not entirely sure.

Comment on lines 16 to 23
{ name ? "npm-deps"
, src ? null
, srcs ? [ ]
, patches ? [ ]
, sourceRoot ? [ ]
, hash ? ""
, ...
} @ args: stdenv.mkDerivation ({
Copy link
Member Author

@winterqt winterqt Sep 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why these are set to their default/empty values in fetchCargoTarball... I did it here anyways, though I'm guessing they can probably be removed (both here and in fetchCargoTarball).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think { foo ? {} } @ args: doesn't keep the {} for foo, args removes the default.

@winterqt
Copy link
Member Author

winterqt commented Sep 3, 2022

A npmTestHook which runs npm test should probably be added.

Comment on lines 85 to 89
let lock: PackageLock = serde_json::from_str(&fs::read_to_string(args.next().unwrap())?)?;

let out = PathBuf::from(args.next().unwrap()).join("_cacache");

let print_hash = args.next().is_some();
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a help message would be good, considering this will be used by update scripts.

, srcs ? [ ]
, patches ? [ ]
, sourceRoot ? [ ]
, hash ? ""
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should support for sha256 be added, to match other fetchers? (As well as adding npmDepsSha256 to buildNpmPackage.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, i think that's a good idea.

Copy link
Member

@IvarWithoutBones IvarWithoutBones left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR is looking great, I'm very happy to see npm build support being improved this substantially :)

Left you some comments/suggestions.

echo "Finished npmSetupHook"
}

configurePhase=npmSetupHook
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think its a good idea to make this overridable.

Suggested change
configurePhase=npmSetupHook
if [ -z "$configurePhase" ]; do
configurePhase=npmSetupHook
fi

Same goes for the other phases.

Copy link
Member Author

@winterqt winterqt Sep 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the only phase that that wasn't added -- I'm not sure when you'd want to override this when using buildNpmPackage (and duplicate the effort in the phase), since it sets up everything, but for consistency I guess that makes sense?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be worth adding corresponding dont options (or options that act as that) to all these phases as well (which I only currently do in two of them).

Though, now that I think about it, having both conditionals creates confusion:

  1. When using buildNpmPackage, setting a custom configurePhase will cause the npm phase not to be used
  2. When using this hook on its own, why would you set dontNpmConfigure instead of just... not including the hook?

I don't know, I'm confused. Cargo does this in its build hook, and I don't really see why both are needed.

@jonringer What do you think about this, as the person who added these options in #114716? It seems like these options don't really provide much benefit when hooks are used properly -- maybe I'm missing something?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be worth adding corresponding dont options (or options that act as that) to all these phases as well (which I only currently do in two of them).

I definitely agree, that's the most consistent with other builders as well AFAIK. Can't hurt to have a mechanism to disable these when using buildNpmPackage :)

When using buildNpmPackage, setting a custom configurePhase will cause the npm phase not to be used

Yes, which might be preferable in some scenarios. That way you can have complete control over the build process, you can override the phase with custom behaviour that produces a similar result yourself.

When using this hook on its own, why would you set dontNpmConfigure instead of just... not including the hook?

That does seem useless yeah, I can't imagine any scenario you'd do that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't hurt to have a mechanism to disable these when using buildNpmPackage

To be clear, what I mean is that all of these should be disabled if a custom phase is already set -- having dont options is useless as long as those conditionals exist.

, srcs ? [ ]
, patches ? [ ]
, sourceRoot ? [ ]
, hash ? ""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, i think that's a good idea.

Comment on lines 16 to 23
{ name ? "npm-deps"
, src ? null
, srcs ? [ ]
, patches ? [ ]
, sourceRoot ? [ ]
, hash ? ""
, ...
} @ args: stdenv.mkDerivation ({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think { foo ? {} } @ args: doesn't keep the {} for foo, args removes the default.

@IvarWithoutBones
Copy link
Member

I just saw this is still marked as a draft, it appears i got a bit excited with the review 😅

Hope you dont mind

@winterqt
Copy link
Member Author

winterqt commented Sep 3, 2022

I just saw this is still marked as a draft, it appears i got a bit excited with the review 😅

These are the kind of reviews I want, hah -- these'll get it ready to actually be merged.

@winterqt
Copy link
Member Author

winterqt commented Sep 4, 2022

@IvarWithoutBones I've migrated to jq -- that wasn't as bad as I thought.

I can't respond to some of your comments for whatever reason... any clue why? This is all I see:
image

These look like responses to my other comments, but GitHub... doesn't properly show them as responses? 🧐

Copy link
Contributor

@yu-re-ka yu-re-ka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I won't have the spoons to look at this in detail anytime soon.

I really like the overall design of the fetcher, it is similar to fetchYarnDeps which is already proven, just for npm. Much less risks for FOD reproducibility compared to running npm ci in an FOD.

Doing the second part - buildNpmPackage - right is pretty hard though. I feel like Javascript builds don't really match the stdenv model with its phases well. I wonder if the fetcher could be useful already without a specialized build environment, or just with an npm install hook. This could reduce the scope of this PR a lot.

Anyways, thanks for working on this!

@yu-re-ka
Copy link
Contributor

yu-re-ka commented Sep 4, 2022

Oh, there is a critical feature for helping debug FOD issues that we have in fetchYarnDeps, and I'd also like to see here:

Copy the package-lock.json to the FOD output.
In npmInstallHook, compare the package-lock.json from the cache FOD to the package-lock.json present in the working directory.
If it doesn't match, warn the user that they might have forgotten to update the FOD hash and don't even attempt to use the cache.

@IvarWithoutBones
Copy link
Member

Doing the second part - buildNpmPackage - right is pretty hard though. I feel like Javascript builds don't really match the stdenv model with its phases well.

Do you have an example as to what could cause problems with our stdenv's phases? I'm not very familiar with Javascript, but the current implementation looks like it fits in fairly well.

I wonder if the fetcher could be useful already without a specialized build environment, or just with an npm install hook. This could reduce the scope of this PR a lot.

I do think using just the fetcher would be more work in the end. Packages could be refactored to use it but would still end up having a fair bit of code duplication. They would have to be refactored again once buildNpmPackage gets merged.

The added complexity of the configure and build hooks is fairly small, I'd say it is definitely worth trying to get those additions merged as well if we're gonna include the install hook.

I do think reducing the scope of this PR is something that needs to happen though. In my opinion we should drop the commits converting packages to this new builder. Those will all need to be reviewed by the individual packages maintainers, which will greatly increase the time and care needed to get this merged.

Without those changes I think the diff is very reasonable.

@IvarWithoutBones
Copy link
Member

IvarWithoutBones commented Sep 4, 2022

@IvarWithoutBones I've migrated to jq -- that wasn't as bad as I thought.

Neat, this looks a lot nicer IMO :)

I can't respond to some of your comments for whatever reason... any clue why? This is all I see:

These look like responses to my other comments, but GitHub... doesn't properly show them as responses? 🧐

I think that's because I was responding to comments you posted in my review. Github just shows my one comment but doesn't display any additional context or let you reply in that case. It's kinda weird. If you scroll up to your original comment you should be able to reply.

@winterqt
Copy link
Member Author

winterqt commented Sep 4, 2022

Doing the second part - buildNpmPackage - right is pretty hard though. I feel like Javascript builds don't really match the stdenv model with its phases well.

Can you provide an example as to what you mean?

I feel like the default phases I have here are pretty representative of what projects that use npm do:

  1. configurePhase: installs dependencies
  2. buildPhase: runs a build script
  3. installPhase: npm pack + wrapping binaries

Of course, these can always be overridden by packages when needed, but I feel like these are good defaults.

@winterqt
Copy link
Member Author

winterqt commented Sep 5, 2022

Copy the package-lock.json to the FOD output.

This is already done, by the way. :)

@DavHau
Copy link
Member

DavHau commented Oct 26, 2022

The cache format has been stable for 5 years, which makes me confident in using it as an output for a FOD.

Still, putting complex logic inside an FOD always comes with some risk. If at any time, there appears to be a bug in your fetcher, you might not be able to fix it without breaking peoples hashes. Also, adding features later might be hard, because those must never change the output for existing users.

It would be good if critical inputs, like the rust code of the fetcher, would trigger a re-build of the FOD once changed.
I think that can be achieved, by hashing the store path names of these inputs via builtins.hashString and putting that hash into the name of the FOD.

@winterqt
Copy link
Member Author

It would be good if critical inputs, like the rust code of the fetcher, would trigger a re-build of the FOD once changed.

I already do this for a cherry-picked set of test cases, which will be run automatically by OfBorg whenever the code is changed (see here).

@winterqt winterqt force-pushed the npm-ng branch 2 times, most recently from 8fc8ac2 to 90250e2 Compare October 27, 2022 00:30
@winterqt winterqt marked this pull request as ready for review October 27, 2022 00:32
@winterqt
Copy link
Member Author

winterqt commented Oct 27, 2022

Hi all.

This push addresses a majority of comments surrounding the docs and flags, as well as fixes using old lockfiles with modern npm. As such, I've marked this PR as ready for review.

I still need to find a clean way to make a npmSourceRoot -- just not sure what to name it. This option would allow specifying a path to cd to, where the package-lock.json is. Thoughts welcome.

For future reference: that one Nix test is failing even when marked as flaky. Triggered a re-run.

@dit7ya
Copy link
Member

dit7ya commented Nov 1, 2022

I am super excited about this getting merged to nixpkgs. Thank you so much for this amazing work! How hard do you think it would be to write something like this but for the pnpm package manager (which uses a pnpm-lock.yaml lockfile)? I can try on my own if you have some pointers.

@happysalada
Copy link
Contributor

I'm also excited to try this tbh, this might need more work, but I think it's ready for testing, we can add subsequent PRs later.

@happysalada happysalada merged commit 1672290 into NixOS:master Nov 9, 2022
@winterqt
Copy link
Member Author

winterqt commented Nov 9, 2022

I would appreciate you having asked me if this was acceptable to be merged, @happysalada. I was waiting on some reviews. Maybe I should have just kept this as a draft? 😕

@happysalada
Copy link
Contributor

My bad, there usually isnt a lot of action on some MRs, i didnt want to let this go to waste.

Lets keep the discussion going here to see if any changes are requested. Ill be sure to ask you before a merge on any follow ups

@winterqt
Copy link
Member Author

winterqt commented Nov 9, 2022

My bad, there usually isnt a lot of action on some MRs

Please be more diligent in reading the (most recent) discussion.

i didnt want to let this go to waste.

I don't see what you mean? This would have been merged in the next few weeks. There is still active interest in it. It wouldn't have gone to waste in any circumstance.

@happysalada
Copy link
Contributor

There were several previous attempts at this that just died out after inactivity.

Its often quite hard to know if something will just die or if there is still legitimate interest.

Being eager about this i didnt read the comments well enough.

Please accept my apology for merging too early.

@fricklerhandwerk
Copy link
Contributor

We can still revert and re-open, to avoid locking users into an unfinished API. Just sad to lose some context of then discussion. It's your PR @winterqt, what do you prefer?

@delroth
Copy link
Contributor

delroth commented Nov 9, 2022

FWIW I would not have found this if it wasn't merged today, and it was exactly what I needed for a new package (etherpad-lite) that just wasn't going to work with node2nix. Leaving some feedback here from my experiments, lmk if you'd rather collect this somewhere else. Ordered from most important (bugs -> missing features) to least important (docs/confusion -> notes):

  • I think there's an issue with OldPackage -> Package conversion where a dependency with bundled: true (and thus no resolved/integrity) can take precedence on a dependency declaring resolved/integrity, which causes packages to be missing from cache. I've hack-patched this locally in the Cargo code by skipping OldPackage objects with bundled: true, and it seems to work. You can check for fast-deep-equal in etherpad-lite's package-lock.json for an example.

  • If a dependency's version is a git URL (e.g. "https://github.com/mapbox/node-sqlite3#593c9d498be2510d286349134537e3bf89401c4a") npm install tries to use git and contact the remote (in the main derivation, not the fixed output deps one):

    Installing dependencies
    npm ERR! Error while executing:
    npm ERR! /nix/store/86vv52z14rp30rdw0jpyadspf4ii4psb-git-2.38.1/bin/git ls-remote -h -t https://github.com/mapbox/node-sqlite3.git
    npm ERR!
    npm ERR! fatal: unable to access 'https://github.com/mapbox/node-sqlite3.git/': Could not resolve host: github.com
    

    Using "codeload.github.com" URLs in the package-lock.json fixes that, but I'm not sure why :)

  • The current npm-install-hook does not work with npm v6, it doesn't support --omit. Instead it's very confused at the "dev" on the command line and tries to ask the registry about it :P

    npm ERR! code ENOTCACHED
    npm ERR! request to https://registry.npmjs.org/dev failed: cache mode is 'only-if-cached' but no cached response available.
    

    Doing instead NODE_ENV=production npm prune should be equivalent and work on both v6 and v8.

  • The npm prune seems to have another weird issue where it can't find something from the cache which worked fine for all prior npm invocations. In my case:

    npm ERR! request to https://codeload.github.com/mapbox/node-sqlite3/tar.gz/593c9d498be2510d286349134537e3bf89401c4a failed: cache mode is 'only-if-cached' but no cached response available.
    

    For now I've disabled the "npm prune" until I can figure out what's going on. This happens to also be a package that needs to be node-gyp built, so maybe this has something to do with it.

  • fetchNpmDeps does not currently handle URLs like "github:user/repo#rev" which are apparently valid for npm. I worked around this with a patch to rewrite these to "https://github.com/user/repo#rev" in the src's package-lock.json, but the Rust code should likely take care of that instead.

  • I hit this in the npm ci phase:

    npm WARN old lockfile The package-lock.json file was created with an old version of npm,
    npm WARN old lockfile so supplemental metadata must be fetched from the registry.
    npm WARN old lockfile
    npm WARN old lockfile This is a one-time fix-up, please be patient...
    

    Which obviously fails because this is not in the fixed output derivation anymore. I worked around this by overriding nodejs = nodejs-14_x; but I wonder if this could be handled better (either by doing the requisite package-lock massaging in the fixed output derivation, or better diagnostics/error messages).

  • nativeBuildInputs = [ python3 ]; needs to be manually added if a package requires a node-gyp build. I don't know if that's good or bad or just needs to be documented, just making a note of it in case it's not a deliberate decision.

  • The error message when npm build fails should likely mention dontNpmBuild, I had to dig it up from the hook's source code. That should probably be more documented.

  • This new buildNpmPackage seems like it completely obsoletes applications/version-management/sourcehut/fetchNodeModules.nix by doing the same thing better :-) Yay for less custom "frameworks" in nixpkgs.

I'm super excited to see something making nodejs/npm app packaging in nixpkgs better! Let me know if any of my previous points are unclear and if I can help with repro or anything. https://gist.github.com/delroth/e7b67c8cc514a9de62f6c53656d1c96a is my current draft package with which I found these issues :)

@winterqt
Copy link
Member Author

@happysalada It's alright.

@fricklerhandwerk I'm going to keep it like this, but I'd really appreciate a review of the documentation (as you didn't respond to my previous request for one). I want it to be ~perfect before branch-off/before 21.11, hopefully.

@delroth I've addressed your issues in #200470, but I also wanted to comment on a few things that we didn't fix in #dev or the linked PR:

nativeBuildInputs = [ python3 ]; needs to be manually added if a package requires a node-gyp build. I don't know if that's good or bad or just needs to be documented, just making a note of it in case it's not a deliberate decision.

This was deliberate. I don't see a reason for packages that don't use node-gyp to needlessly depend on Python 3 (yay, closure size).

This new buildNpmPackage seems like it completely obsoletes applications/version-management/sourcehut/fetchNodeModules.nix by doing the same thing better :-) Yay for less custom "frameworks" in nixpkgs.

:) I'll get this switched over.

I'm super excited to see something making nodejs/npm app packaging in nixpkgs better!

❤️ Yeah, me too.

@winterqt winterqt deleted the npm-ng branch November 10, 2022 03:45
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/announcing-js2nix-scale-your-node-js-project-builds-with-nix/23111/8

@reckenrode reckenrode mentioned this pull request Nov 12, 2022
13 tasks
@dhess
Copy link
Contributor

dhess commented Nov 22, 2022

Thanks for this. Sorry if I'm missing something obvious, but if I'd like to use this not just to package someone else's Node package for Nix, but also to develop/maintain my own, how does one get a Nix shell with the package's dependencies in it?

For example, if I've got a Node package in a git repo, and I'd like to use buildNpmPackage to build it and test it using Nix, how would I go about creating a Nix shell for the package such that I could do:

% nix develop
...
bash-5.1$ npm build

where all the Node dependencies needed for the npm build step are either copied or symlinked to ./node_modules?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.