put application source and dependencies in separate nix packages

[adrian-gierakowski]

Currently, it seems, all node deps are bundled together with application source in a single package. This leads to wasted disk space and network bandwidth when only application code changes. Would it be possible to separate those, for example by symlinking the packages into .yarn/cache during nix-build instead of copying them?

[stephank]

I'd really like this, but unfortunately there were issues with that last I tried. (Possibly it broke module resolution at run-time, or something along those lines.) But I haven't tried in a while, certainly not since Yarn 3.

Even if it breaks things, maybe it could be made to work with some upstream changes.

I'm currently not working on this, though. 🙂

[adrian-gierakowski]

From my limited tests, I found that replacing cp here with ln -s works in conjunction with nodeLinker: pnp, but breaks with nodeLinker: node-modules.

I think it could be beneficial to add an argument to the generate yarn-project.nix allowing the user to choose between copying and linking.

I'm currently not working on this, though. slightly_smiling_face

Have you're needs changed, or have you found a better solutions for packaging node applications with nix?

[stephank]

Sorry, I meant this specific feature. I do use the plugin in general at work, but admittedly not as much as I had planned. We use it for projects that drive our deployments and require integration with Nix.

We did make an attempt at deploying client projects, but the developer experience wasn't great compared to Docker, which is what everyone is used to. (Perhaps extra difficult to sell because we're a Mac shop.) So I'm pretty much the only one enthusiastic about Nix in the team. 🤷‍♂️

[adrian-gierakowski]

cool, happy to submit a PR for this. How would you like me to name the argument? Would a boolean symlinkPackages work?

[stephank]

Sounds good!

Even if you'd rather have it default false for now, maybe we can make it default true for PnP in the future with more testing. I think Yarn's config system allows default: null along with a type definition of boolean | null, where we use null to detect PnP or not. 🙂

[mmahut]

This would be very beneficial!

[adrian-gierakowski]

Hi all

I spent quite a bit of time on this since it turned out that what I wrote in the comment above was wrong. I had to jump through a bunch of hoops, including making changes to yarn itself to make it work nicely with pnp (not sure about the node_modules mode since I'm not interested in it).

You can see all the work I've done so far on dev branch of my fork. The initial implementation resorted to patching .pnp.cjs, converting paths of individual packages in /nix/store to relative. I then made a few modifications to yarn to simplify nix build script (the changes to yarn are referenced from the commit message).

I haven't opened a PR with these changes here due to the fact that I don't like the first solution and the second one requires changes to yarn which have not been upstreamed yet (I initiated the process with this issue however I haven't heard back from the maintainers after initial exchange.

Note, the dev branch of my fork also contain support for private npm registries and git repos which I'd be happy to submit as a separate PR.

[adrian-gierakowski]

One other feature I've got in the pipeline is support for file: protocol, which however requires additional changes to yarn due to the way they calculate directory hashes: write permissions affect the hash, so the hash changes when content of the directory referenced with file: protocol is copied to nix store (nix wipes all write permissions).

[nicknovitski]

This is exciting!

One other feature I've got in the pipeline is support for file: protocol, which however requires additional changes to yarn due to the way they calculate directory hashes: write permissions affect the hash, so the hash changes when content of the directory referenced with file: protocol is copied to nix store (nix wipes all write permissions).

Couldn't nix ignore the provided hash for a file: dependency and import it as a local path, no hash needed?