Incremental builds of dependencies

[Ma27]

First of all, thanks a lot for this! This is a big step towards how I'd like to see Nix being used for development!

I have a question and I'd like to get some feedback on whether this is actually reasonable. If I have the time for that, I may file a PR on my own later on if that's the case.

When I e.g. add a dependency to my Cargo.toml and run cargo build, only the newly added (or existing, but updated) crates will be rebuilt. When I do a nix build instead, all of the deps are rebuilt in the pname-deps drv. I'm wondering if it'd be somehow reasonable & possible to split the pname-deps derivation into even more derivations, i.e. for each crate my project depends on. That means, we'd need to represent the entire dependency graph from cargo as graph of derivations.

Benefits would be (1) faster rebuilds while developing and (2) being able to share built crates between projects (e.g. if both projects are using the exact same version of a dependency), IIRC even cargo can't do that currently (though I'm not sure if that'll actually be possible).
Downsides are most likely (1) more expensive evaluations and (2) higher maintenance workload in this project because we may need to adapt on newer Cargo releases if certain internals change.

Do you think that'd be a reasonable thing to do? As I said, this is for now a hypothetical question because I need a reality check whether that's actually a good idea or completely stupid, haven't done a deep-dive into Cargo internals yet :D

Follow-up question: do you think it'd be possible to actually build all files in their own derivations in craneLib.buildPackage? That'd be the holy grail (for me at least :p) considering that we'd have completely incremental Rust builds with Nix then.

See for related work in Haskell e.g. https://github.com/matthewbauer/ghc-nix (I know that it's using a slightly different approach - i.e. "injecting" Nix builds into ghc, but the end-result is pretty similar to what I'd like to have).

[dpc]

There's a substantial overhead of the derivation containing results of dependency build. Splitting builds of dependencies could make sense if you one could reliably partition the set of dependencies into ones that take a long time to build and change rarely, and ones that are quick to build and change more often.

In practice however this would be extremely tedious and complex, and in many scenarios the changes in the project would invalidate all deps builds anyway.

do you think it'd be possible to actually build all files in their own derivations in craneLib.buildPackage

This is not very practical/possible as compilation of each file in a crate require (potentially) all other files in a crate, and changes in one would invalidate builds in other. Rust compiles the crate as a whole.

Building each crate separately is possible, but requires reproducing all the rules of crate building of cargo. There are project that try to do that like cargo2nix. The complexity will be higher and the whole approach more fragile, but per-crate derivation is possible. In practice I doubt it's worth it, but that's just my opinion.

[ipetkov]

Hi @Ma27 thanks for the discussion!

I do want to say that I definitely wouldn't want to discourage the exploration of new ideas or possibilities of improving caching. That said, I believe that granular level of caching becomes impossible in practice without either changing how cargo itself behaves, or moving entirely away from cargo. Let me elaborate further:

---

Part of the complexity around cargo's caching has to do with the interaction of feature flags across crates. Adding a new crate to the dependency closure can result in rebuilding other, previously existing crates depending on how features are unified, meaning we can't just compile the new crate and its (new) dependencies and call it a day.

Suppose we have the following:

• a workspace we wish to build with a top level crate A
• A depends on crate B
• B depends on crate C and turns on the foo feature flag
• suppose now we introduce a change where A now depends on crate D
• D however depends on C and turns on the bar feature flag
• now cargo will need to recompile crate C with both the foo and bar features enabled
• This in turn would need to recompile B because implementations and functions could have changed

Thus it is only possible to (naively) cache cargo artifacts as long as the entire dependency closure remains intact. The Cargo.toml, Cargo.lock, and any enabled workspace feature flags are witnesses/lead to that specific closure, which is why updating Cargo.lock (even to add a new dependency) results in having to rebuild all dependencies.

(Technically cargo doesn't rebuild everything like this when you add a new dependency and build outside of Nix because it happens to reuse whatever artifacts on disk end up being valid, but then you're out there in the non-functional-mutable-state world)

---

Like @dpc mentioned above cargo2nix is another project which eschews relying on cargo directly in favor of generating explicit derivations of each crate being built. It does incur the complexity of having to parse out and fully understand what crates are present, along with any enabled features and rustflags and so on.

Obviously this comes with the benefit of being able to "observe" whether adding a new crate to the dependency closure influences having to rebuild other crates. If not, great, the artifacts can be reused!

The really big downside is the additional complexity of having to understand all about cargo's behavior and re-implement that on top of Nix, especially if new features are added to cargo which change this completely. Granted I've never deeply used cargo2nix but I'm not sure how easy it is to integrate other cargo-based tools with it (e.g. cargo-nextest, or cargo clippy, etc.) since it takes over managing the build directly.

If you don't care about using cargo then cargo2nix might be a great option! But given that the Rust community has standardized around using cargo proper my goal with crane was to embrace cargo instead of trying to re-implement it in Nix ;)

---

It's worth mentioning that I do remember there being some proposals for having cargo generate and output build-plans: rather than have cargo do the actual build, it could spit out a plan of what it would do (e.g. compile these crates in this order, run that build script, etc.). The idea was that other build systems could consume this and do the actual orchestration and artifact management themselves.

Unfortunately I haven't heard very much about it since then. If that does ever land I think it could lead to a whole bunch of optimizations and improvements! But until then we're back to the current status quo so long as we want to continue using cargo without reimplementing its behavior

---

Now to answer your questions more specifically:

Do you think that'd be a reasonable thing to do? As I said, this is for now a hypothetical question because I need a reality check whether that's actually a good idea or completely stupid, haven't done a deep-dive into Cargo internals yet :D

I personally think the maintenance burden of keeping up with cargo's internals would be too much for me (though if it could produce some kind of build plan with a stable format I'd change my mind). But if you are motivated enough to explore it I'd say go right ahead :)

Follow-up question: do you think it'd be possible to actually build all files in their own derivations in craneLib.buildPackage? That'd be the holy grail (for me at least :p) considering that we'd have completely incremental Rust builds with Nix then.

Like @dpc mentioned I don't think this is possible. Rust's compilation unit is the whole crate and you cannot compile individual files even if you try to invoke rustc directly

[drakon64]

It's worth mentioning that I do remember there being some proposals for having cargo generate and output build-plans: rather than have cargo do the actual build, it could spit out a plan of what it would do (e.g. compile these crates in this order, run that build script, etc.). The idea was that other build systems could consume this and do the actual orchestration and artifact management themselves.

cargo build --build-plan does this, but currently requires a Nightly build and for unstable options to be enabled. It's something though, so maybe incremental builds will be a possibility one day.

cargo build --build-plan is due to be removed but cargo build --unit-graph might do the same job?