Interface for system package manager to execute build plan

[Ericson2314]

Original Post

I wrote a fairly long email to the nix-dev list (http://lists.science.uu.nl/pipermail/nix-dev/2016-September/021765.html) proposing that cabal2nix work from cabal-install's build plans instead of the package cabal files themsevles. This would obviated needing to teach it to resolve configuring options etc, basically replicating Cabal. [The idea is of cabal-install resolving the plan and the system package manager executing the plan generalizes for any system package manager, not just Nix, hence not singling Nix out in the issue title.]

Reading the cabal-install section of the componentized Cabal proposal leads me to further propose that plan.json (or whatever should be the interface between the system package manager and cabal-install) contain exactly the component dependency graph proposed in the RFC. This granularity would be nice for Nix for the same reason it is nice for cabal and stack---I personally will contest that playing around with some packages and then having to rebuild them to enable tests is quite annoying.

A final extension to that would be to extend Cabal itself to also work from these a single-component manifests. The default Setup binary would gain the ability to work from either a manifest--corresponding to a single component, or a bunch of manifests--corresponding to that component and its dependency closure. [I say either because I am not sure whether the new-* work affect Cabal or just cabal-install. If Cabal too, I assume the closure is needed, if just Cabal then the single component's manifest is probably fine.]

There are two advantages of this final bit. The first is that we currently only use Cabal, not cabal-install, in Nix Haskell packages, and this would allow us to continue doing that. That said, I'm not sure what the advantages of this current approach so this may not be good motivation. The second is were we also wanted to use stack to generate build plans, the component dependency dag is probably a good format for both to support (I speculate given that the componentization proposal was originally @snoyberg's idea). stack --nix already exists but for reasons given in my mailing list post, I think the mechanism I'm describing would work strictly better.

Anyways, after writing the proposal for the Nix list, I thought it would be good to get the attention and opinion of upstream.

Current Plan for Nix

For non-local dev (packaging executables for nixpkgs), here is a sketch of the entire process:

1. Make the cabal.project
   • Use extra-packages: for the executables, since they come from hackage
   • Use compiler: to specify the compiler so we don't it built at config time.
   • Ignore native deps
2. cabal new-configure without any network access and limited FS access only to cabal-install's runtime deps + downloaded index
   • Neither the compiler, executable deps, or native library deps need to be built/exposed at this time.
   • TODO some way to give cabal-install a nix-downloaded index, ideally without weakening with the TUF stuff. Maybe .cabal/config already handles this? I assume we can put the config elsewhere with a CLI flag?
3. Resulting plan.json contains "install items" each with:
   • Basic metadata like package name, component name, and version
   • Relevant calls to ./Setup with all flags
   • TODO: pkg-config deps relevant to just this component. (./Setup configure is still responsible for checking whether those libraries are present, but cabal configure can anticipate when they will be looked-for.)
   • TODO: Deps on other "install items"
   • TODO: hackage urls + hash sufficient for Nix to fetch
4. Turn plan.json into a package set where each "install item" becomes a derivation. Either this comparatively simple task is the new job of cabal2nix, or this is so simple we can just use builtins.fromJSON and write in Nix.
   • Use Nix-style uniq ID's to replicate dependency graph structure.
   • Fetch and unpack Hackage tarballs.
   • Put the./Setup commands specified in the proper phases. Hopefully we pass cabal new-configure enough so that the ./Setup flags do not need to be modified.

Work to be done

• build Setup.hs with local compiler when cross-compiling #1493: host vs target tools

[23Skidoo]

/cc @ttuegel

[23Skidoo]

Related: #3651.

[Ericson2314]

So it's my opinion that (to borrow the git terminology) stack --nix implements porcelain without the plumbing. I.e, it looks nice but in my experience doesn't really work properly outside of more trivial scenarios, so this isn't a purely theoretic design disagreement.
What I'm proposing here is strictly not Nix-specific, but IMO gives Nix or any other auxiliary package manager the interface needed to do things right. After this is done, I think the time would be ripe to do a more slick CLI like stack --nix. With this plan, the implementation of #3651 would be quite different, but if we go this route we can redesign that when we get there.

[ezyang]

I wrote a fairly long email to the nix-dev list (http://lists.science.uu.nl/pipermail/nix-dev/2016-September/021765.html) proposing that cabal2nix work from cabal-install's build plans instead of the package cabal files themsevles.

That sounds fine, but it does mean we have to commit to a more stable and documented plan.json format. (We still don't know if we've put enough stuff in yet, if it actually is usable by people ;) It also means we have to commit to a specific interface for configuring and building packages. That also should be fine; the Cabal command line API moves very slowly.

A final extension to that would be to extend Cabal itself to also work from these a single-component manifests.

Are you sure you want to add a new single component manifest format, as opposed to plan.json recording a pile of configure/build flags/arguments that you need to use? That would be more backwards compatible and you could start using it today. And as I said above, the configure/build API is pretty stable.

Next we instruct cabal-install to decompose the plan into "sub plans" for each dependency/would-be system package---this is the biggest change needed for cabal-install, but one that any system package manager could leverage, not just us.

I don't know what you mean by this. Could you maybe write some concrete JSON pseudocode describing what a sub plan looks like?

Finally, each of those derivations can use cabal new-install to install their root package

Did you mean cabal install, or did you mean calling the Setup script above (which is what you mentioned earlier?)

since the Nix derivations will also depend on each other, cabal new-build's own caching should kick in so each Nix derivation does in fact correspond to each Haskell package being built.

new-build caching works in a very specific way: for any specific unit identifier we depend on, we look in .cabal/store to see if the directory for that library already exists (and it is in the package database). If so we improve into the installed version. Is that interface sufficient for Nix or do you need something else.

I didn't read the section on local dev with Nix, I know very little about how it works.

[ezyang]

@Ericson2314 are you interested in contributing and maintaining the necessary patches to cabal-install to make this work? I think it would be much appreciated by the Nix community.

[Ericson2314]

[On subway so limited Internet.] I'd hope since this isn't Nix-specific I wouldn't be solely on the hook for maintenance :), but I'm certainly down to implement.

[ttuegel]

Despite appearances, #3651 is actually orthogonal to this issue. What is proposed here is really an alternative to cabal2nix, which itself produces a poor approximation of a build plan. Even with #3651, Nix users would benefit from this proposal. And, even with this proposal Nix users would need #3651 to manage non-Haskell dependencies, which is actually my sole motivation for implementing it.

I don't think this proposal offers any features beyond what new-build does, just a different mechanism.

[ttuegel]

I want to single-out one statement from the mail to nix-dev:

During this stage it in principle only needs access to .cabal files / the Hackage index and a project config---certainly not any already built binaries like today.

The build-type: Custom interface makes this untrue, unfortunately.

[Ericson2314]

but it does mean we have to commit to a more stable and documented plan.json format. (We still don't know if we've put enough stuff in yet, if it actually is usable by people ;)

Well, if there is a helper executable to call GHC etc on behalf of the system package (e.g. stock ./Setup as I tentatively proposed only in this thread), then plan.json can still be somewhat of a black box.

It also means we have to commit to a specific interface for configuring and building packages. That also should be fine; the Cabal command line API moves very slowly.

Because all the configuration should be done beforehand by cabal-install, the CLI for this should be very simple and need not look nice so as to not sqat on valuable CLI real-estate should it need to be deprecated.

Did you mean cabal install, or did you mean calling the Setup script above (which is what you mentioned earlier?)

In the mailing list, I erred on the side referring to cabal-install, as I am not sure quite what the division of labor is for new-build. Here, I went on a limb and proposed using ./Setup, hoping Cabal is enough. Sorry for the inconsistency + confusion.

I didn't read the section on local dev with Nix, I know very little about how it works.

That is icing on the cake / more #3651's territory, so feel free to ignore it for now :).

I don't know what you mean by this. Could you maybe write some concrete JSON pseudocode describing what a sub plan looks like?

new-build caching works in a very specific way: for any specific unit identifier we depend on, we look in .cabal/store to see if the directory for that library already exists (and it is in the package database). If so we improve into the installed version. Is that interface sufficient for Nix or do you need something else.

Well, to get the easy concerns out of the way, we would have one new-style .cabal/store per system package, analogous to what we do with GHC_PACKAGE_PATH and legacy DBs today.

Now the complex bit. Well, what, if anything in .dist-new is used? Ignoring local builds for now, that directory will not be accessible for the easiest method on Nix's end, or we can rebuild part of it. The key issue is that Nix will be more conservative about hashing, and so we don't want irrelevant parts of .dist-new affecting the current build.

Extrapolating from Nix, I assume building needs more than the .cabal/store and the current *.cabal because then we'd be back to square one configuring plans biased by what's been built---and doing so per package/component not per cabal build which is even worse. So parts of the plan are needed. Now packages that aren't a (transitive) dependency can definitely be pruned from the plan so as not to trip of Nix's conservative hashing. The question remains do we need just the plan for the current package/component, or also the plan for each of it's transitive dependencies? Ideally, since the current plan includes the unique identifier (with hash), enough metadata can be put in Cabal(-install)'s store so only the JSON object (/binary equivalent) for the current package/component is needed.

[One final hiccup is I'd like not to have all non-haskell deps build and exposed at cabal new-configure-time, but if deferring that part of configuring until the packages/components that need them are being built is hard, let's forget about that for the first version of this.]

[Ericson2314]

@ttuegel Hmm? it was my understanding that we pray assume that custom setups don't affect the build plan, so cabal2nix certainly ignores them. Does cabal new-configure not?

[Ericson2314]

[edit pipeline sketch is now at the top of this thread]

[Ericson2314]

CC @peti as this is more active than the nix-dev thread.

[Ericson2314]

Come to think of it, this may also benefit getting rid of ghc-cabal in Hadrian (snowleopard/hadrian#18). The analogy is Nix : Nixpkgs : cabal2nix :: Shake : Hadrian : ghc-cabal.

[ezyang]

This thread is sprawling so I'm going to respond comment by comment. But maybe there is some big picture comments that need to be made.

Well, if there is a helper executable to call GHC etc on behalf of the system package (e.g. stock ./Setup as I tentatively proposed only in this thread), then plan.json can still be somewhat of a black box.

At the very least, the tools should realize that plan.json defines a number of install items which each can be built and installed individually. After all, you want a one to one correspondence between install item and Nix derivation. (I don't call these packages because they are not necessarily packages; e.g., with per-component builds you'll get an install item per component.) And you also need to know the dependencies for Nix derivations. So I'm very doubtful you want to black box this.

Because all the configuration should be done beforehand by cabal-install, the CLI for this should be very simple and need not look nice so as to not sqat on valuable CLI real-estate should it need to be deprecated.

The only "configuration" cabal-install does today is compute the set of flags to pass to ./Setup configure and other commands.

Well, to get the easy concerns out of the way, we would have one new-style .cabal/store per system package, analogous to what we do with GHC_PACKAGE_PATH and legacy DBs today.

So, I am still not sure if this is what you actually want, but we'll have to do some work in this case. Maybe you just don't ever want improvement to take place for Nix-style, in which case you just don't need the store at all.

Now the complex bit. Well, what, if anything in .dist-new is used? Ignoring local builds for now, that directory will not be accessible for the easiest method on Nix's end, or we can rebuild part of it.

If you are never building any inplace packages, as would be the case for non edit-recompile, then there is NOTHING in dist-newstyle that actually gets used. Certainly cabal new-build will cache the result of dependency solving so that it runs faster, but you could safely just always blow away dist-newstyle before computing one of these plans.

assume building needs more than the .cabal/store and the current *.cabal because then we'd be back to square one configuring plans biased by what's been built---and doing so per package/component not per cabal build which is even worse.

But obviously you need the source for the Cabal package, to actually build?

I mean, you also need to have the dependencies in a visible package database for the build. But if you're calling Setup directly you can just pass in all the package databases of each of the dependencies that were managed by Nix and that would be fine.

I wonder if there is some architectural description that is missing here but would help guide the discussion

So parts of the plan are needed. Now packages that aren't a (transitive) dependency can definitely be pruned from the plan so as not to trip of Nix's conservative hashing.

I'm still confused. Aren't you going to make a derivation per install item, which depend on each other?

The question remains do we need just the plan for the current package/component, or also the plan for each of it's transitive dependencies? Ideally, since the current plan includes the unique identifier (with hash), enough metadata can be put in Cabal(-install)'s store so only the JSON object (/binary equivalent) for the current package/component is needed.

I think that's true? But it's still hard for me to understand what gave you the idea that you need a plan for each transitive dependency.

[ezyang]

Just don't deal with native deps because it's ok to only expose those for the packages that need them? Or provide a "pkg-config name / executable name -> external-tool-specific opaque string" mapping. Cabal-install assumes every native library / executable in the set of keys is exists when configuring, and forall p: package. k: lib/exe. v: opaque-blob. pneedsk => kmapTov => v inPlanJsonFor p` to make the new reduced cabal2nix's life easier.

OK, so maybe we can do something to make Nix's job easier here. The way that pkg-config deps are handled is Cabal just calls pkg-config to figure out what information is necessary. In fact, this call doesn't happen until we actually are configuring with ./Setup configure. Maybe you can just use the normal thing that happens in this case?

We also need the source of packages to be downloaded by Nix. Worse-case does this mean hot-patching every hackage dep into a local dep?

But if you're calling the Setup script directly, you can just download the tarball as you want it, unpack it, and then you're off to the races. But note that you may need to build the Custom setup script.

Split the build plan up so that for the Sake of Nix's hashing no package knows more than it needs to. Per the previous post I hope that's just each/component's own plan, as it can get the rest from the (cabal(-install)) store.

See my comments above. Can't you just build the derivations as you traverse the build plan.

new-install isn't implemented, but new-build already uses cache for external packages, but nix-provided source means we have a local package? These are my conflicting thoughts

The point is that you're not going to use new-build to actually build, right? So you're just going to call the Setup script directly, so you don't have to worry about any caching business.

OVERALL: So, maybe it would be helpful if there was some architecture document describing how all the pieces of Cabal/cabal-install fit together?

[Ericson2314]

Thanks, these comments help a lot.

I wonder if there is some architectural description that is missing here but would help guide the discussion

For reference, https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/generic-builder.nix is how we currently invoke ./Setup today.

[Ericson2314]

The only "configuration" cabal-install does today is compute the set of flags to pass to ./Setup configure and other commands.

Ok good, that shrinks the difference between how things already work and how I want them to work. This project basically becomes shrinking that file I linked.

So I'm very doubtful you want to black box [plan.json]

Ok I'm convinced. I'll reduce my claim to that the flags for each ./Setup command can be a plan list—we shouldn't need to know why they are passed.

[Ericson2314]

So, I am still not sure if [multiple caches] is what you actually want, but we'll have to do some work in this case. Maybe you just don't ever want improvement to take place for Nix-style, in which case you just don't need the store at all.

What is "improvement"? Finding something already cached? I'm fundamentally relying the nix derivations to be the same whenever cabal-install would use its cache on its own. So maybe you're right in that we don't rely on the new caching, just the ability to write down the plan up front.

[Ericson2314]

If you are never building any inplace packages, as would be the case for non edit-recompile, then there is NOTHING in dist-newstyle that actually gets used. Certainly cabal new-build will cache the result of dependency solving so that it runs faster, but you could safely just always blow away dist-newstyle before computing one of these plans.

I mean what must be preserved in between the creation of the plan and it's execution—and remember cabal-install won't be present to execute the plan. So it's sounds like all we need to extract and preserve from that directory after cabal-install is run is plan.json?

[Ericson2314]

But obviously you need the source for the Cabal package, to actually build?

Oh yes, duh. I probably meant in addition to that.

I mean, you also need to have the dependencies in a visible package database for the build. But if you're calling Setup directly you can just pass in all the package databases of each of the dependencies that were managed by Nix and that would be fine.

I was trying to get at what I just commented above. Without cabal-install computing the grand plan on the fly, we need to get those flags for ./Cabal into the nix derivation. .cabal/store contains/is a package db right?

[Ericson2314]

I'm still confused. Aren't you going to make a derivation per install item, which depend on each other?

I was thinking if we needed to compute the flags on the fly from the plan for transitive deps. But if the plan already contains all the flags scratch this. All we need is the JSON object for the current install item and the flags it contains.

[Ericson2314]

I think that's true? But it's still hard for me to understand what gave you the idea that you need a plan for each transitive dependency.

Yeah, so as far as invoking ./Setup, the plan itself is enough, and as far as ./Setup doing the right thing with those flags, metadata in the cache/package dbs is enough. This matches how things already work in nixpkgs.

[Ericson2314]

In fact, this call doesn't happen until we actually are configuring with ./Setup configure. Maybe you can just use the normal thing that happens in this case?

Oh great! Perhaps cabal-install could just copy any relevant native library deps to plan.json, so the new cabal2nix just needs to look at plan.json and not also *.cabal files?

[Ericson2314]

But if you're calling the Setup script directly, you can just download the tarball as you want it, unpack it, and then you're off to the races.

That's fine--great if the plan.json can contain everything we need (including hashes) to do this download. I assume cabal new-configure does not download any hackage sources because that is what the index is for---I hope that is the case as we do need it to work without network access.

But note that you may need to build the Custom setup script.

Heh, that's the last hitch. I'd love the plan to build custom setups as a separate item. Also for cross compiling (including GHCJS) we use a native compiler to build Setup scripts, so it would be cool if cabal-install could work that way too.

[Ericson2314]

The point is that you're not going to use new-build to actually build, right? So you're just going to call the Setup script directly, so you don't have to worry about any caching business.

As long as extra-packages: * making Hackage packages "local packages" doesn't change the flags passed ./Setup different, we're good. (If the roots of the plan are built differently, we won't be able to cache as well.)

[Ericson2314]

OVERALL: So, maybe it would be helpful if there was some architecture document describing how all the pieces of Cabal/cabal-install fit together?

Yes! All I knew going in to this is a) cabal-install is the executable b) Cabal is the library that cabal-install and ./Setup uses, and c) how the nixpkgs Haskell infra works without cabal-install.

Also, while I need to stop thinking about this right now, I'll eventually go back and edit my summary comment to reflect what's been cleared up.

[peti]

I have been trying to understand this proposal from a Nix perspective for a while now, but the purpose of these suggestions continues to elude me. In particular, I don't understand what problem this proposal is supposed to solve. Can someone please give me concrete example of a use-case that doesn't work in Nix today but that would work if this proposal were to be implemented?

[Ericson2314]

The primary purpose is to make our Haskell infrastructure smaller and simpler by leveraging cabal-install, not to add more features.

The ghcjs problem I mentioned in the mailing list thread has been fixed: NixOS/cabal2nix#230 but with these changes such problems would never occur because no nix-specific code (cabal2nix, nixpkgs, or otherwise) would need to evaluate cabal files.

I've talked about a nicer workflow for local development, but haven't been able to articulate that clearly, and as I say above it icing on the cake, so let's just ignore that extra work + extra benefits for the time being.

[ezyang]

I'll reduce my claim to that the flags for each ./Setup command can be a plan list—we shouldn't need to know why they are passed.

Yep I agree.

What is "improvement"? Finding something already cached? I'm fundamentally relying the nix derivations to be the same whenever cabal-install would use its cache on its own. So maybe you're right in that we don't rely on the new caching, just the ability to write down the plan up front.

Yeah. So the way that cabal new-build works is that first computes a plan from first principles; i.e., it doesn't look at the store at all. Once it has the plan and computed all the hashes, only THEN does it "improve" the plan, replacing the packages it planned to do with the ones that are actually stored.

I mean what must be preserved in between the creation of the plan and it's execution—and remember cabal-install won't be present to execute the plan. So it's sounds like all we need to extract and preserve from that directory after cabal-install is run is plan.json?

Yeah. Setup script doesn't use any of the cabal-install infrastructure; it creates dist directories on its own.

.cabal/store contains/is a package db right?

It contains a package database.

But if the plan already contains all the flags scratch this.

The plan already contains flags.

Perhaps cabal-install could just copy any relevant native library deps to plan.json, so the new cabal2nix just needs to look at plan.json and not also *.cabal files?

I guess we could record the build-tools? Put that in the summary. (see more below)

if the plan.json can contain everything we need (including hashes) to do this download

It probably has enough info? (Maybe it's missing which Hackage revision was used?) In any case if it doesn't we should add it.

I'd love the plan to build custom setups as a separate item

Yep, that's something I want too. Not implement yet, alas.

As long as extra-packages: * making Hackage packages "local packages" doesn't change the flags passed ./Setup different, we're good.

Hmm, actually, that is not going to work, because if you build it inplace it will get inplace package IDs, which is not what you want. Maybe make a fake package which build-depends on the stuff you actually want to build? (I wonder if cabal new-configure pkg1 pkg2 pkg3 works...)

Also, while I need to stop thinking about this right now, I'll eventually go back and edit my summary comment to reflect what's been cleared up.

Yes please do. Actually, put it at the top of the ticket, please!

[ezyang]

@Ericson2314 I would seriously consider opening up a https://github.com/ghc-proposals/ghc-proposals for your summary. It will make following the convo easier, as you can just update the proposal doc as you learn more.

[Ericson2314]

@ezyang Cool, it sounds like we are on the same page now.

Yeah. So the way that cabal new-build works is that first computes a plan from first principles; i.e., it doesn't look at the store at all. Once it has the plan and computed all the hashes, only THEN does it "improve" the plan, replacing the packages it planned to do with the ones that are actually stored.

Ah, OK. I hope improving just means skipping/removing install items that are built, not modifying any of the others?

Hmm, actually, that is not going to work, because if you build it inplace it will get inplace package IDs, which is not what you want.

Oh well, makes sense.

Maybe make a fake package which build-depends on the stuff you actually want to build? (I wonder if cabal new-configure pkg1 pkg2 pkg3 works...)

I don't think the fake package would if the root set includes executable, however. Even putting them as a tool dependency would fail when cross compiling as those are (or ought to be) built natively. However cabal new-configure pkg1 pkg2 pkg3, or even one invocation for each package, works fine.

I do wonder if a better strategy is to build anything that transitively depends only on Hackage sources in the deps, and anything else in the local cache. This is only slightly different than currently as external deps that depend on local deps are already built in the local cache. A local package would then probably be redefined as just anything not from Hackage.

Yes please do. Actually, put it at the top of the ticket, please!

I would seriously consider opening up a https://github.com/ghc-proposals/ghc-proposals for your summary. It will make following the convo easier, as you can just update the proposal doc as you learn more.

Ok put it at the top for now, but I'm happy to write a formal proposal too when I have the time. I want to do this with Cargo too eventually, so I imagine I'd get some pretty sweet prose reuse :).

[ezyang]

OK. I hope improving just means skipping/removing install items that are built, not modifying any of the others?

No modification occurs.

However cabal new-configure pkg1 pkg2 pkg3, or even one invocation for each package, works fine.

Good. Note that you probably don't want to call individually, because then the dep solver has latitude to pick different deps for each package in each case. Probably not what you want.

I do wonder if a better strategy is to build anything that transitively depends only on Hackage sources in the deps, and anything else in the local cache. This is only slightly different than currently as external deps that depend on local deps are already built in the local cache.

I don't understand. If a Hackage source depends on local source code (not on Hackage) of course it can't be installed globally.

[Ericson2314]

Note that you probably don't want to call individually, because then the dep solver has latitude to pick different deps for each package in each case.

Oh yeah, I was forgetting that while I can use the same project file that is not enough to guarantee same constraint resolution unless I lock down every package.

I don't understand. If a Hackage source depends on local source code (not on Hackage) of course it can't be installed globally.

Yeah that part remains the same. Basically I am saying make extra-packages not affect external vs local. Right now a hackage package is built in the local store if a) it is in fact a local package because of extra-packages or b) it depends on other local packages, right? I'm proposing throwing out a) leaving just b).

[ezyang]

Yes, that seems like a reasonable design. @dcoutts what did you think extra-packages would be used for when you added it? Can we just change its semantics?

[hvr]

To which components does extra-packages refer to btw?

[ezyang]

Mu. The question is equivalent to saying what components does packages refer to. In principle, all of them, but tests/benchmarks are only enabled on a best effort basis, and the actual ones that get used depend on the actual build targets you subsequently pass.

[Ericson2314]

Bump. @dcoutts do those semantics for extra-packages sound good? Should I break this up into bite-sized issues like makes Setup a separate component?

[Ericson2314]

Now a few year's later, there's https://github.com/input-output-hk/haskell.nix which gets the division of labor just about right. (Any further improvements would involve big changes to Cabal itself; stuff I won't propose just yet :).)