cabal new-install UI design concept (WIP)

[hvr]

Status

• Executables
  • Local
  • Nonlocal new-install: nonlocal exes #4825 (see also new-install symlink location #5000)
• Libraries
  • Local
  • Nonlocal

---

There's at least two axes for use-cases which need to be considered for the UI design: Most importantly is the distinction between

• making executables available
• making libraries available in GHC_ENVIRONMENTs (for GHC 8.0.2+)

And then sources can be distinguished by being either

• Non-local packages
• Local packages/projects

With the new nix-store paradigm, the idiomatic way to "install" things would be to first populate the requested target in the nix-store, and install a reference to the materialised artifact in a "view" (this signficantly differs depending on whether it's an executable or a library).

In this paradigm, the "installation" rather refers to installing a symlink or adding a reference to a package-env, than to build the actual artifacts. In some cases, all a cabal new-install operation does degenerates into replacing the target of an existing symlink (e.g. if you first "install" alex-3.2.0, and then you install alex-3.2.1, and then decide to install alex-3.2.0 again; the 3rd operation would merely update the symlink, as the store would already contain both versions of alex)

Basic UI

The basic UI would look like

$ cabal new-install <build-target-spec> [<install-location>]

(NOTE: as of #4825, the syntax is cabal new-install [--symlink-bindir=<install-location>] <build-target-spec>)

A build-target would be something like

Omission of <install-location> would denote a default location.

For libraries the default would be the current value of GHC_ENVIRONMENT, or otherwise the default environment (this matches the ghc-env lookup logic of GHC 8.0.2+).

For executables, the install location denotes a bin folder; by default this would be the value configured in ~/.cabal/config (e.g. ${HOME}/.cabal/bin) or possibly in the in-scope cabal.project.

Executables

Non-local packages

Installing executables from non-local packages is the easiest case; in fact, this can be emulated already now via script like https://gist.github.com/hvr/c77c54d682555b7dd4fe1248732fe978

Which first causes the requested executable to be materialised (if it wasn't already cached) in the nix store via a dummy package description containing build-tool-depends: ${PKG}:${EXE}, and then
locates the resulting executable to install a symlink to the desired folder-location.

Local packages

This case is currently less straightforward when the non-local components have associated package-data (#4120).

However, this use-case is quite important, e.g. when a cabal project is embedded into a larger build-system which needs to invoke cabal to make available some "local" executable to the rest of the
build-system, then such a project is often just part of the source-tree and as such comprised of local
unpublished) packages.

Libraries

Libraries are a bit more complicated, as package-environments should ideally represent views into the nix-store providing a flat consistent install-plan where each package exposed shall be compatible with each other package in the same view.

The simple case is when creating a new package-environment, and specify /all/ libraries to be installed therein upfront. This is the known situation as handled already by cabal new-build via build-depends speecifications.

However, cabal new-install should also support installing new packages into an existing view in an incremental way. But this means that if we may get into a situation where the "old" cabal install
would warn about running into a "reinstall"-situation.

One way to design the UI would be to ask the user whether we should try to resolve, and come up with a new consistent monolithic install-plan for the set of packages that were already available in the package-environment plus the newly to be added packages.

This would take into account the original version constraints imposed if there were any specified for the pre-existing packages. We may need something like per package-environment "world" files, to basically
keep track of an incrementally growing list of build-depends lines (and also possibly constraint-lines and other flags such as profiling levels). This way the user could have control over the package environment in a direct way by editing the world-files in order to manually resolve hard conflicts.

Non-local packages

To some degree, this could be emulated right now with an external shell (similiar to the executable-install-case mentioned above) by using a dummy .cabal file where the requested libraries are
specified via build-depends, and the automatically resulting .ghc.environment.* file which is copied over to the package-environment specified as the install-location.

Local packages

For one, parts of this is already automatically provided by the automatic creation of .ghc.environment.* files, but you'd currently need to manually copy them to their <install-location>. But this would also be limited to single cabal.projects currently.

This is more complicated just as for the local executable case, as associated package-data of local libraries is not handled conveniently for this use-case currently (#4120).

---

TODO:

• describe how extra-packages: fits into this scheme (see this comment)

[BardurArantsson]

I'm not sure I really understand the use case for installing libraries this way. Can you explain what the actual end-user's use case would be? (Is it REPL use without having to have a .cabal file or something like that?)

[hvr]

@BardurArantsson Yes, it's exactly that; we want to be able to support .cabal-less workflows and most importantly, without having to go through cabal new-repl or cabal exec. This new paradigm replaces the previous single user package db which caused all sorts of usability issues (and ultimately incompatible with nix-style local builds) with the new approach of having the ability to manage multiple "views" into your nix store (one of which is the special default view which GHC looks up by default).

[BardurArantsson]

I wonder, would it be too onerous to just require users to specify the "world" file themselves instead of cabal maintaining it (statefully)? AFAIUI that's pretty similar to how stack does it, and it seems pretty reasonable, both for the REPL case (stack.yaml) and the script case... since all you have to do is specify the list of packages+versions you want.

[hvr]

@BardurArantsson Well, we can also support that workflow of course (we actually have something like that already: if you work in the context of a cabal.project file, that's what the extra-packages field is meant for -- to state deps beyond those needed by your local packages to make available to local GHCi sessions)... as long as we also provide the more interactive one... as for the script case, see #3843 (there I agree that it makes more sense to have the dependency spec stated upfront)

[BardurArantsson]

I'm not sure I see any advantage whatsoever for the stateful/interactive one, but maybe that's just me. It seems more difficult conceptually to have to explain "views", how to update them, etc. that to just say "plop a file in your working directory" (or wherever).

/me shrugs :)

[ezyang]

I'm pretty OK with not supporting local libraries (and even exes) in the beginning. If you have a source tree, it's not a far step to just working with it directly, no need for install. cabal install somelib to play around with it in GHCi; cabal install for a random development copy of a library lying around my file system... that's a lot more questionable.

@BardurArantsson I think it is reasonable for a user to say, "I want package X installed on my system, and I want it installed ASAP." Because life is too short to muck around with package systems all the time.

[hvr]

If you have a source tree, it's not a far step to just working with it directly, no need for install

Try hooking up cabal new-build to a Makefile to build a local executable and update some symlink according (including proper dep-tracking and minimal rebuilds - and don't cheat by using cabal-plan), and tell me again that it's not a far step working directly with it... ;-)

(Here's one helper script I currently use for that; but I'd rather have this supported directly in cabal)

[ezyang]

Fair enough!

[23Skidoo]

#4120 is relevant here:

20:50 < refold> do you guys think that copying data files to somewhere inside dist-newstyle and setting env vars in new-run to point there is an 
                acceptable solution?
20:51 < fgaz> the advantage i see is that modifications to the source don't propagate to the built file until rebuild
20:51 < fgaz> BUT new-run implies a rebuild
20:52 < dmwit> refold: I think it depends a little bit on what new-install is going to do.
20:52 < dmwit> So let me comment on why I think that.
20:52 < dmwit> Currently, .cabal/config gives you lots of control over where binaries and data files go.
20:53 < dmwit> I suspect this makes it kind of tricky for the final binaries to use relative paths to get from "where the binary is" to "where the 
               data files are".
20:53 < dmwit> (I don't know how that's currently handled. Maybe installed executables always use absolute paths?)
20:54 < fgaz> i think so
20:54 < dmwit> If new-install is going to respect those same configurations, *and* we want the filepath discovery to use relative paths, that means 
               we are basically giving the user carte blanche to write anywhere inside dist-newstyle while copying files.
20:54 < fgaz> teated and yes, absolute path and the data files actually get installed
20:55 < dmwit> That seems like a dangerous plan.
20:56 < dmwit> So if that's what new-install is going to do (and I think it probably should), I think it probably makes sense to pick a distinguished 
               location in dist-newstyle (or in the package root) for files to go, and point at them with environment variables.
20:57 < dmwit> On the other hand, if it's okay for new-install to wrest a little bit of control from the user, and choose a distinguished relative 
               path between executables and data files, then I think it would make sense to copy the files into the appropriate place in 
               dist-newstyle and not set any environment variables.
20:58 < dmwit> Now that I've laid out what I think the alternatives are, it seems clear we should be setting the environment variables. If you take 
               away fine-grained control over binary location and data-files location, people who package Haskell programs are probably going to be 
               quite annoyed.

I should note that new-build already takes some control from the user, i.e. the store location doesn't seem to be configurable. But @dmwit makes a good point that we should consider the needs of packagers and distributors.

[23Skidoo]

Another interesting issue raised by @dmwit is the interaction with new-run and whether new-run should run exes from non-local dependencies (which is useful, as those may have different versions from the ones accessible via symlinks in ~/.cabal/bin).

[tom-bop]

One way to design the UI would be to ask the user whether we should try to resolve, and come up with a new consistent monolithic install-plan for the set of packages that were already available in the package-environment plus the newly to be added packages.

Other than asking the user, isn't this just what happens when the build-depends are changed between cabal new-builds?

(Very eager for this feature, btw!)

[typedrat]

#4120's semi-solution is not enough if we want to support local installing.

I can see an easy solution for executables (instead of trying to use a symlink, generate a tiny script that sets the correct environment variables and stuff, basically cd project/dir && cabal new-run thing) but there's no way to easily embed that kind of patch-up for libraries.

Either we decide on a more permanent/invasive solution that is strong enough that the library can just Know™ how to find the data files or we just refuse to support that because it's of semi-questionable usefulness.

[fgaz]

@typedrat
I think the right thing to do is to treat the local package like a remote one and build&install it in the store. Then the actual installation/symlinking can be done by the current implementation. This also has the advantage of freezing the installed package to the version on which you ran the new-install command, and ensuring it can alwayse be run regardless of the status of the project's source. It also avoids triggering unexpected rebuilds when running the installed program.

This is much like how nix itself does it. You nix-build a package, and then you can nix-env --install the result, which is a fixed path in the store and will not change unless you rebuild and reinstall it.

[typedrat]

Currently blocking progress with @fgaz's idea is that building out of line requires a SourceHash, and I don't know how to intelligently define that.

[typedrat]

I'm half tempted to just put this on the back burner until I get sdist up and going, then have part of the process be "do a sdist". 😂