Separate cabal install --lib into own command (cabal env)

[phadej]

... e.g. cabal env.

People do confuse what "installing libraries" and "installing executables" means, and after answering on few issues, I think stronger that separation of commands will make UX clearer.

An alternative is to drop --lib flag, and teach cabal install to do the right thing every time. I don't see how it would work.

---

edit by @fgaz: I'm hijacking the OP to raise the visibility of the cabal env prototype: https://github.com/phadej/cabal-extras/tree/master/cabal-env
help & feedback are welcome

[typedrat]

The only alternative that I can see is to make it so that cabal install just installs every public executable or library (or foreign library if someone who actually has a project to test with wants to land that nascent patch). I don't like that, but personally I don't like the idea of adding a whole new command, either.

[phadej]

@typedrat the drawback of install everything, is that when you e.g. cabal install hlint, or cabal install stylish-haskell (which has more dependencies), all these will be in the default environment, therefore causing the "environment cabal hell" earlier.

That said, I guess there's a way to tell solver to solve for all components in the target package. I don't fully understand what will happen if the component sets change across versions: e.g. yaml package, with public libraries they might come and go: these are major version changes, but if you say cabal install lens it's unclear whether you want the latest version or just some version.

One way is to have cabal install pkg to always default to the library, and if you want executable, then you must say cabal install pkg:exename, which is "just" flipping the --lib default; but it's more explicit (and also related to #6369). One variation is to default to executable installation, if that's the only component; but that's not true for many exes (e.g. mentioned hlint and stylish-haskell, and cabal-install-2.2.0.0, so cabal install cabal-install could not default to installing executable: there is a library).

This is a tough UX question.

Also I motivate cabal env command because then you can have subcommands, to

• list what's in the environment
• list available environments
• manage them in some other ways

One could argue that "it's just moving and editing files", but editing files is very bad UX; nor is well scriptable (c.f. cabal user-config for updating ~/.cabal/config; way better than seding).

So I think that cabal env command is motivated already alone, the same way as cabal sandbox was "just ghc-pkg and pass few flags to cabal install"; but we need good UX for a features to be used. (Relatedly: when there would be good cabal env command, and cabal store gc, then @Mistuke's argument will become void).

[danidiaz]

One argument for keeping both behaviors under the same command is that, conceptually, both installing an executable and installing a library take something that lives safely in the store under the control of Cabal, and put it "somewhere else" not directly under Cabal control, where it can be useful to other tools but also potentially overwrite stuff that shouldn't be overwritten. That said, if we had multiple commands that did that, a sentence like "Warning: this command can alter global settings not directly under control of Cabal" could simply be added to each command's help.

One argument for splitting the command: I have the impression that some newcomers to cabal have the idea that cabal v2-install is used to install the dependencies required by their package (when they really should be using cabal v2-build instead). What's worse, v2-install actually tries to install libraries somewhere. The newcomer will get supremely confused. Having a separate command for libraries, with a name less suggestive than "install", could help with this "UX failure mode".

Not to mention that the notion of "global package environment" is not very clear for newcomers (and it isn't very clear to me, either). In what sense is the global environment different from the Nix-like store? Are they the same thing? They are both "global", after all. A whole separate command for managing them would be a better place from which to hang documentation about the concept and its relationship to Cabal.

A further hint that the command should be split is that there are flags that only seem to affect one of the behaviors. --install-method is only for executables, isn't it? And --package-env only makes sense for libraries.

[m-renaud]

Proposed cabal env API:

Subject to bikeshedding, of course.

Base command: cabal env

Flags:

• --env=<ENV> - the environment to operate on (defaults to default)
• --overwrite-existing-entries=[pkglist] - overwrite existing entries in <ENV> if there is an existing version with a different hash (to resolve Reinstall of package breaks things #6391).
  • [pkglist] is a comma separated list of packages names to overwrite (could accept all if we forbid a package named all on hackage /shrug)
  • Alternatively, just boolean --overwrite-existing-entries flag to overwrite all.

Sub-commands:

• list - list the current environments (eg. default)
• info - information about the specified <ENV> (such as packages in the environment and their versions)
• add [<packages...>] - Add all <packages> to the specified <ENV>
• remove <ENV> [<packages...>] - Remove all <packages> from the specified <ENV>
• delete --env=<ENV> - Delete the specified <ENV> (--env must be explicitly specified)

/cc @phadej @yaxu

[phadej]

@m-renaud That is good start.

Let me comment based on cabal-env experience.

• list and info are good.

• Same comment about delete and remove. Specifically as these
  commands are destructive we should name environment deletion command
  somehow that you don't invoke it by accident thinking you'll
  remove a package.

  Use-case: I'd advice tidal users to use tidal environment for tidal
  stuff, so it's not affected by whatever else they might do.

• I'd have cabal env remove [--env=ENV] package [package2...]

  There is no point to remove all packages from the environment,
  It is "easier" to remove whole environment.

cabal-env works so that it explicitly stores which packages
are asked to be added to the environment:

-- This is GHC environment file written by cabal-env
--
clear-package-db
global-package-db
package-db /home/ogre/.cabal/store/ghc-8.8.3/package.db
package-id base-4.13.0.0
package-id some-1.0.1-6ae17ba0ce2d6d4e39d44a0cb504c88eaeb5f1021bb9d0c11208fd57b99020d1
-- cabal-env packages: some -any
-- cabal-env plan:
-- cabal-env     /Td6WFoAAATm1rRGAgAhARYAAAB0L+Wj4ActAq5dAD2IiGZT0UYdkKRpnoLIXvjSQtxNmBJAdmbOadFu
-- cabal-env     YeYvCmrbLipDaWlevQY6O15sAH0//RR1U2U3r3L3iNuCtsO4ulgbIB+DPxWzZebpraqywXx7veuy+94D

When new package is asked to be added to the environment
explicitly added packages are required to be there also,
(IIRC TODO: and versions are marked as preferred), so silent upgrades
don't happen. This is to prevent "rebuilding everything" after
cabal update.

cabal-env also stores a serialised version of plan.json data,
such that when new package is requested, and if it's in the plan already,
the operation is quick: just regenerate the environment file.
cabal-env doesn't expose all packages by default,
but that can be configured per environment.

Local packages. I understood that's what Tidal users need.
I.e. git clone tidal && cd tidal && cabal env --env=tidal add tidal

cabal-env doesn't currently support local packages, but it's not complicated
to add. As the cabal install --lib currently, one would need to
sdist the local project packages, but also copy them somewhere
in ~/.cabal/... (folder to be configurable?) so one could have

-- cabal-env packages: tidal -any
-- cabal-env tarballs: tidal-1.4.9-sha256hash.tar.gz

we need hash to not destroy other environments.
We store the actual tarballs, so temporary cabal.project can be recreated
completely.

[m-renaud]

Same comment about delete and remove. Specifically as these
commands are destructive we should name environment deletion command
somehow that you don't invoke it by accident thinking you'll
remove a package

We could do something with an interactive prompt for any destructive operations:

a) Before deleting/removing, have a prompt which reiterates what the command is about to do, and prompt for y/n (see examples below)
b) Add a -f option which overrides the prompt.

ex.

$ cabal env remove some-package
You are about to remove 'some-package' from the 'default' environment, this cannot be undone.
Would you like to continue? (use -f to skip prompt)
y/n: y

$ cabal env delete-environment --env=myenv
You are about to delete the 'myenv' environment and all associated files, this cannot be undone. Would you like to continue? (use -f to skip prompt)
y/n: y

[m-renaud]

Another thought: for the case of overwriting locally installed packages with same name and version but different hash (local development case), we could split out the overwrite case from add into re-add so you need to be explicit to overwrite a previously added package in a destructive manner:

$ cabal env --env=tidal add tidal
$ touch some-file-which-changes-tidal-hash
$ cabal env --env=tidal add tidal
Error: Cannot add `tidal-version` to the `tidal` environment. The environment already
contains an entry for `tidal-version` but with a different hash. This can happen if you
are developing `tidal` locally and trying to update an environment with a new version
of the package. To overwrite the existing entry in the environment file with the new
version use the following command:
  cabal env --env=tidal readd tidal
$ cabal env --env=tidal readd tidal
You are about to replace the existing entry for `tidal-version` in the `tidal` environment with a
different package, this cannot be undone.
Would you like to continue? (use -f to skip prompt)
y/n: y

[fgaz]

I've been using @phadej's cabal-env, and it's definitely the right path imo. Here's some feedback based on my experience.

• I found it useful to create local (.ghc.environment...) envs (I had to move the env file) cabal-env: Allow creating local .ghc.environment.* files. phadej/cabal-extras#17
  • I'm not sure whether the implicit environment should be default or a local one
• matching the -package-env ghc interface (like install --lib does) by allowing paths would be good. I was surprised when this did not work
• --transitive is quite useful
• some other stuff that is missing, but I assume it'll be present in cabal
  • removing a package from the env
  • unhiding a package (add/install could do it)

Other ideas:

• cabal env could be integrated with write-ghc-environment-files somehow, so that generated envs contain the plan and can be manipulated

EDIT:

• --only-dependencies --enable-tests for cabal new-install --only-dependencies --enable-tests doesn't work #5650

[fgaz]

More positive feedback: #5650 (comment)

[danidiaz]

@fgaz

I'm not sure whether the implicit environment should be default or a local one

Personally, I almost always use local environments, and I would perhaps expect local as the default

For example in npm one needs to use --global or -g for global installs.

[danidiaz]

As a side note, global environments currently don't work on Windows. I filed the bug in the GHC repo because the behavior seem to contradict GHC's own user guide.

[Mikolaj]

I think the only way you can use world is via cabal install world. So if you don't, your workflow should not be affected.

Citing some things removed in that PR:

     -- | A special virtual package that refers to the collection of packages
     -- recorded in the world file that the user specifically installed.
     --
     -- > cabal install world

-- To rebuild/upgrade the packages in world (e.g. when updating the compiler)
-- use
-- # cabal install world

And the changelog of this PR:

synopsis:  Remove World file functionality
packages: cabal-install
issues: #6767
prs: #7746
description: {
In v1-install, 'world' was used to trace what packages have been
installed, and re-install everything that is listed in it on-demand. However, in
v2-install, this is no longer needed, and outdated.
Additionally, 'world' code-path is probably not up-to-date, remove it instead of
having partial features.

- Don't generate `world` file in `~/.cabal` when `v1-install` is executed.

- Don't ever read `world` from `~/.cabal`.

- Remove meta-target `world` from `v1-install`.
}

[yaxu]

@Mikolaj Thanks, no I'm not using that!

[tomsmeding]

(This issue is linked from the cabal install --help output for the --lib flag.)

The semantics of cabal install --lib continues to confuse people: e.g. in this irc discussion (links to this paste), before reading on towards the resolution of the problem, and trying to forget the context of this issue, I challenge you to find out what the problem is for this user.

I get that there are many colourable panels on the bikeshed that is cabal env, and even many real engineering choices and challenges, but can we please, at least, have some warning if cabal install --lib creates a new environment file? Perhaps the warning should not be displayed (or should not be very loud) if an environment file already exists, to acknowledge the small (?) set of people who are actually using this functionality, but I suspect that the vast majority of people who run cabal install --lib were not intending to create an environment file, with all the action-at-a-distance consequences that entails.

Currently there is no output at all when you run cabal install --lib that indicates that something potentially dangerous happened. Can we change that?

[ulysses4ever]

Great idea, I think. I'd open a separate issue though.