stack init --resolver ghc-7.10.3 creates incomplete stack file

[soenkehahn]

General summary/comments (optional)

When using stack init --resolver ghc-7.10.3 stack sometimes produces a valid stack file, except that there's one dependency missing: hex-0.1.2. Why is that? Am I doing something wrong?

Steps to reproduce

$ git clone https://github.com/haskell-servant/servant
$ cd servant
$ stack init --force --resolver ghc-7.10.3
$ stack test

This will fail, recommending to add hex-0.1.2 to the extra-deps. After doing that, stack test works.

Stack version

$ stack --version
Version 1.1.2 x86_64 hpack-0.14.0

Method of installation

stack upgrade

[sjakobi]

Strange!

I can reproduce the issue with stack-HEAD:

~/s/servant (master|✚1) $ stack test
...
While constructing the build plan, the following exceptions were encountered:

In the dependencies for http2-1.6.1:
    hex must match -any, but the stack configuration has no specified version (latest applicable is 0.1.2)
needed due to tutorial-0.8 -> http2-1.6.1

Recommended action: try adding the following to your extra-deps in /home/simon/src/servant/stack.yaml:
- hex-0.1.2

You may also want to try the 'stack solver' command
Plan construction failed.

Executing stack solver ends with the message No changes needed to stack.yaml.

Strangely the http2-1.6.1 library doesn't depend on hex at all! Only the test suites and executables do.

(Pinging @harendra-kumar)

EDIT: The solver seems to be correct, but why does stack build think it needs hex?

[soenkehahn]

Aren't the executables of all dependencies built?

[sjakobi]

Aren't the executables of all dependencies built?

Oh, you're right. By default, they're built. It's even mentioned in the docs.

But the http2 executables should only be built if the devel flag is enabled which shouldn't be the case here.

BTW, this issue can be reproduced in a smaller project by simply adding http2 to the library's build-depends.

[mgsloan]

Strange! At first I thought that the issue was that stack test is used, and the http2 test-suites have a dependency on hex. This doesn't make sense, though, because test-suites for deps don't get built.

I cannot reproduce the issue. The init command adds hex-0.1.2 to the extra-deps list. Which cabal-install version do you have? I have cabal-install-1.22.6.0.

From the output of stack dot --external, it does indeed seem that stack thinks http2 depends on hex. This older cabal-install seems to think so too. My guess is that cabal-install-1.24 changed what's considered to be a dep of a package and not.

[harendra-kumar]

Oh, you're right. By default, they're built. It's even mentioned in the docs.

Why do we build the executables of dependencies?

[mgsloan]

Why do we build the executables of dependencies?

It is a little weird, but I think it's because usages of the library might depend on the executable at compile-time (TH or Setup.hs). I can't immediately think of any good examples of that, though.

It's also due to build-tool dependencies. Before proper build-tool deps were added, I think folks just relied on getting them via presence in build-deps.

It's also done because it's a reasonable thing to do with stack - the executables for deps get installed in sandboxed locations that're only on the PATH when they should be.

[sjakobi]

I was using cabal-install-1.24.0.0. Switching to 1.22.9.0 made stack init add hex to the extra-deps.

I guess this makes for a nice argument for using cabal-install's solver via the library!?

[sjakobi]

Is this really an upstream issue? The true problem seems to be that stack pulls in dependencies for (dependency) executables that aren't supposed to be buildable.

[mgsloan]

Yes, this would be addressed by having the solver in a library (if that existed!). However, that's less flexible for stack users to use the external process, at the cost of being more knobs to fiddle with.

I think I figured it out. cabal-install-1.24 fixed a bug with considering buildability of components in dep resolution. Stack had correct code for this, but we had to make it match cabal's behavior -

stack/src/Stack/Package.hs

Line 523 in 4be2916

-- | This is a copy-paste from Cabal's @allBuildInfo@ function, but with the

haskell/cabal#1725

I think the solution is to:

1. Properly consider buildability for computing deps if Cabal >= 1.24

2. Warn if stack init is run with a cabal-install version that has a major version mismatch with the Cabal library in use.

[mgsloan]

So yeah, not an upstream issue, instead it's an upstream fix. Our code assumed Cabal was still broken in this way.

[mgsloan]

I started working on a fix, and found that cabal version info would need to be plumbed through a lot of code that didn't previously care about the version. I've created an issue that details an approach to solving this - #2479 . Marking this as blocked on that.