generic-haskell-builder: Overridden Cabal package to be used by Setup.hs is ignored

[nh2]

Issue description

I found a subtle issue with the generic Haskell builder that can make things confusing and the wrong Cabal version to be chosen.

Let's say you have an overridden version of the Cabal library and (this is an extract from here) a function to use that for a package:

  useFixedCabal = drv: pkgs.haskell.lib.overrideCabal drv (old: {
    setupHaskellDepends = (if old ? setupHaskellDepends then old.setupHaskellDepends else []) ++ [ Cabal_patched ];
    libraryHaskellDepends = (if old ? libraryHaskellDepends then old.libraryHaskellDepends else []) ++ [ Cabal_patched ];
  });

Then you use

      hsyslog = useFixedCabal super.hsyslog;

You would expect hsyslog's Setup file to be linked against the version of Cabal you specified.

However, that is not the case.

It gets linked against the GHC-builtin Cabal in the "global package database".

This is because a dependency of hsyslog, cabal-doctest, was not built against the custom cabal version. hsyslog uses cabal-doctest in its Setup.hs file.

(Note in newer versions of hsyslog this is not the case any more but the issue in general remains for many other packages that have custom Setup.hs files.)

The generic builder builds the Setup executable from Setup.hs by directly invoking ghc:

nixpkgs/pkgs/development/haskell-modules/generic-builder.nix

Line 330 in e7e5aaa

${nativeGhcCommand} $setupCompileFlags --make -o Setup -odir $TMPDIR -hidir $TMPDIR $i

For my useFixedCabal super.hsyslog, this passes a -package-db=... that contains the overridden Cabal. But GHC will see both the builtin Cabal and the one in the given package DB, see that cabal-doctest was built against the non-overridden one, and silently choose the non-overridden one for hsyslog too.

There is no warning, see ticket https://ghc.haskell.org/trac/ghc/ticket/15425 which I filed for the GHC side.

What can we do about it?

A workaround is:

      cabal-doctest = useFixedCabal super.cabal-doctest;
      hsyslog = useFixedCabal super.hsyslog;

but for coming up with this this you really need to know the dependencies of all packages so it isn't really feasible.

Ideally, the generic Haskell builder would refuse to build / fail loudly when the Cabal versions passed for packages in dependency chains like hsyslog and cabal-doctest are not the same.

---

CC @peti

[nh2]

Why is this important / how does this issue affect us?

Because in static-haskell-nix I override Cabal to add a new flag --enable-executable-static so that Haskell executables can be built statically (see also this overview on the effort.

I do this on all Haskell executables.

For many of them, it complains:

unrecognized 'configure' option `--enable-executable-static'

because even though I have explicitly told nix to use my version of Cabal, GHC eventually does not link that version in.

[peti]

I don't see how the generic builder could detect this case. The problem is that you have both versions of Cabal in scope, so Cabal is free to choose which version of the library it's going to use. The outcome is not what you intended it to be, but it's a perfectly legitimate outcome according to the rules. First of all, you really need to use overrideScope in a case such as this one to make sure the override applies to all build inputs of hsyslog, too. This would probably fix the issue already because then Cabal would consistently choose the Cabal library with the newer version string. (I trust your Cabal_patched build has a version that's higher than the default 2.2.0.1, right?)

If you want to be absolutely sure, then you could probably pass an appropriate -hide-package flag to ghc in configureFlags to make sure the bullt-in version isn't visible.

Alternatively, passing --constraint="Cabal >2.3" (or something along those lines) in the configure stage would get the job done, too.

P.S.: Instead of

if old ? setupHaskellDepends then old.setupHaskellDepends else []

it's easier to write:

old.setupHaskellDepends or []

[nh2]

@peti

Cabal is free to choose which version of the library it's going to use

I'm a bit confused by this statement, as Cabal is not involved choosing versions. We compile Setup.hs directly with ghc so GHC is the only thing picking versions (directly from two competing package DBs).

I trust your Cabal_patched build has a version that's higher than the default 2.2.0.1, right?

No, it has exactly the same version, it has a few patches cherry-picked.

old.setupHaskellDepends or []

Ah thanks, that's great to know.

but it's a perfectly legitimate outcome according to the rules

I'm not convinced of that; the feedback on https://ghc.haskell.org/trac/ghc/ticket/15425 suggests that GHC should really complain about it.

I don't see how the generic builder could detect this case.

If there's no trick we can use to protect the user against such mistakes, we may have to wait until the linked GHC issue implements it to be an error.

[nh2]

First of all, you really need to use overrideScope in a case such as this one to make sure the override applies to all build inputs of hsyslog, too

@peti This is a great idea, I didn't know about overrideScope.

It looks like just the tool I want which could simplify what I'm doing.

But I found this tangential issue:

For 1 package (darcs) overrideScope seems to have the weird side effect that it resets a lot of overrides that I'm doing for native dependencies like curl.

See nh2/static-haskell-nix@52416d2

Do you understand what's going on there?

This is currently stopping me from using overrideScope.

[peti]

I'm a bit confused by this statement, as Cabal is not involved choosing versions.

Oh, you are right, of course. I misunderstood the problem. I thought about a Setup-driven build where there are multiple versions of a package and then cabal configure picks one and makes it visible but not the others. But that's clearly not happening when building Setup.hs, Cabal is not involved there.

[Ericson2314]

That @kirelagin + @matthewbauer work of separating the compiler from core libraries should solve this by ensuring there really is only one Cabal visible.

[kirelagin]

Yes, this is pretty similar to #42069, but, I think, in this specific case there is really nothing that could help. Overriding a dependency version for a single Haskell package is pretty much impossible in the current implementation of Haskell infra, what you should do is override it globally in your entire package set.

The reason is simply that all transitive dependencies get unconditionally propagated into the pkgdb, so, e.g. if you are trying to override some package with an older version and one of your dependencies depends on a more recent version, you’ll end up having two config files for the same package in your pkgdb and the newer version will always win.

Dependency propagation logic should really be Haskell-specific instead of relying on default setup.sh and it should take into account the fact that our dependencies can depend on the same package that we depend on and put only one config file in pkgdb, the one we asked for.

[Ericson2314]

I'm not quite sure what the specific issue here is you are referring too? a) it seems the only issue is wired-in package weirdness; there is no intentional mixing of Cabal versions b) setup dependencies are not propagated.

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.