Cabal ignores hs-source-dirs: if main-is: is a C source file

[mpilgrem]

Describe the bug

After GHC 9.8.2/Cabal-3.10.2.0, and by GHC 9.10.1/Cabal-3.12.0.0, Cabal (the library) ignores hs-source-dirs if main-is is a C source file. Previously, Cabal respected hs-source-dirs.

To Reproduce
Imagine a small package. In directory app it has Main.hs with contents:

module Main ( main ) where

import Lib ( myMax )

main :: IO ()
main = print $ myMax 10 100

In directory src it has Lib.hs with contents:

module Lib ( myMax ) where

myMax :: Int -> Int -> Int
myMax x1 x2 = if x1 > x2 then x1 else x2

foreign export ccall myMax :: Int -> Int -> Int

In directory c-app it has main.c with contents:

#include <stdio.h>
#include <HsFFI.h>
#ifdef __GLASGOW_HASKELL__
#include "Lib_stub.h"
#endif

int main(int argc, char *argv[]) {
  hs_init(&argc, &argv);
  printf("%lld\n", myMax(10,100));
  hs_exit();
  return 0;
}

It has the usual Setup.hs:

import Distribution.Simple
main = defaultMain

and it has a Cabal file:

cabal-version: 1.12

name:           haskell-c-tests
version:        0.1.0.0
build-type:     Simple

library
  exposed-modules: Lib
  hs-source-dirs: src
  ghc-options: -stubdir autogen-stubs
  build-depends: base
  default-language: Haskell2010

executable c-exe
  main-is: main.c
  hs-source-dirs: c-app
  ghc-options: -no-hs-main
  include-dirs: autogen-stubs
  build-depends: base, haskell-c-tests
  default-language: Haskell2010

executable haskell-exe
  main-is: Main.hs
  hs-source-dirs: app
  ghc-options: -threaded -rtsopts -with-rtsopts=-N
  build-depends: base, haskell-c-tests
  default-language: Haskell2010

With GHC 9.8.2/Cabal-3.10.2.0 all is fine (extracts only):

❯ stack --stack-yaml stack-ghc-9.8.2.yaml runghc --no-ghc-package-path -- Setup.hs configure
Configuring haskell-c-tests-0.1.0.0...

❯ stack --stack-yaml stack-ghc-9.8.2.yaml runghc --no-ghc-package-path -- Setup.hs build -v
...
Preprocessing executable 'c-exe' for haskell-c-tests-0.1.0.0..
Building executable 'c-exe' for haskell-c-tests-0.1.0.0..
creating dist\build\c-exe
creating dist\build\c-exe\c-exe-tmp
Building C Sources...
creating dist\build\c-exe\c-exe-tmp
Running: "C:\Users\mike\AppData\Local\Programs\stack\x86_64-windows\ghc-9.8.2\bin\ghc.exe" "-c" "-odir" "dist\build\c-exe\c-exe-tmp" "-Idist\build\c-exe\autogen" "-Idist\build\global-autogen" "-Idist\build\c-exe\c-exe-tmp" "-Iautogen-stubs" "-Idist\build\autogen-stubs" "-optc-O2" "-pgmc" "C:\Users\mike\AppData\Local\Programs\stack\x86_64-windows\ghc-9.8.2\lib\../mingw/bin\clang.exe" "-hide-all-packages" "-no-user-package-db" "-package-db" "dist\package.conf.inplace" "-package-id" "base-4.19.1.0-6554" "-package-id" "haskell-c-tests-0.1.0.0-2Qfk7puHfWxBnTSX2kZfQz" "c-app\main.c" "-no-hs-main"
Linking...

Note that Cabal has passed "c-app\main.c" to GHC 9.8.2 (the second from last argument).

However, with GHC 9.10.1/Cabal-3.12.0.0, Cabal passes only "main.c" to GHC 9.10.1 and, consequently, fails to find main.c:

❯ stack --stack-yaml stack-ghc-9.10.1.yaml runghc --no-ghc-package-path -- Setup.hs configure
Configuring haskell-c-tests-0.1.0.0...

❯ stack --stack-yaml stack-ghc-9.10.1.yaml runghc --no-ghc-package-path -- Setup.hs build -v
...
Preprocessing executable 'c-exe' for haskell-c-tests-0.1.0.0...
Building executable 'c-exe' for haskell-c-tests-0.1.0.0...
creating dist\build\c-exe
creating dist\build\c-exe\c-exe-tmp
Wanted build ways: [StaticWay]
Building C Sources...
creating dist\build\c-exe\c-exe-tmp
Running: "C:\Users\mike\AppData\Local\Programs\stack\x86_64-windows\ghc-9.10.1\bin\ghc.exe" "-c" "-odir" "dist\build\c-exe\c-exe-tmp" "-Idist\build\c-exe\autogen" "-Idist\build\global-autogen" "-Idist\build\c-exe\c-exe-tmp" "-Iautogen-stubs" "-Idist\build\autogen-stubs" "-optc-O2" "-pgmc" "C:\Users\mike\AppData\Local\Programs\stack\x86_64-windows\ghc-9.10.1\lib/../mingw//bin\clang.exe" "-hide-all-packages" "-no-user-package-db" "-package-db" "dist\package.conf.inplace" "-package-id" "base-4.20.0.0-30dc" "-package-id" "haskell-c-tests-0.1.0.0-BiMnNkNFjFZ2oxYL44TYmr" "main.c" "-no-hs-main"
<command line>: does not exist: main.c

(stack --stack-yaml stack-ghc-9.10.1.yaml runghc --no-ghc-package-path -- etc runs runghc in the correct environment - that is, one that provides the relevant version of GHC and its Cabal boot package. The --no-ghc-package-path is because GHC_PACKAGE_PATH and Cabal are incompatible.)

[ulysses4ever]

Have you checked GHC 9.10/Cabal-3.10? That may be helpful.

[mouse07410]

Have you checked GHC 9.10/Cabal-3.10?

With Cabal-3.10.3.0, Stack-2.15.7, and GHC-9.10.1 the build seems to succeed.

With Cabal-3.12.1.0 and GHC-9.6.5 the build fails (Cabal cannot find the .c file).

[mpilgrem]

Have you checked GHC 9.10/Cabal-3.10?

As the problem is what different versions of Cabal (the library) are, or are not, passing to GHC, I reason that changing GHC version for a given Cabal version has no effect. But I've not tested that empircially.

[mouse07410]

But I've not tested that empiricially

I have. The problem is with Cabal. :-(

[mpilgrem]

I agree with @mouse07410. I was able to try the GHC 9.10.1/Cabal 3.10.2.0 combination by adding to the Cabal file:

build-type:     Custom

custom-setup
  setup-depends: base, Cabal ==3.10.2.0

and to stack-ghc-9.10.1.yaml:

snapshot: nightly-2024-04-04 # GHC 9.8.2
compiler: ghc-9.10.1
extra-deps:
- Cabal-3.10.2.0
- Cabal-syntax-3.10.2.0
# Specifying a boot package as an extra-dep requires its boot package dependencies to be
# specified also
- Win32-2.13.4.0@sha256:6a1299d051c514aa5ac7b77ef5f86be6c0aa6940b00302c6dc246192c7a97d99,5769
- binary-0.8.9.2@sha256:03381e511429c44d13990c6d76281c4fc2468371cede4fe684b0c98d9b7d5f5a,6611
- containers-0.6.8@sha256:bb2bec1bbc6b39a7c97cd95e056a5698ec45beb5d8feb6caae12af64e4bd823c,2670
- directory-1.3.8.5@sha256:fbeec9ec346e5272167f63dcb86af513b457a7b9fc36dc818e4c7b81608d612b,3166
- filepath-1.4.300.2@sha256:345cbb1afe414a09e47737e4d14cbd51891a734e67c0ef3d77a1439518bb81e8,5900
- parsec-3.1.17.0@sha256:8407cbd428d7f640a0fff8891bd2f7aca13cebe70a5e654856f8abec9a648b56,5149
- process-1.6.20.0@sha256:2a9393de33f18415fb8f4826957a87a94ffe8840ca8472a9b69dca6de45aca03,2790
- text-2.1.1@sha256:78c3fb91055d0607a80453327f087b9dc82168d41d0dca3ff410d21033b5e87d,10653
- time-1.12.2@sha256:88e8493d9130038d3b9968a2530a0900141cd3d938483c83dde56e12b875ebc8,6510

[mouse07410]

Actually, things are even more interesting. Below is a proof that the regression is in Cabal 3.12.0.0 library, with which Stack-2.15.7 is built.

• With Cabal-3.12.1.0, both cabal build and stack build commands fail.
• With Cabal-3.10.3.0, cabal build (which uses cabal 3.10 library) works, but stack build (which uses cabal 3.12 library) fails:

$ cabal build
Resolving dependencies...
Build profile: -w ghc-9.10.1 -O2
In order, the following will be built (use -v for more details):
 - h-tst5-0.1.1.1 (lib) (configuration changed)
 - h-tst5-0.1.1.1 (exe:h-tst5-exe) (configuration changed)
 - h-tst5-0.1.1.1 (exe:h-tst5-c-exe) (configuration changed)
Configuring library for h-tst5-0.1.1.1..
Preprocessing library for h-tst5-0.1.1.1..
Building library for h-tst5-0.1.1.1..
Configuring executable 'h-tst5-exe' for h-tst5-0.1.1.1..
Configuring executable 'h-tst5-c-exe' for h-tst5-0.1.1.1..
Preprocessing executable 'h-tst5-c-exe' for h-tst5-0.1.1.1..
Building executable 'h-tst5-c-exe' for h-tst5-0.1.1.1..
Preprocessing executable 'h-tst5-exe' for h-tst5-0.1.1.1..
Building executable 'h-tst5-exe' for h-tst5-0.1.1.1..
[3 of 3] Linking /Users/ur20980/src/h-tst5/dist-newstyle/build/x86_64-osx/ghc-9.10.1/h-tst5-0.1.1.1/x/h-tst5-exe/opt/build/h-tst5-exe/h-tst5-exe [Library changed]
$ stack build

Warning: Stack has not been tested with GHC versions 9.10 and above, and using 9.10.1, this may fail.

Warning: Stack has not been tested with Cabal versions 3.12 and above, but version 3.12.0.0 was found, this may fail.
h-tst5> build (lib + exe) with ghc-9.10.1
Preprocessing library for h-tst5-0.1.1.1...
Building library for h-tst5-0.1.1.1...
Preprocessing executable 'h-tst5-c-exe' for h-tst5-0.1.1.1...
Building executable 'h-tst5-c-exe' for h-tst5-0.1.1.1...
<command line>: does not exist: test.c

Error: [S-7282]
       Stack failed to execute the build plan.
       
       While executing the build plan, Stack encountered the error:
       
       [S-7011]
       While building package h-tst5-0.1.1.1 (scroll up to its section to see the error) using:
       /Users/ur20980/.stack/setup-exe-cache/x86_64-osx/Cabal-simple_HwdwpEmb_3.12.0.0_ghc-9.10.1 --verbose=1 --builddir=.stack-work/dist/x86_64-osx/ghc-9.10.1 build lib:h-tst5 exe:h-tst5-c-exe exe:h-tst5-exe --ghc-options " -fdiagnostics-color=always"
       Process exited with code: ExitFailure 1 
$

[mpilgrem]

@mouse07410, the official release of Stack 2.15.7 has a dependency on Cabal=3.10.3.0, but Stack does not use that dependency to build. For Simple it uses the boot package of the specified GHC. That can be overriden using a Custom build.

[mouse07410]

@mouse07410, the official release of Stack 2.15.7 has a dependency on Cabal=3.10.3.0, but Stack does not use that dependency to build. For Simple it uses the boot package of the specified GHC. That can be overriden using a Custom build.

Ok, in that case why does "cabal-install"-v3.10.3.0 build succeeds when "stack" build fails? Does GHC-9.10.1 use boot package with cabal-3.12?

[ulysses4ever]

Thanks everyone for helping to isolate the problem to Cabal-3.12. Next step would be bisect it, perhaps.

[mpilgrem]

@mouse07410, yes GHC 9.10.1 comes with Cabal-3.12.0.0. This is a handy source: https://gitlab.haskell.org/ghc/ghc/-/wikis/commentary/libraries/version-history. This is another: https://www.snoyman.com/base/.

[mouse07410]

Thanks everyone for helping to isolate the problem to Cabal-3.12. Next step would be bisect it, perhaps.

Good idea, but infeasible for me, as I can't build Cabal locally. :-(

[mpilgrem]

On isolating the code, my current assumption is that it is something to do with Distribution.Simple.GHC.Build.ExtraSources.buildSources:

buildCSources =
  buildExtraSources
    "C Sources"
    Internal.componentCcGhcOptions
    ( \c -> do
        let cFiles = cSources (componentBuildInfo c)
        case c of
          CExe exe
            | let mainPath = getSymbolicPath $ modulePath exe
            , isC mainPath ->
                cFiles ++ [makeSymbolicPath mainPath]
          -- NB: Main.hs is relative to hs-source-dirs, but Main.c
          -- is relative to the package.
          _otherwise -> cFiles
    )

EDIT: That is, it seems (from the code comment above) to be a deliberate decision to change Cabal's design, but one that was undocumented.

EDIT2: The previous corresponding code appears to me to be Distribution.Simple.GHC.gbuildSources:

gbuildSources verbosity pkgId specVer tmpDir bm =
    case bm of
      GBuildExe  exe  -> exeSources exe
      ...
  where
    exeSources :: Executable -> IO BuildSources
    exeSources exe@Executable{buildInfo = bnfo, modulePath = modPath} = do
      main <- findFileEx verbosity (tmpDir : map getSymbolicPath (hsSourceDirs bnfo)) modPath
      ...
      if isHaskell main || pkgId == fakePackageId
        then
          ...
        else let (csf, cxxsf)
                   | isCxx main = (       cSources bnfo, main : cxxSources bnfo)
                   -- if main is not a Haskell source
                   -- and main is not a C++ source
                   -- then we assume that it is a C source
                   | otherwise  = (main : cSources bnfo,        cxxSources bnfo)

             in  return BuildSources {
                            cSourcesFiles      = csf,
                            cxxSourceFiles     = cxxsf,
                            inputSourceFiles   = [],
                            inputSourceModules = exeModules exe
                        }

EDIT3: Some commits of interest:

• 2decb0e committed by @alt-romes
• 7b90583 committed by @sheaf (related PR Make Cabal agnostic about working directory #9718)

[mpilgrem]

If the design choice is as intended, the 'problem' is that the change in the Package Description Format Specification was not documented in the following places:

• https://cabal.readthedocs.io/en/3.12/file-format-changelog.html#cabal-version-3-12
• https://cabal.readthedocs.io/en/3.12/cabal-package-description-file.html#pkg-field-executable-main-is
• https://cabal.readthedocs.io/en/3.12/cabal-package-description-file.html#pkg-field-hs-source-dirs

[alt-romes]

Thanks for investigating @mpilgrem.
I'm fairly certain this change wasn't intended, and I think it should be fixed to regardless.

[sheaf]

Thanks for your continued investigation @mpilgrem. I don't believe this change was intended.

My working directory patch didn't make it into Cabal 3.12, so I don't think that is the issue. I think 2decb0e looks like a possible culprit.

[alt-romes]

-- NB: Main.hs is relative to hs-source-dirs, but Main.c
-- is relative to the package.

I think this comment simply means that if main-is is a haskell file, it'll be looked up in hs-sources-dir(where haskell sources live). If it is a C file, it's looked up in the package root (ie not in the haskell sources dir).

[alt-romes]

I'm going to look into this.

[mouse07410]

I think this comment simply means that if main-is is a haskell file, it'll be looked up in hs-sources-dir(where haskell sources live). If it is a C file, it's looked up in the package root (ie not in the haskell sources dir).

If hs-source-dirs: is not supposed to apply to non-haskell files (like main.c), IMHO there has to be a way to specify a directory other than the package root. Name it source-dirs: or whatever, but it has to be there.

[alt-romes]

I think this comment simply means that if main-is is a haskell file, it'll be looked up in hs-sources-dir(where haskell sources live). If it is a C file, it's looked up in the package root (ie not in the haskell sources dir).

If hs-source-dirs: is not supposed to apply to non-haskell files (like main.c), IMHO there has to be a way to specify a directory other than the package root. Name it source-dirs: or whatever, but it has to be there.

Indeed. I've realised that I was being contradictory in saying we should fix this but asserting that C mains should not be found in hs-source-dirs, not realising that this is what the OP literally intended to do.

The patch me and @sheaf have put together restores the existing behaviour of looking for non-haskell mains in hs-source-dirs, as there's no clear advantage to breaking this existing behaviour. This regressed because of an oversight in the 2decb0e refactor.

[mouse07410]

being contradictory in saying we should fix this but asserting that C mains should not be found in hs-source-dirs, not realising that this is what the OP literally intended to do.

What the OP (and I ;) need is a way to specify a directory, other than the package root, for non-Haskell files - but it does not have to be exactly hs- source-dirs.

If you replace hs-source-dirs: for non-Haskell files with something else - IMHO it would be OK, not risking to break a lot of existing packages. The main point of this issue (and of this problem) is not to lose this capability altogether (as the 2decb0e apparently and unfortunately does).

[alt-romes]

What the OP (and I ;) need is a way to specify a directory, other than the package root, for non-Haskell files - but it does not have to be exactly hs- source-dirs.

Yes, indeed. But we also don't want to inadvertently break existing packages out there whose maintainers don't follow the latest in Cabal development ;), as this was previously the only way to set a source directory for a C main it seems.

I think it would be a welcome feature if someone implemented, say, non-hs-source-dirs, or other-source-dirs, ...., but it's not critical (I don't have the bandwidth to do this).

In the meantime, here's the fix which restores the existing behaviour, once again looking up C main files in hs-source-dirs: #10171

[ulysses4ever]

I first thought: "Umh, we broke something, we should fix it ASAP!" After reading more about it, I've started wondering: do we want to preserve this somewhat unintuitive behavior -- looking into hs-source-dirs for non-Haskell sources?

One way to answer this is to first ask: what would you choose if designing Cabal from scratch today? My feeling is that I'd not want that behavior.

Now, to the brakages. We probably don't want more brakages than necessary. But, first, we already caused breakage for people who relied on this behavior and are trying to update to cabal-install-3.12.1.0. And second, more importantly, do we understand how much of a breakage this causes?

In no way do I want to block #10171, just wanted to make sure that we make an informed choice with some reasoning under it (unlike the initial change that happened by accident).

[mpilgrem]

The history is:

• March 2005: GHC 6.4 is released, introducing Cabal-1.0. That documents:

  main-is: filename (required)
  The name of the source file containing the Main module, relative to the hs-source-dir directory. 
  

• September 2013: Cabal-1.18.0 is released. That documents:

  main-is: filename (required)
  The name of the .hs or .lhs file containing the Main module. Note that it is the .hs filename
  that must be listed, even if that file is generated using a preprocessor. The source file
  must be relative to one of the directories listed in hs-source-dirs.
  

• August 2019: Cabal-3.0.0.0 is released. It documents - I think for the first time - that Cabal 1.18 had "Add[ed] support for specifying a C/C++/obj-C source file in executable:main-is field." It also documents

  main-is: filename (required)
  The name of the .hs or .lhs file containing the Main module. Note that it is the .hs filename
  that must be listed, even if that file is generated using a preprocessor. The source file
  must be relative to one of the directories listed in hs-source-dirs. Further, while the name
  of the file may vary, the module itself must be named Main.
  
  Starting with cabal-version: 1.18 this field supports specifying a C, C++, or objC source
  file as the main entry point.
  

[mouse07410]

Thank you for the patch, @alt-romes and @geekosaur !

Would you have any idea when is it likely that we'd see a released version of Cabal with this patch/PR incorporated?

[hasufell]

Thank you for the patch, @alt-romes and @geekosaur !

Would you have any idea when is it likely that we'd see a released version of Cabal with this patch/PR incorporated?

I'm not involved in cabal release management, but I would suggest to wait a little longer in case there are more regressions coming up, before cutting a new point release.

Otherwise you can always downgrade to 3.10.3.0.

[geekosaur]

That was our conclusion in our dev meeting yesterday, yes. There's also the complication that this is the Cabal library, and many things will stick to the version that ships with ghc (whether it should is another question) even if we make a minor release.