code-generators fails to find executable defined in same package as test suite

[RyanGlScott]

I recently attempted to make use of the code-generators feature that debuted in Cabal-3.8. Unfortunately, I ran into an apparent limitation in which the code generator executable cannot be defined in the same package as the test suite that makes use of it.

To Reproduce

1. Clone https://github.com/RyanGlScott/code-generators-issue, which contains a minimal reproducer for the issue.

2. By default, this repo makes use of a code generator defined in a separate package, which is named codegen-different-package. To observe that this works as expected, run cabal v2-test:

   $ cabal v2-test
   Resolving dependencies...
   Build profile: -w ghc-9.2.4 -O1
   In order, the following will be built (use -v for more details):
    - codegen-different-package-0.1 (exe:codegen-different-package-exe) (first run)
    - code-generators-issue-0.1 (test:code-generators-issue-test) (first run)
   Configuring executable 'codegen-different-package-exe' for codegen-different-package-0.1..
   Preprocessing executable 'codegen-different-package-exe' for codegen-different-package-0.1..
   Building executable 'codegen-different-package-exe' for codegen-different-package-0.1..
   [1 of 1] Compiling Main             ( src/CodegenDifferentPackage.hs, /home/ryanglscott/Documents/Hacking/Haskell/code-generators-issue/dist-newstyle/build/x86_64-linux/ghc-9.2.4/codegen-different-package-0.1/x/codegen-different-package-exe/build/codegen-different-package-exe/codegen-different-package-exe-tmp/Main.o )
   Linking /home/ryanglscott/Documents/Hacking/Haskell/code-generators-issue/dist-newstyle/build/x86_64-linux/ghc-9.2.4/codegen-different-package-0.1/x/codegen-different-package-exe/build/codegen-different-package-exe/codegen-different-package-exe ...
   Configuring test suite 'code-generators-issue-test' for code-generators-issue-0.1..
   Preprocessing test suite 'code-generators-issue-test' for code-generators-issue-0.1..
   Building test suite 'code-generators-issue-test' for code-generators-issue-0.1..
   [1 of 2] Compiling GeneratedFile    ( /home/ryanglscott/Documents/Hacking/Haskell/code-generators-issue/dist-newstyle/build/x86_64-linux/ghc-9.2.4/code-generators-issue-0.1/t/code-generators-issue-test/build/code-generators-issue-test/code-generators-issue-test-gen/GeneratedFile.hs, /home/ryanglscott/Documents/Hacking/Haskell/code-generators-issue/dist-newstyle/build/x86_64-linux/ghc-9.2.4/code-generators-issue-0.1/t/code-generators-issue-test/build/code-generators-issue-test/code-generators-issue-test-tmp/GeneratedFile.o )
   [2 of 2] Compiling Main             ( test/Test.hs, /home/ryanglscott/Documents/Hacking/Haskell/code-generators-issue/dist-newstyle/build/x86_64-linux/ghc-9.2.4/code-generators-issue-0.1/t/code-generators-issue-test/build/code-generators-issue-test/code-generators-issue-test-tmp/Main.o )
   Linking /home/ryanglscott/Documents/Hacking/Haskell/code-generators-issue/dist-newstyle/build/x86_64-linux/ghc-9.2.4/code-generators-issue-0.1/t/code-generators-issue-test/build/code-generators-issue-test/code-generators-issue-test ...
   Running 1 test suites...
   Test suite code-generators-issue-test: RUNNING...
   Test suite code-generators-issue-test: PASS
   Test suite logged to:
   /home/ryanglscott/Documents/Hacking/Haskell/code-generators-issue/dist-newstyle/build/x86_64-linux/ghc-9.2.4/code-generators-issue-0.1/t/code-generators-issue-test/test/code-generators-issue-0.1-code-generators-issue-test.log
   1 of 1 test suites (1 of 1 test cases) passed.
   

3. To observe the bug, patch code-generators-issue.cabal to make use of a code generator defined in the same package (codegen-same-package):

   diff --git a/code-generators-issue.cabal b/code-generators-issue.cabal
   index 4317924..7d26933 100644
   --- a/code-generators-issue.cabal
   +++ b/code-generators-issue.cabal
   @@ -27,7 +27,5 @@ test-suite code-generators-issue-test
        ghc-options:        -Wall
        default-language:   Haskell2010
        build-depends:      base >= 4 && < 5
   -    build-tool-depends: codegen-different-package:codegen-different-package-exe
   -    code-generators:    codegen-different-package-exe
   -    -- build-tool-depends: code-generators-issue:codegen-same-package-exe
   -    -- code-generators:    codegen-same-package-exe
   +    build-tool-depends: code-generators-issue:codegen-same-package-exe
   +    code-generators:    codegen-same-package-exe

   Starting from a clean slate:

   $ cabal v2-clean
   

   This time, if you run cabal v2-test, it fails:

   $ cabal v2-test
   Resolving dependencies...
   Build profile: -w ghc-9.2.4 -O1
   In order, the following will be built (use -v for more details):
    - code-generators-issue-0.1 (exe:codegen-same-package-exe) (first run)
    - code-generators-issue-0.1 (test:code-generators-issue-test) (first run)
   Configuring executable 'codegen-same-package-exe' for code-generators-issue-0.1..
   Preprocessing executable 'codegen-same-package-exe' for code-generators-issue-0.1..
   Building executable 'codegen-same-package-exe' for code-generators-issue-0.1..
   [1 of 1] Compiling Main             ( codegen-same-package/CodegenSamePackage.hs, /home/ryanglscott/Documents/Hacking/Haskell/code-generators-issue/dist-newstyle/build/x86_64-linux/ghc-9.2.4/code-generators-issue-0.1/x/codegen-same-package-exe/build/codegen-same-package-exe/codegen-same-package-exe-tmp/Main.o )
   Linking /home/ryanglscott/Documents/Hacking/Haskell/code-generators-issue/dist-newstyle/build/x86_64-linux/ghc-9.2.4/code-generators-issue-0.1/x/codegen-same-package-exe/build/codegen-same-package-exe/codegen-same-package-exe ...
   Configuring test suite 'code-generators-issue-test' for code-generators-issue-0.1..
   Preprocessing test suite 'code-generators-issue-test' for code-generators-issue-0.1..
   /home/ryanglscott/Documents/Hacking/Haskell/code-generators-issue/dist-newstyle/build/x86_64-linux/ghc-9.2.4/code-generators-issue-0.1/t/code-generators-issue-test/build/codegen-same-package-exe/codegen-same-package-exe: createProcess: posix_spawnp: does not exist (No such file or directory)
   

Expected behavior
I would expect cabal v2-test to succeed regardless of which form of code generator is used.

System information

• Ubuntu 20.04 (64-bit)

• $ cabal --version
  cabal-install version 3.8.1.0
  compiled using version 3.8.1.0 of the Cabal library
  
  $ ghc --version
  The Glorious Glasgow Haskell Compilation System, version 9.2.4
  

[andreabedini]

I spent a couple of hours looking at this. To me it looks like the bug is in the addInternalBuildTools function, which was not touched by #7688.

cabal/Cabal/src/Distribution/Simple/Build.hs

Lines 930 to 944 in 51da23d

	addInternalBuildTools
	:: PackageDescription
	-> LocalBuildInfo
	-> BuildInfo
	-> ProgramDb
	-> ProgramDb
	addInternalBuildTools pkg lbi bi progs =
	foldr updateProgram progs internalBuildTools
	where
	internalBuildTools =
	[ simpleConfiguredProgram toolName' (FoundOnSystem toolLocation)
	| toolName <- getAllInternalToolDependencies pkg bi
	, let toolName' = unUnqualComponentName toolName
	, let toolLocation = buildDir lbi </> toolName' </> toolName' <.> exeExtension (hostPlatform lbi)
	]

That function is taking the name of a executable component (toolName) and declaring its location is buildDir lbi which is the build directory of the component which needs the tool, i.e. the test component in this case.

You can see in the error above that cabal is looking for codegen-same-package-exe in the build directory of the test component dist-newstyle/build/x86_64-linux/ghc-9.2.4/code-generators-issue-0.1/t/code-generators-issue-test/build.

I tried few things but I cannot find a way to obtain the build directory of the tool exe component from generateCode.

But that function has not been touched for years ... 🤔

[andreabedini]

AH! That's correct, that function does not work with per-component builds. Run cabal test --disable-per-component and it works (because now everything is in the same builddir).

[gbaz]

Should we try to fix addInternalBuildTools to handle per-component builds?

[andreabedini]

@gbaz

Should we try to fix addInternalBuildTools to handle per-component builds?

That would be awesome. I am not very familiar with that part, but if you do a bit of a brain dump (just here perhaps?) I can take note and do a bit of an investigation.

[gbaz]

I don't have any more knowledge than you to brain-dump sadly. I'd add just that generateCode is about generating the modules for the test suite. Its not about generating the tool exe component, which it presumes already exists.

I think maybe getAllInternalToolDependencies is where we need to give more attention?

But it may be that addInternalBuildTools in general is the right place to look -- perhaps it should make use of targetBuildDepends -- not sure?