local packages with c-sources give runtime errors in stack ghci

[vagifverdi]

if you point to a package source code located on the hard drive and that package has c source files, it will give runtime errors in stack ghci.

For example hdbc-odbc errors out in stack ghci when trying to connect to db. It gives following error:

/usr/local/lib/ghc-7.10.2/bin/ghc: symbol lookup error: /tmp/ghc25980_0/libghc_1.so: undefined symbol: SQLSetConnectAttr

The function SQLSetConnectAttr is calling a c module hdbc-odbc-helper.c that is bundled with hdbc-odbc package.

Loading hdbc-odbc from hackage works fine in stack ghci.

Stack build also works fine.

[borsboom]

@chrisdone, any ideas?

[mgsloan]

I cannot reproduce this issue with another package which has C source files, process. My steps:

1. Clone https://github.com/haskell/process
2. autoreconf -i
3. stack ghci

*Main> rawSystem "ls" []
aclocal.m4  cbits         config.log     configure     include  process.buildinfo  prologue.txt  Setup.hs    System  tests
autom4te.cache  changelog.md  config.status  configure.ac  LICENSE  process.cabal      README.md     stack.yaml  test
ExitSuccess

@vagifverdi Can you please give a set of steps for reproducing the problem?

[vagifverdi]

@mgsloan You need to create a separate project that refers to the cloned package.
Create a new folder test
In it add stack.yaml and refer to the package you cloned:

packages:

• ../process

Also, Not every call fails. Only the ones that call a code from the c files in that cloned package.

[mgsloan]

Even with process as a separate package, I don't get this error. Being in a separate package really has no affect on GHCI, since it loads all the modules together anyway.

rawSystem is definitely calling into code in the c files of process.

It'd be great to have a repro that's more minimal than a usage of hdbc. Or if you could point me to your project (if it has instructions for setting it up). As is, I keep on spending time attempting to construct a repro for this, and not getting anywhere.

Also, does this happen with the development version of stack (stack upgrade --git to install)? There have been a number of fixes to GHCi.

[vagifverdi]

i'll try to build a minimal example.

[vagifverdi]

You are right. It is not related to c-sources generally. process package works for me too. But hdbc-odbc does not. It works fine if stack build or in cabal repl, but not in stack ghci

Here are the steps to reproduce:

clone git https://github.com/hdbc/hdbc-odbc

remove all test build targets from cabal file, just leave library.

Put this stack.yaml in there:

extra-deps:
- HDBC-2.4.0.1

resolver: lts-3.9

stack build works fine.

Then do stack ghci
and load this file:

module Main
       where

import           Database.HDBC
import           Database.HDBC.ODBC

test_foo = do
  cn <- connectODBC ""
  rs <- quickQuery' cn "select 1" []
  print rs

main = test_foo

Inside stack ghci trying to run test_foo will give you first the error that it could not connect to database (which is normal since you did not supply connection string), But then it will give you the symbol lookup error: /usr/local/lib/ghc-7.10.2/bin/ghc: symbol lookup error: /tmp/ghc25980_0/libghc_1.so:

[mgsloan]

Thanks for the repro! I installed libodbc via sudo apt-get install unix-odbc-dev

I do indeed get the following output:

Main> main
*** Exception: SqlError {seState = "[\"IM002\"]", seNativeError = -1, seErrorMsg = "connectODBC/sqlDriverConnect: [\"0: [unixODBC][Driver Manager]Data source name not found, and no default driver specified\"]"}
*Main> /usr/local/lib/ghc-7.10.2/bin/ghc: symbol lookup error: /tmp/ghc21814_0/libghc_1.so: undefined symbol: SQLDisconnect

Very strange! I'll look into this more later, but I'm starting to think that this is a GHC issue. In particular, this is probably the case if cabal repl has the same behavior.

[vagifverdi]

I've been using hdbc-odbc in haskell-mode with repl for years. It works fine. As far as i remember haskell-mode uses cabal repl for its session. Anyway, testing cabal repl should be quite easy, just to rule it out.

[spencerjanssen]

I work with @vagifverdi and we did some troubleshooting of this today.

As confirmed by @mgsloan stack ghci does indeed fail with symbol errors when working inside a Git checkout of hdbc-odbc. However, cabal repl has the same behavior. I suspect both stack and cabal are failing to honor extra-libraries: odbc in the Cabal file when starting ghci.

However, there is another scenario that we're interested in. We have another project that depends on hdbc-odbc and we're able to run cabal repl there without any issues. However, we experience linking errors with a stack.yaml similar to:

resolver: lts-5.0

packages:
- '.'
- '../hdbc-odbc'

However, this stack.yaml works fine with stack ghci:

resolver: lts-5.0

packages:
- '.'
- location: /home/sidev/hdbc-odbc
  extra-dep: true

I don't know enough about stack to know whether this is a bug or intentional behavior.

[mgsloan]

Interesting! I haven't checked, but I'm thinking that in the first case, the code gets loaded into ghci, whereas in the 2nd case it gets used as a dependency. What happens if you do stack ghci PKG, where PKG is the name of the package at .?

[dpwiz]

I'm fighting some like that for a few weeks already. But my struggle is with atomic-primops and ghc-mod instead. ghci works fine, ghc-mod works fine on the package itself and some others that depend on it, but on main project it crashes with

.stack-work/install/x86_64-linux/lts-5.2/7.10.3/lib/x86_64-linux-ghc-7.10.3/atomic-primops-0.8.0.2-Il4AcfudM89CXhIIq1LN3T/libHSatomic-primops-0.8.0.2-Il4AcfudM89CXhIIq1LN3T.a: `unknown symbol 'store_load_barrier'.

[mgsloan]

@wiz Sounds like something that should get reported to the ghc-mod tracker. It does something quite different from stack ghci, in some ways more correct, in other ways not so much.

[cocreature]

In case another example is helpful https://github.com/cocreature/nanovg-hs suffers from the same problem although in this case the unknown symbol is __glewBindBuffer. Again if I use it as an extra-dep in another project it works. Cabal repl suffers from the same problem.

[mboes]

I can repro this using the network package as well:

$ stack ghci --ghc-options -DCALLCONV=ccall
[...]
ByteCodeLink: can't find label
During interactive linking, GHCi couldn't find the following symbol:
  sendFd
This may be due to you not asking GHCi to load extra object files,
archives or DLLs needed by your current session.  Restart GHCi, specifying
the missing library using the -L/path/to/object/dir and -lmissinglibname
flags, or simply by naming the relevant files on the GHCi command line.
Alternatively, this link failure might indicate a bug in GHCi.
If you suspect the latter, please send a bug report to:
  glasgow-haskell-bugs@haskell.org

sendFd is defined in a cbits/ancilData.c file local to the package.

[cocreature]

@mboes You might want to try using --whole-archive as I did in cocreature/nanovg-hs@05441ee That fixed the issue for me.

[mboes]

@cocreature interesting. Doesn't seem to help in this case.

[mboes]

@cocreature I'm not sure I understand why --whole-archive helped in the case of nanovg. Could you explain how you got to that solution? In the case of network, the missing symbol is in one of the local C files, not an external library. Could there be a way to apply a similar trick?

[cocreature]

@mboes I stole that solution from deech’s fltkhs bindings. I sadly can’t tell you what exactly it does or why it works, I just know that it did :)

[mboes]

@mgsloan @chrisdone so I found a hacky workaround, if that can help point towards a solution. The idea is to load the object file corresponding to the local C file with a missing symbol manually:

$ stack ghci --ghci-options .stack-work/dist/x86_64-linux-nix/Cabal-1.22.5.0/build/cbits/ancilData.o

Perhaps the right logic to add to Stack is: "for each file in c-files, add an extra argument to the GHCi call"? Much like runhaskell Setup.hs build adds such object files to the ghc --make call.

[mgsloan]

@mboes That sounds like a quite likely solution!

[mboes]

Aha! It turns out stack ghci is already doing the right thing: https://github.com/commercialhaskell/stack/blob/master/src/Stack/Package.hs#L344-L348. So in my case at least, and possibly in OP's case too, the problem is really with the network package: it's not listing all C files it wants linked in the c-files stanza.

[mgsloan]

Good news! @mboes committed an elegant PR resolving this. Great stuff! I didn't realize it could be so straightforward.

[mboes]

To be clear, I'm not sure that PR #2242 fixes all instances of this error in stack ghci. @cocreature's nanovg example in particular won't be improved by that PR. But it sounds like @cocreature worked out a reasonable workaround for his case, and if @vagifverdi still has issues with latest Stack feel free to reopen.