Upgrade to GHC 9.4

[tbidne]

Resolves #6.

Hello, thought I'd take a crack at this. Draft because not everything builds on my machine (see below), but perhaps this is a useful starting point.

Nix build issues

My build issues undoubtedly stem from the nix flake I attempted to use. I see that it is unmaintained -- which is fine -- so I'm opening this PR in case either someone understands the nix problem, or perhaps someone else can verify that it builds without nix, in which case the nix issues are presumably not a blocker.

Click to expand Nix build issues

The error I get is, predictably, related to C dependencies:

Preprocessing library for hw-json-simd-0.1.1.2..
creating dist/build/HaskellWorks/Data/Json/Simd/Internal
creating dist/build/HaskellWorks/Data/Json/Simd
creating dist/build/HaskellWorks/Data/Json
creating dist/build/HaskellWorks/Data
creating dist/build/HaskellWorks
creating dist/build/HaskellWorks/Data
creating dist/build/HaskellWorks/Data/Json
creating dist/build/HaskellWorks/Data/Json/Simd
creating dist/build/HaskellWorks/Data/Json/Simd/Internal
/home/tommy/.cabal/store/ghc-9.4.4/c2hs-0.28.8-e-c2hs-794bf7fa772b7fd373e82fffb8097c4e9a55c573ac24bfb064cce888f38f27a5/bin/c2hs '--cpp=/nix/store/l0fvy72hpfdpjjs3dk4112f57x7r3dlm-gcc-wrapper-12.2.0/bin/cc' '--cppopts=-E' '--cppopts=-D__GLASGOW_HASKELL__=904' '--cppopts=-Dlinux_BUILD_OS=1' '--cppopts=-Dx86_64_BUILD_ARCH=1' '--cppopts=-Dlinux_HOST_OS=1' '--cppopts=-Dx86_64_HOST_ARCH=1' '--cppopts=-Icbits' '--cppopts=-includedist/build/autogen/cabal_macros.h' '--include=dist/build' '--cppopts=-I/home/tommy/.cabal/store/ghc-9.4.4/vector-0.12.3.1-6cb3178a425e01e80112f8106cf2b77b9683c47d60c49844b7234a5c61d06e2d/lib//include' '--cppopts=-I/home/tommy/.cabal/store/ghc-9.4.4/primitive-0.7.4.0-aaba494f9dd7d87204674efe59e6dfee75cbdcc9ce63c07aea9772661852caf3/lib//include' '--cppopts=-I/nix/store/4y9vrwgb31689cfhjshnp374mf52lvxw-ghc-9.4.4/lib/ghc-9.4.4/bytestring-0.11.3.1/include' '--cppopts=-I/nix/store/4y9vrwgb31689cfhjshnp374mf52lvxw-ghc-9.4.4/lib/ghc-9.4.4/base-4.17.0.0/include' '--cppopts=-I/nix/store/wxdjiw2k4257rww31b794vi6ijgzn87w-gmp-with-cxx-6.2.1-dev/include' '--cppopts=-I/nix/store/4y9vrwgb31689cfhjshnp374mf52lvxw-ghc-9.4.4/lib/ghc-9.4.4/ghc-bignum-1.3/include' '--cppopts=-I/nix/store/4y9vrwgb31689cfhjshnp374mf52lvxw-ghc-9.4.4/lib/ghc-9.4.4/rts/include' '--cppopts=-I/nix/store/kzr9q2mpcinlc6w7mgqmh6prsl8kl0x3-libffi-3.4.4-dev/include' '--output-dir=dist/build' '--output=HaskellWorks/Data/Json/Simd/Internal/Foreign.hs' src/HaskellWorks/Data/Json/Simd/Internal/Foreign.chs
c2hs: C header contains errors:

/nix/store/9pgq84sf921xh97gjj2wh7a7clrcrh4m-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0/include/avx512fp16intrin.h:38: (column 18) [ERROR]  >>> Syntax error !
  The symbol `__v8hf' does not fit here.

This error comes from trying to build hw-json-simd. From what I can tell, that __v8hf symbol comes from libc's intrinsics, so my guess is we need a different libc. I'm not really sure what the right version would be (I get the same error trying to build master with ghc 9.2), nor is it clear to me how to override the C library that gets linked (naively changing the buildInputs doesn't appear to do anything; the same <hash>-gcc-12.2.0 is used). I found these issues (1, 2) but I wasn't able to get anything working.

Commenting out the offending libs leads to this classic error:

/nix/store/qmmd097h4rwh2pwgz9l9i0byxb0x8q8l-binutils-2.40/bin/ar: creating dist-newstyle/build/x86_64-linux/ghc-9.4.4/clc-stackage-0.1.0.0/build/objs-2179391/libHSclc-stackage-0.1.0.0-inplace.a
gcc: fatal error: cannot execute ‘/nix/store/9pgq84sf921xh97gjj2wh7a7clrcrh4m-gcc-12.2.0/libexec/gcc/x86_64-unknown-linux-gnu/12.2.0/collect2’: execv: Argument list too long
compilation terminated.
`cc' failed in phase `Linker'. (Exit code: 1)

Also note I updated the flake's nixpkgs to explicitly pin the current version and get a more recent version for the new cabal. Obviously it'd be nicer to have a single pinned nixpkgs, but I figured it was easier to change as few things as possible to try to get something working first.

Bounds

I generated the packages' bounds via a CLI tool. This was a conservative change, merely updating every package's bounds in the current build-depends, removing packages that were dropped from stackage nightly. Of course we'd probably want to include new packages as well, but this seemed like an easier starting point. wc suggests that this PR results in 300 fewer packages.

Note I also added a .gitignore and index-state. I'm happy to remove any of these commits if desired, but I left them in for now as they were useful to me and might be for others.

[Bodigrim]

Thanks, great job @tbidne! I'll take a closer look later next week.

@googleson78 could you possibly look at Nix-related issues here please?

[googleson78]

I'll try building on my end as well.

[googleson78]

Same issue reproduces

@tbidne are you running NixOS?

[tbidne]

Same issue reproduces

@tbidne are you running NixOS?

@googleson78 Yes.

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.15, NixOS, 23.05 (Stoat)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.3`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

I wonder if anyone who has had a successful build recently could share what C libs they are using.

[googleson78]

I had a friend do a successful build of hw-json-simd only, with a nix-provided cabal and ghc94, here's an ldd on the resulting binary:

[nix-shell:~]$ ldd ~/.cabal/bin/hw-json-simd
	linux-vdso.so.1 (0x00007ffcad097000)
	libm.so.6 => /nix/store/yzjgl0h6a3qh1mby405428f16xww37h0-glibc-2.35-224/lib/libm.so.6 (0x00007f429e66c000)
	libgmp.so.10 => /nix/store/iqpqnl4fz14xsyfhv6950cxw10yfm04c-gmp-with-cxx-6.2.1/lib/libgmp.so.10 (0x00007f429e5cc000)
	libc.so.6 => /nix/store/yzjgl0h6a3qh1mby405428f16xww37h0-glibc-2.35-224/lib/libc.so.6 (0x00007f429e200000)
	librt.so.1 => /nix/store/yzjgl0h6a3qh1mby405428f16xww37h0-glibc-2.35-224/lib/librt.so.1 (0x00007f429e5c7000)
	libdl.so.2 => /nix/store/yzjgl0h6a3qh1mby405428f16xww37h0-glibc-2.35-224/lib/libdl.so.2 (0x00007f429e5c0000)
	libffi.so.8 => /nix/store/fynz7l60dnqphimnm6l090pckv5scx7b-libffi-3.4.4/lib/libffi.so.8 (0x00007f429e5b3000)
	/nix/store/yzjgl0h6a3qh1mby405428f16xww37h0-glibc-2.35-224/lib/ld-linux-x86-64.so.2 => /nix/store/yzjgl0h6a3qh1mby405428f16xww37h0-glibc-2.35-224/lib64/ld-linux-x86-64.so.2 (0x00007f429e74e000)

[googleson78]

The glibc in the flake/my glibc appears to be the same, which is a pretty bad sign. Not sure what's going on here.

[tbidne]

Same. With the current diff, ldd --version shows glibc 2.34, which I thought might be the culprit. So locally I used the new nixpkgs hash:

diff --git a/flake.nix b/flake.nix
index 51906aa..62b067e 100644
--- a/flake.nix
+++ b/flake.nix
@@ -19,9 +19,9 @@
       pkgs = import nixpkgs {
         inherit system;
       };
-    in with pkgs-old;
+    in with pkgs;
     {
-      devShell.x86_64-linux = pkgs-old.mkShell {
+      devShell.x86_64-linux = pkgs.mkShell {
         buildInputs = [
           pkgs.cabal-install
           libffi.dev

Which changed ldd --version to 2.35, like your friend, but I still get the same result 😕.

[googleson78]

Ah sorry, I totally forgot to mention that I also did that change before testing. But it seems that didn't really help much.

In any case, this seems to be nix(os) exclusive, so maybe we should just merge this with a disclaimer on the flake?

[tbidne]

Fine by me, assuming this works for other users. For the record, I'd like to highlight the message here:

c2hs: C header contains errors:

/nix/store/9pgq84sf921xh97gjj2wh7a7clrcrh4m-gcc-12.2.0/lib/gcc/x86_64-unknown-linux-gnu/12.2.0/include/avx512fp16intrin.h:38: (column 18) [ERROR]  >>> Syntax error !
  The symbol `__v8hf' does not fit here.

@googleson78 Do you know where this 9pgq84sf921xh97gjj2wh7a7clrcrh4m-gcc-12.2.0 comes from? Do you have the same error? It seems to be independent from the nix shell...I played around with some minimal shells, changing the nixpkgs hash, adding --pure, yet it is always the same hash. I wonder if hw-json-simd is getting linked to a system library somehow.

[googleson78]

Can you build from https://github.com/googleson78/clc-stackage/tree/ghc944? Just so we're 100% on the same page. I'll post my exact error as well a bit later.

[tbidne]

Sure, I get the same error.

Also, I figured out the origin of that referenced gcc in the error. Somewhere in the logs there are lines like:

Using gcc version 12.2.0 found on system at:
/nix/store/l0fvy72hpfdpjjs3dk4112f57x7r3dlm-gcc-wrapper-12.2.0/bin/cc

That gcc, l0fvy72hpfdpjjs3dk4112f57x7r3dlm-gcc-wrapper-12.2.0, references 9pgq84sf921xh97gjj2wh7a7clrcrh4m-gcc-12.2.0 that appears in the final error. I.e.

$ cat /nix/store/l0fvy72hpfdpjjs3dk4112f57x7r3dlm-gcc-wrapper-12.2.0/nix-support/orig-cc 
/nix/store/9pgq84sf921xh97gjj2wh7a7clrcrh4m-gcc-12.2.0

I believe that l0fvy72hpfdpjjs3dk4112f57x7r3dlm-gcc-wrapper-12.2.0 comes from whatever GHC is built with. I believe this because these gcc versions (l0fvy72hpfdpjjs3dk4112f57x7r3dlm and 9pgq84sf921xh97gjj2wh7a7clrcrh4m in this example) depend on the GHC you're using. In this case it was some GHC 9.4.4 that exists in my nix store. But when I used a GHC 9.4.4 I built from source, I see two different versions (h5003wsy3qqimqvrkn3bc5mwq4hhidag-gcc-wrapper-12.2.0 and p975i9blgmkjfxpnlvdmm0xvjg573b6l-gcc-12.2.0). Crucially, that first one, h5003wsy3qqimqvrkn3bc5mwq4hhidag, is what I used to build GHC.

Thus it appears the error message is coming from c2hs trying to use whatever gcc was used to build GHC. Is that significant? No idea. This is outside my area of expertise.

[Bodigrim]

@tbidne could you please update this line:
https://github.com/Bodigrim/clc-stackage/blob/2e36d7e1010a1c5edd00d98dba5a672f892b9dff/clc-stackage.cabal#L9
?

[tbidne]

Ah right, one sec.

[Bodigrim]

Of course we'd probably want to include new packages as well, but this seemed like an easier starting point.

I'm happy with the current state of the PR. Would you like to add new packages in a separate PR?

[tbidne]

Sure, I can do that in a separate PR.

[Bodigrim]

Merged, huge thanks!

[tbidne]

No problem!