Upgrading to nixpkgs master #6

nh2 · 2018-09-10T13:53:13Z

The latest nixpkgs master has all fixes I had in my nixpkgs commit 50677e464359947f2a71cfd497c4022e3cdf8c7d.

So I'd like to just switch to a nixpkgs master commit.

The text was updated successfully, but these errors were encountered:

nh2 · 2018-09-10T13:55:02Z

Trying nh2/nixpkgs@efb033a, I get an issue compiling e2fsprogs because apparently the musl-based GCC doesn't understand __GNUC_PREREQ:

niklas@ares:/tmp/nix-build-e2fsprogs-1.44.4.drv-0/e2fsprogs-1.44.4/lib/e2p$ gcc -I. -I../../lib -I../../lib -I../../intl -I../../intl -g -O2 -DHAVE_CONFIG_H -c feature.c -o feature.o
In file included from ../../lib/ext2fs/ext2fs.h:97:0,
                 from feature.c:20:
../../lib/ext2fs/hashmap.h:20:19: error: missing binary operator before token "("
 #if __GNUC_PREREQ (4, 8)
                   ^
../../lib/ext2fs/hashmap.h:25:19: error: missing binary operator before token "("
 #if __GNUC_PREREQ (4, 8)
                   ^

My system gcc understands it:

niklas@ares:/tmp/nix-build-e2fsprogs-1.44.4.drv-0/e2fsprogs-1.44.4/lib/e2p$ /usr/bin/gcc -I. -I../../lib -I../../lib -I../../intl -I../../intl -g -O2 -DHAVE_CONFIG_H -c feature.c -o feature.o

CC @dtzWill

nh2 · 2018-09-11T02:32:00Z

gcc -I. -I../../lib -I../../lib -I../../intl -I../../intl -g -O2 -DHAVE_CONFIG_H -c feature.c -o feature.o in the build environment reproduces the issue immediately.

nh2 · 2018-09-11T02:34:22Z

Pretty sure the reason is the upgrade to musl 1.1.20 (NixOS/nixpkgs#46064); the commit NixOS/nixpkgs@e37c21c itself fails with that message but the commit before it, NixOS/nixpkgs@88ae8f7, doesn't.

dtzWill · 2018-09-11T05:10:30Z

Oh, I don't see this locally because I'm using the fix I committed to staging that addresses this :P.

Sorry about that. Can you try using that patch (however works best for you) and LMK if that fixes the issue for you (especially if it doesn't!).

This PR: NixOS/nixpkgs#45703

nh2 · 2018-09-11T11:18:03Z

@dtzWill I see. Why did it work before though? Why does the musl update make this difference?

nh2 · 2018-09-11T14:31:51Z

On NixOS/nixpkgs@88ae8f7 I also get a Cython build failure due to failing tests (when building sphinx, a dependency of servant):

======================================================================
ERROR: runTest (__main__.CythonRunTestCase)
compiling (cpp) and running parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1117, in run
    self.run_tests(result, ext_so_path)
  File "runtests.py", line 1132, in run_tests
    self.run_doctests(self.module, result, ext_so_path)
  File "runtests.py", line 1142, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 1202, in run_forked_test
    (module_name, result_code & 255))
Exception: Tests in module 'parallel' were unexpectedly killed by signal 11

----------------------------------------------------------------------
Ran 2589 tests in 938.163s

FAILED (errors=1)
Following tests excluded because of missing dependencies on your system:
   Cython.Coverage
   run.asyncio_generators
   run.coverage_api
   run.coverage_cmd
   run.coverage_nogil
   run.numpy_pythran
   run.py35_asyncio_async_def

Note this takes quite long to run: 3066.69s user 191.27s system 220% cpu 24:36.07 total

nh2 · 2018-09-11T14:41:13Z

Perhaps related: https://github.com/NixOS/nixpkgs/blob/288f9c648f2e145f6af913b88eba0696aa288df7/pkgs/development/python-modules/Cython/default.nix#L15-L24

dtzWill · 2018-09-11T16:11:57Z

Don't suppose bumping to latest cython fixes things? O:)

Given the comments in the nix expression you linked disabling tests doesn't seem so bad BUT it'd be good to check the reasons for the failures first... :/.

nh2 · 2018-09-11T22:52:10Z

I can reproduce it in the build environment; it looks a bit cleaner there:

niklas@ares:/tmp/nix-build-python2.7-Cython-0.28.3.drv-0/Cython-0.28.3$ HOME="$NIX_BUILD_TOP" /nix/store/z722f9n43493n99xwjr545s0h39x2b33-python-2.7.15/bin/python2.7 runtests.py -j1

======================================================================
ERROR: runTest (__main__.CythonRunTestCase)
compiling (c) and running longlongindex
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1117, in run
    self.run_tests(result, ext_so_path)
  File "runtests.py", line 1132, in run_tests
    self.run_doctests(self.module, result, ext_so_path)
  File "runtests.py", line 1142, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 1195, in run_forked_test
    cid, result_code = os.waitpid(child_id, 0)
OSError: [Errno 4] Interrupted system call

======================================================================
ERROR: runTest (__main__.CythonRunTestCase)
compiling (cpp) and running parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1117, in run
    self.run_tests(result, ext_so_path)
  File "runtests.py", line 1132, in run_tests
    self.run_doctests(self.module, result, ext_so_path)
  File "runtests.py", line 1142, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 1202, in run_forked_test
    (module_name, result_code & 255))
Exception: Tests in module 'parallel' were unexpectedly killed by signal 139

----------------------------------------------------------------------
Ran 10798 tests in 2871.950s

FAILED (errors=2, skipped=50)
Following tests excluded because of missing dependencies on your system:
   Cython.Coverage
   run.asyncio_generators
   run.coverage_api
   run.coverage_cmd
   run.coverage_nogil
   run.numpy_pythran
   run.py35_asyncio_async_def

ALL DONE

I get 2 failures here though, and now the signal number is different.

nh2 · 2018-09-11T23:05:40Z

The longlongindex fails nondeterministically.

The other one can be reproduced quickly like this (the last CLI argument is a match on a test name):

niklas@ares:/tmp/nix-build-python2.7-Cython-0.28.3.drv-0/Cython-0.28.3$ HOME="$NIX_BUILD_TOP" /usr/bin/time /nix/store/z722f9n43493n99xwjr545s0h39x2b33-python-2.7.15/bin/python2.7 runtests.py -j1 --verbose parallel 
...
compiling (cpp) and running parallel
----------------------------------------------------------------------
Traceback (most recent call last):
  File "runtests.py", line 1117, in run
    self.run_tests(result, ext_so_path)
  File "runtests.py", line 1132, in run_tests
    self.run_doctests(self.module, result, ext_so_path)
  File "runtests.py", line 1142, in run_doctests
    run_forked_test(result, run_test, self.shortDescription(), self.fork)
  File "runtests.py", line 1202, in run_forked_test
    (module_name, result_code & 255))
Exception: Tests in module 'parallel' were unexpectedly killed by signal 139

----------------------------------------------------------------------
Ran 110 tests in 45.450s

That's much quicker than waiting 40 minutes for all the tests.

nh2 · 2018-09-12T02:50:46Z

Workaround for the Cython issue: Disabling tests with dontCheck.

And this other workaround for another issue also works: pyca/pyopenssl#768 (comment)

nh2 · 2018-09-12T02:53:14Z

Getting some opendjk issues when trying to evaluate allStackageExecutables, because of:

assert stdenv.hostPlatform.libc == "glibc";

(CC @Ericson2314 @dtzWill)

niklas@ares ~/src/haskell/static-haskell-nix (git)-[master] % NIX_PATH=nixpkgs=https://github.com/NixOS/nixpkgs/archive/88ae8f7d.tar.gz nix-build --no-link survey/default.nix -A allStackageExecutables --show-trace
error: while evaluating the attribute 'nativeBuildInputs' of the derivation 'sparkle-0.7.4' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating the attribute 'buildInputs' of the derivation 'inline-java-0.8.4' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating 'getOutput' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/attrsets.nix:452:23, called from undefined position:
while evaluating anonymous function at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/stdenv/generic/make-derivation.nix:127:17, called from undefined position:
while evaluating 'overrideCabal' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/lib.nix:37:24, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/configuration-nix.nix:83:9:
while evaluating 'override' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:73:20, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/lib.nix:37:28:
while evaluating 'makeOverridable' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:67:24, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:73:29:
while evaluating 'drvScope' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/make-package-set.nix:88:18, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:69:12:
while evaluating anonymous function at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/hackage-packages.nix:122015:6, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/make-package-set.nix:88:27:
while evaluating 'mkDerivation' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/lib.nix:38:20, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/hackage-packages.nix:122018:6:
while evaluating 'override' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:73:20, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/lib.nix:38:25:
while evaluating 'makeOverridable' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:67:24, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:73:29:
while evaluating anonymous function at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/generic-builder.nix:13:1, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:69:12:
while evaluating 'optionalAttrs' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/attrsets.nix:322:25, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/haskell-modules/generic-builder.nix:474:4:
while evaluating the attribute 'buildInputs' of the derivation 'openjdk-8u181b13' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/stdenv/generic/make-derivation.nix:179:11:
while evaluating 'getOutput' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/attrsets.nix:452:23, called from undefined position:
while evaluating anonymous function at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/stdenv/generic/make-derivation.nix:127:17, called from undefined position:
while evaluating 'callPackageWith' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:108:35, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/top-level/all-packages.nix:6823:13:
while evaluating 'makeOverridable' at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:67:24, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:112:8:
while evaluating anonymous function at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/compilers/openjdk/bootstrap.nix:1:1, called from /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/lib/customisation.nix:69:12:
assertion failed at /nix/store/rvfl5ywiv60qlh7r7yc103g13iwk0ld5-88ae8f7d.tar.gz/pkgs/development/compilers/openjdk/bootstrap.nix:7:1

Note that we can't upgrade to the latest master yet because musl 1.1.20 breaks things vs 1.1.19 (see #6).

nh2 · 2018-09-16T23:31:02Z

I'm also having a trouble the tests for python3.6-Cython-0.28.3, which seems to be some dependency of static-stack (perhaps via sphinx for GHC 8.2?), hangs forever in its test suite with musl.

As opposed to the above Cython test issues with Python 2.7, it doesn't show test failures, only hangs forever.

nh2 · 2018-09-16T23:48:19Z

I've now worked around all of these issues affecting the upgrade to musl-1.1.19 on my branch upstream-nixpkgs-musl-1.1.19, commit 2fe0db9 as of writing.

I can build the working set, most of stackage exacutables (see here) and Stack 1.7 and 1.9 prerelease.

I haven't put in time for support for newer nixpkgs with musl-1.1.20 based on nixpkgs staging yet.

Ericson2314 · 2018-09-17T14:55:19Z

@nh2 you can just remove that assert. I think i was generalizing whatever was there before. A git blame should answer conclusively.

More generally, any chance of being able to get this onto 18.09 too? I might have a few cycles to help now, finally.

dtzWill · 2018-09-18T22:58:30Z

I'm not sure openjdk was working with musl previously--something must be requiring it now that didn't previously (not at all related to 1.1.19 vs 1.1.20). If you have any additional problems you're seeing that appear related to musl (or if you're not sure, of course!) please liberally cc me so I can take a look. Hope to get things working for you soon!

nh2 · 2018-09-20T15:06:02Z

More generally, any chance of being able to get this onto 18.09 too? I might have a few cycles to help now, finally.

@Ericson2314 Which part would you like to get into 18.09? An overlay ala https://github.com/nh2/static-haskell-nix/blob/7ad63b1ab4168e4a97396dddfe59a11a5e3c6f43/survey/default.nix so that Hyda would build at least some static executables from the working attrset?

If yes, sounds like a good idea to ensure that things keep working as stuff like musl is upgraded. Where in nixpkgs should that be placed?

Ericson2314 · 2018-09-20T23:09:27Z

@nh2 Well even better than that overlay incorperate things into the packages proper; I guess I just disagree with those saying it isn't worth it to make a "static nixpkgs": there's tons of enableStatic arguments and the like already, it makes no sense not to hook them up to e.g. hostPlatform.linkage == "static" or something.

nh2 · 2018-09-20T23:50:35Z

@Ericson2314 I agree with you, and am happy to help with that effort. I think it would be awesome if I could request static exes from all kinds of executables easily via nix, and have cache.nixos.org deliver them trivially.
But I don't know how to kickstart that or where it should be placed into nixpkgs.

Ericson2314 · 2018-09-22T04:07:40Z

@nh2

Go to https://github.com/NixOS/nixpkgs/blob/master/lib/systems/default.nix and add some sort of linkage with "static" or "shared". (Feel free to bike shed.)
Grep for enableShared options and make them enableShared ? stdenv.hostPlatform.linkage.
There's this --disabled-shared logic we should probably move out of stdenv/generic/setup.sh into a setup hook (that would prepend a preConfigureHook or something). (Honestly we could maybe remove it altogether, but that's a larger job.)

nh2 · 2018-09-25T09:40:08Z

Note for myself: Probably also want NixOS/nixpkgs#47278 via NixOS/nixpkgs#46923 (comment).

nh2 · 2019-08-14T19:05:34Z

nh2 · 2019-10-27T14:05:30Z

I did another big master-pin update in #61.

nh2 · 2021-07-23T00:49:27Z

Another big update to nixpkgs-unstable in #107.

nh2 self-assigned this Sep 10, 2018

nh2 mentioned this issue Sep 11, 2018

musl: 1.1.19 -> 1.1.20 NixOS/nixpkgs#46064

Merged

9 tasks

nh2 mentioned this issue Sep 11, 2018

flaky test: Interrupted system call in longlongindex cython/cython#2602

Open

nh2 mentioned this issue Sep 12, 2018

Building all Stackage executables #4

Closed

nh2 added a commit that referenced this issue Sep 12, 2018

Upgrade to nixpkgs master instead of nh2's fork. See #6.

338d136

Note that we can't upgrade to the latest master yet because musl 1.1.20 breaks things vs 1.1.19 (see #6).

dtzWill mentioned this issue Sep 25, 2018

gcc*: fix libgomp (OpenMP) to work w/musl, don't build DSO using initial-exec NixOS/nixpkgs#47309

Merged

9 tasks

This was referenced Aug 14, 2019

Couldn't build hmatrix due to a curl build error #46

Closed

nixpkgs: Update submodule to latest master #44

Merged

nh2 mentioned this issue Aug 17, 2019

Fully static Haskell executables - overview issue NixOS/nixpkgs#43795

Open

60 tasks

nh2 closed this as completed Jul 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrading to nixpkgs master #6

Upgrading to nixpkgs master #6

nh2 commented Sep 10, 2018

nh2 commented Sep 10, 2018

nh2 commented Sep 11, 2018

nh2 commented Sep 11, 2018 •

edited

Loading

dtzWill commented Sep 11, 2018

nh2 commented Sep 11, 2018

nh2 commented Sep 11, 2018 •

edited

Loading

nh2 commented Sep 11, 2018

dtzWill commented Sep 11, 2018

nh2 commented Sep 11, 2018

nh2 commented Sep 11, 2018

nh2 commented Sep 12, 2018

nh2 commented Sep 12, 2018 •

edited

Loading

nh2 commented Sep 16, 2018

nh2 commented Sep 16, 2018 •

edited

Loading

Ericson2314 commented Sep 17, 2018

dtzWill commented Sep 18, 2018

nh2 commented Sep 20, 2018

Ericson2314 commented Sep 20, 2018

nh2 commented Sep 20, 2018

Ericson2314 commented Sep 22, 2018

nh2 commented Sep 25, 2018

nh2 commented Aug 14, 2019 •

edited

Loading

nh2 commented Oct 27, 2019

nh2 commented Jul 23, 2021

Upgrading to nixpkgs master #6

Upgrading to nixpkgs master #6

Comments

nh2 commented Sep 10, 2018

nh2 commented Sep 10, 2018

nh2 commented Sep 11, 2018

nh2 commented Sep 11, 2018 • edited Loading

dtzWill commented Sep 11, 2018

nh2 commented Sep 11, 2018

nh2 commented Sep 11, 2018 • edited Loading

nh2 commented Sep 11, 2018

dtzWill commented Sep 11, 2018

nh2 commented Sep 11, 2018

nh2 commented Sep 11, 2018

nh2 commented Sep 12, 2018

nh2 commented Sep 12, 2018 • edited Loading

nh2 commented Sep 16, 2018

nh2 commented Sep 16, 2018 • edited Loading

Ericson2314 commented Sep 17, 2018

dtzWill commented Sep 18, 2018

nh2 commented Sep 20, 2018

Ericson2314 commented Sep 20, 2018

nh2 commented Sep 20, 2018

Ericson2314 commented Sep 22, 2018

nh2 commented Sep 25, 2018

nh2 commented Aug 14, 2019 • edited Loading

nh2 commented Oct 27, 2019

nh2 commented Jul 23, 2021

nh2 commented Sep 11, 2018 •

edited

Loading

nh2 commented Sep 11, 2018 •

edited

Loading

nh2 commented Sep 12, 2018 •

edited

Loading

nh2 commented Sep 16, 2018 •

edited

Loading

nh2 commented Aug 14, 2019 •

edited

Loading