Update to nixos-23.11 #9546

thufschmitt · 2023-12-06T13:14:22Z

Motivation

Update to the latest stable NixOS release.

I didn't try to deeply check anything, just updated the flake input (and removed a redundant fetch) and saw that the build and tests didn't break.

Priorities

Add 👍 to pull requests you find important.

Ericson2314 · 2023-12-06T14:38:45Z

I think I would like to do #9535 first. Some of the more exotic things have regressions in 23.11 I need to fix also.

edolstra · 2024-01-12T16:28:03Z

This is failing because the bear package is marked as broken. (Don't know what that is, but we're using it in the clang devshell.)

cole-h · 2024-01-12T20:43:30Z

It's used for LSP (if you run your make commands with bear, it'll generate compile_commands.json that LSP servers use to find e.g. headers and whatnot).

It was marked broken in NixOS/nixpkgs#250435 (comment) because it fails to build on x86_64-darwin (it built fine on aarch64-darwin for me).

Ericson2314 · 2024-01-25T17:02:15Z

I guess let's adjust that meta.broken to be more fine-grained? And then the dep here can be corresponding adjusted.

SuperSandro2000 · 2024-01-25T18:49:19Z

That was already started in NixOS/nixpkgs#279035

lf- · 2024-02-27T06:29:41Z

@thufschmitt can you have another go at this? bear is fixed now, and I now have a PR blocked on the C++ compiler being antique.

thufschmitt · 2024-02-27T12:35:18Z

@thufschmitt can you have another go at this? bear is fixed now, and I now have a PR blocked on the C++ compiler being antique.

The fix hasn't been backported to 23.11 (I've just opened NixOS/nixpkgs#291814 for that).
But that shouldn't block us, we can just disable bear on Darwin. Trying to build the thing right now

thufschmitt · 2024-02-27T13:21:28Z

@Ericson2314 the evaluation failures on Darwin (https://github.com/NixOS/nix/actions/runs/8064971474/job/22029891317?pr=9546) are above my pay grade, do you think you can have a look?

thufschmitt · 2024-02-27T19:04:45Z

OK, so I managed to fix the evaluation errors on Darwin (by just disabling everything that failed).

I'm quite bewildered by the evaluation performance between 23.05 and 23.11 though: evaluating the flake on 23.11 takes nearly 3x longer than on 23.05 (despite having removed some outputs)

$ hyperfine --warmup 1 \
    "nix flake show nix/nixos-23.11 --option eval-cache false --all-systems" \                                                            
    "nix flake show nix --all-systems --option eval-cache false"
Benchmark 1: nix flake show nix/nixos-23.11 --option eval-cache false --all-systems
  Time (mean ± σ):     27.201 s ±  0.065 s    [User: 19.490 s, System: 2.875 s]
  Range (min … max):   27.100 s … 27.310 s    10 runs
 
Benchmark 2: nix flake show nix --all-systems --option eval-cache false
  Time (mean ± σ):      9.398 s ±  0.216 s    [User: 7.261 s, System: 1.046 s]
  Range (min … max):    9.054 s …  9.843 s    10 runs
 
Summary
  nix flake show nix --all-systems --option eval-cache false ran
    2.89 ± 0.07 times faster than nix flake show nix/nixos-23.11 --option eval-cache false --all-systems

any one knows what that could be due to?

thufschmitt · 2024-02-27T19:05:18Z

@infinisil there's a failure in the fileset tests on darwin: https://github.com/NixOS/nix/actions/runs/8069576505/job/22044909419?pr=9546 does that match anything already known?

infinisil · 2024-02-27T21:36:11Z

Oof this is weird, it looks like a bash regex matching bug. The behavior of matching \s (any space character) using =~ differs between Linux and Darwin:

# default.nix
{ system ? builtins.currentSystem }:
let
  pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/tarball/931ee755b560f905775a11b3a719a303c8f992e1") {
    inherit system;
    config = {};
    overlays = [];
  };
in
pkgs.runCommandNoCC "bash-regex-${system}" {} ''
  r="^\s$"
  if [[ " " =~ $r ]]; then
    echo "\s working"
  else
    echo "\s NOT working"
  fi > $out
''

❯ nix-build --argstr system x86_64-linux && cat result
/nix/store/w0dlf0zzg4n8s8ga9axicbw2axdpinqv-bash-regex-x86_64-linux
\s working
❯ nix-build --argstr system x86_64-darwin && cat result
/nix/store/ikwd9iqq44pbilrbca7lk6zhh6q4ykry-bash-regex-x86_64-darwin
\s NOT working
❯ nix-build --argstr system aarch64-darwin && cat result
/nix/store/mwpgr44xk5bvsr71mjs8cfcqm6x4krdn-bash-regex-aarch64-darwin
\s NOT working

(can't test on aarch64-linux)

infinisil · 2024-02-27T22:46:57Z

Turns out \s is not POSIX.. See NixOS/nixpkgs#291933 for a fix

This was found when trying to run the fileset tests on Darwin (NixOS/nix#9546 (comment)), which mysteriously fail on Darwin: test case at lib/fileset/tests.sh:342 failed: toSource { root = "/nix/store/foobar"; fileset = ./.; } should have errored with this regex pattern: lib.fileset.toSource: `root` \(/nix/store/foobar\) is a string-like value, but it should be a path instead. \s*Paths in strings are not supported by `lib.fileset`, use `lib.sources` or derivations instead. but this was the actual error: error: lib.fileset.toSource: `root` (/nix/store/foobar) is a string-like value, but it should be a path instead. Paths in strings are not supported by `lib.fileset`, use `lib.sources` or derivations instead. After dissecting this, I find out that apparently \s works on Linux, but not on Darwin for some reason! From the bash source code, it looks like <regex.h> with `REG_EXTENDED` is used for all platforms the same, so there's nothing odd there. It's almost impossible to know where <regex.h> comes from, but it looks to be a POSIX thing. So after digging through the almost impossible to find POSIX specifications (https://pubs.opengroup.org/onlinepubs/007908799/xbd/re.html#tag_007_003_005), I can indeed confirm that there's no mention of \s or the like! _However_, there is a mention of `[[:blank:]]`, so we'll use that instead.

This was found when trying to run the fileset tests on Darwin (NixOS/nix#9546 (comment)), which mysteriously fail on Darwin: test case at lib/fileset/tests.sh:342 failed: toSource { root = "/nix/store/foobar"; fileset = ./.; } should have errored with this regex pattern: lib.fileset.toSource: `root` \(/nix/store/foobar\) is a string-like value, but it should be a path instead. \s*Paths in strings are not supported by `lib.fileset`, use `lib.sources` or derivations instead. but this was the actual error: error: lib.fileset.toSource: `root` (/nix/store/foobar) is a string-like value, but it should be a path instead. Paths in strings are not supported by `lib.fileset`, use `lib.sources` or derivations instead. After dissecting this, I find out that apparently \s works on Linux, but not on Darwin for some reason! From the bash source code, it looks like <regex.h> with `REG_EXTENDED` is used for all platforms the same, so there's nothing odd there. It's almost impossible to know where <regex.h> comes from, but it looks to be a POSIX thing. So after digging through the almost impossible to find POSIX specifications (https://pubs.opengroup.org/onlinepubs/007908799/xbd/re.html#tag_007_003_005), I can indeed confirm that there's no mention of \s or the like! _However_, there is a mention of `[[:blank:]]`, so we'll use that instead. (cherry picked from commit 3429594)

About time :) This required disabling `bear` on darwin as it's currently broken (fixed on master, but not yet on 23.11).

The required version check was a bit too lenient, and `nixpkgs#nixUnstable` was considered valid while it didn't have the fix.

Just `stdenv.isDarwin` isn't enough because it doesn't apply to the build platform, which mean that cross packages building from darwin to another platform will have `isDarwin` set to false. Replace it by `stdenv.buildPlatform.isDarwin`.

Don't evaluate, and probably not really useful (if at all)

Flake lock file updates: • Updated input 'nixpkgs': 'github:NixOS/nixpkgs/4dd376f7943c64b522224a548d9cab5627b4d9d6' (2024-02-26) → 'github:NixOS/nixpkgs/b550fe4b4776908ac2a861124307045f8e717c8e' (2024-02-28)

I hate this. We should have it, but for now we can't.

The required version check was a bit too lenient, and `nixpkgs#nixUnstable` was considered valid while it didn't have the fix.

Apparently gcc is able to implicitly cast from `FileIngestionMethod` to `ContentAddressMethod`, but clang isn't. So explicit the cast

`NIX_HARDENING_ENABLE` causes `_FORTIFY_SOURCE` to be defined. This isn't compatible with `-O0`, and the compiler will happily remind us about it at every call, spamming the terminal with warnings and stack traces. We don't really care hardening in that case, so just disable it if we pass `OPTIMIZE=0`.

roberth · 2024-02-28T13:08:21Z

flake.nix

-            ++ lib.optional stdenv.cc.isClang pkgs.buildPackages.bear
+            # TODO: Remove the darwin check once
+            # https://github.com/NixOS/nixpkgs/pull/291814 is available
+            ++ lib.optional (stdenv.cc.isClang && !stdenv.buildPlatform.isDarwin) pkgs.buildPackages.bear


Indeed #9900 did not need it. 👍

infinisil · 2024-02-28T18:11:23Z

Docker push is failing: https://github.com/NixOS/nix/actions/runs/8084099897/job/22090423845

This is caused by using Nix 2.13.3:

nix/.github/workflows/ci.yml

Line 119 in 587c7dc

install_url: https://releases.nixos.org/nix/nix-2.13.3/install

Which doesn't have builtins.readFileType, causing lib.fileset to not work correctly in pure evaluation mode, see also NixOS/nixpkgs#266356 (comment)

Ericson2314 · 2024-02-28T21:33:02Z

Clang shell is now broken because a change in how fileset works

error: access to absolute path '/nix/store' is forbidden in pure eval mode (use '--impure' to override)

infinisil · 2024-02-28T21:42:10Z

@Ericson2314 As above, that only happens with Nix versions <2.13.5, though no idea why such an old version is used there

This reverts commit 587c7dc, reversing changes made to 864fc85.

This reverts commit d6d7d2c.

This was found when trying to run the fileset tests on Darwin (NixOS/nix#9546 (comment)), which mysteriously fail on Darwin: test case at lib/fileset/tests.sh:342 failed: toSource { root = "/nix/store/foobar"; fileset = ./.; } should have errored with this regex pattern: lib.fileset.toSource: `root` \(/nix/store/foobar\) is a string-like value, but it should be a path instead. \s*Paths in strings are not supported by `lib.fileset`, use `lib.sources` or derivations instead. but this was the actual error: error: lib.fileset.toSource: `root` (/nix/store/foobar) is a string-like value, but it should be a path instead. Paths in strings are not supported by `lib.fileset`, use `lib.sources` or derivations instead. After dissecting this, I find out that apparently \s works on Linux, but not on Darwin for some reason! From the bash source code, it looks like <regex.h> with `REG_EXTENDED` is used for all platforms the same, so there's nothing odd there. It's almost impossible to know where <regex.h> comes from, but it looks to be a POSIX thing. So after digging through the almost impossible to find POSIX specifications (https://pubs.opengroup.org/onlinepubs/007908799/xbd/re.html#tag_007_003_005), I can indeed confirm that there's no mention of \s or the like! _However_, there is a mention of `[[:blank:]]`, so we'll use that instead.

nixos-discourse · 2024-03-08T08:55:53Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/tweag-nix-dev-update-55/40996/1

thufschmitt requested a review from edolstra as a code owner December 6, 2023 13:14

infinisil mentioned this pull request Dec 7, 2023

Simplify flake.nix by creating generic callPackage-able package.nix #9535

Merged

roberth mentioned this pull request Jan 4, 2024

flake: Go back to regular nixos-23.05-small #9690

Merged

lf- mentioned this pull request Feb 27, 2024

refactor: move the repl input code to its own file #10083

Merged

thufschmitt force-pushed the nixos-23.11 branch from 8fa8b60 to 3626935 Compare February 27, 2024 12:47

github-actions bot added the with-tests Issues related to testing. PRs with tests have some priority label Feb 27, 2024

infinisil mentioned this pull request Feb 27, 2024

lib.fileset: Fix tests on Darwin, more POSIX NixOS/nixpkgs#291933

Merged

1 task

roberth mentioned this pull request Feb 28, 2024

Add nixos-unstable based outputs, bdwgc 8.2.4, Nixpkgs lib tests, cleanup #9900

Open

3 tasks

This was referenced Feb 28, 2024

Git object hashing in libstore #8918

Merged

docker: Allow building for non-root user #9854

Open

thufschmitt and others added 6 commits February 28, 2024 07:08

flake: Update to NixOS 23.11

44f10f0

About time :) This required disabling `bear` on darwin as it's currently broken (fixed on master, but not yet on 23.11).

Bump the required daemon version for the impure-env test

bbef038

The required version check was a bit too lenient, and `nixpkgs#nixUnstable` was considered valid while it didn't have the fix.

Disable the “static” darwin stdenvs

cf3ef06

Don't evaluate, and probably not really useful (if at all)

nixpkgs: nixos-23.11-small -> release-23.11

945940f

Flake lock file updates: • Updated input 'nixpkgs': 'github:NixOS/nixpkgs/4dd376f7943c64b522224a548d9cab5627b4d9d6' (2024-02-26) → 'github:NixOS/nixpkgs/b550fe4b4776908ac2a861124307045f8e717c8e' (2024-02-28)

flake.lock: Strip out treeHash. Too soon...

8dc4b41

I hate this. We should have it, but for now we can't.

thufschmitt force-pushed the nixos-23.11 branch from a37370c to 8dc4b41 Compare February 28, 2024 06:10

thufschmitt added 2 commits February 28, 2024 07:11

Bump the required daemon version for the git hashing tests

6147d27

The required version check was a bit too lenient, and `nixpkgs#nixUnstable` was considered valid while it didn't have the fix.

Fix a too smart implicit cast

da90be7

Apparently gcc is able to implicitly cast from `FileIngestionMethod` to `ContentAddressMethod`, but clang isn't. So explicit the cast

github-actions bot added the store Issues and pull requests concerning the Nix store label Feb 28, 2024

thufschmitt removed the store Issues and pull requests concerning the Nix store label Feb 28, 2024

github-actions bot added the store Issues and pull requests concerning the Nix store label Feb 28, 2024

roberth reviewed Feb 28, 2024

View reviewed changes

roberth approved these changes Feb 28, 2024

View reviewed changes

roberth merged commit 587c7dc into master Feb 28, 2024
15 checks passed

roberth deleted the nixos-23.11 branch February 28, 2024 16:51

roberth mentioned this pull request Feb 28, 2024

actions docker_push_image: Update nix 2.13.3 -> 2.20.3 #10110

Merged

Ericson2314 added a commit to NixLayeredStore/nix that referenced this pull request Feb 29, 2024

Revert "Merge pull request NixOS#9546 from NixOS/nixos-23.11"

d6d7d2c

This reverts commit 587c7dc, reversing changes made to 864fc85.

Ericson2314 added a commit to NixLayeredStore/nix that referenced this pull request Feb 29, 2024

Revert "Revert "Merge pull request NixOS#9546 from NixOS/nixos-23.11""

5a29854

This reverts commit d6d7d2c.

thufschmitt mentioned this pull request Mar 2, 2024

Remove and gitignore the autoreconf generated files #10130

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to nixos-23.11 #9546

Update to nixos-23.11 #9546

thufschmitt commented Dec 6, 2023

Ericson2314 commented Dec 6, 2023

edolstra commented Jan 12, 2024

cole-h commented Jan 12, 2024

Ericson2314 commented Jan 25, 2024

SuperSandro2000 commented Jan 25, 2024

lf- commented Feb 27, 2024

thufschmitt commented Feb 27, 2024

thufschmitt commented Feb 27, 2024

thufschmitt commented Feb 27, 2024

thufschmitt commented Feb 27, 2024

infinisil commented Feb 27, 2024 •

edited

Loading

infinisil commented Feb 27, 2024

roberth Feb 28, 2024

infinisil commented Feb 28, 2024

Ericson2314 commented Feb 28, 2024

infinisil commented Feb 28, 2024

nixos-discourse commented Mar 8, 2024

Update to nixos-23.11 #9546

Update to nixos-23.11 #9546

Conversation

thufschmitt commented Dec 6, 2023

Motivation

Priorities

Ericson2314 commented Dec 6, 2023

edolstra commented Jan 12, 2024

cole-h commented Jan 12, 2024

Ericson2314 commented Jan 25, 2024

SuperSandro2000 commented Jan 25, 2024

lf- commented Feb 27, 2024

thufschmitt commented Feb 27, 2024

thufschmitt commented Feb 27, 2024

thufschmitt commented Feb 27, 2024

thufschmitt commented Feb 27, 2024

infinisil commented Feb 27, 2024 • edited Loading

infinisil commented Feb 27, 2024

roberth Feb 28, 2024

Choose a reason for hiding this comment

infinisil commented Feb 28, 2024

Ericson2314 commented Feb 28, 2024

infinisil commented Feb 28, 2024

nixos-discourse commented Mar 8, 2024

infinisil commented Feb 27, 2024 •

edited

Loading