unstable/unstable-small boot tests blocked due to alsa-firmware failing nix verify

[JJJollyjim]

Describe the bug

nix verify fails in the boot tests, like so:

machine # path '/nix/store/7d37hb96a4m0z7kx0crh7vqyi24r1iwz-alsa-firmware-1.2.1' was modified! expected hash 'sha256:04051aw1fpag65s5nrvaj8bf55x8nzckq6610lz0vfxy7sh9cp7n', got 'sha256:154zsxs2d2zxqgr0mia3yj26m5szwygab3z122ws9ydzq9ilglkw'
machine: output: 
error: 
Traceback (most recent call last):
  File "/nix/store/mb9lhv3n6h20ybbpr2j8xdhbv5jgijwb-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 943, in run_tests
    exec(tests, globals())
  File "<string>", line 1, in <module>
  File "<string>", line 10, in <module>
  File "/nix/store/mb9lhv3n6h20ybbpr2j8xdhbv5jgijwb-nixos-test-driver/bin/.nixos-test-driver-wrapped", line 483, in succeed
    raise Exception(
Exception: command `nix verify -r --no-trust /run/current-system` failed (exit code 1)
cleaning up
(0.00 seconds)

See e.g. the history here: https://hydra.nixos.org/job/nixos/unstable-small/nixos.tests.boot.biosCdrom.x86_64-linux/all

Steps To Reproduce

On master, nix-build nixos/release.nix -A tests.boot.biosCdrom.x86_64-linux. This failed for me on the first try:

machine # path '/nix/store/fyx0rpdshfb9np7sb2dqr50rm72xzwkh-alsa-firmware-1.2.1' was modified! expected hash 'sha256:04051aw1fpag65s5nrvaj8bf55x8nzckq6610lz0vfxy7sh9cp7n', got 'sha256:154zsxs2d2zxqgr0mia3yj26m5szwygab3z122ws9ydzq9ilglkw'

Interestingly, that path is fine on my host machine:

$ nix verify -r --no-trust /nix/store/fyx0rpdshfb9np7sb2dqr50rm72xzwkh-alsa-firmware-1.2.1
$ echo $?
0
$

Expected behavior

The boot tests succeed, and channels are unblocked.

Notify maintainers

Unclear who to ping here, since I'm not sure if this is a nix daemon bug, a corruption issue on cache.nixos.org, or an alsa-firmware problem...

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: alsa-firmware
# a list of nixos modules affected by the problem
module:

[FRidh]

@roberth I think you referred to a related Nix issue earlier when I asked, but I can't recall what it was.

[vcunat]

It was this PR: #123943 (but not in the sense that it fixes this issue)

[vcunat]

It would be nice to understand why the modification happens.

I tried just dumb bisection. The problem seems reliably reproducible on my machine, and I arrived at c5114b3. (retried multiple times on the commit and its parent) But it doesn't really make sense, and reverting that merge on master does not fix it for me anyway.

[JJJollyjim]

I grabbed the nar out of a running test VM with nix-store --dump /nix/store/fyx0rpdshfb9np7sb2dqr50rm72xzwkh-alsa-firmware-1.2.1.

Here's the diff-of-xxds:
vimdiff <(xxd fyx0rpdshfb9np7sb2dqr50rm72xzwkh-alsa-firmware-1.2.1.nar-from-vm) <(curl https://cache.nixos.org/nar/1qsqiayqgg7y3pvzrsg1mkxbq2y33mbbwv6wc315b434l9vh002i.nar.xz | unxz - | xxd)

[JJJollyjim]

Checking edolstra's PHD thesis to find out what this actually means :)

[vcunat]

Maybe it would be easier to somehow "unpack" it into a directory and compare those.

[JJJollyjim]

It's the sizes of the files /nix/store/fyx0rpdshfb9np7sb2dqr50rm72xzwkh-alsa-firmware-1.2.1/share/alsa/firmware/usx2yloader/us224.prepad and /nix/store/fyx0rpdshfb9np7sb2dqr50rm72xzwkh-alsa-firmware-1.2.1/share/alsa/firmware/usx2yloader/us428.prepad.

In the cache version, they're 70 and 72 bytes respectively, in the vm version, they're 71 each.

[JJJollyjim]

If I'm understanding correctly, this is a simple non-reproducable build, and due to some messy nix behavior, that is manifesting as a failed verify, since the narinfos come from two places with two different ideas of what the final output hash should be?

I don't believe enforcing true reproducibility as a channel blocker is the intention of this test, right?

[vcunat]

I had checked that first: on my machine the nix-build -QA alsa-firmware --check always succeeds (i.e. reproduces the hash from cache.nixos.org). Of course, in the VM it might be different, but so far I suspect the problem is somewhere else.

[JJJollyjim]

Ah of course, the VM isn't actually doing the build it's presumably getting it from the same place as the host. Which means something spookier is going on...

[vcunat]

Maybe it's some bug related to those files containing just zero bytes.

[JJJollyjim]

Ah yeah, looking at the alsa-firmware sources these files do seem to just be static files, so the build being truely reproducable seems unlikely. The correct sizes are:

.rw-r--r-- 71 jamie 14 Nov  2019 us122.prepad
.rw-r--r-- 70 jamie 14 Nov  2019 us224.prepad
.rw-r--r-- 72 jamie 14 Nov  2019 us428.prepad

So cache.nix.org is correct and the VM is incorrect. I assume the VM is getting it from the iso file it's booting? Will check that next.

Cosmic rays? :3

[JJJollyjim]

Yep, I have mounted the squashfs from the ISO and it is indeed wrong:

-r--r--r-- 1 root root 71 Jan  1  1970 share/alsa/firmware/usx2yloader/us122.prepad
-r--r--r-- 1 root root 71 Jan  1  1970 share/alsa/firmware/usx2yloader/us224.prepad
-r--r--r-- 1 root root 71 Jan  1  1970 share/alsa/firmware/usx2yloader/us428.prepad

[vcunat]

I suspect plougher/squashfs-tools@19b161c I'll verify.

[ncfavier]

That commit doesn't seem to be part of squashfs-tools 4.5 though.

[JJJollyjim]

No, the bug was introduced in 4.5, and the fix is not yet part of a release, see the top of the readme:

NEWS
----

2021-07-25 Important bug found in release.

A new point release will be forthcomming in the
next couple of days.  Sooner if no other release
bugs are reported.

[ncfavier]

Ah, thanks

[vcunat]

Yes, that patch fixed the bug. At least for me.

I assume it was elusive because sparsity of files isn't perfectly reproducible (especially when you have /nix/store on a different FS, or maybe the order of files mattered).