nixos-unstable blocked on evaluation error “error: unexpected EOF reading a line”

[andersk]

nixos-unstable has been blocked for a week, and currently fails to evaluate with the dreaded error: unexpected EOF reading a line, which gives no information about where the real failure originated.

Previous/related issues:

• Evaluation error is blocking jobsets #83647
• nixos-unstable evaluation broken on Hydra #99236
• Evaluation error on master since after 2021-01-02 09:24:30 #108304
• Evaluation error blocking nixos-unstable #143937
• hydra-eval-jobs masking real errors hydra#728
• hydra-eval-jobs masks error messages from Nix (again) hydra#822
• Produce better errors for failing jobsets hydra#1025
• hydra-unstable: 2021-08-11 -> 2021-10-27 #143216
• hydra-unstable: 2021-08-11 → 2021-10-27 #144044
• hydra-unstable: 2021-08-11 -> 2021-12-17 #152427
• Hydra updates #160202

[vcunat]

The failure is almost surely running out of memory. At least that's what was reported several days ago.

[vcunat]

Hydra has done many evaluation attempts in the past few days, as I increased the frequency to 10/day in hope that at least sometimes we could get lucky and fit the RAM, but apparently we're too high now. Now I also tried with empty limitedSupportedSystems.

/cc @grahamc

[vcunat]

I wonder if that limitedSupportedSystems change broke it to

{UNKNOWN}: Died at /nix/store/19xkba2a2psaknwn09yi76sdmxm4l0d9-hydra-0.1.20220222.a254612/bin/.hydra-eval-jobset-wrapped line 802. at /nix/store/2d7iizyv1mkg6wklbsv92qzkq0vizn7j-hydra-perl-deps/lib/perl5/site_perl/5.34.0/Catalyst/Model/DBIC/Schema.pm line 526

[vcunat]

I suspect that some success exceptions a week ago were thanks to lower RAM usage soon after evaluator restart. No idea why it would happen, but that's what I thought after looking at the graphs of ceres https://monitoring.nixos.org/grafana/d/hkRCcV0mk/instance-metrics?orgId=1&from=1645236595934&to=1645487941388 with usage only very rarely getting low (<20%), like some kind of "memory leak".

[andersk]

On my system, with the version of hydra-unstable in nixpkgs, hydra-eval-jobs -I . nixos/release-combined.nix --option allow-import-from-derivation false --verbose dies with

error: aggregate job 'tested' references non-existent job 'nixos.tests.nano.x86_64-linux'

presumably due to

• nixosTests.nano: replace with script using GNU expect #161707

With the newer version of hydra-unstable from #160202, this isn’t fatal to hydra-eval-jobs (NixOS/hydra#1025), though I’m guessing it would still be channel-blocking.

[vcunat]

Thanks, dropped in 18bd82e.

[vcunat]

Wow, we got a successful evaluation: https://hydra.nixos.org/eval/1746348

It's a shame that the "Evaluation Errors" tab is so often unusable (in our practice).

[vcunat]

Well, we got a channel bump, but it still won't evaluate successfully in default setting.

[vcunat]

OK, so I dropped aarch64-linux from limitedSupportedSystems, as it seems better than not getting any evaluation, but we're apparently near our RAM limits. Locally the eval succeeded with 32G RAM:

$ hydra-eval-jobs nixos/release-combined.nix -I . --verbose

[leiserfg]

Is it happening again? one week without unstable updates already.

[drupol]

Is it happening again? one week without untable updates already.

I got this update yesterday, and for today, I don't know what is happening, it's seems to be stuck indeed (https://hydra.nixos.org/job/nixos/trunk-combined/tested#tabs-constituents).

[vcunat]

I can't see anything stuck. nixos-unstable is on a two-days old commit, evals happen every day. You need to be more specific about what you think is wrong.

[drupol]

You need to be more specific about what you think is wrong.

I'm not saying something is wrong on my side. However, from my NixOS rookie eye, I just have the impression that sometimes builds are really fast to be done (like the last 3 successful builds 2 days ago) and sometimes they stay much longer and I have the impression that something is stuck somewhere.

[leiserfg]

Sorry, is the nixpkgs-unstable the one blocked.
https://status.nixos.org/

[vcunat]

Yes, that's because our darwin resources were low, so it's been taking longer.

[drupol]

Yes, that's because our darwin resources were low, so it's been taking longer.

Yes indeed, it's extremely slow.

[leiserfg]

8 days already, versionitis is killing me 😄 .

[vcunat]

Thanks to the foundation and Graham, the evaluator was migrated to a much more powerful machine. So these RAM issues shouldn't happen anymore, and I lifted the temporary restriction of aarch64-linux from the primary jobset, since https://hydra.nixos.org/eval/1753830

[vcunat]

Let's close this, so that we don't accumulate more not-that-much related issues into the same thread.

[nh2]

the evaluator was migrated to a much more powerful machine. So these RAM issues shouldn't happen anymore

@vcunat @grahamc Out of interest, how much RAM does that machine have? I would like to know how much one should provision for this type of task.

[vcunat]

The old not-quite-sufficient one was PX62-NVME: 6 cores, 64G ram, https://www.hetzner.com/dedicated-rootserver/matrix-px

Note that the postgres DB runs on a different machine.

[nh2]

And the new sufficient one?

[vcunat]

I don't know exactly, but the RAM insufficiency came just from the large number of NixOS tests in a single evaluation. 100k normal builds in a single evaluation was OK.

[ncfavier]

I think I saw Hetzner AX101 mentioned on Matrix? (that would be 128 GB)