ZERO Hydra Failures 21.11

[tomberek]

Mission

Every time we branch off a release we stabilize the release branch.
Our goal here is to get as little as possible jobs failing on the trunk/master jobsets.
I'd like to heighten, while it's great to focus on zero as our goal, it's essentially to
have all deliverables that worked in the previous release work here also.

Please note the changes included in RFC 85.

Most significantly, branch off will occur on 2021 Nov 19; prior to that date, ZHF will be conducted
on master; after that date, ZHF will be conducted on the release channel using a backport
workflow similar to previous ZHFs.

Jobsets

nixos:release-21.11 Jobset
nixpkgs:nixpkgs-21.11-darwin Jobset

How many failing jobs are there?

At the opening of this issue we have

• the main x86_64-linux jobset at 653 failing jobs
• x86_64-darwin at 1449
• aarch64-linux at 782

Thanks to nix-review-tools we know which dependencies are causing the most jobs to fail in these individual jobsets:

• nixos:trunk-combined
• nixpkgs:trunk
• https://tomberek.info/eval_reports/1723307.html

Previous releases first evals

20.09 had 1153 failing jobs
21.05 had 789 failing jobs

How to help (textual)

1. Select an evaluation of the trunk jobset
   

2. Find a failed job ❌️ , you can use the filter field to scope packages to your platform, or search for packages that are relevant to you.
   
   Note: you can filter for architecture by filtering for it, eg: https://hydra.nixos.org/eval/1719540?filter=x86_64-linux&compare=1719463&full=#tabs-still-fail

3. Search to see if a PR is not already open for the package. It there is one, please help review it.

4. If there is no open PR, troubleshoot why it's failing and fix it.

5. Create a Pull Request with the fix targeting master, wait for it to be merged.
   If your PR causes around 500+ rebuilds, it's preferred to target staging to avoid compute and storage churn.

6. (after 2021 Nov 19) Please follow backporting steps and target the release-21.11 branch if the original PR landed in master or staging-21.11 if the PR landed in staging. Be sure to do git cherry-pick -x <rev> on the commits that landed in unstable. @jonringer created a video covering the backport process.

Always reference this issue in the body of your PR:

ZHF: #144627

Please ping @NixOS/nixos-release-managers on the PR.
If you're unable to because you're not a member of the NixOS org please ping @jonringer, @tomberek , @nrdxp

How can I easily check packages that I maintain?

You're able to check failing packages that you maintain by running:

# from root of nixpkgs
nix-build maintainers/scripts/build.nix --argstr maintainer <name>

New to nixpkgs?

• Packaging a basic C application
• Python nix packaging
• Adding a package to nixpkgs
• other resources at: https://github.com/nix-community/awesome-nix
• https://nix.dev/tutorials/

Packages that don't get fixed

The remaining packages will be marked as broken before the release (on the failing platforms).
You can do this like:

meta = {
  # ref to issue/explanation
  # `true` is for everything
  broken = stdenv.isDarwin; 
};

Closing

This is a great way to help NixOS, and it is a great time for new contributors to start their nixpkgs adventure. 🥳

cc @NixOS/nixpkgs-committers @NixOS/nixpkgs-maintainers @NixOS/release-engineers

Related Issues

• Timeline: Discourse: 21.11 Release Timeline
• Feature Freeze Items: #21.11 Feature Freeze #140168

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixos-21-11-zero-hydra-failures/15904/1

[nixinator]

Be nice to get @ryantm to run this report,, so we can find out the packages/dependencies that are that are causing the most failures.

https://discourse.nixos.org/t/finding-most-depended-upon-packages-that-fail-to-build-in-hydra/10090

EDIT: I missed it, been a long day in the data mines.

[nrdxp]

@nixinator, @tomberek already ran this report during a meeting we just had and posted it under his personal domain. Not sure if he posted it up above and you just missed it or if he forgot to post it though.

edit

It is posted above:

[fabianhjr]

Create a Pull Request with the fix targeting master, wait for it to be merged. If your PR causes around 500+ rebuilds, it's preferred to target staging to avoid compute and storage churn.

Hey, there had been some discussion about non-breaking (particularly unbreaking/fixing) commits going directly to staging-next rather than staging. Mentioning since any unbreaking changes targeting staging might miss the branch off of the 19th of November.

Eg, #137501

[tomberek]

Thank you everyone for a wonderful ZHF!

[roberth]

Do we have staging jobsets for 21.11? I don't see them in https://hydra.nixos.org/project/nixpkgs. Iirc, this has been improved in the last release, but maybe it hasn't been documented as part of the release process yet? cc @jonringer

[siraben]

Any chance we can get a final detailed eval report? I'm curious now the deltas are for each platform.

[tomberek]

Based on
https://hydra.nixos.org/eval/1727142?compare=1718945
https://hydra.nixos.org/eval/1727142?filter=x86_64-linux&compare=1718945

x86_64-linux jobset at 653 182 failing jobs
x86_64-darwin at 1449 1208 failing jobs (some aborted)
aarch64-linux at 782 505 failing jobs

https://tomberek.info/eval_reports/1727142.html

[jonringer]

Do we have staging jobsets for 21.11? I don't see them in https://hydra.nixos.org/project/nixpkgs. Iirc, this has been improved in the last release, but maybe it hasn't been documented as part of the release process yet? cc @jonringer

we should

[jonringer]

done, just created it, and added notes about its creation to the update wiki PR

[vcunat]

Was there supposed to be a pass where (most of) the remains get marked with meta.broken or similar?

[risicle]

Note there are still a few outstanding unresolved PRs ☝️ #148045 #147405 #147332 #147543 at least

[jonringer]

Was there supposed to be a pass where (most of) the remains get marked with meta.broken or similar?

This is meant to be done on the release branch, but it is a PITA

cc @nrdxp @tomberek