nixos: reusable assertions

[roberth]

Description of changes

Cleaning up some tech debt that we bumped into in #207095 (comment)

Manually tested by editing simple.nix

nix-repl> nixosTests.simple
trace: warning: The option `machine' defined in `makeTest parameters' has been renamed to `nodes.machine'.
«derivation /nix/store/x43w4wab5n15q8jb0ivk1q3raycdk8n9-vm-test-run-simple.drv»

And of course, the classic:

nix-repl> (nixos {}).toplevel
error:
       Failed assertions:
       - The ‘fileSystems’ option does not specify your root file system.
       - You must set the option ‘boot.loader.grub.devices’ or 'boot.loader.grub.mirroredBoots' to make the system bootable.

cc @infinisil for a humble improvement in an area where we had more ambitious ideas.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[ncfavier]

Nice. Could we call _module.assertAndWarn in the module system directly, so that assertions get triggered when evaluating the final config? (It seems like that's where assertions and warnings belong eventually)

[roberth]

Nice. Could we call _module.assertAndWarn in the module system directly, so that assertions get triggered when evaluating the final config? (It seems like that's where assertions and warnings belong eventually)

This is actually hard, because assertions and warnings tend to add strictness.

• We don't really know whether a config is supposed to be final or not. If it is final, we can add strictness, but otherwise it must remain lazy. Submodules tend not to be final, but even this is not a certainty, because the evalModules caller may interlink configurations.
• We don't know which submodules are meant to be checked, and which ones should be ignored.
• We don't yet have the means to find all the submodules and retrieve the assertions that are defined inside. We'd have to wire it through the type somehow, but types assume 1 return values, not a return value that must be split into two (config and _module)

For context, the discussion of such an attempt:

• [Reverted] Module-builtin assertions, disabling assertions and submodule assertions #97023

[infinisil]

Suggested change

	_module.assertAndWarn = mkOption {
	assertAndWarn = mkOption {

The _module namespace should be reserved for module-internal options

[ncfavier]

Quoting you:

The main insight I had was that _module doesn't have a _ at the start because it's for module-internal things, but rather to avoid clashing with option names. While @roberth pointed out that we don't have a checks option at the root, these options are added to submodules as well. So introducing sugar for this could cause problems for any module or submodule which has a checks option. While there is no such option in nixpkgs, we can't be sure that users don't have one on their own.

This is module-internal in spirit anyway.

[infinisil]

That PR was a module-internal implementation of assertions and warnings though, it added support for them to any module (including submodules) since _module.checks is added by default. This is in contrast to this PR here, which is opt-in for modules.

[roberth]

Perhaps a middle ground would be for _module.assertAndWarn to be a module-system-standardized interface rather than a full-featured implementation of the concept?

[infinisil]

I'd think it would be better then to make it a full-featured implementation. Essentially resurrect #97023 but with opt-in triggering of checks using _module.triggerChecks (like _module.assertAndWarn).

[roberth]

Shorthand strikes again. Fair enough.

[roberth]

I suppose users such as the NixOS toplevel could alias the checks as an internal config.checks option, at which point we're basically full circle, but at least the intent around the _module.* attributes will be clear?
Seems like a lot of bookkeeping just to create an illusion of a formally approved interface, but ok.

[infinisil]

Another idea I just had: What if we just used #97023 as is with checking done strictly by default, but additionally enable customization of when it trigger (and whether they trigger at all). This way we can just add a couple manual fixes on top of the rare breakages.

[roberth]

Oh no, I forgot to submit my huge response to this. We'll have to make do with a less detailed comment for now.

Laziness is crucial for modularity without infinite recursions. Adding strictness to a module is inherently risky, as we can not know the dependencies that user logic adds. A supposedly helpful warning must never cause evaluations to fail with an infinite recursion. Not even config would be a good enough trigger because the user may need a mutually recursive dependency with another submodule. For the main logic to keep working it must not depend on any checks, because the checks themselves must depend on the main logic. Anything that makes creating such dependencies on checks easy would be a mistake. Instead, we could standardize on specific _module attributes, and leave config itself unencumbered by checks.

This way we can just add a couple manual fixes on top of the rare breakages.

Try to explain this design choice to someone who's confronted with a 120-item eval trace that appears to have no relation to their code, except for two items that are obviously not in a mutually recursive relationship according to any sensible logic. That's if they can even make sense of the data flow in their modules, because yes, that's a plural.

• just - no, this requires a specific skill
• manual fixes - so that will have to be us then
• rare - feel free to play with the crocodile; it rarely bites someone's head off.

Triggering must only happen "automatically" at the ultimate of top level. Anything lower than that is susceptible to infinite recursions when users indirectly create mutual dependencies between, for example, NixOS configurations in a deployment.

[roberth]

We're going to need something outside NixOS. It doesn't have to be perfect, and when the time comes, we deprecate the old opt-in solution.

Encountered it again today

• Group package set related options srid/haskell-flake#72 (comment)

[ncfavier]

This came up again in nix-community/home-manager#3793 (comment)

[ncfavier]

Can we please just merge this

[roberth]

@infinisil The built-in idea is valid but hard and isn't going anywhere anytime soon, unless you commit time (I wish I could). Until then, this is the solution that people want use, and they'll be happy to migrate to something better when it comes along.
Note also that architecturally, the module functions mk*Module are coupled to this module.

TODO

• rebase
• add the module to lib.modules. lib.modules.modules.assertions?
• add to flake as .modules.generic.assertions?

[ncfavier]

Hit this again today, this time we wanted to mkChangedOptionModule settings.foo to settings.bar, where settings is an RFC 42-style submodule. This makes me think that this PR is not ideal, because it would pollute settings with warnings and assertions attributes that would have to be removed. I think those belong under _module as well.

Context: https://github.com/nix-community/home-manager/pull/4086/files#r1235087054

[ncfavier]

I guess another problem with mkRenamedOptionModules in freeformType submodules is that the old option is still left in the resulting config, so you end up with duplicates.

[roberth]

I think those belong under _module as well.

Removing them from config means that we'll have no way (as of yet) to retrieve them for independent evaluation.
Making the checks independent is crucial for avoiding infinite recursions, so without adding a new "out of band" data flow, laziness is in direct conflict with moving them into _module.

As for the reason why, here's a simple example,

settings: submodule, maybe with a freeformType
settings.foo: int
settings.bar: int
settings._module.assertions.foo_is_even: bool

Now a user defines

{ config, ... }: {
  settings.foo = config.settings.bar * 2;
}

So now foo depends on settings.bar, which depends on settings, which depends on the side effects of that submodule, which depend on foo, and that's an infinite recursion.

While you could argue that this situation is avoidable in one way or another, the problem is that the situation will occur, and infinite recursions are very hard to work with for users.

So if you want to have it in _module, you need to at least add a way to retrieve the checks outside of config, which means change the types submodule, attrsOf and listOf to pass _module around, or at least an attribute within it, and then change the options module argument to expose this to the modules.

[ncfavier]

Removing them from config means that we'll have no way (as of yet) to retrieve them for independent evaluation.

What do you mean? evalModules returns both the _module-free config and _module separately.

[roberth]

I guess another problem with mkRenamedOptionModules in freeformType submodules is that the old option is still left in the resulting config, so you end up with duplicates.

What if instead of returning the usual config from a submodule, we could make it return anything. We could make evalModules return { config = _module.return or config; }, such that an assertion module can come in and change things up as needed, ie set { _module.return = { config = removeAliases (config._module.filteredAttrs) config; check = runChecks config._module.checks; }; }.
A module could import that logic into the settings module and then do

config = mkIf cfg.enable {
  _module.checks = [ services.foo.settings.checks ];
  systemd.services.foo = fooService services.foo.settings.config;
}

My main objection with this is that it normalizes the in-band handling of checks, whereas I think handling them outside of config would be nicer.
Maybe treating the checks as first class is actually better, because at least it's not a weird special case where you have to go through options. It all depends on how well we could make the handling of checks through options and types work.

Should that stop us from implementing _module.return to unblock both of your use cases? I don't know.

• Automatically remove aliases from the config file value composably (filteredAttrs?)
• Implement checks|

I have to leave it at this for now, though I feel like I'm going to have to elaborate some things.

[roberth]

Removing them from config means that we'll have no way (as of yet) to retrieve them for independent evaluation.

What do you mean? evalModules returns both the _module-free config and _module separately.

It's more of a submodule problem, and it can be solved, but it's a non-trivial design problem, which is why the built-in assertion feature has been stuck for a long time.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/the-module-system-is-dead-how-to-do-input-validation/30297/1

[piegamesde]

Why is the type for assertions types.listOf types.unspecified instead of something more specific?

[roberth]

Why is the type for assertions types.listOf types.unspecified instead of something more specific?

d5047fa updated the type but didn't model the item type. A submodule for the individual assertion type would be nice.