Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

init Modular Portable Service Layer, proof of concept #267111

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

roberth
Copy link
Member

@roberth roberth commented Nov 12, 2023

Description of changes

It implements a bit more than necessary, allowing an abstract service to potentially create multiple systemd services, if it seeks out options?systemd to figure out the kind of process manager it's loaded into.
I think that's nice functionality, but the proof of concept could be changed to something more restrictive if that's the direction you want to take. (ie don't define process-manager specific options; only read the data from the abstract service)

I hope this gives some insight into what's possible with the module system.
Or maybe you can learn some useful techniques. This kind of module based infrastructure doesn't exist in NixOS yet. Closest thing would be flake-parts, for which I originally wrote types.deferredModule.

Anyway, a good starting point is the example directory. It's the closest thing to actual documentation.
(Again, sorry, proof of concept status, time, etc etc)

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.11 Release Notes (or backporting 23.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@github-actions github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Nov 12, 2023
@roberth
Copy link
Member Author

roberth commented Nov 12, 2023

Rendered docs

(with broken links)

services.abstract

All services that are configured on the system.

Type: lazy attribute set of (submodule)

Default: { }

Declared by:

<nixpkgs/nixos/modules/services/abstract/systemd.nix>
<nixpkgs/nixos/modules/services/abstract/generic.nix>
services.abstract.<name>.enable

Whether to enable the service. A service is enabled by default, because you took the effort to define its attribute.

Type: boolean

Default: true

Declared by:

<nixpkgs/nixos/modules/services/abstract/instance/generic.nix>
services.abstract.<name>.args

Arguments to pass to the foreground.process or daemon.process. In other words, these arguments are passed regardless of process manager choice.

Type: list of (string or path convertible to it)

Default: [ ]

Declared by:

<nixpkgs/nixos/modules/services/abstract/instance/generic.nix>
services.abstract.<name>.daemon.args

Arguments to pass to the daemon.process.

Type: list of (string or path convertible to it)

Default: config.args

Declared by:

<nixpkgs/nixos/modules/services/abstract/instance/generic.nix>
services.abstract.<name>.daemon.process

Path to an executable that launches a process in daemon mode.

Type: string or path convertible to it

Default: config.process

Declared by:

<nixpkgs/nixos/modules/services/abstract/instance/generic.nix>
services.abstract.<name>.foreground.args

Arguments to pass to the foreground.process.

Type: list of (string or path convertible to it)

Default: config.args

Declared by:

<nixpkgs/nixos/modules/services/abstract/instance/generic.nix>
services.abstract.<name>.foreground.process

Path to an executable that launches a process in foreground mode.

Type: string or path convertible to it

Default: config.process

Declared by:

<nixpkgs/nixos/modules/services/abstract/instance/generic.nix>
services.abstract.<name>.process

When this property is specified, it translates both to: foreground.process and daemon.process.

Type: string or path convertible to it

Declared by:

<nixpkgs/nixos/modules/services/abstract/instance/generic.nix>
services.abstract.<name>.systemd.services

This module configures systemd services, with the notable difference that their unit names will be prefixed with the abstract service name.

This option’s value is not suitable for reading, but you can define a module here that interacts with just the unit configuration in the host system configuration.

Type: lazy attribute set of module

Declared by:

<nixpkgs/nixos/modules/services/abstract/instance/systemd.nix>

@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 labels Nov 12, 2023
@ghost
Copy link

ghost commented Nov 13, 2023

I can't spend much time on this at all

Unfortunately the devil really is in the details on these things; until you've built and booted a complete example it's impossible to know that you've solved (or even encountered!) all the hard problems.

@roberth
Copy link
Member Author

roberth commented Nov 13, 2023

Yes, it's only a proof of concept at this stage.

Some next steps could be

  • add more of the service properties
  • write a module for a process manager that isn't the NixOS systemd module
  • port a few services

Would you be interested to collaborate on this?

EDIT: I've removed the quoted text about time from the PR description, because I can make time for some collaboration. It'd be stupid not to.

services.abstract = mkOption {
type = types.lazyAttrsOf (types.submoduleWith {
modules = [
./instance/generic.nix
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hardcodes the generic options, but that's not actually necessary and creates some confusion about what the actually used interface really is, in a way that isn't forward compatible.

It should be up to the actual service to decide in which format it wants to be loaded, and process managers can decide which formats they support.

This could be negotiated through specialArgs and may look like:

# postgresql.nix
{ serviceInterface, ... }: {
  imports = [
    serviceInterface.generic_1_0
    serviceInterface.systemd_nixos_23_11 or { }
  ];
  options = {
    # ...
  };
}

This says that a postgresql service expects the process manager to support the generic_1_0 interface and load that.
The systemd_1_0 interface is optional and is only loaded if the process manager supports it.

imports can of course only depend on specialArgs arguments, but that is sufficient for this purpose.

Note that services can already introspect e.g. options?systemd to figure out in which kind of process manager they are loaded. So the addition of such a mechanism is primarily important for forward compatibility as we'll inevitably evolve the taxonomy of services.

This also allows the creation of helper modules that for instance provide polyfills, emulate one process manager's features on top of another, etc. These could be distributed independently, or as part of the suite of modules and interfaces that are built in to the process manager integrations.

Even just within the NixOS context, such versions let use eat the cake and have it too:

  • NixOS/systemd can make breaking changes in new versions
  • third party service modules (e.g. flakes, etc) have stable targets that they can support

Arguably there's a non-zero cost to having a compatibility layer or keeping old code around, but at least it becomes a solvable problem, and the old solution doesn't come with the expectation of parity with the new solution.

@roberth roberth changed the title init Modular Service Abstraction Layer, proof of concept init Modular Portable Service Layer, proof of concept Nov 20, 2023
@roberth
Copy link
Member Author

roberth commented Nov 20, 2023

Renamed the pr: Abstract -> Portable.
It matches the RFC and is more accurate. These services are not abstract.
Layer feels a bit redundant to me. We don't need to think of parts of the NixOS option tree as layers, but it's not entirely wrong, in the sense that an evaluation order is implicit in the dependencies created by various option definitions (ie the expressions that are merged into the option value, and cause ordering by referencing config).
services.abstract also needs a rename. Not sure which

  • service (top level, singular): nice and simple, but perhaps easy to forget or mistype the s
  • services.portable: wrong in the sense that once the definitions have ended up there, they're not portable anymore; also part of the reason why abstract was wrong
  • serviceGroups: leaning into the fact that in this proof of concept, a Portable Service can define multiple daemons, units, etc.

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/community-calendar/18589/99

@wegank wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 5, 2024
@wegank wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 4, 2024
Comment on lines +40 to +57
daemon = {
process = lib.mkOption {
type = pathOrStr;
description = ''
Path to an executable that launches a process in daemon mode.
'';
default = config.process;
defaultText = literalExpression "config.process";
};
args = lib.mkOption {
type = types.listOf pathOrStr;
description = ''
Arguments to pass to the {option}`daemon.process`.
'';
default = config.args;
defaultText = literalExpression "config.args";
};
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think allowing programs to daemonize themselves will be a trap for this. Foremost new programs don't implement this anymore and a generally discouraged to do so. Old programs implement this in completely varying degrees ranging from full signal handling, creating pid files and being their own process manager to basically doing program & and running or anything in between. Also sometimes config options are involved.

IMO we should only take foreground programs and let the service manager handle all of the "daemon" parts. If the service manager doesn't support this, then IMO it is unsuitable.

@stale stale bot removed 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md labels Aug 20, 2024
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/announcing-services-flake/41338/13

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/portable-submodules/54109/5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.status: merge conflict This PR has merge conflicts with the target branch 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants