[RFC 0082] lib.experimental

[fare]

I propose that we create a directory lib/experimental for experimental code,
also available as lib.experimental and pkgs.lib.experimental. Such code is
available for limited use within select parts of nixpkgs, with the explicit
opt-in approval of their respective maintainers, but not for widespread use
in other parts of nixpkgs.

[FRidh]

I suggest participating in #83 which is about creating a common interface for sub package sets, of which gerbil-packages is one.

[roberth]

I appreciate your intent to streamline Nixpkgs development in relation to lib, but I am concerned about the cost of the added process and the invisible cost of missed contributions and collaboration. Perhaps you could provide references for the conflicts or accidents that you're trying to resolve? I am not aware of any, but if they exist in sufficient numbers, that may justify it.

[roberth]

Linux' staging directory comes to mind, although that has stricter rules around it.

[roberth]

This may include functions like cleanSourceWith which have almost no usage in Nixpkgs itself.
While some problems may be solvable outside Nixpkgs, having a blessed implementation in Nixpkgs can help avoid fragmentation.

[roberth]

This seems to contradict the Detailed design section, not being under lib.experimental.

[roberth]

This placement will not allow individual experimental functions to be added to existing lib files, even if they should reuse let bindings from those files. File-wide let definitions aren't used as much as they probably should though.

[roberth]

Ideally, all functions are documented in the Nixpkgs manual. Not all of them are, which is a separate issue, but well documented seems to imply that it's in the manual.
Otherwise, I don't see how they can get years of good use. Years does seem a bit inflexible. I'd hope we'll be able to promote functions sooner than 2 years.

[roberth]

Should I read this as "forbidden"?

Nixpkgs doesn't only define packages. Most of the interesting use cases for lib functions are in nixos/ (calling into lib.modules, lib.types and various functions to support the configuration logic). Some of the functions primarily support users' own projects, or at least that's how I think of lib.cleanSourceWith.

Usually when you choose to rely on someone's open source function, you accept that you may have to help improve it when the need arises. This is a mutually beneficial situation. You can make use of whatever beneficial qualities that code already has and when you need an improvement, the others will benefit as well.

When you decrease the appeal of certain functions, they will receive less use, and therefore less attention.
In my view, use of Nixpkgs is already quite pragmatic, in the sense that contributions arise from users' specific needs. So my questions is, what is the cost of keeping users away?

[fare]

I mean that, assuming my pop library gets in, and someone wants to use it while sending a patch to some recipe that uses a different extension mechanism, I expect the recipe's owner to reject it and request the patch author to use the previously used extension mechanism instead. Alternatively, if daring, he'd ask the patch author to rewrite everything using pop.

[roberth]

What is the cost of these accidents and conflict. Could you provide real-world examples?

Also note that small accidents and conflicts (human or otherwise) are part of how some software evolves.

[fare]

Not sure about real-world examples in nixpkgs... maybe this is all overkill, but I was told not to make my pop.nix library part of the regular lib/ for now, so I wanted a way to get it in without it being "regular".

[edolstra]

This RFC is now open for shepherd nominations!

[lheckemann]

@fare do you plan to continue with this RFC? If not, we would suggest closing it with the possibility of someone (else?) picking it up again later.

[fare]

Yes, I'd like to see it through.

Alternative much simpler proposal: things in lib just get marked ### EXPERIMENTAL! at the top of the file and their contents don't get merged at toplevel.

[FRidh]

The Motivation lacks some rules as to what should not go in lib.experimental, and the Detailed design states some rules on when code from lib.experimental can be considered "stable", however, I am missing when we consider code to be a sufficiently good additional that it can go in lib.experimental. The Motivation states when its not badly written, unused or not fitting. But what if similar kind of code exists? Clearly a motivation needs to be given in the PR (as in every PR) and I think there still also needs to be ownership over the whole of lib.experimental, not just owners of the specific pieces.

[Mic92]

To move this to the next phase we would need shepherds. Anyone open to?

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/rfc-82-lib-experimental-is-looking-for-shepherds/12257/2

[Mic92]

If there are no shepherds for this RFC within two weeks, the rfc steering committee team will close this RFC.

[lheckemann]

@fare or anyone else: if you wish to continue work on this RFC, please feel free to nominate shepherds and request a reopen :)

[roberth]

I nominate myself as shepherd.

Any other takers? Perhaps @sternenseemann?

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/friday-hackday-project-nix-filter/12648/6

[sternenseemann]

Something we should look into as well is how to enforce this. There's probably a nice way to do so using scopedImport.

[sternenseemann]

Any other takers? Perhaps @sternenseemann?

Sure.

[sternenseemann]

Given the recent row about flakes in lib, it'd be great if we could revive this RFC and find the third required sheperd: Then it'd be possible to have a flakes utility library in nixpkgs as many have wanted, but have the benefit that the ability to add experimental code to nixpkgs isn't exclusive to flake related functions.

@zimbatm are you be interested / do you have the time?

[edolstra]

The whole point of flakes is that experimental, controversial or hacky stuff doesn't need to go into Nixpkgs but can live in its own flake. A function like callLocklessFlake seems like a bit of a workaround for the current lack of builtin support for subflakes, but that's not something we want to be stuck with in the standard library forever.

[fare]

Are flakes now considered production quality? If I have packages in main NixOS and they use an experimental library, should I remove all those packages from NixOS and create a flake for them? Create that library in an absurd place close to some of my packages rather than in lib/experimental?

[roberth]

It's not feasible for nixpkgs to depend on flakes, certainly for a long time, so if we truly want to innovate in the beating heart of the ecosystem, we'd better not tell people to go off and do it elsewhere.

The whole point of flakes is that experimental, controversial or hacky stuff doesn't need to go into Nixpkgs but can live in its own flake.

Is that really the whole point? If so, that only stresses the fact that Nixpkgs will remain at the center of the ecosystem and that we need to care for it. Caring includes having a means to deal with non-trivial changes!

Flakes will do more than just that, I know, but that doesn't diminish Nixpkgs' unique position as the ecosystem monorepo or the advantages that this architecture brings.
However green the grass looks on the polyrepo side, let's focus on our monorepo in this RFC discussion.

[lheckemann]

Are flakes now considered production quality?

no, they're still guarded behind experimental-features and have no interface stability guarantees.