Proposal: Subdivide “Experimental” Status

[GeoffreyBooth]

Building off of the discussion in #45711, I think we should split “experimental” status into a few distinct levels to reflect how we actually behave in practice:

• Experimental: Alpha. Early stage features that may be incomplete and subject to substantial change. Both a build flag and a runtime flag are required to enable and a warning is printed. Not all new experimental features need to start as alpha; this is only for unfinished work that isn’t yet ready for end users to test, but for which we want to land PRs iteratively.

• Experimental: Beta. Features that are complete enough for MVP for potential stability, with only minor changes expected. A runtime flag is required to enable and a warning is printed.

• Experimental: Release Candidate. Features that we think are ready to become stable, that we want to allow time to “bake”; only bugfixes are expected. No flags are necessary to enable but a warning is printed.

What problems this solves:

• Currently what would be considered “alpha” or unfinished features are very difficult or impossible to land in reasonably sized PRs. Such features either never land at all or need to gestate somewhere else such as in a node fork (for the case of experimental-modules) or as a library (such as undici for fetch). This high barrier to entry discourages contributions.

• Because we have no gating requirements around experimental features currently, experimental features that land without enabling flags run the risk of becoming popular and entrenched before we’re ready for them to be such; and then these unfinished features become difficult to remove or ship breaking changes for. The classic example of this is async_hooks.createHooks, the focus of Async hooks and CommonJS loaders #45711. By requiring flags at early stages of development, we ensure some level of maturity before features can become popular.

• Because all experimental features are presented as having the same level of stability in our docs, we don’t get as much real-world testing from users for nearly-stable experimental features. Many users presumably ignore all experimental features, rather than draw inferences from whether flags are required or not; but there likely exists some subset of users out there willing to use both stable and “release candidate” features, who could help us shore up nearly-stable features during their “baking” phase.

• If a warning can’t be removed before a feature graduates to “stable,” contributors have an incentive to actually finish their features rather than letting them languish as experimental forever.

What this would mean in practice is that any new experimental features would use the rubric above to decide whether to initially land as alpha or beta with appropriate flags; and we would need to review all existing experimental features and and classify them and update the docs accordingly. I think just about all existing experimental features would be classified as “beta” (the ones that require a flag) or “release candidate” (the ones that don’t require a flag). I can’t think of any current experimental features that we would classify as “alpha;” async_hooks.createHooks is a special case in that it lacks flags or warnings but isn’t “release candidate” and never will be. We would have to decide case-by-case for other exceptions whether a flag or warning should be added.

[jasnell]

I'm not a fan of introducing the additional complexity here. I don't believe it's going to add any actual value for users or make anything easier overall.

[aduh95]

Alpha features wouldn’t even exist on LTS lines

You might want to disambiguate what you mean here, are you saying that alpha features on a release line would need to be reverted when said release line reaches LTS? That seems a lot of work for little benefit. Maybe you meant "alpha features are more likely to graduate before reaching LTS", so if the concern is to protect users from unstability, maybe a runtime flag would work better than blocking backports.

[GeoffreyBooth]

I’ve never had anyone else express concern to me that directly that the way handle experimental this is challenging.

#45711 (reply in thread):

I would really like for us to be much clearer about which features are “stable” and which are not through actions like more aggressive use of experimental-flagging and describing non-LTS releases more like what most other runtimes would call beta. In the early days of Node.js it made sense to encourage people to actively use the latest releases so we’d have a quick enough feedback cycle to catch issues before any of the major users got to it, but with the number of users we have at this point I don’t think that’s necessary anymore. We should be encouraging the use of LTS releases over “current” releases.

If we communicate “current” as more of a beta that would enable us to continue keeping the entry-level low for new contributors to feel comfortable contributing while also being able to gate what is enabled by default in LTS to ensure everything in an LTS line must be fully supported. Rather than every new thing landing in LTS after two weeks regardless of if that was enough time for it to bake fully, we should instead hold things back from LTS until we’re sure it’s fully supported.

There are two important components of any OSS projects which can often be at odds, development velocity and stability. By splitting more completely into two release lines we can not only achieve both but also better focus the differing goals of LTS and current. The state we’re in right now is really just current and slightly older but probably not much more stable current because barely anything gets gated behind flags or anything like that even when landing in an LTS.

#45711 (reply in thread):

What if we had another status in between experimental and stable, like “beta”?

I don’t mean a separate status for shipped code, I mean fully separate release lines. As in code considered “beta” doesn’t even exist in the LTS release.

#45711 (reply in thread):

I do have a concern about a feature becoming relied upon by users while never graduating to stable, like async hooks. That’s a problem I would like a solution for, whether it’s different release lines or a “beta” status between experimental and stable or something else. Criteria for a feature to become stable are meaningless if a feature can stay experimental forever and get wide adoption as such.

I don’t think it’s fair to say that we have no problems to solve. You could argue that my proposal above is worse than the status quo, and that’s fair, or you could suggest improvements to it (like if you don’t like the “backport” part, or if you want to shuffle flags and warnings between levels or reduce levels or whatever) but I think it’s worth discussing whether the status quo is really better than alternatives.

[GeoffreyBooth]

are you saying that alpha features on a release line would need to be reverted when said release line reaches LTS? That seems a lot of work for little benefit. Maybe you meant “alpha features are more likely to graduate before reaching LTS”, so if the concern is to protect users from unstability, maybe a runtime flag would work better than blocking backports.

I wasn’t thinking that we would revert anything. As far as I’m aware we don’t have any features in the codebase that I would classify as “alpha” (other than arguably async hooks, but that’s a special case from a different era) because they essentially can’t be landed right now. When we prepared the new ES modules implementation that shipped with Node 12, we had to build it in a fork of node over years with many PRs, and land it as one huge PR back onto the node repo. It wasn’t an experience that anyone wants to repeat, and I don’t want that to be the recommended workflow for anyone trying to land a new feature that needs room for iteration.

So yes, the concerns are to protect users from instability (or worse, from us being prevented from making changes to things that are ostensibly experimental, like async hooks) but also to lower the barrier to entry for new features that are too large to land directly as what I would call “beta” status.

[jasnell]

I don't think it's fair to say that we have no problems to solve.

Ok, let me be clearer: I don't think the way we currently handle experimental features is the problem that needs solving for. I think we need a different way of managing the release train in general, as a whole. I think your proposal here is looking at the wrong thing and would cause more friction than it would solve. The backport issue is just one example of what I'm talking about. When I say "I disagree that there is a problem to solve here" I mean specifically around experimental things only. The release process as a whole is something I've long thought we need to evolve (see #37495)

[GeoffreyBooth]

When I say “I disagree that there is a problem to solve here” I mean specifically around experimental things only. The release process as a whole is something I’ve long thought we need to evolve (see #37495)

Sure, and I’m not opposed to that; but I’m under the impression that that’s an even heavier lift that we’re even less likely to reach consensus on. If you take the “backport” part out of my proposal, all it really amounts to is:

• New experimental features land behind a flag and with a warning.
• When we think they’re mostly ready, we drop the flag.
• When we’re pretty sure they’re ready, we drop the warning.
• We note each of the above transitions in the docs.

We pretty much already do this for almost all experimental features, including noting in the docs when flags and warnings are dropped; we just don’t describe them as separate stages, so there’s no explanation of what users should be able to expect now that a flag or warning is dropped. I don’t think we get any value out of this ambiguity, and it hurts us in that we’re discouraged from making breaking changes to experimental features (making rough/needs iteration features harder to land) and many users are discouraged from using experimental features at all until they’re stable, depriving us of better real-world testing.

We could also reform the release process, but I see that as separate; and if reforming the release process is impossible, then it doesn’t much matter if that would be a better approach. Some reform is better than none.

[targos]

Features at this stage are not backported.

Anything that we don't backport makes the overall maintenance of LTS lines very difficult, because conflicts accumulate quickly.

[GeoffreyBooth]

But we have the don't land labels and I see them used often; can you point to an example where something that wasn't backported quickly caused problems?

I can imagine what you're saying would be an issue, though I feel like it should be less common for entirely new experimental features. But this is really the crux of the matter: if unfinished work gets released on LTS lines routinely, then I'm less apt to allow the unfinished work to land on main at all; and the more barriers we put on front of code landing, the fewer contributions we'll get.

Is it less of an issue during the part of the year when we have an unstable line, like currently with 19?

[aduh95]

Is it less of an issue during the part of the year when we have an unstable line, like currently with 19?

19.x is not unstable, the community works hard to have it as stable as possible. Node.js follows the semver rules on all release lines, with the exception of security releases (which sometimes introduce a breaking change in a semver-minor) and experimental features.

But we have the don't land labels and I see them used often; can you point to an example where something that wasn't backported quickly caused problems?

Look for all the backport-requested-* labelled PRs, you can clearly see a pattern where when a PR doesn't get backported, it blocks other PRs to be backported. Preparing semver-minor LTS releases are already a lot of work, I think we should not consider lightly adding more work on top of it.

I feel like it should be less common for entirely new experimental features

True, but new features that do not touch any part of the rest of the codebase are pretty rare I think, and maybe setting in stone a rule about backporting is not a good idea, the current status quo of "backport by default, anyone can add the dont-land or backing-for-lts label" works well enough.

[GeoffreyBooth]

Another option would be too allow it to be backported but make it harder to use, like requiring alpha level features to be enabled via build flag. That would solve the backport issue while still ensuring no wide adoption.

[mhdawson]

The current description for alpha already requires a runtime flag, adding a build time flag as well pretty much makes the runtime flag redundant. We might have an earlier stage which requires a build flag, but that is a very high bar to the point were I don't think we'll get end users using it. It would allow build/test in the Ci which could be useful. If we had an option that requires the build flag I think we'd need a separate one with the runtime flag.

[GeoffreyBooth]

The current description for alpha already requires a runtime flag, adding a build time flag as well pretty much makes the runtime flag redundant. We might have an earlier stage which requires a build flag, but that is a very high bar to the point were I don’t think we’ll get end users using it. It would allow build/test in the Ci which could be useful. If we had an option that requires the build flag I think we’d need a separate one with the runtime flag.

I think this might’ve been written before I updated the proposal. Currently it’s:

• Alpha: build flag, runtime flag, warning.
• Beta: runtime flag, warning.
• RC: warning.

Now arguably if we have a build flag we don’t also need a runtime flag, so I could see making the runtime flag only for beta. But the point of alpha is that we don’t want users using it: these are features so unstable that they’re not ready for testing yet. Not everything will need to go through an alpha phase, but it’s worth having as an option for when we want to land new features in pieces before making them available for testing. Having such a status should make it easier to do such iterative development.

Conversely, I feel like little to nothing should arrive directly in RC status. New experimental features should generally start in alpha or beta.

[mhdawson]

I'll also comment that if we removed Features at this stage are not backported. from the alpha stage, what you have seems to document what we typically do already and I agree with you that documenting it adds value.

[aduh95]

If it's behind a build flag, I don't think there's any issue in backporting it, it's not like a library is going to start requiring its users to use an exotic build of node, right?

[GeoffreyBooth]

Right, that’s what I mean. A build flag would definitively address the concern about an unfinished feature becoming too popular. It doesn’t really matter if a feature is released or not if users are building their own node binary; they could build that from main or a PR for some feature they want that hasn’t landed yet. So a build flag is just as good, I think, as not backporting.

I can update the proposal above unless there’s some other idea that’s even better?

[GeoffreyBooth]

Okay, I’ve updated the proposal.

[JakobJingleheimer]

I think this is a good route to pursue.

RE the additional flag, if we did that, i think it would be better to have a single flag that accepts values, like --enable-experiments=import-meta-resolve,loaders and/or --enable-experimentation=alpha,beta,rc

[joyeecheung]

I think an umbrella flag like --enable-alpha-experimental-features would just be repeating the mistake of --harmony (and --harmony was not even intended for users)

[GeoffreyBooth]

@mcollina in nodejs/TSC#1323 (comment):

I’m -1 to the approach in #45900. I agree some clarification is needed, but I think we have a good pace of shipping new features right now and adding more process is going to stall the project.

I’m curious what clarification you have in mind. More detail in the docs of the relative statuses of experimental features? More clarity on when a feature advances from one stage of experimental to another, or to stable?

Several people have noted the concern about this process slowing the pace of development. Which I feel like could only mean that there are teams that aren’t following this process, for example by adding experimental features without flags or warnings. I can’t think of anything I’ve worked on in modules or loaders that hasn’t followed the process in this proposal, so for me there’s no additional work; even the within-experimental transitions like dropping flags or dropping warnings are noted in the docs, so it’s not like there’s even an additional documentation burden. So this raises a few questions:

• For the teams or features not following this process, is that a good thing? Is the increased velocity worth the risk of annoying users if/when we ship a breaking change to an experimental feature that they didn’t realize was experimental? Maybe it is; I’m not trying to imply one way or the other.
• Does it matter that we don’t have a uniform process for advancing experimental features? I would argue that what’s lost in the heterogenous approach we have now is that some features probably aren’t getting all the real-world testing and user feedback that they might get if they were labeled “release candidate” as opposed to simply “experimental.” So again it’s a question of tradeoff: is the increased velocity worth this loss of user feedback? (And maybe it is.)
• Is it better to completely reject this proposal rather than adopt it with exceptions? Like maybe certain features or subsystems we deem low-risk and don’t require the protection discussed here, of flags and warnings; but more vital or depended-on ones get this process.

I think it helps to point to real-world examples. async_hooks.createHooks is the classic case I’m trying to avoid a repeat of, where we want to remove a feature but can’t because it’s achieved mass adoption; I think this proposal would prevent a similar occurrence, and I think any minor loss of velocity would be worth that protection. (After all, it isn’t that much work to add a flag or warning.) Conversely, I understand the point about --test that its use of child processes for running tests would mean lots of extra complexity with warnings (though I feel like surely there must be a technical way to ensure a warning is printed only once, even with child processes). But just because we might have to make an exception for exceptional cases doesn’t mean that a process like this shouldn’t be the default that should apply most of the time. I think the drag on velocity suggested here is very minor, and not present at all for most teams. I would be willing to accept exceptions when justified, but I question the claim that the process proposed here would impose a burden that would actually impede any velocity.

[mcollina]

I really have an hard time in collaborating in GitHub threads as they are too many interweaved threads that make it hard to focus for my brain. I would recommend opening a PR instead. (I'm not the only one with this problem).
If you need something more freeform, even a google doc offers a better UI.
I typically miss notifications for the different threads. The fact that you moved my comment here makes it harder for me to collaborate.

This process is way too strict and it would hamper innovation because people don't use things with runtime warnings, and they won't provide feedbacks. In turn, this would increase the time it takes for features to exist experimental status. This would reduce the willingness of volunteers to add new features, stalling the project once again.

Runtime warnings and CLI flags are only needed in a case-by-case basis.

What's your goal with proposal? From my point of view this proposal would guarantee we ship less features. I don't think it's in the best interest of the project.

I will be happy to work on a different proposal in a different medium.

[aduh95]

This process is way too strict and it would hamper innovation because people don't use things with runtime warnings

Or rather, it would need to be way more lax (like making it possible to go from one stage to another in no particular order, and allow to skip steps entirely when it's convenient) to the point where it wouldn't be that useful to have it.

[joyeecheung]

I like the idea of having more specific stages of the experimental status, but I am not a fan of mandating warnings or flags as the ways to communicate this:

1. The emission of warnings and the configuration of flags might already bring complications to the implementations. For example the current warning machinery is mostly implemented in JS and might require event loop iteration when redirection is enabled. That's already a lot of complexity.
2. IMO flags are not necessarily effective to discourage adoption of volatile features (--harmony was a good example). In the extreme case (like --harmony), you end up with an annoying flag that you can't easily get rid of once everyone starts using it. From my experience it's actually not that rare to see people fall into the traps of weird NODE_OPTION values configured in their server scripts.
3. I don't think warnings are effective either as they can be disabled with various flags/environment variables, frequent warnings coming out of dependencies would encourage users to just use these flags/environment variables which is overall worse for everyone. I don't quite remember the specific reason but I remember there was a time when some people, including myself, set NODE_NO_WARNINGS=1 by default to silence the annoying warnings coming out of npm. As a user I don't really care why warnings come out of my dependencies if I have no other alternatives to get things done - I'd just silence the warnings for good. It's already the case for genuine warnings, let alone warnings about "your dependency is using a shiny new experimental feature", which doesn't actually seem to be worrying for a user.

I think flags and warnings should be added on a case-by-case basis, and not tied to the experimentation. Making them part of the experimentation process can create problems of their own, and they don't make that big of a difference in solving the experimentation dilemma for the reasons above.

[joyeecheung]

About the stage names: can we use more readable names than Alpha and Beta? e.g. Experimental: In development and Experimental: Minimal viable product? There can be multiple interpretations to Alpha and Beta in different software lifecycle models, it's better to be clearer.

[GeoffreyBooth]

Sure, what would you propose? We could also follow the TC39 example and have something like Stage 1, 2, 3, 4.

Most features probably won’t have an alpha stage, so it would feel weird if beta was “minimal viable product.” (Edit: I realize that this is exactly what I wrote above. 😄 Maybe we need another stage in between alpha and beta, to denote stuff that’s okay to start testing but not yet MVP?)

[GeoffreyBooth]

In https://github.com/nodejs/node/blob/main/doc/api/documentation.md we have Stability: 0, 1, 2, 3 for our current levels (deprecated, experimental, stable, legacy). Perhaps we make experimental 1.0, 1.1, 1.2 for what I’m currently calling alpha, beta, RC? And they don’t really need names; we can just describe each experimental sub-level. Something like:

Stability: 1 - Experimental. The feature is not subject to semantic versioning rules. Non-backward compatible changes or removal may occur in any future release. Use of the feature is not recommended in production environments. Experimental features are subdivided into stages:

• 1.0. Experimental features at this stage are unfinished and subject to substantial change. They are not yet ready for user testing.
• 1.1. Experimental features at this stage are complete enough to be considered a minimum viable product for potential stability, with only minor changes expected.
• 1.2. Experimental features at this stage are hopefully ready to become stable. No further breaking changes are anticipated and we welcome user testing and feedback so that we can know that this feature is ready to be marked as stable.

Though having names would make them easier to refer to.

[GeoffreyBooth]

Per TSC meeting today, the next steps on this:

1. I’ll open a PR on the node repo to update https://github.com/nodejs/node/blob/main/doc/api/documentation.md to list the new subdivisions within “experimental” proposed here. In this first PR I’ll leave out any discussion of flags or warnings.
2. Once ☝️ lands, I’ll open a follow-up PR adding requirements for each level, such as flags or warnings.
3. Concurrent with the second PR, we need to review the docs and update all the currently experimental features to distinguish which experimental level they are. I don’t anticipate us adding any flags or warnings to experimental features that currently lack them, but we can consider doing so if we think they might be warranted.

Once the above is complete, this would become the process for new experimental features going forward, that would be considered the “default” process unless we decide to grant exceptions on a case-by-case basis.

[GeoffreyBooth]

I did a survey of modules features added since 12.0.0 (when we merged in the new ESM implementation) and what their trajectories looked like. This was compiled from our docs and a few searches of PRs, so could be incomplete; we also haven’t been good about documenting warnings (whether added or dropped) so I don’t have that data.

• ES modules (experimental modules)

  • Added in 12.0.0 behind --experimental-modules flag.
  • Unflagged in 12.17.0 / 13.2.0.

• CommonJS-style resolution for import

  • Added in 12.0.0 behind --experimental-specifier-resolution flag.
  • Removed in 19.0.0.

• Package exports (“exports” in package.json)

  • Added in 12.7.0 behind --experimental-exports flag.
  • Unflagged in 12.15.0 / 13.0.0.

• Conditional exports

  • Added in 12.15.0 / 13.2.0 behind --experimental-conditional-exports flag.
  • Unflagged in 12.16.0 / 13.7.0.

• Package self-reference (use “name” in package.json)

  • Added in 12.16.0 / 13.1.0 behind --experimental-resolve-self flag.
  • Unflagged in 12.16.0 / 13.6.0.

• JSON modules / import assertions

  • Added in 12.0.0 with --experimental-json-modules flag.
  • Assertions added in 16.14.0 / 17.1.0 behind existing --experimental-json-modules flag.
  • Unflagged in 16.15.0 / 17.5.0.

• Loaders

  • Added in 12.11.1 with --experimental-loader flag. (A version for the pre-12.0.0 ES modules implementation existed behind a --loader flag which was preserved as an undocumented alias.)

• Wasm modules

  • Added in 12.3.0 with --experimental-wasm-modules flag.

• import.meta.resolve

  • Added in 12.16.2 / 13.9.0 with --experimental-import-meta-resolve flag.

• Top-level await

  • Added in 14.3.0 with --experimental-top-level-await flag.
  • Unflagged in 14.8.0.

• HTTPS and HTTP imports

  • Added in 16.15.0 / 17.6.0 with --experimental-network-imports flag.

• data: imports

  • Added in 12.10.0 behind --experimental-modules flag.
  • Unflagged in 12.17.0 / 13.2.0 as part of overall ESM implementation unflagging.

• node: imports

  • Added in 12.20.0 / 14.13.1 for import
  • Added in 14.18.0 / 16.0.0 for require

So of all the new features that I could find, it seems like only the node: prefix landed without a flag. It’s a fairly minor improvement that was possibly considered more of an enhancement to resolution generally rather than a new feature in its own right.

I think this starts to suggest a few guidelines: flags are important for core/vital APIs, like modules; and improvements can be small enough to not need to go through an experimental process at all.

I also don’t think that all these flags contributed to any reduced velocity for modules development. If anything I think they accelerated it, as we would have been too nervous to land many of these features without flags.