Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 0144] Versioned Flake References #144

Closed
wants to merge 5 commits into from
Closed

[RFC 0144] Versioned Flake References #144

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5319e93
[RFC 0144] Versioned flake references
figsoda Mar 19, 2023
75b3946
add flags to `nix flake update as an alternative to the upgrade command
figsoda Mar 20, 2023
44262e3
`git+` URLs are possible
figsoda Mar 21, 2023
c5f2ddd
add not having the upgrade option as an alternative
figsoda Mar 21, 2023
18c2f0c
fixup alternatives
figsoda Mar 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 120 additions & 0 deletions rfcs/0144-versioned-flake-references.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
---
feature: versioned-flake-references
start-date: 2023-03-19
author: figsoda
co-authors: None
shepherd-team: None
shepherd-leader: None
related-issues: None
---

# Summary
[summary]: #summary

Flake references can have a special placeholder that will specify the version
requirement of the flake, which `nix flake update` can use to update the pin to
the latest version compatible with the version specification. A new Nix
command, `nix flake upgrade` will upgrade the version requirement in
`flake.nix` to the latest possible version without regard to compatibility.

# Motivation
[motivation]: #motivation

This will allow Nix libraries to be versioned without requiring their users
to manually update them. Some package managers (e.g. cargo) for more
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here you cite cargo, but the specification they employ to do those machine-driven updates is very dumb - by design. Cargo can update patch version, but minor version has restrictions and major is almost impossible.

It looks so full of restrictions that it is easier to edit by hand.

Copy link
Member Author

@figsoda figsoda Mar 25, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you elaborate on what "full of restrictions" means? I don't think attacking a reference I made is a good explanation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example tool you gave, Cargo, is purposefully very limited. This design decision conveys the idea that major and minor versions should not be upgraded by a machine, but by a human being.

But your specification is too complex (maybe a consequence of Nix flakes being too complex), and it feels problematic to me.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This design decision conveys the idea that major and minor versions should not be upgraded by a machine, but by a human being.

The whole point of defining a set of rules for the version specification is to make them upgradable by a machine, hence the strict rules on what version bumps can introduce breaking changes. This is so libraries can follow a predictable pattern, instead of their own preference, so updating can be automated.

Your response still doesn't explain what is limited, and I don't see why Cargo being supposedly limited has anything to do with my proposal being problematic. The purpose of this proposal is to give library authors the freedom to version their library, instead of the current situation of users following the main branch and library authors not having a good communicable way to introduce breaking changes.

Copy link
Member

@AndersonTorres AndersonTorres Mar 25, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your response still doesn't explain what is limited

I said, and I will say again: Cargo (or more precisely cargo.toml) is limited. Being limited, it is easy to understand and implement.

Your proposal is too wide. It introduces more syntatical and semantical rules - implying more burden for the compiler writer (of a language with no formal specification yet, by the way).

Further,

give library authors the freedom to version their library

Goes against

strict rules on what version bumps can introduce breaking changes.

The design decision should find the equilibrium between these two points.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cargo (or more precisely cargo.toml) is limited.

If you are talking about the expressiveness of Cargo.toml vs flake.nix, I agree. However, I don't think the difference applies to this specific feature.

It introduces more syntatical and semantical rules.

Cargo defines much more rules in this aspect and also more strictly enforces it, as it will fail if the version is absent or not a valid semver.

Implying more burden for the compiler writer

I agree.

freedom goes against strict rules

It doesn't. They are talking about completely different things. This feature will always be opt-in. Library authors can still not version their libraries, or define their own versioning scheme - they just wouldn't be benefited by the change. The only way tooling like this can happen is if people following a predictable set of rules when versioning their libraries, which this proposal will allow by defining strict rules.

conventional programming languages have this functionality built-in, allowing
library authors to introduce breaking changes in a communicatable way that will
not break their downstream dependents's code.

# Detailed design
[design]: #detailed-design

## Syntax
Version placeholders will be in `{}`, with a list of version requirements
separated by `,` (commas), e.g. `{^1.0,<3}` will expand to a version that
specifies the version requirements `^1.0` and `<3`

Version requirements consists of a comparator and version. The version does not
have to be a valid version defiined by the versioning scheme, `1` and `1.0` are
also valid versions for version requirements. The comparator has to be one of
the following options:
- `^` compatible (as defined by the versioning scheme)
- `<` less than
- `<=` less than or equal to
- `>` greater than
- `>=` greater than or equal to
- `=` equal to

A version placeholder with no version requirements will match all valid
versions defined by the versioning scheme.

## Versioning scheme
The versions will follow [semantic versioning] (semver). The tags have to
follow semver, `1.0` and `1.0.0.0` will not match the placeholder `{=1.0}` as
it is not a valid version defined by semver. Build identifiers will not be
taken into consideration when calculating version requirements, and the latest
tag will be selected if multiple tags have the same precedence defined by
semver.

## Upgrading
A new Nix command, `nix flake upgrade`, will edit `flake.nix` and upgrade all
the version placeholders in the inputs. A new flag, `--upgrade-input` will
Comment on lines +60 to +61
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upgrading flake.nix in-place is not really an option since we don't have the ability to do that. But I also don't see why it's necessary. nix flake update can simply update to the latest versions that are consistent with the version placeholders. At most, we want a flag to nix flake update to allow/disallow major upgrades that wouldn't be semantically compatible.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If a flag were to temporarily ignore semantic compatibility, wouldn't that introduce some weird behaviour? Wouldn't the update behaviour on subsequent uses (without the flag) be to downgrade to a compatible version, thus defeating the purpose of initially using the flag? The idea here is to permanently alter the semver to accept the latest version as a convenience (so the user doesn't need to track down the latest tags of every input). I can't see any way of doing that (without potentially confusing behaviour) but inplace changes to flake.nix.

Alternatively, we could have a command/flag which would report the latest tags of the inputs (similar to how update reports lock file changes). This way there's no need to edit flake.nix inplace and the user gets the necessary information at once instead of playing tag whack-a-mole. It's less powerful but still works to make version tracking much less annoying which, I think, is the ulterior motive of this RFC.

upgrade only the specified flake input. Only version placeholders that contain
exactly one version requirement which the comparator is either `^` or `=` will
be upgraded. All other version placeholders will be left unchanged.

# Examples and Interactions
[examples-and-interactions]: #examples-and-interactions

Here are the git tags of a hypothetical Nix flake "github:foo/bar":
- v0.1.0
- v0.1.1
- v0.2.0
- v1.0.0
- v1.0.1
- v1.1.0
- v2.0.0
- unrelated

Flake reference | The tag it points to | The `upgrade`ed flake reference
-|-|-
`github:foo/bar/v{^1}` | `v1.1.0` | `github:foo/bar/v{^2}`
`github:foo/bar/v{=1.0}` | `v1.0.1` | `github:foo/bar/v{=2.0}`
`github:foo/bar/v{^1.0}` | `v1.1.0` | `github:foo/bar/v{^2.0}`
`github:foo/bar/v{=1.0.0}` | `v1.0.0` | `github:foo/bar/v{=2.0.0}`
`github:foo/bar/v{^1.0.0}` | `v1.1.0` | `github:foo/bar/v{^2.0.0}`
`github:foo/bar/v{^0.1.0}` | `v0.1.1` | `github:foo/bar/v{^2.0.0}`
`github:foo/bar/v{<2}` | `v1.1.0` | `github:foo/bar/v{<2}` (no change)
`github:foo/bar/v{>1.0}` | `v2.0.0` | `github:foo/bar/v{>1.0}` (no change)
`github:foo/bar/v{^1,<1.1}` | `v1.0.1` | `github:foo/bar/v{^1,<1.1}` (no change)
`github:foo/bar/v{}` | `v2.0.0` | `github:foo/bar/v{}` (no change)

# Drawbacks
[drawbacks]: #drawbacks

- This is impossible with some types of flake references, such as `path`s and
tarballs, which will make the flake interface less consistent.
- This adds extra complexity to `nix flake update`.
- The difference between `update` and `upgrade` might cause confusion.

# Alternatives
[alternatives]: #alternatives

- Using a different syntax for the placeholders
- Using a version scheme other than semantic versioning
- Adding flags to `nix flake update` instead of creating a new `upgrade` command
- Update to incompatible versions with regard to `flake.lock` instead of mutating
`flake.nix` directly
- Not adding the feature, manually update the flake references

# Unresolved questions
[unresolved]: #unresolved-questions
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not discussed here is the algorithm for selecting versions. In conjunction with follows/overrides, finding a feasible solution could become expensive, e.g. in

inputs.foo.url = "github:bla/foo/{>1.2,<1.5}";
inputs.bar.url = "github:bla/bar/{^1}";
inputs.bar.inputs.foo.follows = "foo";

Nix would need to find a version of bar whose foo input has constraints compatible with the ones given here.

I wouldn't want to have a SAT solver in Nix...

Copy link
Member Author

@figsoda figsoda Mar 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My thought was that bar's requirement for foo would be ignored/overridden, so only {>1.2,<1.5} would be used, and version requirements would always be combined with and

maybe I should clarify that in detailed design

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't want to have a SAT solver in Nix...

We should consider having one outside of Nix, making lockfile generation pluggable.
Perhaps the version constraints should be in a separate attribute that's open for extension?

In fact, we could consider the lock reading and fetching to be part of Nix, and the Flake CLI to be one implementation of the broader Nix package management idea. Dream2nix is also in the business of lock file generation. Perhaps the role of Nix should be limited to the integration of lock file generators, and not actually implementing them. We don't even need a dumb lock file generator like we currently have because that one could just as well live in a flake that's already been locked.

Nix should focus on its core competencies that it does very well.

  • building immutable store paths
  • bootstrapping other software

Anything that's built on top of that functionality is going to feel as good as native, plus have the benefit of pinning, patching, and decentralized development.

Flakes is oddly centralized for a package manager built on Nix...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Late to the party, but let me just write this down just in case. One way to avoid the need for SAT-solving is to allow only semver open (^1.2.3) version requirements. That way, a greedy algorithm which just always picks the latest semver-compatible version works.

Moreover, in practice, in Cargo:

  • 99% of constraints are in fact just ^1.2.3 (it was genius design decision to make version="1.2.3" be a syntactic sugar for version="^1.2.3")
  • people often try to use non-semver open constraints, most of the time that's a wrong thing to do, and leads to problem down the line (eg, their build works, but builds of folks depending on them gridlocks due to unsatisfiable constraints)
  • at some point we even considered deprecating non-^ constraints, but didn't do that, because there are occasionally good reasons for them

So, perhaps a safe move wold be to allow only ^1.2.3 constraints, (and using only implicit 1.2.3 syntax). If that turns out to be expressive enough, that would be a massive simplification. If not, fancier constrains can be added later.


- How will this work in query parameters?
- Should only tags be used, or should branches also be considered?
- Should whitespace be allowed in the version placeholders?

# Future work
[future]: #future-work

[semantic versioning]: https://semver.org/