Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support stable-X as a version descriptor #2291

Closed
thejpster opened this issue Apr 21, 2020 · 42 comments
Closed

Support stable-X as a version descriptor #2291

thejpster opened this issue Apr 21, 2020 · 42 comments

Comments

@thejpster
Copy link
Contributor

Describe the problem you are trying to solve

In Rust-Embedded land, we are struggling with the management of our Minimum Supported Rust Version. That is, the promise we make to our users that compilers all the way back to our MSRV will compile this code, and we won't use any features stabilised after that compiler was released.

Currently we set an MSRV like a stake in the sand, and then time moves on and the stake gets further and further away. One of our crates has an MSRV of 1.30 - thats 72 weeks old at the time of writing. What I'd like to do is bring that MSRV along with us - trailing it behind us on a piece of string.

Describe the solution you'd like

Basically in my CI files I want to "rustup install stable-2", and if stable is 1.40, it would install 1.38. If stable was 1.51 it would install 1.49. Then I can set my Travis CI matrix to build on:

  • stable
  • stable-1
  • stable-2

And I never have to touch the CI file again.

Notes

At the moment, to achieve a rolling MSRV we would have to touch the CI file every six weeks when there's a new stable release. That or rustup install stable, grok the current version with some grep/sed foo, subtract two minor versions (not easy in Bash) and then install that version.

@kinnison
Copy link
Contributor

It's an interesting idea, certainly. However it wouldn't fit directly with rustup's current model. That's not to say it couldn't be made to work, just that it'd be awkward.

If you're insistent on doing it via a Travis matrix then it'd have to be rustup or else travis would have to have some special behaviour to achieve that. Currently rustup is able to just go and ask static.rust-lang.org for channel data. I wonder if, perhaps, it might be sensible to have a rolling set of stable-{1,2,3,4,5} channel symlinks on there?

@kinnison
Copy link
Contributor

At that point, we could add support for the requisite channel name shape to rustup and wouldn't need special handling otherwise.

@thejpster
Copy link
Contributor Author

I guess the Debian equivalent would be old-stable and old-old-stable, if the numeric component causes an issue.

@kinnison
Copy link
Contributor

I think oldstable and oldoldstable are more plausible just because - is used to separate the elements of toolchain names. In theory ~ could be used too but that introduces a whole host of usability issues potentially. @pietroalbini If I were to propose adding some channel symlinks (or however we deal with stable/nightly vs. the numbered/dated equivalents currently) which approach do you think would fly?

@rbtcollins
Copy link
Contributor

Do we know from channel data the stable version? We could do the arithmetic easily enough in rustup to synthesise new pseudo channels, no?

@kinnison
Copy link
Contributor

Such synthesis would change the download-channel-install-channel model which IMO would be moderately awkward; though as I said before yes it'd be possible. However the ideal would be if the pseudochannels were handled on static.rlo, otherwise we have to start faffing with working out which patch level is the newest for a version (e.g. if stable is 1.35.0 and the user wants stable~1 do we install 1.34.0 or the more useful 1.34.2 ?) So many roundtrips would make rustup update expensive and slow. And yes, I know we do that for nightly backtracking but that's only one dimension (date) rather than more (channel / minor / patch)

@pietroalbini
Copy link
Member

I think a good approach would be to create new manifests for 1.MINOR.x (like 1.43.x, 1.34.x...) that always point to the latest patch release of that minor version. This would allow to:

  • Fetch the stable~N in just two requests (as we can subtract from the minor of the current stable, and fetch the .x channel of the resulting version).
  • Pin a minor version in CI while still updating to the latest patch release (for example, if your MSRV is 1.31 you'd use 1.31.x as the MSRV toolchain).

@tesuji
Copy link
Contributor

tesuji commented Apr 25, 2020

Yeah, I like using stable~<number> too. Feels like using git reset HEAD~<number>.

@tesuji
Copy link
Contributor

tesuji commented Apr 25, 2020

if stable is 1.35.0 and the user wants stable~1

The good default to me is using latest patch release. As patch releases often
backport soundness and compatible/build fixes.

Edit: minor -> patch

@thejpster
Copy link
Contributor Author

if stable is 1.35.0 and the user wants stable~1

The good default to me is using latest minor release. As minor releases often
backport soundness and compatible/build fixes.

I'd call that the patch release, as the 35 in 1.35.0 is the minor, and the 1 is the major. But yes, I can't see why anyone would want to not pick up the highest patch release for a given major/minor pair.

@kinnison
Copy link
Contributor

So I like both @pietroalbini 's idea of MAJOR.MINOR.x being the latest PATCH for MAJOR.MINOR though I'd note that might confuse people who do numeric ordering and assume the dots separate numbers. and I like the idea of stable~1 being "one back from current stable". The .x links have the advantage that they're only updated on a release of that MAJOR.MINOR series, but the stable~n would have to be updated every stable release. If we do the latter, I don't think we should do it for beta or nightly though.

@pietroalbini
Copy link
Member

I'd prefer not to have stable~N as actual manifests on static.rlo (keeping them up to date is going to be a bit of a pain). Could we go with MAJOR.MINOR.x on static.rlo, and implement stable~N on the rustup client?

@kinnison
Copy link
Contributor

I guess we could have a go. I don't like the double-downloading of manifests but I suppose if the user is opting in then they're prepared to pay the cost. So we define stable~N to mean major(STABLE).(minor(STABLE)-N).x and then fetch that? Will we backfill static.rlo with the MAJOR.MINOR.x for all the releases? Presumably that'd be fairly cheap.

@pietroalbini
Copy link
Member

Yeah pretty sure we can backfill static.rlo.

@kinnison
Copy link
Contributor

kinnison commented Dec 3, 2020

We now have major.minor channel files present on the dist server, so we're a step closer to being able to do this.

If someone wanted to have a go at implementing the channel parsing and handling the double-manifest-download then we can see how it works.

@rami3l
Copy link
Member

rami3l commented Aug 21, 2023

@rbtcollins This seems a bit complex, and I don't have a concrete timeline, but I still want to give this one a try. Could you please mark this issue as claimed?

@rami3l
Copy link
Member

rami3l commented Aug 23, 2023

Before actually trying to implement this change, I'll try to share my thoughts on this issue below:

De-sugaring

It seems that the main use cases for this stable-DELTA syntax is in CIs, so at which level should we de-sugar it?

IIRC, in today's model, stable, 1.71 and 1.71.0 are 3 separate toolchains (assuming the current stable is on 1.71.0):

  1. stable will become 1.72.0 weeks later.
  2. 1.71 will stay on 1.71 but still upgradable (e.g. from 1.71.0 to 1.71.1).
  3. 1.71.0 will get pinned to 1.71.0 forever.

To me, it will be too weird to actually make stable-DELTA an upgradable toolchain just like stable, so I'm considering to define the semantics of stable-2 to be exactly the same as 1.69.
That is, I don't expect anyone to actually perform a +0.1 upgrade on those.

Does this sound acceptable? (cc @kinnison @joshtriplett)

Abandoned alternative plan

As currently envisioned, stable-2 will be a separate toolchain installation that:
- Are distinguished from major.minor toolchains.
- Will receive +0.1 updates just like stable, but always maintain a -0.2 difference from stable's actual version.
- Can get out of date (if installed before a stable version bump), and thus sometimes require rustup update.

And since

> I'd prefer not to have stable~N as actual manifests on static.rlo (keeping them up to date is going to be a bit of a pain).

This means that we have to locally maintain "virtual channels" (a.k.a. "pseudo channels") based on actual(stable) - 0.2's manifest.

Full syntax

The current syntax for a toolchain spec is:

CHANNEL[-YYYY-MM-DD][-TARGET]

... where:

  • CHANNEL is one of stable, beta, nightly, X.YYY or X.YYY.ZZ.
  • TARGET can be partial, like i386, pc-windows, msvc, i386-pc-windows, i386-msvc or pc-windows-msvc instead of the complete form i386-pc-windows-msvc.

... and I suggest to add a possibility for the 2nd part: an unsigned integer of at most 3 digits, so that it becomes:

CHANNEL[-YYYY-MM-DD|-WWW][-TARGET]

This would then accept the following (still assuming the current stable is on 1.71.0):

  • stable-2 => 1.69
  • stable-31-pc-windows => 1.40-pc-windows
  • stable-999 1
  • beta-2 1
  • nightly-2 1
  • 1.23-4 1
  • 1.23.0-4 1

Footnotes

  1. Allowed by the syntax, but might cause semantic errors. 2 3 4 5

@thejpster
Copy link
Contributor Author

How I would direct rustup/cargo to do a build with the resolved toolchain? Will cargo +stable-2 build work?

@rami3l
Copy link
Member

rami3l commented Aug 24, 2023

How I would direct rustup/cargo to do a build with the resolved toolchain? Will cargo +stable-2 build work?

@thejpster It is still a bit too early to talk about this. As I see it, there are two ways of defining the semantics of stable-2:

  1. Making it a shorthand of 1.69 (if stable is on 1.71), which won't receive any +0.1 updates.
    If this happens, then with stable-2 you have just installed 1.69 and nothing more, since stable-2 is not actually a channel in this case. The actual de-sugaring happens every time the syntax is used, so every time you are using something like cargo +stable-2 build, you are sure that you are actually building with the current stable-2 (but that also means it will instead require 1.70 if the stable version has bumped).

  2. Making it a "virtual channel" installed as separate toolchain.
    If this happens, then stable-2 behaves just like stable (but 2 versions behind the current stable): they are distinguished from major.minor toolchains, will receive +0.1 updates, can get out of date (and if that happens you need to update them with rustup update), and finally, cargo +stable-2 build requires a stable-2 toolchain installation to work.


I'm still hesitant about this, initially I wanted to go with 1. (as posted above), but after some discussions with other devs it seems to me that 2. is a better option. After all, these two semantics should be equivalent on CIs, but what if someone wants to do this on their own dev machine?

But on the other hand, I don't currently know if this virtual channel thing does exist. If it doesn't, I might have to invent the wheel myself :o

There are other similar things to decide, for example:

  • How will toolchain files work in this case?
  • How will stable-2 play with Rust 2.0? Maybe stable-0.2?
  • Will people immediately get that - is minus instead of a connector? stable~2 and stable^2 also seem dangerous... Some suggested something like 2-before-stable or stable-minus-2 for clarity 1.
  • How to ensure the coherence with Allow cargo +rust-version build|test|... #2925? (cc @weihanglo)

Footnotes

  1. I was suddenly reminded of my time spent on SICP... What about -2+stable? 😂

@djc
Copy link
Contributor

djc commented Aug 24, 2023

I think I agree that (2) would provide a better UX, although it would be more work to implement.

(I also agree the minus vs connector thing needs to be thought about.)

@thejpster
Copy link
Contributor Author

Thank you!

IIRC Debian use oldstable and oldoldstable.

@rami3l
Copy link
Member

rami3l commented Aug 25, 2023

Thank you!

IIRC Debian use oldstable and oldoldstable.

@thejpster Thanks for the help!

This is indeed very clear without using any weird characters! I'll consider using this name or at least some variant of it :)

@djc
Copy link
Contributor

djc commented Aug 25, 2023

I think there will also be use cases for old-old-old-old or even old-old-old-old-old-old-old-old, so I'd probably try to get a number in there rather than relying on repetition like this.

@rbtcollins
Copy link
Contributor

There are two design points referenced in the recent message.

  1. what name should rustup accept to specify one of these stable + offset things,

  2. are they real channels or virtual.

  3. I think we should accept stable-${decimal} : debians versioning schemes are very rigorous but also influenced and limited by compatibility with early versions of dpkg and apt, and we are not constrained by them.

  4. per Support stable-X as a version descriptor #2291 (comment) there are now manifest files for major.minor releases which get updated. A bit of history: we used to have channels like stable and exact versions but not major.minor. This meant you couldn't install rust 1.72 via metadata - rustup had to decide you meant 1.72.0 and get you that instead.
    This means that we already have actual real channels for each concrete floating version that might be referenced in an MSRV if we can name them.

The proposal linked above is that when rustup encounters a name with a version offset (@kinnison used ~, I think we should use - as ~ has different semantics in Debian and it would be confusing to use that), rustup will examine the stable version number, subtract the version offset from the minor part, discard the point part, and then use that channel as the toolchain.

The question about whether we end up with e.g. a toolchain on disk called stable-2 is a good one. I propose we don't do that and take the option (1) described in #2291 (comment)

What do you think of this possible bit of documentation for the feature.

# MSRV 'last-N releases support'

Rustup can support MSRV for projects that want to build up to N releases back from the current stable release by building using the syntax `stable-N`. For instance `rustup toolchain install stable-3`. This will:
- install the stable toolchain if not installed (this provides the anchor point by which `stable-3` is calculated
- install a toolchain `MAJOR.MINOR` with the version number determined by subtracting 3 from the stable version numbers minor part and dropping the point part

When building for MSRV support, use a [toolchain override](...url skipped) of `stable-N` to build against the current major.minor toolchain as described above. If your stable toolchain is out of date, this may build with an older major.minor than you might expect. Point releases on the major.minor toolchain are only applied when a `rustup toolchain update` is performed - the same as for any other toolchain.

Reasons why I propose this:

  • reduces the number of distinct toolchains people need: if they have 1.69, and 1.69 is needed for a stable-N scenario, their 1.69 is reused. We have open bugs about preventing additional downloads of toolchains, so it makes sense to me to take that into consideration in new feature development.
  • avoids adding new floating-toolchain-versions: 'stable-2' on disk doesn't refer to any specific version, and we also have bugs around the mismatch between disk and toolchain versions (e.g. that nightly of a particular date often has a toolchain with a build date that is off by one).

@rami3l
Copy link
Member

rami3l commented Aug 26, 2023

  1. I think we should accept stable-${decimal} : debians versioning schemes are very rigorous but also influenced and limited by compatibility with early versions of dpkg and apt, and we are not constrained by them.

I'm still wondering if there will be a 2.0 moment of Rust, and how we might design this feature to be future-proof in that respect.

  • reduces the number of distinct toolchains people need: if they have 1.69, and 1.69 is needed for a stable-N scenario, their 1.69 is reused. We have open bugs about preventing additional downloads of toolchains, so it makes sense to me to take that into consideration in new feature development.

One downside of this approach, I imagine, would be too many old toolchains on one's disk if they choose to install locally. 1.69, and 1.70 after that... There doesn't seem to be a "garbage collection" mechanism.

@weihanglo
Copy link
Member

I'm still wondering if there will be a 2.0 moment of Rust...

When there is Rust 2.0, I believe rustup is very likely to ship breaking changes along with that. Thus not an issue from my pov.

@djc
Copy link
Contributor

djc commented Aug 28, 2023

I wonder if we can support this without mandating that the stable toolchain is installed? Or do we think that ~everyone has stable installed anyway so there's not much point?

@rami3l
Copy link
Member

rami3l commented Aug 28, 2023

@djc I think @rbtcollins wants stable to be installed as an anchor point so that we can effectively know what stable-2 actually means (so that stable-2 is always 2 versions behind the installed stable rather than the real stable).

This IMO is also different from my original proposal, in which I wished to resolve stable on the fly and calculate the actual version based on that, but I think it is somewhat more consistent.

Ideally we can do both based on whether stable is installed, but I'm afraid that will cause more confusion than necessary...

@rbtcollins
Copy link
Contributor

I realise I didn't cover one of the design constraints. Because rustup proxies are in the critical path for build time, we need a design that doesn't pay (much) overhead, and that works when the internet is down.

The (much) overhead - its probably ok to run rustc, or parse the stable manifest or both, when a proxy is invoked with no RUSTUP_TOOLCHAIN set. Once set, it needs to be set to a concrete value that won't trigger resolution again.

I'm not sure what I'm proposing is necessarily the best, but it seems ok at current thinking to me.

  • use invokes e.g. RUSTUP_TOOLCHAIN=stable-2 cargo build, the rustup proxy does the translation to get e.g. 1.69, resets RUSTUP_TOOLCHAIN for the child process, and away we go.
  • once installed, offline development will work fine, unless the user updates stable to a new version but doesn't attempt a build with stable-2, when the internet would be needed again.
  • if stable and the matching toolchain version are already installed, cargo +stable-2 build won't download anything.

Using stable-2 as a virtual-channel would have similar properties with the following differences:

  • stable-N requires separate disk space
  • stable-N won't pick up new versions except when explicitly asked

@epage
Copy link

epage commented Aug 30, 2023

Something I've adopted in my projects and am also starting to use in cargo is to have RenovateBot automatically update my MSRV.

e.g. see rust-lang/cargo#12381

Would something like that existing lower the priority of implementing a feature like this directly in rustup?

@djc
Copy link
Contributor

djc commented Aug 30, 2023

An alternative possible design is that rustup should expose +msrv which could read the rust-version from the project's Cargo.toml and run that. Maybe simpler and more directly solves the use case? I suppose this would be roughly analogous to using the rust-toolchain file.

@rami3l
Copy link
Member

rami3l commented Aug 30, 2023

An alternative possible design is that rustup should expose +msrv which could read the rust-version from the project's Cargo.toml and run that. Maybe simpler and more directly solves the use case? I suppose this would be roughly analogous to using the rust-toolchain file.

@djc Yes, that looks interesting, which could even free us from caring about stable.

@djc
Copy link
Contributor

djc commented Aug 30, 2023

The tricky part is how this works in workspaces, which don't natively have a rust-version. They could have a workspace.package.rust-version, which could alleviate the problem, but the interactions with --manifest-path might be surprising? I guess cargo +rust-version --manifest-path=foo/Cargo.toml could not pick up the rust-version from foo/Cargo.toml -- but then we already have the same problem today with rust-toolchain files?

@thejpster
Copy link
Contributor Author

Forgive me if I quote myself:

Currently we set an MSRV like a stake in the sand, and then time moves on and the stake gets further and further away. One of our crates has an MSRV of 1.30 - thats 72 weeks old at the time of writing. What I'd like to do is bring that MSRV along with us - trailing it behind us on a piece of string.

@djc
Copy link
Contributor

djc commented Aug 30, 2023

@thejpster I'm not sure exactly what you are trying to convey. Is it that you want the +stable-X to be trailing along without you having to update the rust-version? If so, don't you want your Cargo manifest metadata to be correct as the MSRV increases over time?

@thejpster
Copy link
Contributor Author

My OP I think dates before rust-version existed so yes, we wanted MSRV to move without changing anything in the repo.

@djc
Copy link
Contributor

djc commented Aug 31, 2023

So how/when are you going to the rust-version in your crate's Cargo.toml? Or are you going to publish your crate without rust-version even though there is some limited MSRV support?

@thejpster
Copy link
Contributor Author

I don't know that anyone has thought about it. I would probably publish without a rust-version and say in the README "If you're more than three versions behind stable, this might not work"

@epage
Copy link

epage commented Sep 4, 2023

If the answer is "don't set a rust-version", then this feels too specialized of a feature.

The ideal we should be working towards is setting rust-version

  • crates.io will report this information
  • Users get errors to help them with the problem rather than leaving them confused
  • cargo-add has unstable support for choosing an MSRV-compatible version requirement, on the path to stabilization
  • cargo upgrade (and the part that will get merged into cargo) will prevent upgrades across MSRV
  • There is unstable support for MSRV-aware resolver (long road to stabilization

@thejpster
Copy link
Contributor Author

Then I'm happy to declare this 3 year old ticket as OBE and close it.

@djc
Copy link
Contributor

djc commented Sep 5, 2023

IIRC @epage has used Renovate to send automated PRs to advance the rust-version? Might be a good option.

@rbtcollins
Copy link
Contributor

Yah - rustup toolchain files are not a good fit for MSRV management; that belongs with cargo.

@rbtcollins rbtcollins closed this as not planned Won't fix, can't repro, duplicate, stale Sep 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants