Implement top-level overrides #2385
Conversation
|
r? @huonw (rust_highfive has picked a reviewer for you, use r? to override) |
|
r? @wycats |
|
☔ The latest upstream changes (presumably #2328) made this pull request unmergeable. Please resolve the merge conflicts. |
alexcrichton
force-pushed
the
top-level-overrides
branch
from
78f3919
to
fb9e451
Compare
Is it possible to specify a git branch or commit here? |
|
@SimonSapin yes, as mentioned in the description the syntax in |
|
After our conversation on #2375, it sounds like this is going to be fantastic for us! Thanks a ton, @alexcrichton. I think it's also going to make some of our really massive efforts (like impactful rust upgrades & adding new platforms) significantly easier to collaborate on across multiple people. |
|
If at some point we wanted to allow additional constraints on the thing being replaced (specifying more exactly it's dependency location or version) what would that end up looking like? Is this scheme extensible for that purpose? Is there a plan to allow the restricted replacement provided by |
|
How does this compare to https://www.haskell.org/cabal/users-guide/installing-packages.html#miscellaneous-options |
|
The syntax in Also yeah it seems like we may want to expand this to also be applicable in I believe this provides something very similar to that functionality. You'd basically be saying that "when you see this package with this version, instead you can use this newer version" |
|
@alexcrichton the way It seems like it helps more with packages whose specifications are not updated quickly enough, rather than the case we're talking about here of simply wanting to force a particular version for development purposes. |
|
My point about more specific overrides is that in contrast with Essentially, how does consideration of #736 play with the proposed syntax here? Is it natural to extend it (at some point in the future, I'd rather like to see this PR merged sooner) to allow greater matching control, or is that something that would need to be added via a separate |
|
@johntyree this also would be a package configuration option, not a user configuration option/flag (like |
|
Ah yeah that's true that this doesn't support "use whatever was listed except the higher version". This sort of mechanism, however, is intended to be a transiently-committed artifact of Cargo.toml rather than a long term component, so perhaps it'd work out? I think that "replace with this dependency" is also more general because you can redirect to a git fork or a locally checked out copy.
Oh sorry, to clarify, the override can look like: "libc:0.2.4" = "0.3"This will only override libc 0.2.4 to the version 0.3. In general the key here is a package id specification, which can be used to select any node within a graph. In other words, this should handle #736 for overrides encoded into Cargo.toml. Is that what you were thinking of? |
|
Ah, that's very nice. For those wondering, the docs for package id specifications appears to be here: http://doc.crates.io/pkgid-spec.html It appears to cover some of the use cases, but doesn't quite cover all of them: One can't specify that a In practice, the need to do this might not occur too often, but it isn't clear to me how easy add it after the fact would be. I suppose it could be possible to (later on) expand the package ID spec grammar if people end up needing it? |
|
Yeah it's a good point that you can only target nodes in the dependency graph, not necessarily edges. We may be able to invent a syntax for that, but perhaps this would be good enough for now? |
|
@alexcrichton yep, this seems like a good step in the right direction (and solves actual issues I have today). +1 |
|
I got a chance to discuss this with @wycats today and the conclusion was that he's a little hesitant to land this just yet. @wycats can probably articulate better than I can, but the gist of it is that this feature has the ability to propagate the idea throughout the Rust ecosystem that versions don't matter so much. A failure mode of this feature is:
This feature can also obviously create very confusing bug reports for crate authors. You could perhaps forget that a Finally, this can hide problems such as X not updating its dependency on Y for a subtle but intentional reason. Applications may then blindly replace Y with a more updated version and everything could appear OK until later when the "subtle bug" was encountered. Overall, @wycats was going to disucss some more with others to get some more insight on this. We discussed how this solves many workflow issues with Cargo today (like cascading updates) and how we can possibly be vigilant to ward off the concerns mentioned above. |
|
Oh and another point brought up is that this does not implement platform-specific replacements, only global replacements. One could imagine a package with an OSX-specific fix but you don't want to replace it on other platforms. |
Forgetting you have overrides in place already happens with "old style" path overrides and causes momentary confusion, but one generally figures it out fairly quickly (before filing bugs upstream) since the error messages from rustc include source file names with paths to your git clones rather than the usual I don’t think it’s been a major problem so far.
That could be done in the same style as #2328 : |
|
Right now people who need to distribute code that requires something like As for making it clear what is happening to the users building code: making sure the output is distinct (as it is when using cargo config path overrides) seems like a straight forward solution. |
|
If the concern is that folks might not notice that the For toplevel apps, it seems like it wouldn't matter as much, if the concern is about libraries needing an override for some dependency of theirs. After all, toplevel apps probably have a Cargo.lock checked in, too, and that can have all sorts of wonky versions in there (see: Servo ). |
|
☔ The latest upstream changes (presumably #2406) made this pull request unmergeable. Please resolve the merge conflicts. |
alexcrichton
force-pushed
the
top-level-overrides
branch
from
fb9e451
to
4833614
Compare
|
I had a somewhat long chat with @alexcrichton about the tension here between maintaining a meaningful concept of versions (if it's easy to say "2.4 really means 2.5", then that becomes an acceptable solution to version mistakes rather than fixing them upstream) and the transient needs you have during upgrading (you really need "libc 0.3" to unify with "libc 1.0" because of transitive dependencies). I want to think about it a little bit more and talk with @aturon (and others) about it, but I'll try to have an opinion I feel confident about by the end of next week. It's definitely a tough problem; sorry this is taking a while to sort out. |
I shall try 😄 The basic idea behind this feature, and the use-case it's most directly targeting is:
The intended workflow for this use-case is:
The fact that only a handful of people need to do (3) above is a feature of this workflow. It gives the community a way to rally around in-progress patches and use them as they land, rather than having everyone do the work themselves.
Also, applying the patch in this way does not introduce a full-on cascade; since the replaced package has the same name and version as the original release tag, so as far as the dependency resolver is concerned, nothing has changed. What about version bumps? The idea of the current approach is that version bumps are a special case of the "short-term patch" workflow. In particular, if there are a number of packages in your dependency graph that depend on
Step (2) above is critical. While it may occasionally work to whole-hog replace a version of a dependency with another version throughout the dependency graph, it quite often will not. The step of confirming that direct dependents of the update works by attempting to compile the crate before moving on helps to identify the precise reason a particular update isn't working, and gives a clear workflow for fixing it. Better yet, once that process is done, any other application that depends on that package can use the branch, knowing that the upgrade will work, and that any necessary fixes have already been incorporated. This approach does not require a big-bang cascading update to the entire dependency graph; instead, it focuses the update effort where it belongs: to confirm that packages that claim to support Our goal right now is to monotonically improve the experience of updating dependencies carefully but steadily. As we learn more about the pain points that are still left once we ship the first set of features, we will continue to improve Cargo to directly support workflows that continue to have too much pain, but it's hard to speculate about precisely what those solutions should be when such a disproportionate amount of the pain is focused on things we know how to improve conservatively. |
This is a misfeature in most organizations that care about versioning, I think. There is assumed to be a one-way mapping from versions to code.
0.2.3
No. Bob's patch broke this other thing. I'm using Jane's 0.2.3.
I don't know. They seem pretty arbitrary.
To me it seems like you're approaching this backwards. Github has always provided a way to share patches before they hit upstream. What this does is prevent everything else. I understand that you're trying to establish what you feel are best practices, I just don't agree that this is the right way to go about it. Rather than having the resolver choose a new, non-broken version of a package, it seems like you're proposing that we each locally rewrite history and pretend that the broken version was never released. How is that better from a complexity point of view? |
It sounds like we have a strong agreement that we want to have, as a core constraint, that the community has a strong sense of what versions mean. What alternative do you have in mind for replacements? |
|
I wanted to elaborate a little more on something that was kind of implicit in my original comment. While my comment explicitly talked about the workflows that I thought would work well with the current status of the design, I was implicitly talking about a possible enhancement: allowing I have some concerns about that direction (more in a sec), and I also feel that the more minimal, conservative version of this feature will improve the status quo and give us a clearer view of how to address the pain points that remain.
The most important concern I have about the version-changing feature is similar to @johntyree's concern: I think it's important that we retain as much community consensus as possible about the meaning of versions. Since applications almost always use dependencies that they don't manage directly, the need to upgrade non-atomically ("I need to upgrade libc because one of my deps requires it, but another one hasn't released an upgrade yet") will come up. To me, the question is just the cost-benefit to the community of the different approaches. In my view, an approach that allows you to temporarily fix up some code without bumping the version deals less damage to the notion of versions that an approach that allows you to say, across your entire graph: "whenever you see libc 0.3, pretend you saw libc 1.0". Mary:
rust-url authors:
Mary, days later and after several back-and-forths:
I think this story is worse than the equivalent story where the user explicitly replaced Some lower-order concerns:
I hope this helps clarify things more. |
|
I was writing something here, but you've hit all the points much more clearly than I was about to. I'll just add that I think our disagreement comes from a difference in philosophy about what the tool should be responsible for doing. I think it should be for helping developers ship applications, rather than teaching them about community management or open source etiquette. That said, you've clearly put a lot of thought into this and there's no reason it can't be revisited if what you're proposing turns out to be insufficient. I say go ahead and let's see how it does. |
|
Trying to force people's work flows by adding an extra All the arguing about making it harder to break things ignores that we already have less controllable replacements via Previous limitations that assumed semver were done on |
|
@bors r+ |
|
📌 Commit 54d738b has been approved by |
Implement top-level overrides
This commit is an implementation of top-level overrides to be encoded into the
manifest itself directly. This style of override is distinct from the existing
`paths` support in `.cargo/config` in two important ways:
* Top level overrides are intended to be checked in and shared amongst all
developers of a project.
* Top level overrides are reflected in `Cargo.lock`.
The second point is crucially important here as it will ensure that an override
on one machine behaves the same as an override on another machine. This solves
many long-standing problems with `paths`-based overrides which suffer from some
level of nondeterminism as they're not encoded.
From a syntactical point of view, an override looks like:
```toml
[replace]
"libc:0.2.0" = { git = 'https://github.com/my-username/libc';, branch = '0.2-fork' }
```
This declaration indicates that whenever resolution would otherwise encounter
the `libc` package version 0.2.0 from crates.io, it should instead replace it
with the custom git dependency on a specific branch.
The key "libc:0.2.0" here is actually a package id specification which will
allow selecting various components of a graph. For example the same named
package coming from two distinct locations can be selected against, as well as
multiple versions of one crate in a dependency graph. The replacement dependency
has the same syntax as the `[dependencies]` section of Cargo.toml.
One of the major uses of this syntax will be, for example, using a temporary
fork of a crate while the changes are pushed upstream to the original repo. This
will avoid the need to change the intermediate projects immediately, and over
time once fixes have landed upstream the `[replace]` section in a `Cargo.toml`
can be removed.
There are also two crucial restrictions on overrides.
* A crate with the name `foo` can only get overridden with packages also of
the name `foo`.
* A crate can only get overridden with a crate of the exact same version.
A consequence of these restrictions is that crates.io cannot be used to replace
anything from crates.io. There's only one version of something on crates.io, so
there's nothing else to replace it with (name/version are a unique key).
Closes #942
|
☀️ Test successful - cargo-cross-linux, cargo-linux-32, cargo-linux-64, cargo-mac-32, cargo-mac-64, cargo-win-gnu-32, cargo-win-gnu-64, cargo-win-msvc-32, cargo-win-msvc-64 |
|
Cargo is effectively a workflow tool, so we're not really forcing a workflow here but rather extending the already existing one? You're also right that |
It isn't clear why you're defining Alternately, I guess you could be saying that because
Even if I keep entirely to myself (via local config files), cargo still believes it should be allowed to arbitrarily restrict my workflow as long as the stated goal is to improve collaboration? I guess I don't agree there either. I suppose I'm operating under the impression that cargo should be a useful tool for everyone working with rust, not just those that fit with some folks idea of how things should work. |
|
@jmesmon I was mostly just responding to your comment about "trying to force people's work flows" by pointing out that Cargo is itself a workflow tool. In general, yes, you're indeed right! Cargo shouldn't be preventing use cases and should be a general tool for all. That doesn't mean, however, that literally every feature should be added to Cargo. For example to work around the restriction of the same version you just need to fork a repo and change the version backwards. The idea is that it's a step up from the normal workflow, not forbidden. |
|
FWIW, I just pushed a branch called 1.0.0-pretending-to-be-0.5.9 to servo/rust-url doing exactly what it says, just to be able to use a |
|
And so that I’m not just complaining, I have to say that this feature allowed me to run Servo’s full test suite on CI servers for a large PR before all changes in dependencies were merged and published. This would have been a lot more painful to pull off before. Thanks a lot @alexcrichton ! |
It seems... wrong though to say that 1.0.0 can masquerade with 0.5.9? Every crate would essentially require source level changes to be compatible with 1.0.0 of rust-url, right? In the case that they're being changed already bumping the version dep seems reasonable?
Awesome! Glad to hear it's working out :) |
I don't know about rust-url specifically, but for many packages bumping from a 0.x.y version to a 1.0.0 version doesn't necessarily mean a breaking change, but can simply mean that you have decided the existing API is expected to be stable for the forseeable future, and want to advertise that via the version number. Also, even if breaking changes are introduced, it's possible for them not to break very much, and you to want to do this replacement in order to judge how much will be broken if they update to see how much work you're inflicting on your downstream, similar to a crater run. For instance, if you add one variant to an infrequently used enum that could be exhaustively matched, that's technically a breaking change, but if only one or two packages in the ecosystem ever actually match against that enum, you may have some application with a dependency graph in which you can substitute in the new version without breakage. I think one of the reasons that this requirement for a strict version match seems wrong is that it makes it harder to identify the actual version of a package in question. As mentioned above, you could have multiple different packages called libfoo 0.2.3 floating around at any given time; while the unique hash can distinguish them if using git dependencies, that's pretty much an opaque blob that's hard for people to mentally compare, and can't be ordered without looking at the Git history. If I need to maintain a longer-term local fork for whatever reason, and so introduce multiple patches to libfoo 0.2.3, it would be hard to tell which of ssh://my-server/my-repo.git#abc123 and ssh://my-server/my-repo.git#456def is actually newer when looking at the Cargo.lock; and the Cargo.toml may only contain I suppose that one workaround would be to establish a convention that you use the Hmm. After reading the semver spec more closely, it appears that semver doesn't support a 0.2.3-patch1 version properly, since that would be seen as being a prerelease of 0.2.3, not later than 0.2.3. It doesn't look like semver has any way of adding additional local patches that compare greater than the upstream version they patch. This is something that, coming from a Debian packaging background, I would expect; in Debian, 0.2.3p1 would compare greater than 0.2.3, but So, maybe given that constraint that you can't really indicate properly in semver a version number for a local patch, that using a convention with git tags or branch names is the best you can do for keeping track of local patches to a particular upstream version. |
It can’t really, but I didn’t manage to get things to build otherwise. Maybe related to #2603. |
Even presuming semver for cases other than the '<1' to '>=1' bump, this isn't true. semver is a promise of compatibility based on version numbers, but is never a promise of total incompatibility. Which is another reason, even if we presume everyone is following semver, that trying to restrict package combinations in something that is already an override of normal resolution is not a good idea. Essentially: cargo can't know better than it's users. |
|
As @wycast has commented above (and expanded on) the use case of upgrades is not the sole purpose of this PR. This is also intended for things like hotfixes and bug fixes. Also, as I explained earlier it definitely is indeed the case that major version upgrades do not cause breakage for all packages, we are assuming that this is not the majority of use cases. |
|
@alexcrichton Does this solve the case in which you'd like to override crate-type .. i.e. you'd like to build a shared library or dylib from the dependency as mentionned in: #1826 (comment) |
|
@traoremp For that use case, would it work to have a two-line wrapper crate like this? extern crate foo;
pub use foo::*;Then you could pick whatever |
|
@SimonSapin Oh I just understood what you meant. Ok but why not just give the ability to do that as a dependency field that would specify how to beuild the dependency ? |
This commit is an implementation of top-level overrides to be encoded into the
manifest itself directly. This style of override is distinct from the existing
pathssupport in.cargo/configin two important ways:developers of a project.
Cargo.lock.The second point is crucially important here as it will ensure that an override
on one machine behaves the same as an override on another machine. This solves
many long-standing problems with
paths-based overrides which suffer from somelevel of nondeterminism as they're not encoded.
From a syntactical point of view, an override looks like:
This declaration indicates that whenever resolution would otherwise encounter
the
libcpackage version 0.2.0 from crates.io, it should instead replace itwith the custom git dependency on a specific branch.
The key "libc:0.2.0" here is actually a package id specification which will
allow selecting various components of a graph. For example the same named
package coming from two distinct locations can be selected against, as well as
multiple versions of one crate in a dependency graph. The replacement dependency
has the same syntax as the
[dependencies]section of Cargo.toml.One of the major uses of this syntax will be, for example, using a temporary
fork of a crate while the changes are pushed upstream to the original repo. This
will avoid the need to change the intermediate projects immediately, and over
time once fixes have landed upstream the
[replace]section in aCargo.tomlcan be removed.
There are also two crucial restrictions on overrides.
foocan only get overridden with packages also ofthe name
foo.A consequence of these restrictions is that crates.io cannot be used to replace
anything from crates.io. There's only one version of something on crates.io, so
there's nothing else to replace it with (name/version are a unique key).
Closes #942