RFC: Deduplicate Cargo workspace information

[alexcrichton]

This proposal is aimed at deduplicating dependency and metadata
directives amongst a set of workspace crates in Cargo with extensions to
the [workspace] section in Cargo.toml and package manifests to
reference this [workspace] section.

Rendered

[shepmaster]

Each workspace member can then reference this section in the workspace with a new dependency directive

This makes me remember the on-again off-again feature request of "use the same version of X as a dependency". Would it be worth thinking about leaving room in the syntax to support that?

(replied)

[sfackler]

It seems like the value of inheriting the documentation field in particular will be limited without the ability to template-expand the crate name into the URL, right?

[alexcrichton]

I agree yeah that the usefulness in all cases is pretty limited, but that's somewhat intentional as well. I suspect this will largely be used for homepage/license/etc information, but there wasn't much reason to leave out keys like documentation as well. It may be useful for 2-workspace crates though where one's a macro internals and the other is the real crate, where both effectively are documented through the same url.

[sfackler]

Is the workspace = true entry in a dependency table actually necessary? It seems like that could be inferred if the dependency had no version, git, or path entry, right? So the log case would just look like

[dependencies]
log = {}

[alexcrichton]

I don't think we have a current technical reason requiring something to be here in that we could likely assign workspace = true meaning to an empty table. That being said though I would personally find the log = {} a bit too "small" in that it's sort of weird seeing no other indication of where the dependency comes from. I'd hope that workspace = true wouldn't be too burdensome though to list on lots of repeated dependencies.

Put another way, I think that this is a technically viable option, but I would say that for learnability and skimming crates it'd be nice to have something listed there to remind readers to look at the workspace manifest. I'm curious what others think though!

[petrochenkov]

Seems like the common case is going to be

[dependencies]
foo1 = { workspace = true }
foo2 = { workspace = true }
foo3 = { workspace = true }
foo4 = { workspace = true }
foo5 = { workspace = true }
...

with all or almost all crates being redirected to the workspace.

Which kind of asks for refactoring it into something looking like

[dependencies.workspace]
foo1
foo2
foo3
foo4
foo5
...

(but with valid syntax etc.)

[alexcrichton]

One of the more compact ways I've thought of is

[dependencies]
foo = "ws"

[Nemo157]

Note that the main syntax can also be written

[dependencies]
foo.workspace = true

[pickfire]

I like the foo = "ws".

[pksunkara]

I like the foo = "workspace". Elaborate and concise at the same time.

[hbina]

[dependencies]
foo1 = { workspace = true }
foo2 = { workspace = true }
foo3 = { workspace = true }
foo4 = { workspace = true }
foo5 = { workspace = true }

I disagree, I like this. I think this allows for future updates to this features. Frankly, using strings are arbitrary and will possibly make for a mess down the line when people try to extend this stuff. How about supporting something like

[dependencies]
foo1, foo2, foo3, foo4, foo5 = { workspace = true }

Which is a lot better imo because it solves the problem of repetition.

[Eh2406]

Is that valid toml?

[davidbarsky]

Overall, this is a great RFC that addresses some papercuts I've seen in https://github.com/tokio-rs/tracing. I do have a question: does this RFC make a cargo publish --all (that publishes all unpublished crates in a workspace) easier to support or is RFC orthogonal to that feature?

[alexcrichton]

I don't think this makes that either harder or easier per se, since this is, internally in Cargo at least, mostly just "syntax sugar". I would guess that an implementation of publish --all wouldn't have to specifically handle anything in this rfc (in that it doesn't make it harder), and it would also have to handle everything still (this doesn't make it easier)

[pksunkara]

@davidbarsky I implemented a nice way of doing that (using dep graph of only the crates you are publishing) in https://github.com/pksunkara/cargo-workspaces

[joshtriplett]

When defining the workspace table keys, and paths in workspace dependencies, you should explicitly state at that point that all paths are relative to the workspace Cargo.toml and will be adjusted to be relative to where they're substituted.

[joshtriplett]

Couple of typos.

[joshtriplett]

Thank you for working on this!

[joshtriplett]

On the question of whether to explicitly mark inherited metadata, I can see reasons for both directions there.

I do absolutely think that publish should be inherited by default; that provides substantial value.

One thought, though: suppose you specify a field in the workspace metadata, and a crate in the workspace wants to omit that field. Not fill it in with a blank string, omit it. How could it do so? (Example: description, or license-file.)

[alexcrichton]

@joshtriplett

When defining the workspace table keys, and paths in workspace dependencies, you should explicitly state at that point that all paths are relative to the workspace Cargo.toml and will be adjusted to be relative to where they're substituted.

Done now! (I think)

Good suggestion!

One thought, though: suppose you specify a field in the workspace metadata, and a crate in the workspace wants to omit that field. Not fill it in with a blank string, omit it. How could it do so?

That's a good point yeah and one that isn't supported with the proposal as-is. TOML doesn't have a natural "null" value so we don't have an easy opt-in, but we could perhaps co-opt documentation = {} if this is necessary.

One alternative to what's written down is that we could only allow the documentation = { workspace = "other-member" } form and disallow inheritance from [workspace] entirely. It does mean that you'll still have a big header in Cargo.toml with lots of workspace = "..." pieces, but it would solve this issue.

Do you have a feeling for how often you think this'll come up in practice?

[joshtriplett]

@alexcrichton I do like the workspace = true syntax to get a value from the workspace metadata rather than a specific other crate. That feels more natural to me.

Regarding how often workspace crates will not want the "automatic inheritance" behavior: more often with some fields than others. The pattern of "most crates have this value, but a few don't" is not at all uncommon, particularly for internal usage. Internal-only crates may have publish = false and a different license value, for instance. A large workspace may have the same repository value for most crates, but a few might get pulled in by submodule or subtree and also not specify repository. Or a workspace using edition = "2018" for almost everything may vendor a crate that implicitly uses the 2015 edition.

Here's another alternative proposal, which has the advantage of minimizing boilerplate while still being explicit. What if we default to requiring you to write field = { workspace = true } to get metadata from the workspace, but allow you to explicitly write things like [workspace] field = { value = "field value", inherit = true } if you want auto-inheritance? That way, if you have a workspace where you want the same values everywhere, you can easily get that by writing all the metadata with inherit = true in your workspace Cargo.toml once, but if you have some values you want to deduplicate in your workspace but not use unless explicitly specified, you won't accidentally get a value inherited without expecting that behavior.

How does that sound? That would avoid requiring people to type field = { workspace = true } everywhere (which may be longer than the actual value, partially defeating the purpose).

[alexcrichton]

@shepmaster

This makes me remember the on-again off-again feature request of "use the same version of X as a dependency". Would it be worth thinking about leaving room in the syntax to support that?

Oops sorry I forgot to reply to this earlier!

I didn't really consider this at all when writing up this proposal, and thinking about it a bit now I would say that we probably don't need to consider it much here. I would be pretty hesitant to try to lump it in and I suspect that the desired syntax for doing this would likely not conflict with the syntax we choose here, or we could always design it later to not conflict.

[folex]

workspace = true is an amazing feature! It allows to keep all inner dependencies definitions (specified by path) in one place!

But if I understand correctly, it leads to some repetitions between workspace.members and workspace.dependencies. Here's an example that I've stumbled upon:

Crate configuration

/Cargo.toml

[workspace]
members = [
    "crates/libp2p",
    "crates/waiting-queues",
    "spell-storage",
    "foo"
]

[workspace.dependencies]
libp2p = { path = "crates/libp2p" }
waiting-queues = { path = "crates/waiting-queues" }
spell-storage = { path = "spell-storage" }
foo = { path = "foo" }

/foo/Cargo.toml

libp2p = { workspace = true }
waiting-queues = { workspace = true }
spell-storage = { workspace = true }

Problem

You can see that workspace.members and workspace.dependencies repeat each other for the inner (local) crates.

Here's a PR that moves to workspace = true for all inner dependencies. You can see it has a lot of path duplication in /Cargo.toml fluencelabs/nox#1388

Possible solution

One solution I see is to have workspace.members automatically among workspace.dependencies. Something like that:

/Cargo.toml

[workspace]
members = [
    "crates/libp2p",
    "crates/waiting-queues",
    "spell-storage",
    "foo"
]

[workspace.dependencies]
# empty!

/foo/Cargo.toml

libp2p = { workspace = true }
waiting-queues = { workspace = true }
spell-storage = { workspace = true }

[epage]

@folex

Before answering Looking over my comments below, it feels like there is enough to discuss that burying this in a closed RFCs comments might make it harder to follow. It might be good for us to move to internals or a cargo issue

Instead of that proposal, the RFC included

This behavior should mean that you no longer need to write version = "..." with path dependencies if you publish to crates.io. Coupled with the workspace-level dependencies above this means you never have to write the version of a path dependency anywhere!

which was rejected in prep for stablization. see my comment

It doesn't look like your proposal runs into similar issues.

The questions that would need to be answered

• What do the implicit workspapce.dependencies look like?
  • I'm assuming member = { version = "<x>.<y>.<z>", path = "...." }
  • Is always requiring the latest of a workspace member a good enough default? I assume so as you can't test with prior versions
  • Should it be context-sensitive and we skip the version field on dev-dependencies to allow publishing with circular dependencies, like what cargo add does?
  • Is default-features = true a good enough default?
  • Does this break down for alternative registry users and is that sufficient enough to block crates.io users getting this?
• What happens when an explicit entry is created?
  • I assume it overrides the implicit entry
• What happens when a package name is ambiguous?
  • I hadn't thought of this before but I think I recently saw a discussion of someone who had reused package names within a workspace, so I guess its possible
• Seeing as the generated result isn't easily viewable (cargo package and then break apart the package), are there enough opportunities for users to feel uncertainty or confusion over what is happening from the lack of transparency? How do we help these users?
  • This would be a point against being context-dependent
  • Without context-dependence, it feels like this would be harder to for someone to discover the root cause of their circular dependency publishing issue and know how to resolve it (or understand what someone's explanation says)

Use cases we need to consider besides the standard one:

• clap needs a = dependency on clap_derive because clap_derives proc macro generates code for a specified version of clap
  • Can be handled by an explicit override or not using dep.workspace = true
• clap_complete doesn't want to force clap features on dependents, so uses default-features = false and specifies only what is needed
  • Can be handled by an explicit override or not using dep.workspace = true
  • Concern: While internal and binary packages don't matter, public libs packages likely should be doing this (when features are available) which makes me wonder if we would be encouraging wrong practices by having this feature.
• At one point clap_derive had tests that required clap despite clap depending on clap_derive. To publish, we had to make clap_derives dev-dependency exclude the version field so it'd be ignored on publish. While we no longer have that situation (tests moved to clap), other crates do
  • Can be handled by an explicit override or not using dep.workspace = true
  • See also the "questions" section for the idea of it being context-dependent