mini-RFC: Finalize syntax of impl Trait and dyn Trait with multiple bounds

[petrochenkov]

Priority of + in impl Trait1 + Trait2 and dyn Trait1 + Trait2 is inconsistent with the rest of the grammar.
This RFC lists possible solutions to this problem and chooses one of them for future stabilization.

Note that the chosen alternative is not unambiguously better than others, so please bikeshed the syntax!
This was previously submitted as a PR, but reformatted as an RFC specifically to get feedback from wider audience.

Rendered
Tracking issue: rust-lang/rust#34511

[Centril]

( The rendered link should probably point to: https://github.com/petrochenkov/rfcs/blob/impldyn/text/0000-finalize-impl-dyn-syntax.md to sync with additional future commits if there are any )

[Ixrec]

It seems like the proposed alternative of always needing parentheses is the only option that allows us to backwards-compatibly tweak these rules later and allow dropping parenthesis in certain situations. Is this true? (the fact that the RFC never mentions this makes me think I'm missing a reason why it doesn't actually work)

[ExpHP]

Seems like a home run to me. Even if one could argue that the difference between &(A + B) and &dyn A + B is okay, I simply cannot imagine anyone still saying the same for &mut (A + B) and &mut dyn A + B.

I guess the worst we can expect from something like this is

impl (Fn() -> impl (Iterator<Item=u8> + 'a) + 'a)

[ExpHP]

Actually, I too am wondering along the same lines as @Ixrec especially for types like Box<dyn (A + B)> (which is something that most assuredly will be cropping up!).

Of course, whatever rules are drawn up (if any) for allowing these parentheses to be omitted should apply equally to & and friends as well.

[petrochenkov]

@Ixrec

It seems like the proposed alternative of always needing parentheses is the only option that allows us to backwards-compatibly tweak these rules later and allow dropping parenthesis in certain situations.

Always requiring parentheses like this (impl A + B) (I) or like this impl (A + B) (II)?

---

I is indeed the most forward compatible, but it's really weird, so I didn't listed it in the RFC.

// WHY!?
type A = dyn A + B; // ERORR, interpreted as `(dyn A) + B`
type B = (dyn A + B); // OK, dyn is immediately in parentheses

---

II is a bit more complex, it introduces new syntax - parentheses after impl / dyn - dyn (A + B).
If the rules are relaxed in the future this syntax will become unnecessary legacy dead weight.

Suppose we always require parens now:

fn f() -> impl (A + B) { ... }

but then relax them in the future:

fn f() -> impl A + B { ... } // OK

What happens? impl becomes a special prefix form again, not unary operator, but we have to support additional special production in the grammar for compatibility with impl (A + B).

---

As a result, if rules for the "Alternative 3" are going to be relaxed in the future, then it will become "Alternative 2" + additional legacy syntax. It's better to immediately start with "Alternative 2" in this case.

[ExpHP]

What happens? impl becomes a special prefix form again, not unary operator, but we have to support additional special production in the grammar for compatibility with impl (A + B).

Using your example, is there any reason we could not also support this:

fn f() -> &A + B { ... }

That is to say, these extensions don't have to be a hack exclusively for dyn and impl. They can be done by extending the grammar of unary type operators in general. (or can they not?)

[petrochenkov]

@ExpHP

Using your example, is there any reason we could not also support this:

The same reason why there's desire to change the status quo syntax - operator priorities.
Unary operators have higher priorities than binary operators generally, if this is reverted, then &A + B would mean the opposite of what it means in expressions and what most people would expect it to mean.

[ExpHP]

I guess that I feel that having consistency between expressions and types is not quite as critical as having consistency within types. My thought is, because there are no valid types that contain a unary operator nested within a binary operator (i.e. (&A) + B, where & and + may be any type operators, and the parentheses denote nesting within the AST), it does not seem unreasonable to optimize their grammar for this.

---

To make Box<dyn A + B> work I was considering making a special grammar for types that appear as type arguments or inside the parentheses of Fn(); basically, when they are terminated by a specific token (one of , ) >). Something like (informally)

# Types in most contexts
# parses unary/binary ops like an expression
ty = ty_expr

# Types inside `<>` and `Fn()` parentheses
# Superset of ty that accepts e.g. `&mut dyn A + B`
# (NOTE: it occurs to me that ty_expr would also accept `&mut dyn A + B`,
#        but I guess what I'm trying to convey is that the AST will be different)
ty_arg = [ unop ]* [ ty_without_unop binop ]* ty_without_unop
       | ty

ty_without_unop = ident generics
                | ty_fn # handles `Fn(A) -> B`
                        # A would be ty_arg, B would be ty
generics = [ '<' [ ty_arg ',' ]* '>' ]?

That said, I can think of a few issues with my approach:

• It will be difficult to prevent scenarios where copying and pasting a type somewhere silently changes its meaning.
• Just because we currently have no valid types that contain a unary operator nested in a binary operator does not mean we will never want to.
• Having specialized grammars in certain contexts like this can leak implementation details of macros. Box< $T:ty > would not match Box<dyn A + B> the same way as $T:ty would.

Again, I feel the details of an extension like this could be ironed out later. Perhaps it's a terrible idea that's just not possible; and if this is the case, then all else considered, I'd rather have to write Box<dyn (A + B)> than to have dyn be "this weird special syntax".

[cramertj]

Thanks @petrochenkov for putting this together! It's really helpful seeing all the different options (and their effects) laid out clearly in the RFC.

Unfortunately, I'm personally frustrated by all of the options-- clearly there's no perfect solution. Alternative 3 feels the most logically consistent to me, but it's also the most verbose, and I'm sure I'd quickly become confused and annoyed at the visual clutter it adds. After trying out a few examples, I personally prefer Alternative 1. The vast majority of the time impl or dyn is used, there's some sort of surrounding syntactic element which makes the grouping clear. For example, Box<dyn A + B>, x: impl A + B,, x: &dyn A + B,, and -> impl A + B are all "visually obvious" to me, and parentheses only make them noisier: Box<dyn(A + B)>, x: impl(A + B),, x: &dyn(A + B),, -> impl(A + B).

The drawbacks: + after a type in an expression context is confusing, and -> impl Fn() -> impl A + B is unclear.

It usually doesn't make sense to see + after a type in an expression context-- this really only happens when adding to the result of an as cast. Adding to the result of an as &dyn cast is usually not possible because the Add trait takes self by-value, so you'd need to be casting to a trait that has an implementation of Add for &dyn Trait, which is extraordinarily unusual.

Nested impl Trait cases such as fn foo() -> impl Fn() -> impl A + B are more concerning and harder to visually parse, but I read this naturally as "foo returns a Fn which implements A + B", which is the correct interpretation under Alternative 1.

You might well ask, "Why not Alternative 2? Most of the clear cases you brought up would be handled properly there, and most of the unclear cases would require parentheses?" I've waffled about this a bit, and in the end I decided against Alternative 2 because of how it handles &dyn Foo + Bar. Unlike Alternative 1, it requires parenthesis for this case. However, it does not place the parenthesis where I would expect. &(dyn Foo + Bar) looks odd to me in comparison with &dyn (Foo + Bar), and it's certainly not what I would try first. Because of this, I think it's simpler to just allow &dyn Foo + Bar.

Edit: I think I would also be in support of a variant of Alternative 2 which added (Bound) as a variant in the bounds grammar so that T: (Bound1 + Bound2) would work, along with (more importantly) &dyn (Bound1 + Bound2).

[rolandsteiner]

It's been a while since I looked at the details of types parsing, but isn't there also a potential for ambiguity with tuples when adding/requiring brackets in some of the examples?

I.e., wouldn't -> (impl A + B) rather mean "returns a 1-tuple of impl A + B" (however the latter is then parsed)?

[Centril]

@rolandsteiner A 1-tuple (why do we have such a thing in the type-system anyways...) is always ( T , ) and never (T), which is just T, wherefore -> (impl A + B) is unambiguously the same as -> impl A + B and never -> ( impl A + B , ).

[kornelski]

Hmmm, I assumed dyn/impl is a modifier for the trait name alone, so dyn Trait1 + Trait2 should actually be written as dyn Trait1 + dyn Trait2.

Can this work? It doesn't need parenthesis.

[eddyb]

@pornel No, then it wouldn't really mean the same thing. The + is between trait (or lifetime) bounds, whereas dyn applied to the set of bounds is shorthand for a dynamic existential over a type which implements those bounds (we might eventually add a way to write that explicitly).

[petrochenkov]

@ExpHP

My thought is, because there are no valid types that contain a unary operator nested within a binary operator (i.e. (&A) + B, where & and + may be any type operators, and the parentheses denote nesting within the AST), it does not seem unreasonable to optimize their grammar for this.

(&A) + B as the correct priority can arise in larger context - Fn(&A) -> &A + B.
In addition, bounds have "unary operators" with priorities higher than +:

for<'a> A<'a> + B // (for<'a> A<'a>) + B, not for<'a> (A<'a> + B)
?A + B // (?A) + B, not `?(A + B)`

so

&A + B // (&A) + B, not &(A + B)

is consistent with them as well.

[petrochenkov]

So, I wanted to look how these alternatives look on real code and tested them on rustc and libstd - https://github.com/petrochenkov/rust/commits/impldyntest.

The first commit is the baseline (Alternative 1) and the next commits show changes required to move to Alternatives 2 and 3.
Alternative 2 requires fewer changes compared to Alternative 3.

[petrochenkov]

From #2250 (comment) I'm pretty much convinced that the Alternative 3 is just impractical, Boxes appear too often, too many unnecessary parens are required.
In addition, the parens required by Alternative 2 seem... appropriate (probably due to "normal" operator priorities).

So, I'm going to change the RFC to use the Alternative 2 instead.

@cramertj

Edit: I think I would also be in support of a variant of Alternative 2 which added (Bound) as a variant in the bounds grammar so that T: (Bound1 + Bound2) would work, along with (more importantly) &dyn (Bound1 + Bound2).

I don't want this because it's technically unnecessary and creates more (resolvable) parsing ambiguities (it's basically repeating the mistake from rust-lang/rust#41077), but it's a fully backward-compatible extension that can be implemented later on top of the Alternative 2 if necessary.

[petrochenkov]

I'm going to change the RFC to use the Alternative 2 instead.

The RFC is updated.

[nikomatsakis]

@petrochenkov

I don't want this because it's technically unnecessary and creates more (resolvable) parsing ambiguities (it's basically repeating the mistake from rust-lang/rust#41077), but it's a fully backward-compatible extension that can be implemented later on top of the Alternative 2 if necessary.

Can you say a bit more about these ambiguities? I am trying to page that back in -- maybe some links to prior conversations? :)

[cramertj]

@nikomatsakis There was some previous discussion in this thread.

[nikomatsakis]

OK, I didn't find that especially enlightening. I'd like some examples of the sort of conflict we expect.

So I could get behind alternative 2, though like @cramertj I really feel like dyn (A+B) should parse.

I've also been pondering a few questions. What do people expect from something like Box<impl (A + B) + C>... should that parse?

I tried making a little toy model of types plus bounds here https://github.com/nikomatsakis/rust-grammar-toy using LALRPOP. This is intended to model Alternative 2 except that we do permit parens in lists of bounds too (e.g., dyn(A+B) works). As an interesting side-effect, dyn(A+B)+C also parses, and it is equivalent to (dyn A + B + C). (In other words: dyn takes a +-separated list, and that list may have internal parens in it.)

I'd like, before we settle on something, to be able to express what we expect in fairly concrete terms like this.

The grammar as I wrote it there is a bit peculiar in that it does't support parens like (A) + B in a list of bounds. This is because it creates an annoying LR conflict -- I could resolve it using LALRPOP macros, but I exceeded my quota for toying with this just now.

[cramertj]

@rfcbot fcp merge

While I would prefer that we end up with a grammar that supports &dyn (Foo + Bar) (like the one @nikomatsakis proposed above), we can backwards-compatibly transition to that syntax from the proposed Alternative 2 (low-priority binding).

[rfcbot]

Team member @cramertj has proposed to merge this. The next step is review by the rest of the tagged teams:

• @aturon
• @cramertj
• @eddyb
• @nikomatsakis
• @nrc
• @pnkfelix
• @withoutboats

No concerns currently listed.

Once these reviewers reach consensus, this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[nikomatsakis]

OK, so, we had a discussion about this in the lang team meeting. There was general consensus, I believe, on a few points:

First, the basic idea of "alternative 2" seems like the right starting point.

Second, for those cases where disambiguation is needed, everybody prefers impl (Foo + Bar) to (impl Foo + Bar). This implies that we would accept

However, there was some disagreement about what to do when impl trait appears in the context of bounds lists, in particular in conjunction with the Fn trait. @joshtriplett argued pretty strongly that it was best to avoid potential ambiguity, and hence things like the following ought to be errors:

where F: Fn() -> impl Write + Send

fn foo(arg: impl Fn() -> impl Write + Send)

@cramertj felt like we should interpret impl Fn() -> impl Write + Send in the same way that we would interpret fn foo() -> impl Write + Send, though perhaps we could have rustfmt introduce parentheses (or lint against this structure).

That said, we don't currently accept impl trait in the return position anyway (for other reasons), so in a way this dispute is moot, right?

[withoutboats]

Don't have strong opinions about the remaining controversy & see both arguments. Reading it, interpreting it as grouped with the return type seems obvious and natural, so parens seem superfluous. However, I could see someone getting a compiler error, adding + Send to try to fix it, and then just getting the same error and being flummoxed as to why it didn't work.

Possibly just linting could solve that. I think maybe we should conservatively start with requiring the parens, and see if we can introduce a lint someday that would make requiring them unnecessary.

[stuhood]

Has using angle brackets already been discussed at some point?

Given the keywordness of impl, and the fact that is lowercase, it doesn't seem like it would be too confusing to use a syntax like:

impl<Write + Send>

This would seem to align with the idea that a usage of impl Trait is akin to an anonymous generic type (since the traits listed inside the angle brackets are bounds on some unnamed/anonymous type).

There would also be alignment with Box, which has similar semantics in practice. The keywordness also sends the appropriate signal relative to Box: that the result is more efficient.

(EDIT: A downside, of course, would be that as with Box the brackets would always be required.)

[eddyb]

@mark-i-m It takes a type, and right now if you use a trait name in the type position it can only name the trait object - there's no such thing as "being generic over a trait" so far, the Trait in T: Trait has to be a path to a trait definition, it can't come from generics.

What @scottmcm wrote makes dyn<X> always X - which works for trait objects, but I think the risk here is without it being a lang item, it'd accept e.g. dyn<u8> which wouldn't want to.

[mark-i-m]

Thanks for the explanation :) that's makes sense.

[withoutboats]

Does that mean something like the following in the prelude would get us much of dyn with no grammar changes and no new keyword?

I don't think this is meant to be a serious suggestion, but I think the main advantages of dyn are not supported by something like this:

• You still get bad error messages about "Trait does not impl Sized", because the trait type is still just the trait name. These error messages almost always lead the user to add : Sized to the trait definition.
• A surprise advantage of dyn has been that its introduced a keyword we can use to conceptualize syntaxes in other contexts, like the alloca's dyn length syntax, or the idea that's been floated to have dyn patterns to downcast trait objects.

[stuhood]

I don't think this is meant to be a serious suggestion

I'm not sure whether it was intended as a serious suggestion, but the ease with which it would slide in is interesting for other reasons.

In particular: it demonstrates that dyn and impl could act like types (wrapper types / smart pointers of a sort where deref is more nuanced) rather than as modifiers.

As a thought exercise: what if the way to accomplish the requirement that Box must contain a dyn type was... just a trait bound? The definition of Box would be something like:

pub struct Box<T: ?Sized, D: dyn<T>>(Unique<D>);

the idea that's been floated to have dyn patterns to downcast trait objects.

If dyn were trait-like, downcasting a dyn could maybe be a trait method?

[petrochenkov]

@nikomatsakis

Can you say a bit more about these ambiguities?

Parentheses on the first bound in a trait object, for example, especially if the first bound is a lifetime.

((...(u8,)...)) + Trait // Not a legal type
((...('a)...)) + Trait // Legal type
((...('a)...)) // Not a legal type

We start parsing this as a type and only after burrowing deep enough into the parens we realize this is actually a bound and not a type, and we still don't know if this bound represents a legal type or not until we parse all the closing parentheses.

Can it be disambiguated in principle? Yes.
Does it worth it? Not at all. (Especially with a manually written parser.)

---

I wish bound lists used some other separator and not + so people wouldn't come up with silly ideas of placing parentheses in there.
Nobody asks for parentheses in function parameters, for example:

fn f((a: u8, b: u16), c: u32) {} // Such a beautiful sight

---

If parentheses in impl / dyn will need to be supported in the future, I'd go with simply

DYN_TRAIT_SYNTAX = `dyn` BOUNDS | `dyn` `(` BOUNDS `)` 

without changing grammar for bounds in other contexts:

fn f<T: (A + B)>() {} // Still an error

[petrochenkov]

@stuhood

AFAIK, parentheses are not used anywhere else to group types or trait bounds (except perhaps accidentally, here: rust-lang/rust#39318 ?), whereas angle brackets are.

They are used, they are just not needed very often, for example:

&(Write + Send)

I see mandatory angle brackets as an analogue of Alternative 3 (too noisy) and optional angle brackets as an analogue of dyn (BOUNDS) extension to Alternative 2 (possible, but not technically necessary extension), except for the compatibility issue.

@cramertj

dyn(T) is only valid in expression position, while dyn is already valid in type position.

What if dyn is a fn-like trait? :)
Fn(T) -> U => dyn(T) -> U.
There's even a test ensuring that this is not broken.
Fortunately, such traits are unstable and nobody in the right mind would use Fn as dyn; to make the syntax usable on stable.
I think we should actually break it to reserve for future use, the sooner the better.

@scottmcm

#[allow(non_camel_case_types)] type dyn<#[allow(the_trait_as_type_lint)] T: ?Sized> = T;

This is great 😄
This is still equivalent to Alternative 3 "Mandatory parentheses" though (too noisy).
(Also needs a lint or a magic trait bound to reject non-traits in dyn.)

[stuhood]

@petrochenkov : Thank you for the thorough response!

I agree that Alternative 3 is noisier, although "too noisy" is subjective, and whether the alignment with similar concepts (dyn is type-like/trait-like) is worth paying that cost is subjective as well.

If impl and dyn are inherently type-like/trait-like (rather than necessarily behaving like modifiers), then paying the noisiness-cost to align them with their closest conceptual neighbors could make sense. Unknown.

Thanks again.

[bstrie]

@cramertj

While I would prefer that we end up with a grammar that supports &dyn (Foo + Bar) (like the one @nikomatsakis proposed above), we can backwards-compatibly transition to that syntax from the proposed Alternative 2 (low-priority binding).

I strongly object to this rationale for triggering a final comment period. Why would we choose to accept this now only to force our users to suffer a transition sometime in the future? I agree with the apparent consensus that impl (Foo + Bar) is far preferable to (impl Foo + Bar), so for what reason are we rushing ahead with this proposal instead?

[cramertj]

@bstrie Everyone on the lang team is in consensus that we should include &impl (Foo + Bar). I'm very interested in seeing that it gets added ASAP, but I don't want it to be "the thing" that keeps us from stabilizing impl Trait this cycle. If @petrochenkov Is willing to amend the RFC and his implementation PR to support it, I'd be fully in support of that decision. Otherwise, I will plan to make a PR after this RFC lands to make the change.

[petrochenkov]

@cramertj

If @petrochenkov Is willing to amend the RFC

I won't amend the RFC because it's a completely unnecessary feature complicating the grammar.
(I can implement it though, so it would't end up as a subtly buggy completely unnecessary feature complicating the grammar.)

[cramertj]

@petrochenkov

I can implement it though, so it would't end up as a subtly buggy completely unnecessary feature complicating the grammar.

SGTM 😄

[rfcbot]

The final comment period is now complete.

[nikomatsakis]

So, during implementation, some further questions arose. The TL;DR is that we feel like we ought to be somewhat more conservative than this RFC suggests, and in particular try to make it an error to use a dyn A + B or impl A + B expression in a place where there is ambiguity. I am going to copy and paste [my original comment]rust-lang/rust#45294 (comment)):

---

Let me try to clarify what I think we want. We already have two grammatical precedence levels for types. I'll them T0 and T1. T0 includes all types, including the "n-ary" operators like dyn A + B and A + B. T1 excludes those n-ary operators, and is used primarily for & types (which take a T1 argument).

Currently, in the parser, if we are parsing a T1 type and we see a + we stop, leaving it to be consumed from some enclosing context.

I think that we want to have a rule such that T1 = dyn Identifier parses, but if we see a + after that, we error out (in the parser). This means that &dyn A + B is a parse error, as is where F: Fn() -> dyn A + B, and (I imagine) dyn Fn() -> dyn A + B.

I think of this as being analogous to non-associativity: there are basically multiple interpretations of the + operator. It can be adding bounds to a type parameter in a where-clause list; it can be adding types in a (old-style, A+B) sum-type; and it can be adding bounds to a dyn or impl type.

In a fn item context, or in the T in Box, once we see dyn or impl, there is only one valid interpretation (since we are not in a where clause list) -- . These are exactly the cases (I believe) where we use the grammatical nonterminal T0. We can parse eagerly there.

In other contexts, there are multiple contending interpretations. We are aiming not to choose between them (hence "non-associative").

[bstrie]

If the concern is getting impl Trait stabilized ASAP (which I empathize with), then I agree that it's far better to take the conservative path and simply forbid the ambiguous case rather than to stabilize a short-lived hack. If it really turns out to be such a problem in practice then that also implies that far more people would otherwise be using the temporary bad syntax, which further implies a much more painful transition than expected.

[nikomatsakis]

@bstrie

If the concern is getting impl Trait stabilized ASAP (which I empathize with), then I agree that it's far better to take the conservative path and simply forbid the ambiguous case rather than to stabilize a short-lived hack.

That is the priority. That said, it's unclear that we would ever want to permit

where F: Fn() -> impl Send + Sync

since it seems like it will be always be somewhat unclear what it means, leading to style guides that recommend using parentheses to clarify. In that case, perhaps we should just make it an error and hence require you to rewrite to one of the following:

// unambigious: Sync applies to `F`
where F: Sync + Fn() -> impl Send

// unambiguous: Sync applies to the returned type
where F: Fn() -> (impl Send + Sync)

// these variants are also unambiguous, but require parser to support
// parens in lists of bounds. I personally think we should but we don't
// today and let's leave that aside for now:
where F: Fn() -> impl (Send + Sync) 
where F: (Fn() -> impl Send + Sync) 
where F: (Fn() -> impl Send) + Sync

I could go either way on this point. As @petrochenkov put it at some point, nobody really knows the precedence of most binary operators but we still have precedence rules rather than requiring parens to clarify; so the precedent (pun intended) is to define some rule and stick with it. But I think @joshtriplett's feeling is that we don't have to repeat the mistakes of the past here.

In any case, it's a conservative choice to error out, leaves us room to revisit in the future, so it's obviously the way to go imo.

[stuhood]

Given the challenges encountered, is it worth re-evaluating whether Alternative 3 with required brackets/parens would be a better path forward? Using keywords in a type position seems very error prone, and none of the examples in #2250 (comment) (even the unambiguous ones) seem as readable as impl<Send> or impl<Send + Sync> would be.

As much as we want this feature, it really seems like caution is in order.

[nikomatsakis]

@stuhood

It was certainly discussed. I think that nobody found a syntax we were happy with. For example, I was considering impl<Trait> and dyn<Trait> (mandatory <>) for a while, but the "counter example" was basically that things like Box<Write + Send> now becomes Box<dyn<Write + Send>> and that felt overall too heavy.

This isn't something I really know how to "rigorously evaluate" -- of course, technically, we could always move towards such a syntax in the future.

[stuhood]

was basically that things like Box<Write + Send> now becomes Box<dyn<Write + Send>> and that felt overall too heavy.

I agree that it is heavier, but... only by one character overall. The result of both dyn and impl is in essence a synthetic type, parameterized on some bounds... so it really does seem like a useful alignment to have its syntax evoke a synthetic type.

[scottmcm]

we don't have to repeat the mistakes of the past here

And with epochs we could plausibly even fix them in new code, making confusing cases non-associative.

[petrochenkov]

Note: for trait objects without dyn/impl prefix "refuse to disambiguate" (#2250 (comment)) is not a viable choice due to examples like this:

// ambiguous, `(x as usize) + y` or `x as (usize + y)`,
// but code like this is very common and we must accept it (with low priority `+`)
x as usize + y

[Centril]

Huzzah! The RFC is merged!

Tracking issue: rust-lang/rust#34511