Remove TokenStream::JointTree.

[nnethercote]

This is done by adding a new IsJoint field to TokenStream::Tree,
which simplifies a lot of match statements. And likewise for
CursorKind.

The commit also adds a new method TokenTree:stream() which can replace
a choice between .into() and .joint().

[rust-highfive]

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

[nnethercote]

After these commits, TokenStream looks like this:

pub enum TokenStream {
    Empty,
    Tree(TokenTree, IsJoint),
    Stream(Lrc<Vec<TokenStream>>),
}

I wonder if the recursion via the Stream variant is necessary. Perhaps it could instead be this:

    Stream(Lrc<Vec<TokenTree>>),

To break it down further, perhaps TokenStream is conceptually just this:

Vec<(TokenTree, IsJoint)>

(Possibly with the use of Lrc and Empty/singleton variants for performance
reasons.)

There's a comment at the top of tokenstream.rs that suggests this is true:

TokenStreams represent syntactic objects before they are converted into
ASTs. A TokenStream is, roughly speaking, a sequence (eg stream) of
TokenTrees, which are themselves a single Token or a Delimited
subsequence of tokens.

[petrochenkov]

perhaps TokenStream is conceptually just this: Vec<(TokenTree, IsJoint)>

More that that, IsJoint is not even a property of TokenTree, but only punctuation tokens, so TokenStream is conceptually just Vec<TokenTree>.
See stable public interfaces in proc_macro for what "canonical" token streams look like in terms of interface.
In general, it would be nice to reuse types from proc_macro in the compiler if possible, or at least align with them.

[nnethercote]

@petrochenkov: I tried looking at those public interfaces but I'm having trouble understanding them, particularly the bridge and define_handles! stuff. Can you give me some hints? Thanks.

[petrochenkov]

Oh, there's the bridge thing now.
Then my first hint would be to look at the snapshot of libproc_macro preceding merge of #49219.

[petrochenkov]

(libproc_macro was much more straightforward and closer to its "conceptual" form before all that machinery for client-server communication was added.)

[eddyb]

You can still look at the public API, IMO, it's not like the old code was self-contained.

Group, for example, used syntax::tokenstream::TokenStream before (back then wrapped by proc_macro::TokenStream), and it still uses that today, albeit the actual definition of Group now lives in syntax_ext::proc_macro_server (as does the rest of the compiler side of the implementation).

[petrochenkov]

r=me with #56964 (comment) addressed.
@bors delegate+

[bors]

✌️ @nnethercote can now approve this pull request

[nnethercote]

@bors r=petrochenkov

[bors]

📌 Commit e7c5146 has been approved by petrochenkov

[nnethercote]

perhaps TokenStream is conceptually just this: Vec<(TokenTree, IsJoint)>

More that that, IsJoint is not even a property of TokenTree, but only punctuation tokens, so TokenStream is conceptually just Vec<TokenTree>.

Perhaps it's more like this: (Vec<TokenTree>, IsJoint). The IsJoint applies only to the final TokenTree, and should be used if the Vec gets extended. (All the TokenTrees prior to the last one should have already been joined if necessary.)

At least, that's the case for any TokenStream that may still be extended. A TokenStream that is known to be complete can drop the IsJoint.