Grouping hygiene broken when copying TokenTrees to a new TokenStream

[de-vri-es]

We encountered an issue with the assert2 crate where grouping hygiene did not survive a proc macro.

use assert2::assert;

fn main() {
    macro_rules! bla {
        ($e:expr) => { assert!($e * $e == 4); }
    }
    bla!(1 + 1); // Fails, compares 1 + 1 * 1 + 1 == 4, should be (1 + 1) * (1 + 1) == 4
}

See de-vri-es/assert2-rs#14 for the original issue.

---

We digged a little further and came up with a minimal example here: de-vri-es/proc-macro-grouping-hygiene-bug. It contains two proc macros, one that just clones a TokenStream, and one that creates a new one from the input:

use proc_macro::TokenStream;

#[proc_macro]
pub fn identity(tokens: TokenStream) -> TokenStream {
	tokens.clone()
}

#[proc_macro]
pub fn manual_identity(tokens: TokenStream) -> TokenStream {
	let mut output = TokenStream::new();
	output.extend(tokens);
	output
}

Then we run these tests:

use proc_macro_grouping_hygiene_bug::*;

macro_rules! foo1 {
	($expr:expr) => { identity!($expr * $expr) };
}

macro_rules! foo2 {
	($expr:expr) => { manual_identity!($expr * $expr) };
}

#[test]
fn test_identity() {
	assert_eq!(foo1!(1 + 1), 4); // OK, gives 4.
	assert_eq!(foo2!(1 + 1), 4); // FAILS, gives 3.
}

We expected both assertions to pass, but the second one fails. The same problem occurs when converting a syn::Expr into a token stream again (which is how we originally found it).

This seems like a very serious issue to me.

[petrochenkov]

This is an instance of #67062.

[de-vri-es]

Ah, you are correct. Should I close this issue?

[petrochenkov]

Yes, I've left a comment on that issue, closing.