macros: preserve none-delimiters in function body #3583
Conversation
Kestrer
force-pushed
the
preserve-none-delimiters
branch
from
8c4a931
to
8c2a3b0
Compare
When user-generated token streams are returned from proc macros, all none-delimited groups are flattened. This can cause compilation errors as shown in the issue below. By instead passing the function body token stream straight through from the input of the proc macro to its output, it does not get deconstructed and reconstructed, and so the none-delimited groups are not lost. Fixes: tokio-rs#3579
Kestrer
force-pushed
the
preserve-none-delimiters
branch
from
8c2a3b0
to
41329d7
Compare
|
Well that's weird. It fails on 1.45.2 but not on 1.47.0, and only when |
If this fix only works with the new compiler, I think we probably don't need to block this PR (because it doesn't work with older versions in either way). Or does this mean this PR introducing some new regression with the old compiler? (If so, we need to postpone until MSRV increases.) |
|
Oh yes, that's true. Although I'd have to figure out how to gate a test's compilation based on Rustc version so CI doesn't break. If that's not trivial it might be easier to just wait until the bump, since it's not urgent. |
|
Rust 1.47 is now more than six months old. |
|
|
|
@taiki-e The problem is not |
|
Seems this is not a bug of syn as @Kestrer said, but None-delimited token spans are actually preserved. |
There was a problem hiding this comment.
The implementation looks good to me. That's a bit hacky, but there's no other workaround that looks good, and the problem is unlikely to be fixed on the rustc side soon.
I don't think that's true; the bug was reported before 1.49 existed. They wouldn't've reported it if it worked without the patch. But either way the behaviour I'm observing is that this workaround only worked in 1.47.0 and 1.48.0. Given that @Aaron1011 mentioned that the workaround working at all was only due to a hack, I think it's unlikely this will ever work again, so I'm going to close this PR. |
| // groups inside the function body; see <https://github.com/tokio-rs/tokio/issues/3579>. | ||
| let body = | ||
| proc_macro2::TokenStream::from(TokenStream::from(input_tokens.into_iter().last().unwrap())); | ||
|
|
||
| input.block = syn::parse_quote! { |
There was a problem hiding this comment.
The reason test failed after the master was merged is that body was converted to syn ast again here (confirmed locally: 38326a5)
rust-lang/rust#77135 was merged in 2020-10-14, 1.49 was released in 2020-12-31, #3579 was reported in 2021-03-05.
I think this PR's hack is still usable at this point (#3583 (comment)), but I think it's likely that it won't work in the future. |
Motivation
When user-generated token streams are returned from proc macros, all none-delimited groups are flattened. This can cause compilation errors as shown in the issue below.
Solution
By instead passing the function body token stream straight through from the input of the proc macro to its output, it does not get deconstructed and reconstructed, and so the none-delimited groups are not lost.
Fixes: #3579