Reimplement the lexing of c"…" string literals with backward compatibility in mind

[fmease]

The lexing of c"…" string literals as implemented in #108801 had to be reverted in #113334 since it isn't backward compatible.
It breaks code that uses a pre-2021 edition (i.e. the 2015 or the 2018 edition).
See #113235 (comment) for details.

@rustbot label C-bug T-compiler F-c_str_literals

[SUPERCILEX]

Could c strings be added to 2021 editions+ only?

[dead-claudia]

Could c strings be added to 2021 editions+ only?

@SUPERCILEX On paper, this sounds like a good idea, but in practice, one does not merely do that. Proc macros could use an older version while the source using it could use a newer version.

In this case, the newer source code, when parsed, will yield one AST, while the proc macro may break due to those unfamiliar features.

I'm not familiar enough with the compiler to know how it addresses it, so don't take that as authoritative.

[fee1-dead]

I think the plan is to only support this on 2021+ editions while treating it as an unknown prefix in 2018 or older, which would allow compatibility with macros that want c"xx" to be treated as two separate tokens.

As for macros across crates, I think it wouldn't be an issue? There should already be mechanisms to prevent this sort of thing from happening. We should definitely add a test case to test this behavior.

[nagisa]

We do already recombine some tokens in the parser (I believe & and & is an example of this? Or maybe the other way around where && is contextually interpreted as a double reference? I don’t remember exact details.)

Seems like it wouldn’t be terribly hard to do it for c"" either, as long as care is taken to ensure that no whitespace is between the two tokens.

(edit: this is in response to #113235 (comment))

[fmease]

Yup, there's break_and_eat (and its various convenience methods). However, as already mentioned, every user of the lexer (ra-ap-rustc_lexer to be more precise) would need to remember this forgettable step.

FYI, there's already a PR that would fix this issue (by mainly modifying the lexer, not the parser; at the time of this writing), namely #113476 (although it doesn't mention this issue).

[dead-claudia]

We do already recombine some tokens in the parser (I believe & and & is an example of this? Or maybe the other way around where && is contextually interpreted as a double reference? I don’t remember exact details.)

IIRC from the spec it's && being contextually interpreted as a double reference much like >> as closing two generic types.