Clarify "string continue" for (byte) string literals

[LukasKalbertodt]

The previous version just said "whitespace at the beginning of the next
line is ignored", but that is not quite correct. Currently, exactly four
characters are ignored in that position. This is different from the
definition of char::is_whitespace and char::is_ascii_whitespace.

Additionally "at the beginning of the next line" is confusing as
additional \n are also ignored.

See the lexer implementation:

https://github.com/rust-lang/rust/blob/595088d602049d821bf9a217f2d79aea40715208/compiler/rustc_lexer/src/unescape.rs#L281-L287

https://github.com/rust-lang/rust/blob/595088d602049d821bf9a217f2d79aea40715208/compiler/rustc_lexer/src/unescape.rs#L300-L307

[ehuss]

Thanks for the PR! This behavior is a little surprising to me, since that is not how it works in other languages (like Python). It looks like this has been in Rust ever since the beginning (rust-lang/rust@df9cf0b), and I'm guessing the initial docs (rust-lang/rust#22748) were just confused?

[LukasKalbertodt]

@ehuss It is a bit surprising to me too. Do you suspect that this might still change in the Rust compiler? Would you suggest not merging this PR until we are sure that this is the behavior we want in the compiler?

[ehuss]

Hm, yea, I'm not sure. I asked on Zulip if anyone has any input.

Usually we lean towards documenting what rustc does, but in the cases of bugs or unintended behavior, it can be difficult to know how to proceed. I would suspect this should be merged as-is, since it has had this behavior for 10 years, and breaking language changes are difficult (and this probably isn't important enough to put any time and energy into).

We'll see if anyone chimes in on Zulip.

[joshtriplett]

Skipping multiple newlines in a row seems like really surprising behavior in the Rust compiler. I don't know if we can fix that easily, but we could at least warn about it, and perhaps improve the behavior over an edition.

[joshtriplett]

We discussed this in the @rust-lang/lang meeting today, and we'd prefer to actually have the Rust compiler warn if it encounters this case, to avoid confusion.

We will leave it up to the reference team how to flag such things; if you'd like to merge this as-is that's fine, or if you'd like to flag this somehow as "potentially surprising/undesirable behavior that the compiler may warn about" that'd be fine too. How, in general, would the reference team like to handle "this is the current behavior but it's weird and we might want to address that"?

Meanwhile, we'd love to see a warning added to the compiler. Once that warning exists, we could consider changing this in an edition, or potentially use a crater run (with the warning made into an error) to determine if this could change in all versions of Rust to minimize confusion.

[joshtriplett]

See rust-lang/rust#87319, which requests a warning (and later error) for this.

Also see rust-lang/rust#87318, which flags a related issue @scottmcm noticed while we poked at this.

[LukasKalbertodt]

What's the status here? Rust now warns about this confusing behavior and there are maybe (?) plans to change the behavior in an upcoming edition. Should this PR still be merged as it described the current behavior? Should I add a note about "this might change later because it's confusing"?

[ehuss]

Sorry for the delay. Yea, can you add a small note? Maybe something like:

Note: For clarity, it is recommended to avoid using multiple line breaks with line continuations. rustc will produce a warning in that situation to point out the possible confusion.

Or feel free to rewrite that from scratch. I'm not sure how to say that succinctly and clearly.

[LukasKalbertodt]

Pushed a commit with a note. I came up with some text on my own now, but if you think it has problems, I can easily just use your text as well.

[ehuss]

Looks good, thanks!