Accept arbitrary expressions in key-value attributes at parse time

[petrochenkov]

Continuation of #77271.

We now support arbitrary expressions in values of key-value attributes at parse time.

#[my_attr = EXPR]

Previously only unsuffixed literals and interpolated expressions ($expr) were accepted.

There are two immediate motivational cases for this:

• External doc strings (#[doc = include_str!("my_doc.md")], eliminating the need in Tracking issue: rfc 1990 - add external doc attribute to rustc #44732) and expanding macros in this position in general. Currently such macro expansions are supported in this position in interpolated $exprs (the #[doc = $doc] idiom).
• Paths (#[namespace = foo::bar] extern "C++" { ... }) like proposed in Allow path as value in name-value attribute #76734.

If the attribute in question survives expansion, then the value is still restricted to unsuffixed literals by a semantic check.
This restriction doesn't prevent the use cases listed above, so this PR keeps it in place for now.

Closes #52607.
Previous attempt - #67121.
Some more detailed write up on internals - https://internals.rust-lang.org/t/macro-expansion-points-in-attributes/11455.
Tracking issue - #78835.

[rust-highfive]

r? @davidtwco

(rust_highfive has picked a reviewer for you, use r? to override)

[SoniEx2]

does this solve the translatable docs problem?

[petrochenkov]

@SoniEx2
No? But it may provide tools for a higher-level system to solve it.
Similarly to #44732, this is a low-level feature, in theory in can be used to document crate with different cfgs and include different documentation depending on that cfg.

[SoniEx2]

#[doc = include_str!("my_doc.md")] looks nice but you'd have to be able to supply the path based on a build setting (uhh like #[doc = include_str!(concat!(env!("doc_lang"), "my_doc.md"))] maybe?)

[petrochenkov]

#[doc = include_str!(concat!(env!("doc_lang"), "my_doc.md"))]

This should already work, I will add a test.

[davidtwco]

I'm not exceptionally confident in my understanding of this, but it looks broadly correct to me. r=me if you don't have other changes to make or are fine with just my review.

[m-ou-se]

This change will allow for finally cleaning up the doc_comment hacks in core. 😃 See #79150.

[jyn514]

External doc strings (#[doc = include_str!("my_doc.md")], eliminating the need in #44732) and expanding macros in this position in general. Currently such macro expansions are supported in this position in interpolated $exprs (the #[doc = $doc] idiom).

Before recommending this, someone should test how this interacts with unindent_comments (#78400).

[jyn514]

(but to be clear: I'm strongly in favor of this, I'm always +1 for having features in rustc instead of special-casing rustdoc.)

[m-ou-se]

Before recommending this, someone should test how this interacts with unindent_comments

@jyn514 Looks like that works just fine:

#![feature(extended_key_value_attributes)]

const _: &[u8] = compile_time_run::run_command!("sh", "-c", "echo '    .second(\"hello\")' > somefile");

/// You are supposed to call it like this
///
/// # Example
///
/// ```
/// let example = Example::new()
///     .first("hello")
#[cfg_attr(not(feature = "one"), doc = include_str!("../somefile"))]
///     .third("hello")
///     .build();
/// ```
pub struct Example {}

This results in:

[petrochenkov]

To lang team: this PR has lower priority than #79078.

[Mark-Simulacrum]

We discussed this during the lang team meeting and felt that if this is just adding unstable surface area we can proceed here without a T-lang FCP. If there is something we're stabilizing then we'd like that called out more explicitly though and we can take another look :)

We also had a question around the hygiene behavior of the passed tokens -- do we handle those equivalently to other macro input? If so then no concerns.

[m-ou-se]

The change this PR makes in compiler/rustc_parse/src/parser/mod.rs somehow makes rustfmt apply this formatting:

-macro foo($type_name: ident, $docs: expr) {
+macro foo($type_name:ident, $docs:expr) {

Without the parser/mod.rs changes of this PR, rustfmt leaves this line untouched and does not remove the spaces.

[m-ou-se]

Ah, apparently rustfmt doesn't format the macro parameters if it cannot format/parse the body. The full test case is:

macro foo($type_name: ident, $docs: expr) {
    #[allow(non_camel_case_types)]
    #[doc=$docs]
    #[derive(Debug, Clone, Copy)]
    pub struct $type_name;
}

The body contained a #[doc=$docs] which it couldn't understand before, but can be handled fine now with this change, causing it to no longer skip formatting of the entire macro definition.

Mystery solved.

[pthariensflame]

I passed over this issue the first time through the list, but…does this need relnotes? On a closer look it seems like people would want to know about new capabilities for proc macros.

[petrochenkov]

@pthariensflame
This is still only available on nightly, relnotes for new features are usually applied to stabilization PRs.

[pthariensflame]

Understood!