RFC: Procedural macros 1.1

[alexcrichton]

Extract a very small sliver of today's procedural macro system in the compiler,
just enough to get basic features like custom derive working, to have an
eventually stable API. Ensure that these features will not pose a maintenance
burden on the compiler but also don't try to provide enough features for the
"perfect macro system" at the same time. Overall, this should be considered an
incremental step towards an official "plugins 2.0".

Rendered

[aturon]

Suggestion: this might want to be titled Plugins 1.1, as it doesn't really impact the "macro" system (if you think of that as macro_rules, which I think many do).

[aturon]

cc @jimmycuadra

[eddyb]

@aturon I believe this is because @nrc wanted to avoid "plugins" for "procedural macros".

[alexcrichton]

@aturon yeah it's true that this only really affects what we call plugins today, but I believe that @nrc wanted to start avoiding the term "plugin" as everything will eventually be a macro in the "macros 2.0" world.

[eddyb]

This name is unfortunate, couldn't we use libmacro?

[alexcrichton]

This is currently done because macro is a keyword, and this doesn't propose changing that just yet. I was thinking that we could perhaps wholesale rename the crate one day as well if macro becomes an available name.

[MovingtoMars]

How about libmacros? This follows the approach that Go takes, ie the strings package.

[aturon]

Thanks @alexcrichton for the awesome writeup! We've been working for a while on some way enable usage of Serde and similar libraries on stable in the near term, without massive stabilization. For example, there's a code generation proposal (which sits entirely on the side from the macro/plugin system), as well as attempts to find a very small increment toward "macros 2.0". But this is the first proposal that can be implemented immediately, has a clear path toward speedy stabilization, and imposes virtually no maintenance burden. I'm incredibly excited to be so close to a solution for the biggest cause of nightly usage today.

One thing that wasn't totally obvious to me, given that the macros 2.0 plan is a bit out of cache: what, if any, parts of what you propose to stabilize here would end up being deprecated once macros 2.0 has fully landed?

(To be clear, I have zero problem stabilizing the small slice of functionality you've outlined here in the near term even if in the long term we plan to deprecate it.)

[eddyb]

I would've expected at least #[macro_use] to be necessary.

[alexcrichton]

cc @nrc, I think that importing custom derive annotations was a somewhat unspecified extension of "macros 2.0" right now, so to me it seems somewhat up in the air about whether we'd require that or not.

I don't mind one way or another, but I'm curious what @nrc thinks as well.

Eventually of course custom-derive will be imported through the normal module system I believe, so this is purely just a temporary stopgap.

[eddyb]

The problem is that if you do anything by default now, you'll need to opt out to be able to use the normal module system.

[alexcrichton]

Aha, an excellent point! You've convinced me, I'll add it.

[alexcrichton]

@aturon

what, if any, parts of what you propose to stabilize here would end up being deprecated once macros 2.0 has fully landed?

This is a good question! Of the pieces here, what I would imagine is:

• If we decide to rename anything, we'd accumulate deprecations for old names.
• The behavior of "go to a string, parse, then go back to a token stream" will likely be deprecated in favor of directly manipulating the token stream.
• The definition form of custom #[derive] will likely change and the proposed form here would be deprecated. We likely eventually want one that's imported via use (somehow).

Other than that though I imagine these pieces to survive stabilization:

• The concept of a "macro" crate type
• The concept of a "macro crate" provided in the standard distribution, providing all details necessary to talk to the compiler.
• The ability to inform Cargo of crates which are macro crates
• Cargo's behavior of compiling macro crates for the host and passing them to the compiler for the next unit of compilation.
• Transformers like #[derive] operating over token streams and allowing customization via attributes that are stripped during the expansion process

[strega-nil]

First, I don't think this RFC has gone through enough design, and fast-tracking it just to get people off of nightly is not a good idea, IMO. However, I am in favor of eventually (soon-ish) getting this sliver.

Some of my specific issues:

First: librustc_macro is a terrible stable name! My vote's in for libmacro. It's not a rustc-specific thing, it's an all-of-Rust thing!

Second: Context seems like an unnecessary implementation detail, and, if they're used the same way as in the original proposal "Procedural macros should report errors, warnings, etc. via the MacroContext [Context]."... why. We already have an existing error reporting mechanism, with Results. This new way seems very un-Rustic.

Third: I don't like how macros are defined. I'd rather they be defined with something like macro instead, especially as we eventually are going to allow proc macros and normal stuff in the same library.

[eddyb]

@ubsan I don't like Context either, but Result won't cut it, macros 2.0 will eventually want a full-blown diagnostics API, where you can emit multiple warnings and non-fatal errors.
If the necessary information is placed in TLS, the API would work without a Context type.

[sfackler]

@ubsan Result is an all-or-nothing thing. In the compilation process, it is very common to be creating many diagnostics during one compilation pass. A compilation error is not necessarily fatal - you (within reason) want to continue as long as possible to avoid forcing the user to go through a recompilation cycle for every error when many could all be reported at once. In addition, warnings, notes, etc are certainly not fatal.

[arielb1]

I think librust_macro could be a fine name.

On the integration level: plugins imply some form of host ABI. Stable plugins mean a stable HOSTRUSTC. I think we can just leave HOSTRUSTC as an implementation-defined thing, but we must be careful not to change its definition on existing platforms.

[strega-nil]

@sfackler makes sense. I still don't think Context is the best, though.

[sfackler]

Do you have any suggestions for a better replacement?

[strega-nil]

@sfackler No, but I think @eddyb does :P

[arielb1]

On naming: I see librust_macro as an ordinary plugin library, and rustc_macro_derive as an exposed proc macro that (unstable) compiler plugins should eventually be able to reproduce (BTW, what is the nomenclature wrt. proc macro vs. plugin vs. compiler extension?)

rustc_macro_derive is a bad name, but we still don't have modular macros. I think pseudo-modular #[WHATEVER_CRATE_NAME_WE_PICK_derive] would work.

As for the API: maybe have #[CRATE_derive_v0] for the Context-free version?

[sfackler]

How does shoving Context into TLS hide implementation details?

[eddyb]

@sfackler The TLS slot wouldn't be exposed, the APIs would just not take any context.

[sfackler]

@ubsan's concern with Context seemed to be that it was an implementation detail, and I'm confused as to how moving all of the methods on Context to free functions hides any details of the implementation. It seems totally isomorphic in that regard.

[arielb1]

Yeah. A Context parameter is easy to ignore and hard to add later.

[nrc]

I'm in favour of this. I'm not sure how I feel about the namespacing issue, I'd like to see how this looks when implemented.

[nikomatsakis]

Huzzah! The @rust-lang/lang team has decided to accept this RFC. We decided to ask @alexcrichton to move some of the detailed matters (e.g., prefixes, etc) into unresolved questions to be addressed during stabilization. Thanks everyone!

[nikomatsakis]

Tracking issue: rust-lang/rust#35900

If you'd like to keep following the development of this feature, please subscribe to that issue, thanks! :)

[llogiq]

I still miss my comment about the inferior traceability of code through macros 1.1 as opposed to the current systems. Could we at least add it to the drawbacks section?

[nikomatsakis]

@llogiq I pushed some text about spans

[llogiq]

Ok then. I'll keep an eye on the implementation. 😉