Macro argument scopes (specifically, Code objects), and augmentation libraries

[jakemac53]

We want to be able to model the Macro output in terms of an augmentation library, and in general we want the code output by macros to not have special powers.

At the same time, we allow Code arguments to macro constructors, and we want the scope of that code to be essentially the same scope as the macro annotation. However, we also possibly want to be able to refer to local variables within those expressions.

Example:

abstract class Thing {
  int get x;
  int get y;
  
  @MemoizedVar(x + y) // `x` and `y` here are local variables. 
  int get z;
}

But what if z is a function, and it has parameters named x or y? We need to clarify whether the macro arguments scope is that of the body of the declaration it annotates, or the class, or something else more custom.

Some factors to be taken into consideration:

• It might be useful for a macro arguments to be able to access the arguments to a function.
• A macro on z can emit code for declarations outside the scope of the body of z (for instance it can add fields and methods to the surrounding class and/or library, or add whole new classes).
• Macros may annotate static methods/fields on classes.

There are a few possible solutions I have been considering to this:

Solution 1: No access to local variables or params in macro argument expressions

The scope is the lexical scope of the body of the class, static members of the class are visible (possibly without being qualified by ClassName.?), but no local variables.

This could be modeled by generating a static getter next to the original annotated declaration:

library augment 'original.dart';

augment class Thing {
  static  get _$someGeneratedName => x + y;
}

Pros

• Easiest to define.
• Valid anywhere in the library (may need some munging to add the qualified ClassName. in front, or just require it).

Cons

• Very limiting, can't access parameters or local variables.

Solution 2: Its own lexical scope, as if the expression was inside the body of a new declaration

We could model this in the same way as Solution 1, except the getter wouldn't always be static - it would only be static if the annotated declaration was static.

Pros

• I think this mostly behaves how users would expect, the scope is essentially the same as the annotated declaration.
• Allows access to local variables.

Cons

• Possibly a bit magical, might be unexpected that expressions in macro annotations on static fields have a different scope.
• These expressions cannot be used in parent scopes (unless the annotated declaration was static).

Solution 3: The scope is the same as the scope at the top of the body of the annotated declaration.

We could again model this in essentially same way, except with a local function at the top of the body of the annotated declaration.

library augment 'original.dart';

augment class Thing {
  int get z {
    // Note that the `Code` object in this case would have to be a function invocation not a reference to a getter.
    _$someGeneratedName() => x + y;
    // Rest of the generated code here (should contain an `augment super`).
  }
}

Pros

• Has access to parameters as well as local variables, could be useful.

Cons

• Breaks lexical scoping, the expression looks like normal code and lives outside the body of the declaration, so it likely isn't intuitive that the parameters are available.
• The expression is only usable within the annotated declaration (can't be used even on other declarations in the class).

[jakemac53]

My personal opinion at this time is to go with solution 2. I think its the least surprising, pretty easy to define and model in terms of augmentation libraries, and still provides a good amount of functionality.

[munificent]

Do we only support macro arguments that are used as expressions? I can imagine a macro like:

@addIntField(foo)
class C {
}

Which expands to:

class C {
  int foo;
}

In this case, the macro argument isn't treat as an expression, it's treated as an identifier that binds a name. That suggests another potential solution:

Solution 3: Macro arguments are unresolved

They are treated as pure syntax. It is only the resulting code where they are interpolated that gets resolved. In fact, the same macro argument could be interpolated into multiple different points in the generated code such that each resolves in different ways.

I don't know if this is the best solution, but I think it's a fairly simple one that's implementable.

[jakemac53]

We could make them unresolved too, but I think its a really nice affordance to have them be resolved. It means you can get code completion and everything else within those expressions.

In this case, the macro argument isn't treat as an expression, it's treated as an identifier that binds a name. That suggests another potential solution:

I think for this case I would probably just ask for a String that is the name? Most of the interesting cases will be more complex expressions, where code completion and error checking etc would be nice.

[munificent]

I think for this case I would probably just ask for a String that is the name?

We could, but that feels weird to me. The macro argument is already lived from being an expression to being a metaexpression that represents an identifier. Why the should macro user have to manually double lift the identifier into a string? If the macro needs an identifier which it will then use to declare something, isn't the natural syntax for that argument... an identifier?

[Levi-Lesches]

I'd argue that by passing an identifier into a macro, I'd expect the identifier to resolve and pass the value instead to the macro. This would make using and writing macros that take parameters more intuitive, since that's how constructors already work. If I wanted to pass in a literal name, it would be pretty obvious to me that I should be using a String, just like I would when passing literal text to a UI framework.

[jakemac53]

@Levi-Lesches ya that is basically my intuition. If I wanted to pass a variable definition as Dart code I would probably expect it to be an entire declaration anyways (final int foo or var foo) etc.

We had discussed only supporting expressions, which I still think is the right choice, but i don't see that codified in the spec yet so I will send a PR for that as well.

@tatumizer please read https://github.com/dart-lang/language/blob/master/working/macros/feature-specification.md#arguments

[TimWhiting]

Personally I think both unresolved and resolved syntax are useful. I appreciate the code-completion of the latter and the flexibility of the former.

Example:

macro MyMacro implements ... {
  MyMacro(LexedTokens tokens, {required Expression initializer});

  /// Implementation uses the resolved expression to get the type parameter for the field
  /// but uses an unresolved identifier for the first parameter.
}


/// Usage

late int x;
late int y;

@addLateField(foo, initializer: x + y);
class C {
}

/// expands to 
augment class C {
  late int foo = x + y;
}

The tokens for unresolved code cannot include a comma notably.

It encourages macro authors to use resolved subclasses of Code for the most part (because of being able to resolve types and have code completion for users), while allowing for simple usage for example of unresolved identifiers.

It is up to the author of the macro to return / throw an error if the first argument is not a simple identifier. But this way you don't have to worry about resolving identifiers in scope when they aren't intended to be in scope to begin with.

[jakemac53]

Hi all, this issue got very derailed, so I have hidden a bunch of comments (including my own :D ) that aren't related to the scope of expressions passed to macros as code.

See #1989 for some discussion around other syntax ideas for these code block arguments, or file new issues if you want to discuss their general usefulness or other topics related to that, thanks!

See also jakemac53/macro_prototype#26 which was one of the original issues discussing why this is a useful feature in general.

[jakemac53]

We discussed essentially this again recently, I think we are going to stick with Option 1. Essentially macro applications should behave exactly like annotations do, they will not have a special scope.

It just gets really weird to look at otherwise, and we have enough other questions to answer regarding scoping/identifiers in macros as it is. Further complicating the matter doesn't seem worthwhile (plus the suggestions here would break the expected and intuitive lexical scoping that most dart users love).