Explicitly pass type parameters when delegating the method

[LukasKalbertodt]

#[auto_impl(&)]
trait Foo {
    fn foo<T>();
    fn bar<U>(&self);
}

Is expanded into:

impl<'a, V: 'a + Foo> Foo for &'a V {
    fn foo<T>() {
        V::foo()         // <-- cannot infer type for `T`
    }
    fn bar<U>(&self) {
        (**self).bar()   // <-- cannot infer type for `U`
    }
}

And results in two compiler errors. So we need to change the method body into an explicit call all of the time. In this case:

• V::foo::<T>(), and
• V::bar::<T>(*self)

---

I can mentor anyone interested in tackling this issue :)
Just ping me (via email, this issue, or in any other way)

Instructions: code generation of methods is done in gen_method_item in gen.rs. The important part for this issue are the last two arms of the last match statement in the function (SelfType::Value and SelfType::Ref | SelfType::Mut). These generate incorrect code. You can see the generated code in the quote! {} macro.

The most difficult part is probably to generate the list of generic parameters to call the other method. In the example above it's simply <T>, but it could be more complicated. In gen_method_item(), we have the variable sig which is a syn::MethodSig. We are interested in sig.decl.generics which stores the generics of the method we are generating. Sadly, we can't just use that: e.g. fn foo<T: Clone, U>() would have <T: Clone, U> as generics and we can't call foo::<T: Clone, U>(), but need to call foo::<T, U>(). So we might have to remove all bounds. But with some luck, we can use syn::Generics::split_for_impl. The second element of the returned tuple should be what we want. But we need to test that!

Finally, one or more compile-pass tests should be added which test this exact code.

If anything is unclear, just go ahead and ask!

[saresend]

Hey, is this issue still available?

[LukasKalbertodt]

@saresend To my knowledge, no one is working on this yet. So yep, it's available! So you want to work on this? :)

You can of course always ask questions in this issue, but you can also contact me via mail (address on my GitHub profile), via Discord (on the Rust server https://discord.gg/2cr8M4D) or via Telegram (username is the same as my GitHub username). I think that often, real time messaging or at least non-public messaging is easier and more effective, if you want to get quick help.

[saresend]

Yep, I'd be interested in taking this issue! Thanks, I'll be in touch!

[LukasKalbertodt]

Hi @saresend, I just wanted to check your status. Are you still working on this? But no worries: I don't want to rush you! If you have problems trying to tackle this issue, just let me know!

[saresend]

Hey! Yeah, I'll likely tackle it this weekend, I'm currently in university and so my availability is quite sporadic :P Expect to hear from me sometime tomorrow!

[LukasKalbertodt]

Fantastic! :)
(and yeah, who doesn't know those time problems...)

[saresend]

Hey, I realize that I really lack understanding of procedural macros and how they function in Rust. Do you know of any resources to get up to speed? Thanks!

[KodrAus]

👋 Hi @saresend! The new procedural macro work doesn't seem to have a lot of documentation just yet, I've set up a gitter channel we can use to help you get started.

[LukasKalbertodt]

@saresend No problem, that's why we offer this mentoring :)

There is actually a section in the book about proc macros. But let me quickly summarize it for you in context of this crate (more explanations never hurt, right?):

There are three different kinds of proc macros: custom derives (#[derive(Serialize)]), function-like macros (foo!()) and -- the one we're interested in -- custom attributes (#[auto_impl()]). All proc macros function very similarly. The creator of the proc macro creates a function that "defines" the proc macro. This function gets one or two TokenStreams and returns a TokenStream. The function needs to be annotated with #[proc_macro_derive(Serialize)], #[proc_macro] or #[proc_macro_attribute] for the three kinds of proc macros, respectively. You can see this function in this crate here:

#[proc_macro_attribute]
pub fn auto_impl(args: TokenStream, input: TokenStream) -> TokenStream {
    ... 
}

So what's a TokenStream? This is simply a list of tokens. But what's a token? It's a very simple unit of source code. This is usually the first step of all compilers: transform the list of characters (the source code) into a list of tokens. Roughly speaking, when you add whitespace everywhere in your code where you're allowed to, source_code.split_whitespace() is the list of tokens. An example:

trait Foo<T> {
    fn foo();
}

This results in the tokens:

• trait
• Foo
• <
• T
• >
• {
• fn
• ...

The type TokenStream is defined in the crate proc_macro, which -- like for example std -- comes with the compiler. You can see the documentation here. Note the IntoIter implementation. It is an iterator over TokenTrees. That's where the TokenStream is a bit different from the "list of tokens" idea: Rust already parses so called "groups". Groups are a list of tokens enclosed by either (), [] or {}. Each groups is represented by one TokenTree.

Let's take the example code above and print the actual TokenTrees we get. Let change the lib.rs to:

#[proc_macro_attribute]
pub fn auto_impl(args: TokenStream, input: TokenStream) -> TokenStream {
    for token_tree in input {
        println!("{:?}", token_tree);
    }
    TokenStream::new() // empty result to make it compile
}

And add an examples/test.rs file:

use auto_impl::auto_impl;

#[auto_impl()]
trait Foo<T> {
    fn foo();
}

fn main() {}

And finally say cargo build --example test. We get this output:

Ident { ident: "trait", span: #0 bytes(43..48) }                                                                                                                                    
Ident { ident: "Foo", span: #0 bytes(49..52) }                                                                                                                                      
Punct { ch: '<', spacing: Alone, span: #0 bytes(52..53) }                                                                                                                           
Ident { ident: "T", span: #0 bytes(53..54) }                                                                                                                                        
Punct { ch: '>', spacing: Alone, span: #0 bytes(54..55) }                                                                                                                           
Group { delimiter: Brace, stream: TokenStream [Ident { ident: "fn", span: #0 bytes(62..64) }, Ident { ident: "foo", span: #0 bytes(65..68) }, Group { delimiter: Parenthesis, stream: TokenStream [], span: #0 bytes(68..70) }, Punct { ch: ';', spacing: Alone, span: #0 bytes(70..71) }], span: #0 bytes(56..73) }

(Side note: yes, we just got our own output while executing cargo build -- without running the actual test. That happens because our proc macro code is executed by the compiler while compiling other crates.)

So that should give you a rough idea what a token stream is.

Lastly, you probably noticed that our function takes two TokenStreams: input and args. The former contains all tokens of the item our attribute is attached to (the trait). args contains the tokens in the actual attribute. So #[auto_impl(&, Box) would contain the tokens &, , and Box.

After the compiler called our function, it takes the token stream we returned and replaces the original trait definition with that token stream. So if we always return TokenStream::new(), we basically just delete the trait definition. (side note: the token stream returned by custom derives is added to the original token stream. So custom derives cannot modify the definition of the item they are attached to.)

---

Usually you want to interpret the token stream as a Rust thing (e.g. a trait). That's what the crate syn is for: it's a crate that can parse a token stream into an AST (abstract syntax tree). This makes it way easier to, say, iterate over all methods of a trait. That's one of the first things we do: we tell syn to parse our token stream as trait.

This results in a ItemTrait (this is one AST node, in our case the root node).

With that information, we can start generating our output. We don't want to modify the trait definition, but only add tokens. For each of our so called "proxy types", we need to generate one impl block. So in #[auto_impl(&, Box), we have two proxy types: & and Box. So we will emit two impl blocks.

Let's see an example again. The example from above but with & and Box added (and of course, we reset the definition of fn proc_macro in lib.rs):

use auto_impl::auto_impl;

#[auto_impl(&, Box)]
trait Foo<T> {
    fn foo();
}

fn main() {}

We can now use the great cargo expand tool to show the code generated by our macro. cargo expand --example test shows:

#![feature(prelude_import)]
#![no_std]
#[prelude_import]
use std::prelude::v1::*;
#[macro_use]
extern crate std as std;
use auto_impl::auto_impl;

trait Foo<T> {
    fn foo();
}
impl<'a, T, U: 'a + Foo<T>> Foo<T> for &'a U {
    fn foo() {
        U::foo()
    }
}
impl<T, U: Foo<T>> Foo<T> for ::std::boxed::Box<U> {
    fn foo() {
        U::foo()
    }
}

fn main() {}

(You can ignore the stuff at the very top, that's not from us.)

The generation of these impl blocks is what is the complicated part. All of it starts [here, in gen::gen_impls])(

auto_impl/src/gen.rs

Lines 19 to 24 in d7ba83a

	/// Generates one complete impl of the given trait for each of the given proxy
	/// types. All impls are returned as token stream.
	pub(crate) fn gen_impls(
	proxy_types: &[ProxyType],
	trait_def: &syn::ItemTrait,
	) -> Result<TokenStream2, ()> {

)

---

One additional thing: you will see the quote! {} macro a lot in our code. It comes from this crate and it's an easy way to generate a token stream. So instead of writing creating TokenTrees and all of that jazz yourself, you can simply write quote! { fn foo() } and this results in a token stream that contains three token trees (ident fn, ident foo and group ()).

A bit confusing: quote! returns a proc_macro2::TokenStream. The crate proc_macro2 is a crate very very similar to proc_macro which is used to test stuff. Read the README for more information. The important part is, that it's used a lot. So what we do is to create proc_macro2::TokenStreams with quote and at the very end, we convert them back to the expected proc_macro::TokenStream.

---

So I hope that serves as an introduction. For more questions, just ask in the gitter channel :)

[saresend]

Hey, just wanted to ask a couple of things involving how testing is being built. Specifically, since it depends on using the unstable build-plan to establish the binary target, it seems to be failing to execute the tests. I was wondering if there was a way to rely on something more stable for determining the binary target, specifically I was looking at using cargo build --tests --message-format=json, and parsing that to discover the binary file? Thanks!

[LukasKalbertodt]

@saresend I'm not sure if you've seen that but @KodrAus noticed this problem and created rust-lang/cargo#6082. On recent nightlies, our tests fail because of that, yes. But it's already fixed upstream -- now we only have to wait until it lands in nightly.

But your idea is very interesting! Ideally, something like a compile-fail tester should be implemented in an external crate and not for every project that wants such a test. Maybe I'll look into using the --message-format=json -- but I don't have a lot of time right now.

Some background on the current build system can be found here: #17

Lastly, if you want to test your changes now, you can use an older nightly compiler. For example nightly-2018-09-15 works:

$ rustup override set nightly-2018-09-15
$ cargo test

[saresend]

Oh, cool! Thanks for the heads up!

[LukasKalbertodt]

Fixed in #35 :)