Tracking issue for RFC 3519: arbitrary_self_types

[arielb1]

This is the tracking issue for RFC 3519: Arbitrary self types v2.

The feature gate for this issue is #![feature(arbitrary_self_types)].

About tracking issues

Tracking issues are used to record the overall progress of implementation. They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions. A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature. Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Steps

Current plan is:

• Fix the current lifetime elision bugs
• Search for potentially conflicting method candidates
• Block generic arbitrary self types.
• Introduce an arbitrary_self_types_pointers feature gate
• Rename the old Receiver trait -- currently being run through crater
• Land the new Receiver trait without it doing anything.
• Switch over the method resolution to use the Receiver trait, if the arbitrary_self_types feature is enabled. The main event.
• Add diagnostics for the !Sized case and the NonNull etc. cases.
• Update the Rust reference.
• Add documentation to the dev guide. See the instructions.
• Add formatting for new syntax to the style guide. See the nightly style procedure.
• Stabilize!

Unresolved Questions

None.

Notable for Stabilization

• Arbitrary self types v2: no deshadow pre feature. #134524

Related

• Changing the type of self in methods doesn't work #27941

Implementation history

TODO.

---

(Below follows content that predated the accepted Arbitrary Self Types v2 RFC.)

• figure out the object safety situation
• figure out the handling of inference variables behind raw pointers
• decide whether we want safe virtual raw pointer methods

Object Safety

See #27941 (comment)

Handling of inference variables

Calling a method on *const _ could now pick impls of the form

impl RandomType {
    fn foo(*const Self) {}
}

Because method dispatch wants to be "limited", this won't really work, and as with the existing situation on &_ we should be emitting an "the type of this value must be known in this context" error.

This feels like fairly standard inference breakage, but we need to check the impact of this before proceeding.

Safe virtual raw pointer methods

e.g. this is UB, so we might want to force the call <dyn Foo as Foo>::bar to be unsafe somehow - e.g. by not allowing dyn Foo to be object safe unless bar was an unsafe fn

trait Foo {
    fn bar(self: *const Self);
}

fn main() {
    // creates a raw pointer with a garbage vtable
    let foo: *const dyn Foo = unsafe { mem::transmute([0usize, 0x1000usize]) };
    // and call it
    foo.bar(); // this is UB
}

However, even today you could UB in safe code with mem::size_of_val(foo) on the above code, so this might not be actually a problem.

More information

There's no reason the self syntax has to be restricted to &T, &mut T and Box<T>, we should allow for more types there, e.g.

trait MyStuff {
    fn do_async_task(self: Rc<Self>);
}

impl MyStuff for () {
    fn do_async_task(self: Rc<Self>) {
        // ...
    }
}

Rc::new(()).do_async_stuff();

[porky11]

Why would you need this?
Why wouldn't you write an impl like this:

impl MyStuff for Rc<()> {
    fn do_async_task(self) {
        // ...
    }
}

I'd rather define the trait different. Maybe like this:

trait MyStuff: Rc {
    fn do_async_task(self);
}

In this case, Rc would be a trait type. If every generic type implemented a specific trait (this could be implemented automatically for generic types) this seems more understandable to me.

[cuviper]

This could only be allowed for trait methods, right?

For inherent methods, I can't impl Rc<MyType>, but if impl MyType can add methods with self: Rc<Self>, it seems like that would enable weird method shadowing.

[arielb1]

@cuviper

This is still pending lang team decisions (I hope there will be at least 1 RFC) but I think it will only be allowed for trait method impls.

[arielb1]

@porky11

You can't implement anything for Rc<YourType> from a crate that does not own the trait.

[arielb1]

So changes needed:

• remove the current error message for trait methods only, but still have a feature gate.
• make sure fn(self: Rc<Self>) doesn't accidentally become object-safe
• make sure method dispatch woks for Rc<Self> methods
• add tests

[SimonSapin]

I’ll look into this.

[arielb1]

Note that this is only supported to work with trait methods (and trait impl methods), aka

trait Foo {
    fn foo(self: Rc<Self>);
}
impl Foo for () {
    fn foo(self: Rc<Self>) {}
}

and is NOT supposed to work for inherent impl methods:

struct Foo;
impl Foo {
    fn foo(self: Rc<Self>) {}
}

[SimonSapin]

I got caught in some more Stylo work that's gonna take a while, so if someone else wants to work on this in the meantime feel free.

[kennytm]

Is this supposed to allow any type as long as it involves Self? Or must it impl Deref<Target=Self>?

trait MyStuff {
    fn a(self: Option<Self>);
    fn b(self: Result<Self, Self>);
    fn c(self: (Self, Self, Self));
    fn d(self: Box<Box<Self>>);
}

impl MyStuff for i32 {
   ...
}

Some(1).a();  // ok?
Ok(2).b();  // ok?
(3, 4, 5).c(); // ok?
(box box 6).d(); // ok?

[mikeyhew]

I've started working on this issue. You can see my progress on this branch

[mikeyhew]

@arielb1 You seem adamant that this should only be allowed for traits and not structs. Aside from method shadowing, are there other concerns?

[arielb1]

@mikeyhew

inherent impl methods are loaded based on the type. You shouldn't be able to add a method to Rc<YourType> that is usable without any use statement.

[arielb1]

That's it, if you write something like

trait Foo {
    fn bar(self: Rc<Self>);
}

Then it can only be used if the trait Foo is in-scope. Even if you do something like fn baz(self: u32); that only works for modules that use the trait.

If you write an inherent impl, then it can be called without having the trait in-scope, which means we have to be more careful to not allow these sorts of things.

[mikeyhew]

@arielb1 Can you give an example of what we want to avoid? I'm afraid I don't really see what the issue is. A method you define to take &self will still be callable on Rc<Self>, the same as if you define it to take self: Rc<Self>. And the latter only affectsRc<MyStruct>, not Rc<T> in general.

[mikeyhew]

I've been trying to figure out how we can support dynamic dispatch with arbitrary self types. Basically we need a way to take a CustomPointer<Trait>, and do two things: (1) extract the vtable, so we can call the method, and (2) turn it into a CustomPointer<T> without knowing T.

(1) is pretty straightforward: call Deref::deref and extract the vtable from that. For (2), we'll effectively need to do the opposite of how unsized coercions are implemented for ADTs. We don't know T, but we can can coerce to CustomPointer<()>, assuming CustomPointer<()> has the same layout as CustomPointer<T> for all T: Sized. (Is that true?)

The tough question is, how do we get the type CustomPointer<()>? It looks simple in this case, but what if CustomPointer had multiple type parameters and we had a CustomPointer<Trait, Trait>? Which type parameter do we switch with ()? In the case of unsized coercions, it's easy, because the type to coerce to is given to us. Here, though, we're on our own.

@arielb1 @nikomatsakis any thoughts?

[nikomatsakis]

@arielb1

and is NOT supposed to work for inherent impl methods:

Wait, why do you not want it work for inherent impl methods? Because of scoping? I'm confused. =)

[nikomatsakis]

@mikeyhew

I've been trying to figure out how we can support dynamic dispatch with arbitrary self types.

I do want to support that, but I expected it to be out of scope for this first cut. That is, I expected that if a trait uses anything other than self, &self, &mut self, or self: Box<Self> it would be considered no longer object safe.

[mikeyhew]

@nikomatsakis

I do want to support that, but I expected it to be out of scope for this first cut.

I know, but I couldn't help looking into it, it's all very interesting to me :)

[arielb1]

Wait, why do you not want it work for inherent impl methods? Because of scoping? I'm confused. =)

We need some sort of "orphan rule" to at least prevent people from doing things like this:

struct Foo;
impl Foo {
    fn frobnicate<T>(self: Vec<T>, x: Self) { /* ... */ }
}

Because then every crate in the world can call my_vec.frobnicate(...); without importing anything, so if 2 crates do this there's a conflict when we link them together.

Maybe the best way to solve this would be to require self to be a "thin pointer to Self" in some way (we can't use Deref alone because it doesn't allow for raw pointers - but Deref + deref of raw pointers, or eventually an UnsafeDeref trait that reifies that - would be fine).

I think that if we have the deref-back requirement, there's no problem with allowing inherent methods - we just need to change inherent method search a bit to also look at defids of derefs. So that's probably a better idea than restricting to trait methods only.

Note that the CoerceSized restriction for object safety is orthogonal if we want allocators:

struct Foo;
impl Tr for Foo {
    fn frobnicate<A: Allocator+?Sized>(self: RcWithAllocator<Self, A>) { /* ... */ }
}

Where an RcWithAllocator<Self, A> can be converted to a doubly-fat RcWithAllocator<Tr, Allocator>.

[mikeyhew]

@arielb1

Because then every crate in the world can call my_vec.frobnicate(...); without importing anything, so if 2 crates do this there's a conflict when we link them together.

Are saying is that there would be a "conflicting symbols for architechture x86_64..." linker error?

Maybe the best way to solve this would be to require self to be a "thin pointer to Self" in some way (we can't use Deref alone because it doesn't allow for raw pointers - but Deref + deref of raw pointers, or eventually an UnsafeDeref trait that reifies that - would be fine).

I'm confused, are you still talking about frobnicate here, or have you moved on to the vtable stuff?

[arielb1]

I'm confused, are you still talking about frobnicate here, or have you moved on to the vtable stuff?

The deref-back requirement is supposed to be for everything, not only object-safety. It prevents the problem when one person does

struct MyType;
impl MyType {
    fn foo<T>(self: Vec<(MyType, T)>) { /* ... */ }
}   

While another person does

struct OurType;
impl OurType {
    fn foo<T>(self: Vec<(T, OurType)>) {/* ... */ }
}   

And now you have a conflict on Vec<(MyType, OurType)>. If you include the deref-back requirement, there is no problem with allowing inherent impls.

[adetaylor]

Another update relative to the plan:

• Step 7: the main event. All landed, but I made some assumptions in the process which are given below.
• Step 8: diagnostics
  • The Weak and NonNull diagnostics have landed in Arbitrary self types v2: Weak & NonNull diagnostics #134264
  • We decided not to land Arbitrary self types v2: diagnostics in resolution. #134384 as it was more complexity than merited
  • @cramertj has kindly agreed to figure out the diagnostics for the !Sized case, and more generally, whether we can reuse some of the E0277 machinery to give better diagnostics in cases where the self type doesn't implement Receiver. This work is being tracked in Better error reporting for T: ?Sized types when impl Receiver for MyType<T> is implicitly sized #134390 and there are some preliminary nudges in this branch: master...adetaylor:rust:arbitrary-self-types-diagnostics
• Step 9: docs update
  • Per this comment I'll need to update the Rust reference, dev guide and style guide - I'll be working on those things next. And also the unstable book

Assumptions made during landing the main PR which we should consider before stabilization (if you want to discuss these, please open a new issue instead of discussing here). All of these are related to the deshadowing algorithm, which attempts to spot cases where an outer smart pointer type adds a new method that might "shadow" or override some method in a referent. As a reminder, the complexity here is that the shadower and shadowed might take self in different ways, e.g. self vs &self, so we have to do a bit more searching for method candidates than we previously did.

• As noted in this code comment - note you'll have to expand probe.rs to see it in that diff - and this review comment, we don't notice shadowing cases involving double references like this:

  // Existing code
  struct A;
  impl A {
    fn foo(self: &&NonNull<A>) {}  // note this is by DOUBLE reference
  }
  
  // New code added later
  impl<T: ?Sized> NonNull<T> {
    fn foo(&self) {}
  }

• As noted here, the deshadowing algorithm only cares about inherent methods, both in the shadower and shadowee. This is the primary case we are worried about - where an outer smart pointer type adds a method which prevents the inner method being called. Such shadowing errors become less predictable when traits are brought into scope.
• The deshadowing algorithm also ignores the pick_const_ptr_method and pick_reborrow_pin_method parts of probe.rs. This means there are (very) niche cases where we won't spot methods being shadowed. Rather than explain that here, I added a test showing these cases in Arbitrary self types v2: niche deshadowing test #134509. I think both these cases require other unstable features to be enabled, and even if that weren't the case, I think the possibility of such shadowing cases occurring in the real world is low. (Though perhaps when pin_ergonomics is enabled we might want to extend the algorithm to cover that case).

[adetaylor]

A further update on item 9, documentation. All the following PRs are draft and not yet for review, but linking them from here for awareness:

• Stuff that can land before stabilization

  • Unstable doc updates: Arbitrary self types v2: unstable doc updates. #134525

• Stuff that can land only if we stabilize:

  • rustc-dev-guide updates: Arbitrary self types v2: dev guide updates. rustc-dev-guide#2168
  • Rust reference updates: Arbitrary self types v2: update reference. reference#1699
  • Book stuff Arbitrary self types v2: book changes. book#4174

• Other doc stuff

  • I determined that there was no need to update the style guide since there is no new syntax

In the process of preparing these docs, I found a creative, cunning and implausible way to trigger the new deshadowing code without enabling the arbitrary_self_types feature. This wasn't intentional so this is fixed in #134524.

[chriskrycho]

Just for visibility to the team, see my comment on the PR for this in The Book. Short version: not yet sure if it fits in The Book at all, and if it does we’ll need to think about where and how to make it fit! Happy to discuss further… but probably after the holidays!

[adetaylor]

Another periodic update:

• Arbitrary self types v2: unstable doc updates. #134525 and Arbitrary self types v2: no deshadow pre feature. #134524 landed.
• So, as far as I know, everything is now landed into nightly Rust except the !Sized diagnostics which @cramertj is taking care of in Better error reporting for T: ?Sized types when impl Receiver for MyType<T> is implicitly sized #134390.
• If anyone knows anything I've missed, please let me know!

Next steps:

• Wait. Probably a month or so?
• Polish and sort out the doc PRs:
  • rustc-dev-guide updates: Arbitrary self types v2: dev guide updates. rustc-dev-guide#2168
  • Rust reference updates: Arbitrary self types v2: update reference. reference#1699
  • Book stuff Arbitrary self types v2: book changes. book#4174 (per comments from @chriskrycho we might not even want this)
• Prepare a stabilization report which should include:
  • The assumptions and questions noted in Tracking issue for RFC 3519: arbitrary_self_types #44874 (comment)
  • The theoretical possibility of breaking existing code noted in Arbitrary self types v2: no deshadow pre feature. #134524 (comment) (which we don't think can happen in practice because any such code would be pointless)