Immovable types

[Zoxc]

This RFC introduces a design for immovable types based on this idea by @eddyb.

Given that a large motivation of this RFC is to support immovable generators, I suggest that we avoid closing this until we either define a suitable design for immovable generators or find a better approach to immovable types.

Rendered

[burdges]

As I mentioned in #1853 (comment) this might be enough to have types that zero themselves on drop correctly, which should improve doing cryptography in Rust. In particular, this improves using stack variable in cryptography, so crates can more easily support #[no_std].

Right now, you can zero a reference as done in https://crates.io/crates/clear_on_drop but you cannot zero anything that moves. I've some train wreck branches of that crate that demonstrate this, like https://github.com/burdges/clear_on_drop/blob/owned_fail/src/owned.rs

That said, there are reasons for moving things, so maybe wants a built-in trait that goes beyond just being a market, like

trait MoveItMoveIt {
    fn move_nonoverlapping(src: *const Self, dst: *mut Self, count: usize);
    fn move_maybe_overlapping(src: *const Self, dst: *mut Self, count: usize);
    ...
}

All types implement MoveItMoveIt by default like in this RFC, and you can implement !MoveItMoveIt if you want, but you could also provide a MoveItMoveIt implementation that zeros the source after the move. I'd imagine this increases the options for crates that integrate with garbage collected languages too.

I'd think both MoveItMoveIt and this Move RFC will run into the issue with poor LLVM support encountered by the #[clear_on_drop] RFC #1496 (comment) but maybe we can fix that with more interest overall and a narrower set of changes to LLVM.

cc @cesarb @valarauca @tarcieri

[Stebalien]

@burdges that looks like a move constructor. Not having move constructors is one of rust's selling points.

[burdges]

I see, so anything that zeros really seamlessly should be baked deeper into the compiler. There are maybe still parallel limitations in LLVM for both immovable types and self zeroing types though.

[eddyb]

👍 This seems like it represents my original idea pretty well, thank you!

The only thing I might want to add is that one way you might think about it is in terms of "observing the address" - which requires a borrow. Before that first borrow, nothing knows the address, therefore moves can't break third-party accesses.

This also leads to self/existential-borrows as being an excellent way to encode these types (sadly it'd have to be phantom for types like mutexes where a library or the OS is observing the address).
Oh, and nowadays one can just use @Amanieu's parking_lot crate for locks and get less overhead (it doesn't need stable addresses except while it's borrowed).

But the RFC in its current form is great for generators that borrow across a yield, even without the marker trait (so we can make that optional if we want a truly minimal RFC).
The big issue is the ?Move universe and annotating generic types, traits and functions with it, but that again is not that problematic for generator state (or at least async I/O where almost no abstractions are needed on the state itself).

Well, you'd want Vec<Box<Future+?Move>> to work, and I think we can pull it off by having Box always be Move but disallow moving out for !Move types inside of it (so that borrows don't have to last forever but always observe the same address).

cc @tomaka

[tomaka]

One small potential problem is that code like this will not accept !Move types:

fn foo<T: SomeTrait>(t: &T) {}

A lot of code like this exists in the wild, and I guess it will need to be changed for T: ?Move + SomeTrait.

EDIT: To be fair the code above already doesn't accept !Sized types, so I guess it doesn't change much.

[Stebalien]

Would Move: Sized? If so, maybe ?Sized → ?Move.

[eddyb]

You can still move unsized values with unsafe code so I wouldn't bet on being able to do that.

[krstoff]

If accepted, this RFC resolves this two year old issue. Good discussion in there that is relevant to this RFC.

[Diggsey]

Making Box<T: !Move>: Move, but disallowing moving the value out will be an extremely powerful feature to have, 👍

[eddyb]

As explained in #1858 (comment), focusing on mutexes might not be relevant anymore, and I'm not aware of other specific C APIs that need this sort of thing - usually you use an opaque pointer and don't even know the size of the allocation.

[Zoxc]

Parking lots require allocations. It is not unreasonable to have a mutex which does not allocate.

I may drop the note on interacting with C/C++. I haven't come up with any cases either.

[Amanieu]

parking_lot only allocates memory once at startup. Creating, destroying, locking and unlocking a mutex does not allocate memory.

[comex]

It may be useful if for some reason you need to bind a C++ class type by value, as C++ objects generally don't expect to be silently moved. In particular, this applies if you want to subclass a C++ class from Rust (in which case the class has to be embedded at the beginning of the struct).

[rkjnsn]

One example from boost is offset_ptr which stores the pointer as an offset from this. The idea is to be able to put containers and other data structures in shared memory and have it work even if the memory gets mapped to a different address.

[eddyb]

The PhantomData rule might not be the best choice, although all usecases of PhantomData I can think of involve indirection, so they wouldn't hide any capability. I suppose if we need "!Move if T: !Move" we can do:

struct MoveIffMove<T: ?Sized+?Move>(ImmovableData, PhantomData<T>);
unsafe impl<T: ?Sized+Move> Move for MoveIffMove<T> {}

[Zoxc]

Can we have such implementation for built-in traits?

The reason for this rule is to allow PhantomData<T> inBox<T:? Move> (which is needed for dropck), while keeping Box movable.

[eddyb]

Sure, OIBITs have some precedent here. And if we can do it at all then Box could just do that.

[strega-nil]

@eddyb why is Move an unsafe impl?

[cramertj]

@ubsan You could cause unsafety by implementing Move for a type that contains a !Move type. Moving the outer type would cause the inner type to be moved.

[strega-nil]

@cramertj oh, you're right. thanks :)

[cramertj]

E.g.

struct MoveWhatever<T: ?Move>(T);
impl<T: ?Move> Move for MoveWhatever<T> {}

[Zoxc]

@eddyb We could put PhantomData inside MobileCell to ensure it won't cause Box to be immovable

[RalfJung]

I don't think I entirely understand how this is supposed to work e.g. for StaticMutex... the RFC uses the example StaticMutex::new(0), but doesn't explain how that function would be implemented. I assume it would create a struct using a struct literal, and then call a pthread function to initialize the mutex, which needs to take the address -- so, a borrow is happening. Doesn't that mean that StaticMutex::new is not allowed to return the mutex, since that would be moving it?

[Diggsey]

@RalfJung I presume the mutex only gets initialised on first use, at which point the address is fixed.

edit: That doesn't really work for a mutex though... Maybe you would need to explicitly initialise it via a method taking &self?

[Zoxc]

For pthreads, we'd initialize it with PTHREAD_MUTEX_INITIALIZER, which is just plain data and doesn't need any code to run.

[comex]

So, why not make Move: Sized? After all, unsized types cannot be moved, and that way all existing ?Sized bounds would imply ?Move without breaking anything (including Self for traits by default).

EDIT: Durr, I didn't notice the comment from 10 days ago saying the exact same thing.

[comex]

@eddyb

You can still move unsized values with unsafe code so I wouldn't bet on being able to do that.

I don't think that makes sense. Even if unsafe code can do something with references to specific unsized types that's logically equivalent to moving, that doesn't mean the unsized types themselves need to be Move. There's no real move from the language's perspective. If the concern is that you could, say, have an &mut [Immovable] and sneakily get unsafe code to move it for you, that isn't directly related to whether Move: Sized and would be prevented by the default Move bound on type parameters (in this case for the relevant impls on slices).

[comex]

(Sorry to spam, but if I edit then it won't show up in email notifications...)

If Self were ?Move by default (along with ?Sized), then I guess there would have to be some weird magic for traits which add an explicit Sized bound and then try to move self. Which sounds like a bad idea.

However, making traits support immovable types by default would be quite powerful.

What if instead of having a separate Move trait, immovable types were just !Sized? Then for any properties that are common to regular sized types and immovable types, e.g. the ability to move in the restricted manner described, there could be a new supertrait to Sized. In other words, Sized would really mean "sized and movable", while the new trait would mean "really just sized".

That would be an abuse of terminology, but think about it. Depending on Sized currently gives you the ability to move, and that's pretty much all it does. There is no almost other reason to depend on it; if some function or trait depends on T: Sized (implicitly or explicitly) but does not move objects of type T, the Sized bound is probably unnecessary and should be removed. (I can think of one exception, which is that T: Sized also lets you make assumptions about the representation of &T; but that's relatively obscure and uncommon as the sole reason for a Sized bound.) It's as if the Sized trait were a one-method trait with fn move(), except the functionality is language magic rather than an actual method. This RFC essentially proposes removing that "method" and moving it to a new trait Move, and papering over the change by making everything implicitly depend on it. Imagine if we wanted to remove the requirement that Into conversions be infallible, but instead of adding TryInto, we just changed Into to return a result and then added a new trait InfallibleInto and made everything depend on it by default. (The analogy is a bit messy, but I think it's close enough.) Seen that way, it doesn't make sense to mess up the type system just because the name would become a bit inaccurate.

[strega-nil]

@comex There is code that makes the assumption that all T: ?Sized are moveable. See the T: ?Sized => Box<T> -> Rc<T> RFC. That code (and that RFC) would be broken if ?Sized also meant ?Move. They're logically distinct things. In a similar vein, with the alloca RFC you should be able to put ?Sized data onto the stack.

[comex]

Huh, I didn't know there was a generic way to get the size of values of unsized types. That definitely kills my idea then. (I'm not sure what you mean about alloca, though - the stack doesn't move...)

[strega-nil]

@comex You'd be able to take DSTs from one place, and put them onto the stack in another place.

[withoutboats]

The semantics of what types impl Move seem like Move is an auto trait. Is that accurate or is there a difference?

[eddyb]

@withoutboats An unsafe auto trait, perhaps. Given that you need {*{const,mut},&{,mut}} T for indirection, 4 impls (5 if Box is special-cased still) would suffice to describe Move.

[Ericson2314]

With #1909 in the works, I'm pretty sureSize and Move should definitely be orthogonal.

[chriskrycho]

Documentation RFC pedantry hat on here. 🤓

This section could use to be expanded a bit. Specifically, if accepted this RFC would need to include updates:

• to the standard library where the types are implemented
• to the Reference, under 9. Special Traits
• (probably) to the book in the FFI section

I also think the hand-wave that "the concept is likely familiar to users of C, C++, and C FFIs" is insufficient. This is something which people writing FFIs for the first time will need a place to learn, and we can't assume that everyone writing an FFI has prior C or C++ experience. There are many, and an increasing number of, Rust users who are coming to Rust wanting to do FFI for e.g. Python or Ruby and who haven't done it before in other contexts.

That said, discussions of the immovability in the C, C++, and FFI contexts might be really helpful prior art—it could possibly be cited, and it should certainly be considered as useful background for whoever writes the docs if this is accepted!

[Ixrec]

To add to that, despite my experience with C++, I never really had a clear concept of "immovable type" until the need for them arose in the recent Rust coroutine discussions. The closest concept I think I had when doing just C++ was the rather muddy "things that can be invalidated if this other thing moves around, like iterators and their containers, so don't move that other thing until you're done with the things that can be invalidated".

[aidanhs]

I've read the RFC, and I do like the idea that a !Move type is movable until you take the address of it. However, I have a few concerns, some of which may be resolved by this comment from @eddyb:

Well, you'd want Vec<Box<Future+?Move>> to work, and I think we can pull it off by having Box always be Move but disallow moving out for !Move types inside of it (so that borrows don't have to last forever but always observe the same address).

I'm not clear how this would work - borrows of T: !Move must last until drop, unless we can verify the T is contained in a Box? It seems like this would be required if you wanted to write a function that creates a Box<T: !Move>, calls do_something on it and returns it (otherwise to enforce !Move now that the address has been taken for do_something, the borrow of the value inside the box needs to propagate to the caller...which you can't do afaik) so I think I'd want to read about it in the RFC.

More generally, even if Box is special, Container<T: !Move> is much less useful than I'd hope once you've called a function on it - in addition to not being able to return it, you also can't transfer ownership elsewhere (e.g. cross-thread) or do anything useful at all with multi-element containers like Vec<T: !Move>/&[T: !Move].

Just to throw out an alternative design, let's say moves fall into three categories - parameters, return values and assignment (not sure if there are more?). By forbidding all three (similar to !Sized right now) and only permitting creation via placement new we lose the somewhat nice property of being able to move until the address is taken, but gain some much more interesting (to me) things like being able to use a Vec<T: !Move> types without any magic (assuming the addition of Move bounds on things like .remove etc)/create a NoMoveVec<T: !Move> (if there's sufficiently little shared code) and removing the shackles of only-one-borrow-with-drop-at-the-end.

(please let me know if anything above seems misinformed)

[rpjohnst]

&mut self is the correct signature for C++-like move constructors, since they leave the source object in a valid state. Returning Self is perhaps the wrong approach, since even with RVO it doesn't semantically allow for things like fixing up pointers.

How does bindgen wrap constructors? If it provides at least the option to run them directly on some already-allocated memory then matching that would probably be the closest we can get.

[eddyb]

As a name alternative to move_ consider move_out or take (which already exists on Option).

[arielb1]

@fitzgen

That version of move_ won't work for safe C++ move constructors, because safe code can move the return value around as long as it isn't borrowed, e.g.

extern "C++" {
    class NonMoveType;
    fn NonMoveType() -> NonMoveType;
}

fn example() -> Box<NonMoveType> {
    let n = NonMoveType();
    // this moves `n` into a box.
    box n
}

If you want things to work, you'll need to use &move T for the move-out and &out T for the move-in, or just use unsafe pointers everywhere.

[Zoxc]

I updated the RFC to better align with my PR. Move is now a builtin trait again instead of an auto trait for performance reasons.

[nbaraz]

I think that this needs to be explicit. If you have:

struct Bar;
impl Foo for Bar {}

fn baz() -> impl Foo {
    Bar
}

And then change baz() to:

impl Foo for Immovable {}

fn baz() -> impl Foo {
    Immovable
}

Code that assumes that baz() returns a movable value will break.

In order to return an existantial + immovable struct, you should have to specify it as being ?Move:

struct Bar;
impl Foo for Bar {}
impl Foo for Immovable {}

fn baz() -> impl Foo + ?Move {
    Immovable
}

(I haven't used existantial types, I don't know if impl Foo + Baz works).

The other solution would be to have all existantial types be ?Move by default, which wouldn't work.

[nbaraz]

Actually, this example/use case isn't straightforward because returning / passing an argument is a move. But it could interact with placement-new.

[nbaraz]

I'm not sure how useful this is - the main motivation for this RFC is self-referencing generators (or self-referencing structs in general, I assume), and this doesn't apply to them.

[jdm]

This is useful for interacting with libraries in other languages that store pointers to Rust values that would be invalidated by those values being moved.

[Zoxc]

@nbaraz This is the most important part of this RFC. This is the part which enables immovable types to be moved. This is very useful for generators. In particular you can return generators from functions, since returning is a move.

[jpernst]

Any self-referential structure would need to be immediately considered frozen, as it borrows itself from the beginning of its own existence, so the "until it is borrowed" clause would come into effect immediately.

This section does concern me for its implications in rental, however. It's not at all clear to me what exact conditions a value is to be considered "observed". If a function accepts a ?Move type by value, we could presume that it hasn't bee observed yet otherwise passing it would have been illegal in the first place, but what about immovable types behind a Box or Arc. Are they considered implicitly frozen when accepted as arguments or when returned from a function?

[Zoxc]

Any self-referential structure would need to be immediately considered frozen, as it borrows itself from the beginning of its own existence, so the "until it is borrowed" clause would come into effect immediately.

This is not possible in Rust though (even with this RFC). You can only borrow values after they have been constructed.

It's not at all clear to me what exact conditions a value is to be considered "observed".

One way to determine this is to ask if you can get a pointer to it in safe Rust. Since Box implements Deref we can call Deref::deref to obtain a pointer to the inner value, thus the inner value must be counted as observed once its placed in a Box. It is also possible to have some other MovableBox type, which do not allow access to the inner value, except by moving it out again.

In general, we can allow immovable types in an movable container if we either, disallow all methods of accessing the address of the contained immovable types or prevent the type from actually moving once it's inside.

[jpernst]

This is not possible in Rust though (even with this RFC). You can only borrow values after they have been constructed.

Right, I mean in a theoretical future where rust can express a self-referential type, it would need to be aware that it's in an immediate state of borrowed-ness.

Since Box implements Deref we can call Deref::deref to obtain a pointer to the inner value, thus the inner value must be counted as observed once its placed in a Box.

Excellent, that's what I was hoping for. To be able to add support for immovables to rental, I need to ensure that anything I return is already considered observed and frozen, since rust itself is unaware of the self-referentiality and might otherwise allow the type to move when it shouldn't.
Still poses some challenges with how to allow the user to select which container the value should be constructed into, but ATCs are likely the best solution for that.

[nbaraz]

Another motivation could be types whose destructors must run - If it is on the stack and can't move from there, can safe code prevent its destructor from running?

This would also require a way to mark types as only place-able on the stack, but immovability is still a requirement.

[Zoxc]

You may still place your "unleakable" type inside a wrapper type which may not run destructors and cause it to leak.

[nbaraz]

I explained it more clearly here.

The basic idea is adding a way to mark types as immovable + only place-able on the stack. This way they cannot be placed inside a wrapper to be leaked. If the wrapper is completely on the stack (no heap pointer), the unleakable type is not leaked (since !Move propagates from members to containing structs). Since it cannot be placed in a heap pointer, RC's and such can't be used to leak them.

I don't know if such a marker can be created/enforced.

[nbaraz]

I can't think of a scenario where this is actually safe - If a type assumes that it doesn't move in memory because something depends on its address, moving it will always break something.

I think that it makes sense for immovable types to only live on the stack or behind pointers, which are always Move, and for instances to be created with placement-new.

Maybe add an ExplicitMove trait, which implemented for every struct whose members implement ExplicitMove (by default not implemented for immpvable types)? I think that this can be addressed in another RFC.

[Diggsey]

Should probably add "borrowed and mutable references are always Move" for completeness.

[SoniEx2]

JoinGuard, anyone?

[martinhath]

What is the status on this? I've seen there is an internals thread talking about it, but, from what I can tell, we aren't getting any closer to some solution here. I just ran into this issue, and while my problems could be mitigated my macros, it would have been great to have language support for this.

[aturon]

Given where things are headed in this space, I'm going to propose to postpone this RFC, in favor of library-based solutions for the time being.

@rfcbot fcp postpone

[rfcbot]

Team member @aturon has proposed to postpone this. The next step is review by the rest of the tagged teams:

• @aturon
• @cramertj
• @eddyb
• @nikomatsakis
• @nrc
• @pnkfelix
• @withoutboats

No concerns currently listed.

Once these reviewers reach consensus, this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[eddyb]

I expect/hope to fully replace Move in the long-term with a more orthogonal design.

As I mentioned in my reddit comment, you can't use Clone on a !Move variable and then move it, to produce two copies of it, even if it'd be safe. The type isn't really immovable itself, but rather certain uses of it (Generator::resume) may require it becoming immovable to maintain safety invariants.

The only language extension I can think of is a new &pin T reference which can only be used on data you own and causes perma-borrowing (similar to &mut Pin<T> -> Anchor<&mut T>) - but with extra nice things like method call autoref, the ability to have compiler-checked self-referentiality (you can't self-reference without &pin T, with &mut T it wouldn't be allowed).

Then again, we could also just make the library types special to get all of these niceties!

[hadronized]

Oh that’s too bad this gets postponed. It seems to be a very exciting idea! <3

[eddyb]

@phaazon See #1858 (comment) - we may be able to come up with an improvement over ?Move!

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[rfcbot]

The final comment period is now complete.

[Centril]

Closing since FCP with a motion to postpone is now complete.