Extern types v2 #3396
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,242 @@ | ||
| - Feature Name: `extern_types_v2` | ||
| - Start Date: 2023-02-19 | ||
| - RFC PR: [rust-lang/rfcs#3396](https://github.com/rust-lang/rfcs/pull/3396) | ||
| - Rust Issue: [rust-lang/rust#0000](https://github.com/rust-lang/rust/issues/0000) | ||
|
|
||
|
|
||
| # Summary | ||
| [summary]: #summary | ||
|
|
||
| Define types external to Rust and introduce syntax to declare them. | ||
| This additionally introduces the `MetaSized` trait to allow these types to be interacted with in a generic context. | ||
| This supersedes [RFC 1861]. | ||
|
|
||
|
|
||
| # Motivation | ||
| [motivation]: #motivation | ||
|
|
||
| The motivation from [RFC 1861] for why we want extern types at all still applies. | ||
|
|
||
| The summary of that is that when writing FFI bindings, the foreign code may give you a pointer to a type but not provide the layout for that type. | ||
| The current approach for representing these types in Rust, suggested by the 'nomicon, is with a ZST like the following. | ||
| ```rust | ||
| #[repr(C)] | ||
| pub struct Opaque { | ||
| _data: [u32; 0], | ||
| _marker: PhantomData<(*mut u8, PhantomPinned)> | ||
| } | ||
| ``` | ||
| The type is `repr(C)` so that it can be used in FFI. | ||
| The `_data` field provides the alignment of the type, and the `_marker` field causes the correct set of automatically implemented traits to be implemented, in this case `!Send`, `!Sync`, `!Unpin`, but `Freeze` (`UnsafeCell` can be used to remove the last one of those). | ||
| This works, however the compiler believes this type is statically sized and aligned which may not be true. | ||
| This means that the type can be easily misused by placing it on the stack, allocating it on the heap, calling `ptr::read`, calling `mem::size_of`, etc. | ||
| This proposal introduces extern types, a way to accurately represent these opaque types in the Rust type system such that the compiler prevents you from misusing it. | ||
|
|
||
| The motivation for replacing [RFC 1861] is around computing the offset of fields in aggregate types. | ||
| Extern types do not have an alignment known to the Rust compiler and therefore cannot be included as a field in an aggregate type as it is impossible to correctly calculate their offset, however that RFC did not address this. | ||
| This implies that extern types cannot be included in any (non-`repr(transparent)`) structs and so must be prevented. | ||
|
|
||
| Extern types should be able to be used in generic contexts so that they receive blanket trait implementations and can be used inside generic wrapper types. | ||
| Prior to this, all generic types could be included as fields of structs and it was possible to obtain the size and alignment of any type. | ||
| Extern types do not have a size or alignment so this RFC allows users to specify generic bounds sufficiently to statically prevent the above issues. | ||
|
|
||
|
|
||
| # Guide-level explanation | ||
| [guide-level-explanation]: #guide-level-explanation | ||
|
|
||
| When writing bindings to foreign code, any types that are opaque to the Rust type system should be declared as follows: | ||
| ```rust | ||
| extern { | ||
| type Foo; | ||
| } | ||
| ``` | ||
| This is called an extern type and is `!Sized`, `!MetaSized` (explained below), `!Send`, `!Sync`, and is FFI-safe. | ||
| Unlike other dynamically-sized types (DSTs), currently only slices and trait objects, pointers to it (`&Foo`, `&mut Foo`, `*const Foo`, `*mut Foo`, etc) are thin, that is they are 1 `usize` wide not 2. | ||
|
|
||
| These types cannot be included in structs as Rust is not able to compute the offset of the field because it does not know the alignment. | ||
| ```rust | ||
| struct Header { | ||
| data: u8, | ||
| tail: Foo, // Error due to unknown offset | ||
| } | ||
| ``` | ||
|
|
||
| However, these types may be placed in `repr(transparent)` structs: | ||
| ```rust | ||
| #[repr(transparent)] | ||
| struct Wrapper<T> { | ||
| inner: Foo, | ||
| _marker: PhantomData<T>, | ||
| } | ||
| ``` | ||
|
|
||
| The lack of the `Sized` and `MetaSized` traits on these structs prevent you from calling `ptr::read`, `mem::size_of_val`, etc, which are not meaningful for opaque types. | ||
|
|
||
| In the 2021 edition and earlier, these types cannot be used in generic contexts as `T: Sized` and `T: ?Sized` both imply that `T` has a computable size and alignment. | ||
|
There was a problem hiding this comment. What if a new-edition trait has an associated type with Similarly, what if a new-edition trait has a generic function with (Both of these examples came out of boats's recent exploration into adding a |
||
|
|
||
| In the 2024 edition and later, `T: ?Sized` no longer implies any knowledge of the size and alignment so opaque types can be used in generic contexts. | ||
| If you require your generic type to have a computable size and alignment you can use the bound `T: ?Sized + MetaSized`, which will enable you to store the type in a struct. | ||
|
|
||
| The automated tooling for migrating from the 2021 edition to the 2024 edition will replace `?Sized` bounds with `?Sized + MetaSized` bounds. | ||
|
There was a problem hiding this comment. The tooling should only do the replacement if I feel it would be useful to have an idea on how much of an impact this will have on the ecosystem -- how common will it be to need a Doing the review of the standard library mentioned later would be a start. There was a problem hiding this comment.
How do you propose deciding which case is which? Note that (usually) relaxing the bound is non-breaking but adding it in later is breaking so a human decision is often required for what promises the API wants to make. |
||
|
|
||
|
|
||
| # Reference-level explanation | ||
| [reference-level-explanation]: #reference-level-explanation | ||
|
|
||
| Some nomenclature to assist the rest of this explanation: | ||
|
|
||
| Types have a size and alignment that is known: | ||
| * statically - the size/alignment is known by the Rust compiler at compile time. This is the current `Sized` trait. | ||
| Most types in Rust are statically sized and aligned, like `u32`, `String`, `&[u8]`. | ||
|
Comment on lines
+89
to
+90
There was a problem hiding this comment. In a way, this is the same as knowing from metadata -- the metadata is just (I think that this way of wording is still clearer than omitting it, but figured it's worth pointing out somehow.) There was a problem hiding this comment. Yes, I possibly should clarify that all statically sized things are metadata sized and all metadata sized things are dynamically sized (and similarly for alignment). Note there is still a meaningful distinction between these things: you can put as many statically sized&aligned things in a struct but only one metadata aligned thing in. |
||
| * from metadata - the size/alignment can be derived purely from pointer metadata without having to inspect or dereference the pointer. | ||
| All remaining types fit in this category and are DSTs. | ||
| `[u8]` has a statically known alignment but the size can only be determined from the pointer metadata, `dyn Debug`'s size and alignment are both obtained from the vtable in the pointer metadata. | ||
| * dynamically - the size/alignment can only be determined at run time. | ||
| There are no types currently expressible in the language with dynamically known size or alignment. | ||
| The most discussed potential type in this category is `CStr`, which has a statically known alignment but it's size can only be determined by iterating over it's contents to find the position of the null byte. | ||
| Note that these types are odd, for example determining the size of a `Mutex<CStr>` requires taking a lock on the mutex. | ||
| * unknown - the size/alignment is not able to be determined at compile time or run time. | ||
| This is the category that opaque types fall in (and no other existing types occupy), without any additional domain specific knowledge. | ||
| Therefore extern types will occupy this category to allow the most flexibility. | ||
|
|
||
| The rest of this document will refer to types as "statically sized", "metadata aligned", etc. | ||
|
|
||
| "dynamically aligned" (or "unknown aligned") types cannot be placed as the last field of a struct as their offset cannot be determined, without already having a pointer to the field. | ||
|
|
||
| In the Rust 2021 edition and earlier `T: Sized` implies that `T` is "statically sized" and "statically aligned" and `T: ?Sized` implies that `T` is "metadata sized" and "metadata aligned". | ||
|
|
||
| In the Rust 2024 edition and later `T: Sized` means the same but `T: ?Sized` implies that `T` is "unknown sized" and "unknown aligned". | ||
| A `MetaSized` bound can be introduced to regain the previous meaning. | ||
|
|
||
| ## `MetaSized` trait | ||
| [metasized-trait]: #metasized-trait | ||
|
|
||
| This introduces a new trait `core::marker::MetaSized` that represents a type that is "metadata sized" and "metadata aligned": | ||
| ```rust | ||
| #[lang = "meta_sized"] | ||
| trait MetaSized: Pointee { | ||
| fn size_of_val(metadata: <Self as Pointee>::Metadata) -> usize; | ||
| fn align_of_val(metadata: <Self as Pointee>::Metadata) -> Alignment; | ||
| } | ||
| ``` | ||
| This trait is automatically implemented for all types except extern types. | ||
|
|
||
| All types that implement `MetaSized` can be placed as the last field in a struct because the compiler can emit code to determine the offset of the field purely from the pointer metadata. | ||
| At the time of writing all `MetaSized` types are either statically aligned, or are trait objects and have their alignment in their vtable. | ||
| This RFC does not propose changing codegen for computing the offset as it does not introduce any new `MetaSized` types. | ||
|
|
||
| All locations in the standard library that use `?Sized` bounds need to be reviewed when migrating to the 2024 edition and replaced with either `?Sized` or `?Sized + MetaSized` as appropriate. Some examples: | ||
| ```rust | ||
| pub fn size_of_val<T: ?Sized + MetaSized>(val: &T) -> usize | ||
| ``` | ||
| This requires `MetaSized` because the size must be known at run time. | ||
|
|
||
| ```rust | ||
| pub struct Box<T: ?Sized + MetaSized, A: Allocator = Global>(Unique<T>, A); | ||
|
There was a problem hiding this comment. Without implied bounds, this will require the bound be mentioned everywhere pub trait Trait {
fn foo(self: Box<Self>) where Self: MetaSized;
fn bar(self: Arc<Self>) where Self: MetaSized;
}There was a problem hiding this comment. Interesting but probably bad idea: It's worth noting that since receivers are special, we could add special rules to make a There was a problem hiding this comment. Implied bounds could solve this if they are ever implemented… There was a problem hiding this comment. This is really sad. It feels like a blocker that can't be easily skipped. How reasonable would it be to make receivers special until implied bounds is implemented? Or is the only option here to go and implement implied bounds? There was a problem hiding this comment. I just had a thought, doesn't implied bounds for self types arguably already exist given that you can write this: trait Trait {
fn foo(self);
}You don't have to specify There was a problem hiding this comment. well, technically: trait Trait {
fn foo(self);
}doesn't actually imply |
||
| ``` | ||
| This requires `MetaSized` because the `Box` must be able to determine the Layout of the type at run time to allocate and free the backing memory. | ||
|
|
||
| ```rust | ||
| impl<A: ?Sized, B: ?Sized> const PartialEq<&B> for &A | ||
| where | ||
| A: ~const PartialEq<B>, | ||
| ``` | ||
| This does not require `MetaSized` because the references to `A` and `B` always remain behind a pointer and the size and alignment is not required to call a function on the type. | ||
|
|
||
| ## Extern types | ||
| [extern-types]: #extern-types | ||
|
|
||
| Extern types are defined as above, they are thin DSTs, that is their metadata is `()`. They cannot ever exist except behind a pointer, and so attempts to dereference them fail at compile time similar to trait objects. | ||
|
There was a problem hiding this comment. I assume There was a problem hiding this comment. Yes, exactly. I hadn't considered that distinction, I'll try to clarify that later. |
||
|
|
||
| `repr` attributes are not permitted on extern types as none of the existing representations are applicable. | ||
|
|
||
| Extern types do not automatically implement any traits (except `Pointee`), but users can manually implement, for example, `Send`. | ||
| This does mean that all extern types will be `!Freeze` as it is private, however this is a compiler internal that is only used for optimisations, so this only causes some potential missed optimisations. | ||
|
|
||
| An extern type can be included in a `repr(transparent)` struct as it is always at offset 0. | ||
| The transparent struct is then also a thin DST and inherits traits as normal. | ||
|
|
||
|
|
||
| # Drawbacks | ||
| [drawbacks]: #drawbacks | ||
|
There was a problem hiding this comment. Another drawback is that this RFC rejects a type that is used widely inside rustc: extern "C" {
type OpaqueListContents;
}
pub struct List<T> {
len: usize,
data: [T; 0],
opaque: OpaqueListContents,
}This must definitely be mentioned in the RFC. Ideally there is some kind of proposal for what to do with this type... There was a problem hiding this comment. A simple suggestion: allow structs with extern tails, but don't allow computing the offset of (or taking a reference of) the extern tail. This is sufficient for the rustc impl, which uses the A less simple suggestion is a way to specify a known alignment to use for an extern type; this could either be via |
||
|
|
||
| This introduces language complexity for a feature that most users will not use directly. | ||
|
|
||
| (Copied from [RFC 1861]) | ||
| The syntax has the potential to be confused with introducing a type alias, rather than a new nominal type. The use of extern here is also a bit of a misnomer as the name of the type does not refer to anything external to Rust. | ||
|
|
||
|
|
||
| # Rationale and alternatives | ||
| [rationale-and-alternatives]: #rationale-and-alternatives | ||
|
|
||
| ## Do nothing | ||
|
|
||
| Doing nothing is not a compelling option here, [RFC 1861] is merged but is not implementable. | ||
| At minimum we should mark that RFC as deprecated/unimplemented and remove extern types from the compiler and docs. | ||
| However, the fact that RFC was merged is a very strong indication that something is needed here, and the current workaround with sized types is not good enough. | ||
|
|
||
| ## No generics | ||
|
|
||
| This design aims to be minimal while still allowing extern types to feel like a fully supported part of the language, namely retaining the ability for them to be used in generics. | ||
| A simpler alternative would be to not allow extern types in generics, this would mean not adding `MetaSized` and retaining the existing meaning of `?Sized`. | ||
| This would severely restrict the utility of extern types and would prevent them from implementing many useful traits that have blanket implementations in the standard library. | ||
|
|
||
| ## More traits | ||
|
|
||
| We could introduce a large portion of the traits implied by the "dynamically/metadata/statically sized/aligned" nomenclature above. | ||
| This would allow the most flexibility about exactly what a generic needs but existing usage suggests that this is not necessary and would be a significant amount of language complexity. | ||
|
|
||
| ## Post-monomorphisation errors | ||
|
|
||
| We could allow extern types to be used in generics without the `MetaSized` machinery and simply generate a post-monomorphisation error when the compiler is not able to lay out a type or a function attempts to call `size_of_val` or `align_of_val`. | ||
| This would be a significant departure from the compilers approach to generics, but is not entirely unprecedented as these sorts of errors can be generated by const evaluation. | ||
|
|
||
| ## Opt-out trait bound - `?MetaSized` | ||
|
|
||
| This change could be made in an entirely backwards compatible way by leaving the existing meaning of `?Sized` alone and allowing relaxing the implied `MetaSized` bound with `T: ?Sized + ?MetaSized` (or possibly just `T: ?MetaSized`). | ||
| This is not suggested by this document because the lang team has historically said that they do not wish to add any more opt-out bounds. | ||
|
|
||
|
|
||
| # Prior art | ||
| [prior-art]: #prior-art | ||
|
|
||
| There is a lot of prior art in rust itself from previous attempts at custom DSTs, the `DynSized` trait, and some other things related to "exotically sized types". | ||
| - [Lang team design notes on exotically sized types](https://github.com/rust-lang/lang-team/blob/master/src/design_notes/dynsized_constraints.md). | ||
| This document contains notes from the lang team about what `?Sized + DynSized` needs to imply. | ||
| This document outlines `DynSized`, `MetaSized`, and `Sized` which inspired the "metadata sized" and friends in this RFC. | ||
| I believe this RFC satisfies the constraints outlined, mostly by dropping `DynSized` as an available bound, which limits expressiveness in favour of simplicity. | ||
| - [Custom DSTs](https://github.com/rust-lang/rfcs/pull/2594). | ||
| This postponed RFC attempts to introduce a generic framework for DSTs with arbitrary metadata. | ||
| This RFC aims to be compatible with future attempts at custom DSTs, as it is in essence a very restricted form. | ||
| - [DynSized without ?DynSized](https://github.com/rust-lang/rfcs/pull/2310). | ||
| This contains a lot of very useful analysis of what exactly commonly used crates want when they state `?Sized`. | ||
| However, this RFC aims to be simpler than the lint-based solution presented there. | ||
| Additionally, this deals with not being able to place extern types as fields in structs, rather than solely on `size_of_val` and `align_of_val`. | ||
| - [More implicit bounds](https://github.com/rust-lang/rfcs/issues/2255). | ||
| This discusses whether we should be adding more implicit bounds into the language, and there associated relaxations (like `?MetaSized`). | ||
| This RFC aims to sidestep this issue by utilising the edition system to change the definition of `?Sized`. | ||
|
|
||
|
|
||
| # Unresolved questions | ||
| [unresolved-questions]: #unresolved-questions | ||
|
|
||
| - Should `MetaSized` contain methods, or should it be a marker trait with the implementation left to `core::mem::size_of_val` and `core::mem::align_of_val`? | ||
| - Should "metadata sized" imply "metadata aligned" or should we be adding the `MetaAligned` trait rather than `MetaSized`? | ||
| - Should `MetaSized` be a supertrait of `Sized`? All `Sized` things are `MetaSized` but `Sized` doesn't semantically require `MetaSized`. | ||
| - Should users be able to slap a `#[repr(align(n))]` attribute onto opaque types to give them an alignment? | ||
| This would allow us to represent `CStr` properly but would necessitate splitting `MetaSized` and `MetaAligned` as they would not be "metadata sized" in general. | ||
| (We may be able to get away with the [Aligned trait](https://github.com/rust-lang/rfcs/pull/3319)) | ||
|
There was a problem hiding this comment. I've long been told that "extern type is needed to fix CStr and CString", so if this whole thing happens but doesn't definitely fix those types as part of it then things it would be extremely disappointing. |
||
| - Should the `extern type` syntax exist, or should there just be a `repr(unsized)`? | ||
| This would allow headers with opaque tails (which are very common in C code) but is a more significant departure from the original RFC, and looks more like custom DSTs. | ||
|
There was a problem hiding this comment. Why do you need There was a problem hiding this comment. Imagine we have a setup like the following: #[repr(C)]
struct Foo {
a: u32,
b: u8,
foo: String,
}
#[repr(C)]
struct Bar {
a: u32,
b: u8,
bar: u8,
}
extern type OpaqueTail;
#[repr(C)]
struct HeaderWithTail {
a: u32,
b: u8,
tail: OpaqueTail,
}
#[repr(C, unsize)]
struct HeaderUnsized {
a: u32,
b: u8,
}What alignment does There was a problem hiding this comment. Hmm, I suppose you could forbid referencing the field entirely… |
||
|
|
||
|
|
||
| # Future possibilities | ||
| [future-possibilities]: #future-possibilities | ||
|
|
||
| The most obvious future possibility is custom DSTs. | ||
| This would provide a mechanism for allowing users to implement types with entirely custom metadata and therefore custom implementations of `size_of_val` and `align_of_val`. | ||
| This would likely mean that users could implement types that acted like extern types without using the `extern type` syntax. | ||
| This should not be an issue as `extern type` communicates the intent of these types well, and guarantees FFI-safety. | ||
|
|
||
|
There was a problem hiding this comment. One other valuable extension would be to make There was a problem hiding this comment. MaybeUninit currently takes a sized T, why would this enable relaxing that bound? There was a problem hiding this comment. Oh, I misread the proposal and assumed that Without a notion of There was a problem hiding this comment. Sorry to be clear the proposal is that How would you create a |
||
|
|
||
| [RFC 1861]: https://rust-lang.github.io/rfcs/1861-extern-types.html | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the
extern type Fooshorthand still allowed under this proposal or do all extern types have to be inside an extern block?Similarly, how do extern blocks interact with ABI tags? For example, if I have an
extern "C"block, is a type declaration inside it invalid, or just a warning? I would say that adding an ABI to these should be a deny-by-default lint that states that the tag is unused, with the potential for them having some meaning in the future.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note that
extern { type Blah; }will currently be reformatted by rustfmt toextern "C" { type Blah; }. So this question should probably have t-style weigh in.I believe we've guaranteed the default is
"C"anyway, so it's not clear to me that this makes sense.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This question gets at one of the niggles I have with this proposal. The ABI doesn't really mean anything as we're using extern to mean something slightly different to what it normally means. I think this is a point in favour of
repr(unsized)but extern makes so much sense in that it heavily suggests what this feature is useful for.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When it comes to bikeshedding, what about something like the following?
I personally don't like the use of
typeand I thinkopaquecaptures the meaning better than eitherexternorunsized. I also think it just makes more sense as arepr, and it would go nicely with specifying alignment (eg.repr(opaque, align(16))) if we allow that, which I think we should (though that could be added later if necessary).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's your reasoning for preferring
opaqueoverunsized? I'm thinking of the common header use case where you may want:There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are described as opaque types throughout the RFC, which I think describes them well. The thing that makes them unique isn't that you don't know their size, but that you don't know (nor care about) their composition. This is in contrast to types like
[T],str, anddyn Trait, which are unsized, but not opaque.Regarding the example you gave, if I didn't already know what it meant, I would find it very confusing. I wouldn't understand what
unsizedmeant here, especially considering that thestructwould appear to be composed entirely of sized types. It would be a lot less confusing, though, if thestructhad no fields.To create a type with a non-opaque header followed by opaque data, I would imagine creating two separate types: one for the opaque data itself, and a separate type for the opaque data with a header (though that would seem to require
repr(align)on the opaque type). Using the wordopaquein this context (where the opaque type has no fields) would clearly indicate not only that we don't know the size, but that we don't know (nor care about) the composition of the type. Here's how I would imagine doing that:That said, I'm not opposed to either
extern type Tnor therepr(unsized)syntax. I just wanted to express my view thatrepr(opaque)might be a better fit. However, regardless of the syntax used, I would discourage the idea of allowing fields within the opaque type itself, as that would seem quite confusing to me.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One issue I see with the
reprattribute is that it looks too much like it's a unit struct declaration, if we're using a struct declaration I wonder if we could have native syntax for saying there are unknown fields (maybe with thereprtoo)Though, this has been proposed as syntax for
#[non_exhaustive]previously. I think it fits opaqueness better since the fields are unknown even locally, whereas#[non_exhaustive]only affects metadata and the visible struct declaration is what it is known to be locally.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's one place where
extern "C" { type X; }andextern "Rust" { type X; }could differ: the former obviously should be FFI-safe, but the latter doesn't necessarily have to be. If and only if it's not, then it's presumably semver-compatible to replace it with a standardstructin the future.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When I see "opaque (type)", I think of RPIT (return-position impl trait) types, and soon TAIT, RPITIT, etc.