Tracking issue for RFC 1861: Extern types

[aturon]

This is a tracking issue for RFC 1861 "Extern types".

Steps:

• Implement the RFC (Implement RFC 1861: Extern types #44295)
• Adjust documentation (see instructions on forge)
• Stabilization PR (see instructions on forge)

Unresolved questions:

• Rust does not support types that don't have dynamically computed alignment -- we need the alignment to compute the field offset in structs. extern type violates this basic assumption, causing pain, suffering, and ICEs all over the compiler. What is the principled fix for this?

• Should we allow generic lifetime and type parameters on extern types?
  If so, how do they effect the type in terms of variance?

• In std's source, it is mentioned that LLVM expects i8* for C's void*.
  We'd need to continue to hack this for the two c_voids in std and libc.
  But perhaps this should be done across-the-board for all extern types?
  Somebody should check what Clang does. Also see "extern type" should use opaque type in LLVM #59095.
  RESOLVED because all pointer types are ptr now.

• How should this interact with unsized arguments? Currently it ICEs: unsized_fn_params should not accept types that don't have a dynamically fixed size (such as extern types) #115709

[jethrogb]

This is not explicitly mentioned in the RFC, but I'm assuming different instances of extern type are actually different types? Meaning this would be illegal:

extern {
    type A;
    type B;
}

fn convert_ref(r: &A) -> &B { r }

[canndrew]

@jethrogb That's certainly the intention, yes.

[glaebhoerl]

Relatedly, is deciding whether we want to call it extern type or extern struct something that can still be done as part of the stabilization process, or is the extern type syntax effectively final as of having accepted the RFC?

EDIT: rust-lang/rfcs#2071 is also relevant here w.r.t. the connotations of type "aliases". In stable Rust a type declaration is "effect-free" and just a transparent alias for some existing type. Both extern type and type Foo = impl Bar would change this by making it implicitly generate a new module-scoped existential type or type constructor (nominal type) for it to refer to.

[Ericson2314]

Can we get a bullet for the panic vs DynSized debate?

[plietar]

I've started working on this, and I have a working simple initial version (no generics, no DynSized).

I've however noticed a slight usability issue. In FFI code, it's frequent for raw pointers to be initialized to null using std::ptr::null/null_mut. However, the function only accepts sized type arguments, since it would not be able to pick a metadata for the fat pointer.

Despite being unsized, extern types are used through thin pointers, so it should be possible to use std::ptr::null.

It is still possible to cast an integer to an extern type pointer, but this is not as nice as just using the function designed for this. Also this can never be done in a generic context.

extern {
    type foo;
}
fn null_foo() -> *const foo {
    0usize as *const foo
}

Really we'd want is a new trait to distinguish types which use thin pointers. It would be implemented automatically for all sized types and extern types. Then the cast above would succeed whenever the type is bounded by this trait. Eg, the function std::ptr::null becomes :

fn null<T: ?Sized + Thin>() -> *const T {
    0usize as *const T
}

However there's a risk of more and more such traits creeping up, such as DynSized, making it confusing for users. There's also some overlap with the various custom RFCs proposals which allow arbitrary metadata. For instance, instead of Thin, Referent<Meta=()> could be used

[SimonSapin]

I think we can add extern types now and live with str::ptr::null not supporting them for a while until we figure out what to do about Thin/DynSized/Referent<Meta=…> etc.

[plietar]

@SimonSapin yeah, it's definitely a minor concern for now.

I do think this problem of not having a trait bound to express "this type may be unsized be must have a thin pointer" might crop up in other places though.

[SimonSapin]

Oh yeah, I agree we should solve that eventually too. I’m only saying we might not need to solve all of it before we ship any of it.

[plietar]

I've pushed an initial implementation in #44295

[kennytm]

@plietar In #44295 you wrote

Auto traits are not implemented by default, since the contents of extern types is unknown. This means extern types are !Sync, !Send and !Freeze. This seems like the correct behaviour to me. Manual unsafe impl Sync for Foo is still possible.

While it is possible for Sync, Send, UnwindSafe and RefUnwindSafe, doing impl Freeze for Foo is not possible as it is a private trait in libcore. This means it is impossible to convince the compiler that an extern type is cell-free.

Should Freeze be made public (even if #[doc(hidden)])? cc @eddyb #41349.

Or is it possible to declare an extern type is safe-by-default, which opt-out instead of opt-in?

extern {
    #[unsafe_impl_all_auto_traits_by_default]
    type Foo;
}
impl !Send for Foo {}

[eddyb]

@kennytm What's the usecase? The semantics of extern type are more or less that of a hack being used before the RFC, which is struct Opaque(UnsafeCell<()>);, so the lack of Freeze fits.
That prevents rustc from telling LLVM anything different from what C signatures in clang result in.

[kennytm]

@eddyb Use case: Trying to see if it's possible to make CStr a thin DST.

I don't see anything related to a cell in #44295? It is reported to LLVM as an i8 similar to str. And the places where librustc_trans involves the Freeze trait reads the real type, not the LLVM type, so LLVM treating all extern type as i8 should be irrelevant?

[eddyb]

@kennytm So with extern type CStr;, writes through &CStr would be legal, and you don't want that?
The Freeze trait is private because it's used to detect UnsafeCell and not meant to be overriden.

[jmillikin]

We were talking about the question of alignment requirements of references, why are you now deflecting by talking about size_of_val?

My point is that size_of_val() (and align_of_val()) currently accept references to external types, but crash because those operations do not make sense for external types. When C code returns an opaque pointer, that pointer cannot be assumed to meet the requirements of Rust references that existing APIs depend on, so creating a reference to such a type should be considered undefined behavior.

A practical way to solve that problem is to prevent external types from being used as references, because if the only thing they're good for is their bit pattern then they're not actually references in the Rust sense.

why is stabilization blocked on incorrect handling of Box?

I don't know what you are referring to here. Stabilization is blocked on the issue that the Rust type system does not understand the concept of a type that does not have a dynamically computable size.

I'm referring to #115709, in which a Box<T> is allowed to be constructed for an extern type despite that being undefined behavior (because extern types do not have a size or alignment). And then the thing gets passed as a dereferenced T function parameter, which should also be undefined behavior because there's no way to move or copy such a type.

If you change other people's working code then it may not work any more. Not sure which point you are trying to make here.

But if we have a macro that generates the mod hide_the_internals then it won't be possible to do that.

I thought I was clear in my point, but in case not, the RFC for this tracking issue exactly describes the use of a ZST for extern types in the "motivation" section, and also describes why that is not the ideal solution.

Scrapping the extern type implementation and replacing it with ZSTs would be one possible approach, but I think it would be frustrating to the people who are subscribed to this tracking issue in the hopes of one day seeing this feature stabilize.

[crumblingstatue]

A practical way to solve that problem is to prevent external types from being used as references, because if the only thing they're good for is their bit pattern then they're not actually references in the Rust sense.

I have a use case that looks similar to this, that utilizes references to opaque types:

#![feature(extern_types)]

#[repr(C)]
#[derive(Debug)]
struct TextureSize {
    width: u32,
    height: u32,
}

extern "C" {
    type FfiSprite;
    
    fn ffi_sprite_create() -> *mut FfiSprite;
    fn ffi_sprite_get_texture(sprite: *const FfiSprite) -> *const Texture;
    
    type Texture;
    
    fn ffi_texture_get_size(texture: *const Texture) -> TextureSize;
}


struct Sprite<'texture> {
    handle: *mut FfiSprite,
    // The handle to the texture is on the FFI side
    _texture: core::marker::PhantomData<&'texture Texture>,
}

impl<'texture> Sprite<'texture> {
    fn new() -> Self { Self {
        handle: unsafe { ffi_sprite_create() },
        _texture: std::marker::PhantomData,
    }}
    fn texture(&self) -> &Texture {
        unsafe { &*ffi_sprite_get_texture(self.handle) }
    }
}

// Directly implement the opaque type
impl Texture {
    // Method by reference
    fn size(&self) -> TextureSize {
        unsafe { ffi_texture_get_size(self) }
    }
}

fn main() {
    let sprite = Sprite::new();
    let texture = sprite.texture();
    dbg!(texture.size());
}

Is this/should this be undefined behavior because you can't have a reference to an opaque type?

Here is a real life example in rust-sfml (using pseudo-opaque types similar to what bindgen generates, but the principle is the same): https://github.com/jeremyletang/rust-sfml/blob/3e275e06f408343c63133108e65009d1f4e6295c/src/graphics/texture.rs

Is this undefined behavior? In all my testing it worked fine.

[Skepfyr]

I think there are usability problems with what I've described, reborrowing the Ref(Mut)? versions is painful, and it means that no-one else can name a *mut ExternType preventing them from using it directly (which I think might actually be a problem for -sys crates). I agree the fact that you can misuse it from within the module is bad, but we already have the module as the scope of unsafety so I think it's acceptable.

I agree we should consider getting the opaque tail stuff working a separate issue, it's incredibly painful due to alignment, and I haven't seen a working proposal yet.

@jmillikin Box<T> wouldn't be an issue though if we banned extern types from showing up in generic parameters, and it's not constructable anyway so it's of limited concern. On references, I think banning them from generics also solves this problem, as it prevents them from being passed to *_of_val, or any other API that could inspect the type behind the reference. A reference to an extern type does match Rust's existing semantics, if you assume it doesn't satisfy T: ?Sized, that's why rust-lang/rfcs#3396 works, it proposes to add another bound less restrictive than the current meaning of T: ?Sized.

[RalfJung]

My point is that size_of_val() (and align_of_val()) currently accept references to external types, but crash because those operations do not make sense for external types.

Yes. As I said, that's a type system issue. They crash in a safe way (a non-unwinding panic) so this is not unsound.

When C code returns an opaque pointer, that pointer cannot be assumed to meet the requirements of Rust references that existing APIs depend on, so creating a reference to such a type should be considered undefined behavior.

This does not follow, not at all. UB is a very big hammer we use to enable the compiler to optimize code better. "size_of_val would panic" is most definitely not an excuse for introducing UB. The consequences of accidental UB are way too severe to gratuitously use it here.

Ideally we will use the type system to prevent size_of_val on references or pointers to extern types. But if that doesn't work we can use a runtime check like we do right now, with a panic. That's still miles better than Undefined Behavior. There's no need to risk arbitrary memory corruption just because someone created a reference to an extern type.

I'm referring to #115709, in which a Box is allowed to be constructed for an extern type despite that being undefined behavior (because extern types do not have a size or alignment). And then the thing gets passed as a dereferenced T function parameter, which should also be undefined behavior because there's no way to move or copy such a type.

That issue isn't about Box, it is about unsized_fn_params, which (like size_of_val) assumes that all types are DynSized. There are indeed some gaps in the support for the combination of these two unstable features. Whatever type system solution is used to safeguard size_of_val against extern types should also be used to ensure unsized function parameters (and unsized locals) are dyn-sized.

[kennytm]

@crumblingstatue

IIUC #43467 (comment) the proposed min_extern_type feature did not outright "ban" &Texture.

One big issue of &Texture was that align_of_val::<Texture>(_) is undefined, so a Rust program could not properly generate a valid &Texture reference¹. According to https://doc.rust-lang.org/reference/behavior-considered-undefined.html this was one of the UB scenarios:

• Producing an invalid value, … The following values are invalid (at their respective type):
  • A reference or Box<T> that is dangling, misaligned, or points to an invalid value.

But given that Rust can never safely create a Texture/Box<Texture> value or deference a &Texture place this sentence can be relaxed or clarified for extern type (treating them effectively as align-1 behind a pointer, similar to the treatment of dyn Trait in LLVM IR).

@jmillikin

A practical way to solve that problem is to prevent external types from being used as references, because if the only thing they're good for is their bit pattern then they're not actually references in the Rust sense.

I don't see how banning &Texture is the practical solution. Using the min_extern_type restriction i.e.

1. Texture does not implement ?Sized, i.e. can't be substituted into any generic arguments or associated types
2. Texture can't be used as a struct member field (except #[repr(transparent)])

it can already prevent {size,align}_of_val{,_raw}::<Texture>() from being even instantiated, and Texture can't be used as a struct tail either, side-stepping the entire alignment/size question.

I'm not sure what "references in the Rust sense" encompasses. IMO at least calling methods given a &mut Texture / &Texture do make sense.

Footnotes

1. That said, the Texture type in the real-world code already produced an under-aligned structure: declare_opaque! results in an align-1 ZST, but the actual C++ definition of sf::Texture contains an Uint64 m_cacheId field meaning the actual alignment must be at least 8. ↩

[jmillikin]

@kennytm The problem with the proposed min_extern_types is the inability to use such a type as a generic parameter. It's sacrificing an important capability (extern type generics) to retain something unnecessary (extern type references).

Given the following sets of goals for an extern type T:

• Allow *const T and *mut T, which are required in FFI signatures.
• Allow NonNull<T> and Option<NonNull<T>>, for niche optimization.
• Forbid size_of_val::<T>(_) and align_of_val::<T>(_), as extern types don't have sizes or alignments.
• Forbid struct W(T), Option<T>, [T], and &[T], as a consequence of not having sizes or alignments.

... the proposal to forbid their use as references solves all four, and allowing references but forbidding generics would leave two unsolved.

[programmerjake]

• Forbid struct W(T), Option<T>, [T], and &[T], as a consequence of not having sizes or alignments.

the types Option<T>, [T], and [T; N] are forbidden anyway for extern types since extern types aren't Sized.

[kennytm]

@jmillikin Forbidding &ExternType alone will not prevent calling align_of_val_raw which takes a pointer, not a reference.

Meanwhile forbidding reference but not generic meant the following code is still possible:

// crate A
pub struct Tailed<T: ?Sized> {
    a: usize,
    b: T,
}

// crate B
extern { type Texture; }
type TailedTexture = crate_a::Tailed<Texture>;

The NonNull<T> case can be supported once we have sorted out the Sized hierarchy issue. ?Sized could become an alias of DynSized and the bound of NonNull<T> will be changed from ?Sized to Unsized.

[jmillikin]

@kennytm I don't think size_of_val_raw() and align_of_val_raw() matter that much, because they're not stable and therefore their signatures can change to have new trait bounds or whatever.

I don't understand why the code you provided would be valid for extern type -- they don't have a size or alignment, so they can't be struct fields.

[RalfJung]

You can't forbid references without forbidding extern types in generics. Otherwise it's always possible to use generics to bypass the restriction.

[programmerjake]

I don't understand why the code you provided would be valid for extern type -- they don't have a size or alignment, so they can't be struct fields.

I think it should be valid for extern type, since that allows you to declare a struct that you can make references to that is unsized and is a header for whatever unknown larger struct is actually in memory. actually accessing the extern type field shouldn't be valid, unless the extern type has a known alignment, e.g.:

extern {
    #[repr(align(4))]
    type ExternTypeWithKnownAlignment;
}

[kennytm]

@jmillikin

I don't think size_of_val_raw() and align_of_val_raw() matter that much matter that much, because they're not stable and therefore their signatures can change to have new trait bounds or whatever.

What really matters is the intrinsic they are being forwarded to (min_align_of_val) are actually used by the language to compute alignment of a struct tail field.

I don't understand why the code you provided would be valid for extern type -- they don't have a size or alignment, so they can't be struct fields.

1. If it is forbidden from "crate A", how could it know that T is an extern type and has unspecified size and alignment?
2. If it is forbidden from "crate B", how could it know that T is being used as a struct tail (given that all members of Tailed are private)?

The min_extern_type restriction is to forbid such code from "crate B" (since from "crate A"'s perspective, Tailed<[u8]> and Tailed<dyn Trait> must continue to work) by declaring ExternType does not implement ?Sized, simple.

---

@programmerjake

unless the extern type has a known alignment, e.g.:

While it sounds good in theory I doubt in practice if there is any actual scenario where the FFI type specified the exact alignment yet the size and all fields are implementation details. Usually you either have none or all of these 3 details.

Like for the <cstdio> type FILE*, either you just use whatever returned by fopen() and don't care about its alignment/size/fields, or you'll actually use the complete definition like

#[repr(C)]
struct FILE {
    _flags: c_int,
    _IO_read_ptr: *mut c_char,
    ...
}

I see no case that you specifically only want to know align_of::<FILE>() == align_of::<usize>() and nothing else.

The only exception might be dynamic-sized array, like an array tail like xxxx_t tail[0], or C-style string const char* / const wchar_t*, but I think they are a different kind of Custom DST that shouldn't be hacked through extern type.

[nikomatsakis]

@RalfJung

That's not giving the function an extra capability, but gives the caller an extra capability! Those are very different things. In fact they are the exact opposite of each other (in a precise, formal sense even). I think it would be extremely confusing to treat them like the same thing.

I agree they are not the same (and my blog post said that). However, my point was that I do not think that understanding this distinction is as important as you do, at least at first, and I think that by the time it is, people will be able to handle it. In contrast, the ? operator puts that price up front -- it marks this bound as a very different thing -- in a way that I think can intimidate readers and isn't especially necessary for them to understand early on. Now, if ?Sized could scale to all the use cases we need, then maybe that'd be one thing, but it seems to me that it cannot (though maybe there are proposals I've missed, I feel like I have to do a bit of a review, I didn't realize how much conversation had been going on).

Let me explain this a different way. Early on in their Learning Rust journey, I think the main way that people will interact with Unsized or DynSized bounds is when calling library APIs -- i.e., reading them in documentation, not authoring them. They might not realize that Unsized or DynSized is in fact saying that this function accepts a broader set of types (unlike Debug, which narrows it). But so long as they can tell that their types implement those traits, do they care? I don't think so.

Maybe they then go to author an impl that needs T: Unsized to be maximally applicable. My experience in authoring such impls is that I often forget the T: ?Sized initially. I write the code. It type checks. Then I go to use it, and I have to add some bounds. This is ... surprisingly similar to what happens with ordinary bounds. Of course the "compiler complains, add bound, recompile" cycle is happening when I use the impl, not when I author the impl, but the overall interplay feels similar. Especially because there is a kind of reverse thing that sometimes plays out, where I add bounds to write the code, then find out that the types I want to use don't support those bounds (sometimes for good reason), and I have to go back and refactor the code not to require so many bounds.

All that said, I think we would really benefit from doing some experimentation here. It's often hard to judge how something will feel after using it for a while. My hunch is that Unsized bounds will feel pretty natural, but I'd feel better if we had a nightly implementation to play with. And, as I said, I'd like to review the other ideas. I'm open to something better. But mostly I don't want us to delay forever and ever.

[nikomatsakis]

(Has anybody written lints to detect when a type parameter could have been ?Sized?)

[zetanumbers]

I have a concern that drop glue for extern types should not be instantiated, meanwhile it currently resolves with empty drop_in_place. Same goes for needs_drop as it should always return true instead of false, just like with dyn Trait.

P.S. It should probably become some sort of a post-monomorphization (or linker) error, since this feature usually relies on the linker finding symbols anyway. I could try implement that. Perhaps this could also be done to size_of_val? So I would say calling drop_in_place most probably indicates a bug in the code. I wonder what crater run would say about that.

[Skepfyr]

@zetanumbers Could you give a code example of what your talking about? I can't imagine a scenario where drop glue would be emitted for an extern types, because you can't ever have an owned extern type in Rust. Similarly, I'd expect needs_dropto return false, because you can't implement Drop for extern types (which means I'd expect drop_in_place to do nothing).

[zetanumbers]

@zetanumbers Could you give a code example of what your talking about? I can't imagine a scenario where drop glue would be emitted for an extern types, because you can't ever have an owned extern type in Rust. Similarly, I'd expect needs_dropto return false, because you can't implement Drop for extern types (which means I'd expect drop_in_place to do nothing).

As you have said, so that drop_in_place::<ForeignType> is instantiated with empty drop glue.

I'd expect needs_drop to return false, because you can't implement Drop for extern types

You cannot implement Drop for a number of types including dyn Trait for which needs_drop would return true.

I understand you don't see the difference for yourself, I am just looking which behavior would people prefer if they care.

[RalfJung]

You cannot implement Drop for a number of types including dyn Trait for which needs_drop would return true.

dyn Trait forwards Drop to the actual underlying type. So not being able to implement Drop here is an entirely different matter.

Extern types do not have any implicit drop glue, so there's no good reason for needs_drop to return true here. needs_drop() == false just means that the drop shim does nothing, and indeed drop_in_place does nothing for these types, so all works out.

Maybe we want to allow impl Drop for ExternType in the future, but that's a separate discussion and the current behavior is forward-compatible with that.

[zetanumbers]

I am worried if such behavior may affect support for unforgettable types. Extern types seem like very much raw feature for unsafe code only, which is not inherently bad, but I wonder if usage from safe code is planned.

Also to note if unforgettable types are indeed only meaningful when they have a lifetime, as I believe it would be without accounting other hypothetical features, then extern type should probably not be allowed to have lifetime arguments.

[Skepfyr]

I don't see how extern types are particularly interesting here. In terms of needs_drop and drop_in_place, I think extern type Foo acts exactly the same as struct Foo. Currently, you can't implement Drop for an extern type but as Ralf says it's not clear that restriction will stay forever, and if it does drop it will continue to look like struct Foo: needs_drop will return true and drop_in_place will call the drop function.

[crumblingstatue]

Since no one seems to have asked this question yet:

The current "idiom" for opaque types is having a _opaque: [u8; 0] field.
This approach supports having PhantomData fields for supporting things like adding lifetime annotations on the type.
I make very heavy use of this in rust-sfml:

#[repr(C)]
pub struct Shader<'texture> {
    _opaque: [u8; 0],
    // Keeps track of the lifetime of the texture the Shader borrows,
    // but of which Rust has no idea otherwise, since
    // the field that "borrows" the texture is on the C side
    _texture: PhantomData<&'texture Texture>,
}

How would I go about this using extern types?
If extern types cannot support this, then it's not really a universal replacement for the _opaque: [u8; 0] field "workaround".

[programmerjake]

How would I go about this using extern types?

I would assume by doing something like this:

extern {
    type ShaderExtTy;
}

#[repr(transparent)]
pub struct Shader<'texture> {
    inner: ShaderExtTy,
    _phantom: PhantomData<&'texture Texture>,
}

[crumblingstatue]

@programmerjake
Perhaps... Although it would have to be

#[repr(transparent)]
pub struct Shader<'texture> {
    _phantom: std::marker::PhantomData<&'texture Texture>,
    inner: ShaderExtTy,
}

Since only the last field can be unsized.

[CatsAreFluffy]

The third unresolved question shouldn't be a problem anymore, since LLVM now uses opaque pointer types.

[programmerjake]

The third unresolved question shouldn't be a problem anymore, since LLVM now uses opaque pointer types.

it is probably still a problem for rustc_codegen_gcc since libgccjit has separate pointer types for each pointee type.

[RalfJung]

Hopefully, GCC isn't quite so sensitive about which pointee type needs to be picked. (I forgot why it was important for LLVM for void*.) But anyway given that the GCC backend is not complete yet, I don't think we should block this RFC on that -- it's "just" one more item on the list of things whose implementation is still incomplete.