Coherence can be bypassed by an indirect impl for a trait object

[arielb1]

Comments

The check for manual impl Object for Object only makes sure there is no direct impl Object for dyn Object - it does not consider such indirect impls. Therefore, you can write a blanket impl<T: ?Sized> Object for T that conflicts with the builtin impl Object for dyn Object.

Reproducer

edit: minimal reproducer from #57893 (comment)

trait Object<U> {
    type Output;
}

impl<T: ?Sized, U> Object<U> for T {
    type Output = U;
}

fn foo<T: ?Sized, U>(x: <T as Object<U>>::Output) -> U {
    x
}

fn transmute<T, U>(x: T) -> U {
    foo::<dyn Object<U, Output = T>, U>(x)
}

---

I had some difficulties with getting the standard "incoherence ICE" reproducer, because the object candidate supersedes the impl candidate in selection. So here's a "transmute_lifetime" reproducer.

trait Object {
    type Output;
}

trait Marker<'b> {}
impl<'b> Marker<'b> for dyn Object<Output=&'b u64> {}

impl<'b, T: ?Sized + Marker<'b>> Object for T {
    type Output = &'static u64;
}

fn foo<'a, 'b, T: Marker<'b> + ?Sized>(x: <T as Object>::Output) -> &'a u64 {
    x
}

fn transmute_lifetime<'a, 'b>(x: &'a u64) -> &'b u64 {
    foo::<dyn Object<Output=&'a u64>>(x)
}

// And yes this is a genuine `transmute_lifetime`!
fn get_dangling<'a>() -> &'a u64 {
    let x = 0;
    transmute_lifetime(&x)
}

fn main() {
    let r = get_dangling();
    println!("{}", r);
}

Duplicates, see also

• Associated type constraints on super traits allowing for unsound transmutation to trait objects  #114389

[scalexm]

@arielb1 Interesting. I guess we would need a deeper check, something along the lines of:

impl<T> Object for SomeType<T> where WC {
/* ... */
}

// If the following goal has a non-empty set of solutions, reject the impl.
exists<T> {
    exists<U> {
        Unify(dyn Object<Output = U>, SomeType<T>),
        dyn Object<Output = U>: WC
    }
}

cc @nikomatsakis

[arielb1]

Sure.

What makes this non-trivial is auto-traits (dyn Object + Send is a distinct wrt. unification from dyn Object), and to some extent binders (unless you ignore them in unification). i.e., supposing the impl you have is

impl<T> Object<'a> for SomeType<T> where WC

Then the "ideal" lowering would be something of this format:

// If the following goal has a non-empty set of solutions, reject the impl.
exists<T: type, 'a('x): lifetime -> lifetime, U('x): lifetime -> type, AT: list of auto-traits> [
    Unify(dyn for<'s> Object<'a('s), Output = U('s)> + AT, SomeType<T>),
    dyn for<'s> Object<'a('s), Output = U('s)> + AT: WC
}

If you ignore binders in unification (i.e., you consider for<'a> Object<'a, Output=&'a T> to be the same as Object<'x, Output=&'y T> in coherence), then you don't have to deal with the HRTB problem, but you still have to deal with auto-traits:

// If the following goal has a non-empty set of solutions, reject the impl.
exists<T: type, 'a: lifetime, U: type, AT: list of auto-traits> [
    Unify(dyn Object<'a, Output = U> + AT, SomeType<T>),
    dyn Object<'a, Output = U> + AT: WC
}

While I don't think this is insurmountable, it is fairly annoying and places some restrictions on how you encode auto-traits.

[Aaron1011]

I'm interested in working on this.

[Aaron1011]

It turns out that only one trait is necessary to reproduce this: (playground)

trait Object {
    type Output;
}

impl<T: ?Sized> Object for T {
    type Output = &'static u64;
}

fn foo<'a, T: ?Sized>(x: <T as Object>::Output) -> &'a u64 {
    x
}

fn transmute_lifetime<'a, 'b>(x: &'a u64) -> &'b u64 {
    foo::<dyn Object<Output=&'a u64>>(x)
}

// And yes this is a genuine `transmute_lifetime`!
fn get_dangling<'a>() -> &'a u64 {
    let x = 0;
    transmute_lifetime(&x)
}

fn main() {
    let r = get_dangling();
    println!("{}", r);
}

This seems quite bad, as simply writing a blanket impl is enough to expose the issue.

[Aaron1011]

One way to fix this issue would be the following:

If a trait has any associated items, and a blanket impl exists for it, that trait cannot be object-safe.

This is pretty extreme - however, the only other solution I can think of would be to deny all blanket impls of a trait with associated items - which seems even worse, given that the trait might not have even been object-safe to begin with.

[Centril]

Unfortunately, it gets worse. Here's an implementation of mem::transmute in safe Rust:

trait Object<U> {
    type Output;
}

impl<T: ?Sized, U> Object<U> for T {
    type Output = U;
}

fn foo<T: ?Sized, U>(x: <T as Object<U>>::Output) -> U {
    x
}

fn transmute<T, U>(x: T) -> U {
    foo::<dyn Object<U, Output = T>, U>(x)
}

I think blame should be assigned to not normalizing <T as Object<U>>::Output to U in fn foo. If that happened, foo::<dyn Object<U, Output = T>, U>(x) would not type-check.

I checked the code above with godbolt and the problem has existed since Rust 1.0.0.

[jonas-schievink]

The first comments in the thread seem to indicate that the problem is instead with the blanket impl itself being accepted by the compiler.

If I understood correctly, @Centril's impl above can be applied to dyn Object<U, Output=T> for any U and T, and defines Output=U. However, a trait object like dyn Object<U, Output=T> already provides an implicit impl with the same input types, but a different output type, which is incoherent.

[jonas-schievink]

Seems like a good idea to discuss in language and compiler team, nominating.

[Centril]

Notably, in my reproducer above, if you change fn foo to:

fn foo<T: ?Sized + Object<U>, U>(x: <T as Object<U>>::Output) -> U {
    x
}

then you will get:

error[E0308]: mismatched types
  --> src/lib.rs:10:5
   |
9  | fn foo<T: ?Sized + Object<U>, U>(x: <T as Object<U>>::Output) -> U {
   |                                                                  - expected `U` because of return type
10 |     x
   |     ^ expected type parameter, found associated type
   |
   = note: expected type `U`
              found type `<T as Object<U>>::Output`

[jonas-schievink]

Is that not just due to where-clauses having precedence over impls? AFAIK trait selection will prefer the Object<U> impl in the bounds, which doesn't have an assoc. type specified, so the compiler can't normalize it.

[Aaron1011]

I made a first attempt at a fix here: https://github.com/Aaron1011/rust/tree/fix/trait-obj-coherence

My idea was to extend the obligations generated by ty::wf::obligations to include projection predicates of the form:

<dyn MyTrait<type Assoc=T> as MyTrait>::Assoc = T.

This is combeind with an extra flag in SelectionContext to force impl candidates to be chosen over object candidates. In the case of an incoherent blanket impl (e.g. any of the reproduces in this thread), <dyn MyTrait<type Assoc=T> as MyTrait>::Assoc should project to the type specified in the blanket impl. This will fail to unify with T, correctly causing a compilation error.

Unfortunately, I wasn't able to get this working, due to how the well-formed predicate generation is integrated with the rest of the compiler. I would need to significantly refactor FulfillmentContext, SelectionContext, and/or Predicatein order to keep track of the extra flag that needs to be passed toSelectionContext`.

I'm hoping that someone can come up with a better approach. However, any solution will need to deal with the fact that SelectionContext masks attempts to detect the incoherence by discarding the impl candidate in favor of the object candidate.

[RalfJung]

This is interesting! The fact that foo in @Centril's example even gets accepted shows that we aggressively exploit coherence during type-checking: because we saw an impl that uses a certain associated type, we type-check assuming that that is the impl that got used here. Unfortunately, as the example here shows, that is in conflict with the "implicit impl" that we get for trait objects.

I wouldn't be sad if we would fix this by being less aggressive about exploiting coherence during type-checking -- that makes analysis much more complicated, and this example shows why. But there's likely already tons of code out there that exploits this. Another fix is to refuse to create trait objects that "contradict" the kind of coherence knowledge that the type checker might exploit, but that seems extremely fragile.

@Aaron1011 your plan seems to be to encode in the type of foo the fact that we exploited coherence? However this "exploitation" happens in the body of the function, so what exactly gets exploited cannot be known yet when computing the well-formedness predicate for foo. Thus I guess what you want to / have to do is to aggressively look for all things one might be able to exploit up-front, add them all to the well-formedness obligations, and then during type-checking the boy look only at the obligations and not exploit coherence any more (as doing so would introduce scary bugs if the two systems for getting such type equalities out of coherence would ever not agree).

Once again, exploiting assumptions implicitly instead of inferring them to an explicit form proves to be an issue. This reminds me that we still implicitly exploit WF everywhere instead of turning that into explicit assumptions...

[pnkfelix]

triage: P-high due to unsoundness. Leaving nominated in hope of discussing today. Not assigning to anyone yet.

[pnkfelix]

assigning to @nikomatsakis with expectation that they will delegate. (and removing nomination label)

[Centril]

Still nominated for t-lang.

[arielb1]

@RalfJung

I am not sure that giving up on coherence is the right choice here. I think it is too easy to split the code into functions, such that no one function sees the coherence violation:

struct Data<T: ?Sized, U> where T: Object<U> {
    data: <T as Object<U>>::Output
}

trait Object<U> {
    type Output;
}

trait Mark {
    type Output;
}

impl<T: ?Sized, U: Mark<Output=V>, V> Object<U> for T {
    type Output = V;
}

fn iso_1<T, U>(data: T) -> Data<dyn Object<U, Output=T>, U> {
    // in any coherence-less solution, this shouldn't "see" the
    // blanket impl, as it might not apply (e.g., `Local` might
    // be in a third-party crate).
    Data { data }
}

fn iso_2<X: ?Sized, U, V>(data: Data<X, U>) -> V
    where U: Mark<Output=V>
{
    // similarly, this shouldn't "see" the trait-object impl.
    data.data
}

fn transmute_m<T, U, V>(data: T) -> V
    where U: Mark<Output=V>
{
    // This function *also* shouldn't see the blanket impl - `Local`
    // might be in a third-party crate.
    iso_2::<dyn Object<U, Output=T>, U, V>(
        iso_1::<T, U>(data)
    )
}

// These 3 functions could be in a child crate

struct Local<T>(T);

impl<T> Mark for Local<T> {
    type Output = T;
}

fn transmute<T, U>(x: T) -> U {
    // and this function shouldn't see *anything* that looks like a
    // problematic associated type.
    transmute_m::<_, Local<_>, _>(x)
}

[arielb1]

I think that conditioning object safety on an object type being coherent is a reasonable-enough way out of this.

If we want to be tricky, we might put the condition "directly" on the impl - i.e., have ObjectCandidate not apply if an ImplCandidate might "theoretically" apply to any substitution of this type. This would however require doing coherence-style negative reasoning during selection, which would be ugly.

[RalfJung]

@arielb1 your iso_2 exploits coherence, so "giving up on exploiting coherence" would not let that function type-check.

[QuineDot]

I thought I'd just note that versions of the "erased trait" pattern may rely on the specialization discussed in this ticket.

+// relevant diff from the blog post

+    impl<'data, Item> AsyncIter for dyn ErasedAsyncIter<Item = Item> + 'data {
+        type Item = Item;
+        type Next<'me> = Pin<Box<dyn Future<Output = Option<Item>> + 'me>>
+        where
+            Item: 'me,
+            'data: 'me,
+        ;
+
+        fn next(&mut self) -> Self::Next<'_> {
+            self.next()
+        }
+    }

     impl<T> ErasedAsyncIter for T
     where
-        T: AsyncIter,
+        T: AsyncIter + ?Sized,

(Or this example.)

[nikomatsakis]

In today's @rust-lang/types meeting, we discussed this issue and decided we need a deep-dive covering the various details, examples, and attempts that have been made to fix it. Filed rust-lang/types-team#100 to cover that.

[Jules-Bertholet]

• Any is useful only in trait-object form (you can just use TypeId::of otherwise).
• The only way to obtain a dyn Any trait object is by coercing from a Sized type.

Given these facts, might it be justifiable, as a soundness fix, to add a Sized bound to T in the impl<T> Any for T blanket impl? This would make the vtable trait impl and the blanket impl no longer overlap. It would also allow adopting the fix of making blanket impls suppress conflicting vtable impls, as Any's behvior seems to be the main impediment that.

One wrinkle is that downstream impls of Any for custom DSTs should probably be forbidden, but Rust doesn't (yet) have a mechanism for that.

[Jules-Bertholet]

Making dyn Trait not dyn safe if:

• there is a dyn overlapping impl and
  • the trait has associated non-functions
  • that do not have where Self: Sized (not implemented)

This solution is unnecessarily restrictive. The problem is not the dyn-safety of the trait, the problem is the conflict between the blanket and compiler-generated trait impls. Suppressing the latter is sufficient to resolve the incoherence. For example, the trait below could be fully object safe:

trait Foo {
    type Assoc;
}

impl<T: ?Sized> Foo for T {
     type Assoc = i32;
}

[Jules-Bertholet]

(HRTBs complicate the story somewhat. Given trait Foo<'a> {}, an impl<T: ?Sized> Foo<'static> for T should suppress the compiler-generated impl<'a> Foo<'a> for dyn for<'any> Foo<'any> {}.)

[compiler-errors]

Given these facts, might it be justifiable, as a soundness fix, to add a Sized bound to T in the impl Any for T blanket impl?

#115006

I tried that, and it's not really great.

[Jules-Bertholet]

Hmm, looking at those regressions, perhaps edition-dependent name resolution offers us a way out. Potential plan:

1. Split the Any trait into two traits, edition2024#Any and edition2021#Any.
2. On edition >= 2024, Any always resolves to edition2024#Any.
3. On edition <= 2021, Any resolves to edition2021#Any in most cases. But when used as part of a trait object (dyn Any) or supertrait bound (trait Foo: Any), it resolves to edition2024#Any. (The latter condition ensures that upcasts to dyn Any work as expected.)
4. edition2021#Any has a blanket impl for all T: ?Sized (that suppresses the vtable impl). edition2024#Any is implemented for all T: ?Sized + Aligned (so as not to break protobuf<=0.2, which has a trait with an Any supertrait and an impl for str; RFC: Aligned trait rfcs#3319 is therefore a prerequisite).

I think that would fix the vast majority of the crater regressions.

[Jules-Bertholet]

Another possible solution:

• make blanket impls always suppress overlapping vtable impls, and
• make inherent methods on dyn Trait shadow trait methods of the same name, and
• add an inherent type_id() method to dyn Any that uses the vtable impl, and
• make method call resolution on trait objects prefer inherent impls from upcasting over trait impls.

[lcnr]

#57893 (comment) this also affects TryFuture

https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=544e1eb2b706aed44f83f25725253d67

[saethlin]

(already linked above via back-link) FYI the reproducer for this miscompile ICEs Miri because said reproducer currently compiles to invalid MIR: #127667