borrowed referent of a &T sometimes incorrectly allowed

[nikomatsakis]

@jorendorf asks on the users forum about a curious discrepancy around fields. It seems that implicit borrows sometimes seem to get overlooked in the borrow checker. This seems like a kind of bad bug, though it's exact scope is unclear until we investigate a bit more.

Here is a variant of @jorendorf's example which is pretty clearly wrong. Here, the block variable is mutably borrowed into x, so it should not be accessible via let p:

#![allow(dead_code)]
pub struct Block<'a> {
    current: &'a u8,
    unrelated: &'a u8,
}

fn bump<'a>(mut block: &mut Block<'a>) {
    let x = &mut block;
    let p: &'a u8 = &*block.current;
	// (use `x` and `p` so enabling NLL doesn't assign overly short lifetimes)
	drop(x);
	drop(p);
}

fn main() {}

I'm guessing that the problem has to do with the logic around borrowing the referent of an &T (in this case, we are borrowing *block.current). In particular, we deem that to be "safe" for the scope of 'a because the data is independently guaranteed to be valid that long (this is reasonable). But we still need to validate that block.current can be (instantaneously) read. It seems we are not doing that. But this is all a hypothesis: I've not dug into the code to validate it.

[nikomatsakis]

I have a fix locally. I'm going to try and evaluate the impact.

[nikomatsakis]

This is gonna' be bad unless we address the problem that makes vec.push(vec.len()) an error.

[arielb1]

How bad?

[nikomatsakis]

@arielb1 I don't know. I'm working through the bootstrap process now, then I'll try to do some sort of crater run to get an idea.

As an aside, this bug dates back to Rust 1.0.0 from what I can tell.

[nikomatsakis]

Branch in my repo is nikomatsakis/issue-38899 btw. I did a crater run which showed 160 root regressions: https://gist.github.com/nikomatsakis/9279f3023ecbec3d1f09730671c7884b

[nikomatsakis]

I don't know what percentage of those would be fixed by the method call temporary thing. Certainly not all (I just opened one at random and found it would not have been).

[aidanhs]

Not sure why ayzim is listed as a root regression when the log shows the culprit is lazy_static (same is true of a few other crates I clicked at random).

[nikomatsakis]

@aidanhs yeah, it seems like it gets a bit goofy sometimes in that respect.

[arielb1]

@nikomatsakis

If only say 20% of these examples are fixed by method call temporaries, I don't see why we shouldn't do the normal lint -> hard error sequence here.

Sure it's sad to break basically every non-trivial rust project ever, but method call temporaries won't help/

[nikomatsakis]

@arielb1 it seems worth trying to investigate a bit more deeply. However, the experience in rustc was quite different -- the majority of errors were fixed. I think I had only one problem (out of ... 10 or so?) that was fixed a different way.

[arielb1]

The lazy_static example looks like it's tracking a borrow of the dereference of an unsafe pointer *self.0 for some reason - that's a false positive.

In rustc, it seems that:
2 error can't be fixed by multiphase borrows (rustc_resolve::macros, getopts), of them the rustc_resolve::macros issue is a "hard-to-fix" false positive and the getopts fix is a "technically UB" case.
19 errors are of the foo.mutate(&*foo.field) kind.
1 errors will be fixed by making reborrows multiphase, but will not be fixed by changing order of evaluation (rustc_typeck::check::upvar)

[aidanhs]

@arielb1 isn't it tracking the mutable borrow of the pointer itself rather than its dereference? Which then conflicts with dereferencing the pointer later on.

[arielb1]

@aidanhs

After looking at it again, sure. It's just another case of #6393.

[nikomatsakis]

@arielb1 right; so 19/23 (80%) seems pretty significant.

[Ms2ger]

CC @jorendorff

[bstrie]

This is gonna' be bad unless we address the problem that makes vec.push(vec.len()) an error.

@nikomatsakis does this imply that NLL would allow us to fix this without breakage?

[nikomatsakis]

@bstrie

does this imply that NLL would allow us to fix this without breakage?

No, some things would still break.

[bstrie]

@nikomatsakis I see that you removed your nomination from this bug without assigning it a priority, was that intentional?

[arielb1]

@bstrie

This is "waiting for 2φB", which is waiting for an RFC + MIR borrowck.

[bstrie]

This is gonna' be bad unless we address the problem that makes vec.push(vec.len()) an error.

The "Enable nested method calls" RFC (a.k.a. "two-phase borrows", which is what I presume @arielb1 meant by "2φB") was recently accepted: rust-lang/rfcs#2025 .

Even considering MIR borrowck as a blocker, given that that RFC seems to be a high priority for the forthcoming three-month impl period, is it safe to assume that we'll have something in nightly by December with which we can start doing cargobomb runs to determine exactly how painful this bug will be to fix?

Possibly overstepping my bounds my assigning this a P-high, but it really does concern me the most of all the open and in-stable soundness bugs, and I have a longstanding pet peeve concerning unprioritized soundness bugs. :P

[aidanhs]

Per #46557 (comment): I think it's reasonable to say this is a P-medium issue until the fix is in master and enabled by default (I'm open to disagreement though!).

[nikomatsakis]

@aidanhs and I discussed. Since there is a test and this is fixed in MIR borrow check, closing as duplicate of #47366.

[nikomatsakis]

Er, perhaps that was premature. =)

[pnkfelix]

@nikomatsakis so wait, what is the policy then for #47366? Are we closing issues that are fixed by #![feature(nll)] as long as they have a test? Or are we just downgrading them to P-medium?

(And either way, should this bug be linked from checklist in #47366 description?)

[pnkfelix]

Ah according to #46557 (comment) the policy is that we're removing WG-compiler-nll label from such issues? That seems ... reasonable. Since then the work queue (either in-progress or to-do) should be identifiable from that label?

[nikomatsakis]

@pnkfelix yeah

[pnkfelix]

NLL (migrate mode) is enabled in all editions as of PR #59114.

The only policy question is that, under migrate mode, we only emit a warning on this (unsound) test case. Therefore, I am not 100% sure whether we should close this until that warning has been turned into a hard-error under our (still in development) future-compatibility lint policy.