regression: encountered mutable pointer in final value when "outer scope" rule applies in const/static with interior mutability

[Mark-Simulacrum]

• https://crater-reports.s3.amazonaws.com/beta-1.77-3/beta-2024-02-18/gh/adrianromero.rustexamples/log.txt
• https://crater-reports.s3.amazonaws.com/beta-1.77-3/beta-2024-02-18/gh/emirror-de.boa-test/log.txt
• https://crater-reports.s3.amazonaws.com/beta-1.77-3/beta-2024-02-18/gh/glenfordwilliams.rustybrain/log.txt
• https://crater-reports.s3.amazonaws.com/beta-1.77-3/beta-2024-02-18/gh/traxys.pac_proxy/log.txt
• https://crater-reports.s3.amazonaws.com/beta-1.77-3/beta-2024-02-18/reg/Boa-0.13.1/log.txt
• https://crater-reports.s3.amazonaws.com/beta-1.77-3/beta-2024-02-18/reg/epo-1.0.1/log.txt

[INFO] [stdout] error: encountered mutable pointer in final value of constant
[INFO] [stdout]    --> /opt/rustwide/cargo-home/registry/src/index.crates.io-6f17d22bba15001f/Boa-0.13.0/src/builtins/map/mod.rs:341:9
[INFO] [stdout]     |
[INFO] [stdout] 341 |         const JS_ZERO: &JsValue = &JsValue::Rational(0f64);
[INFO] [stdout]     |         ^^^^^^^^^^^^^^^^^^^^^^^

[apiraino]

bisection seems to point to #119044 ? cc @RalfJung @oli-obk

searched nightlies: from nightly-2024-01-01 to nightly-2024-02-26
regressed nightly: nightly-2024-01-24
searched commit range: d5fd099...5d3d347
regressed commit: 6265a95

bisected with cargo-bisect-rustc v0.6.8

Host triple: x86_64-unknown-linux-gnu
Reproduce with:

cargo bisect-rustc --start=2024-01-01 -- test 

[RalfJung]

Oh strange, this landed a while ago and so far we got no reports I think...

Boa-0.13.0/src/builtins/map/mod.rs:341:9

That seems to be a deprecated crate, the repo has moved on to a new version. Not sure how to get source code for it.

The epo crate also regresses via boa

[INFO] [stdout] error: encountered mutable pointer in final value of constant
[INFO] [stdout]    --> /opt/rustwide/cargo-home/registry/src/index.crates.io-6f17d22bba15001f/boa_engine-0.14.0/src/builtins/mod.rs:205:9
[INFO] [stdout]     |
[INFO] [stdout] 205 |         const UNDEFINED: &JsValue = &JsValue::Undefined;
[INFO] [stdout]     |    

That code I found here. (The current version of the crate on master doesn't have this any more, at least not in that file.) But so far my attempts at extracting an example that reproduces the issue failed. I assume it has to do with that JsObject type, that mentions some Gc things so it might be particularly tricky.

[apiraino]

I can reproduce the crash by compiling an old tarball of the boa crate (v0.13). But it also crashes stops compiling on rustc stable (1.76) for other reasons (glob imports), so unsure how helpful can that be.

I have the feeling that trying to get to a reproducer would be expensive in terms of time vs. benefit 🤔 But I am unsure if behind this crash compile error lies something useful for us to know

[RalfJung]

"crash"? It doesn't crash for any of them I hope, just show this error?

I am certainly surprised that there is a regression at all, in particular given that an earlier version of that PR (#116745) has been cratered and didn't find anything. So in that sense it would be interesting to extract a testcase here.

[apiraino]

"crash"? It doesn't crash for any of them I hope, just show this error?

I stand corrected. Compile stops and exits with the error reported in this issues (sorry for mixing up apple and oranges).

[RalfJung]

I copied more of the type definition into its own file, still no luck.

But I think the only way this can happen is if different parts of the compiler disagree on whether a type has interior mutability, which is quite concerning.

[RalfJung]

Aah! I finally got it:

use std::cell::Cell;

pub enum JsValue {
    Undefined,
    Object(Cell<bool>),
}

impl ::std::ops::Drop for JsValue {
    fn drop(&mut self) {}
}

const UNDEFINED: &JsValue = &JsValue::Undefined;

fn main() {
}

The Drop is key. In boa it's generated by a derive which made this hard to spot...

So, I think what happens here is that we don't do promotion because of the Drop, but the "outer scope" rule still applies. The reference is created to JsValue::Undefined and some analysis concludes that this is fine since while the type has interior mutability, the value doesn't. But when we actually evaluate the initializer for UNDEFINED we just see a shared reference created to a type that has interior mutability so we mark it as potentially mutable, and then interning sees that and rejects it.

Honestly my preference would be to reject this in const-checking, IMO we shouldn't allow references to interior mutable types under the "outer scope" rule. Note that both Stacked Borrows and Tree Borrows actually currently allow writing through &JsValue::Undefined (or they would if we turned off promotion, which changes the code in a way that UB arises without even involving the aliasing model). In that sense both promotion and const-checking are really playing with fire when they accept enum variants of enums where other variants have interior mutability.

[RalfJung]

So... actually allowing this code again will be quite tricky. Not sure if we want to do that.

What I would prefer to do is make this check entirely type-based, and thus give a nicer error for when this happens:

rust/compiler/rustc_const_eval/src/transform/check_consts/check.rs

Lines 487 to 491 in 010f394

	let borrowed_place_has_mut_interior = qualifs::in_place::<HasMutInterior, _>(
	self.ccx,
	&mut |local| self.qualifs.has_mut_interior(self.ccx, local, location),
	place.as_ref(),
	);

But that is quite the departure from our attempts to handle interior mutability in a value-based way, i.e. with these "qualifs".

EDIT: That's now implemented in #121786.

[RalfJung]

So... what shall we do here? Retroactively ask T-lang to approve the part of #119044 that is a breaking change? That PR went through FCP but we didn't realize it would be a breaking change.

[oli-obk]

So, I think what happens here is that we don't do promotion because of the Drop,

I'm assuming changing that, specifically for promotion in consts/statics is not gonna fly? We are already trying to remove/limit the method call exception that promotion has there

[RalfJung]

You mean making the code work by having promotion kick in regardless? That sounds very tricky, we'd have to ensure this happens only when it's the final value of the const being promoted, not some intermediate value.

If we absolutely must accept this code then we can make const-eval do a type-driven traversal of the x in &x expressions to check whether there's any UnsafeCell anywhere. But that would introduce UB into code that I don't think should have UB (when x is not valid at the given type) and it could be terrible for performance...

[oli-obk]

If we absolutely must accept this code then we can make const-eval do a type-driven traversal of the x in &x expressions to check whether there's any UnsafeCell anywhere. But that would introduce UB into code that I don't think should have UB (when x is not valid at the given type) and it could be terrible for performance...

Yea, i discarded this option right away and was looking for alternatives

[RalfJung]

Yea, i discarded this option right away and was looking for alternatives

I think that option is still better than making promotion cover this.^^

[oli-obk]

I guess we each hate each other's preferred solution, and neither of us love our own preferred solution. So...

@rust-lang/lang We did an accidental breaking change in #119044

The following example does not compile on beta anymore, but does compile on stable:

use std::cell::Cell;

pub enum JsValue {
    Undefined,
    Object(Cell<bool>),
}

impl ::std::ops::Drop for JsValue {
    fn drop(&mut self) {}
}

const UNDEFINED: &JsValue = &JsValue::Undefined;

with

error: encountered mutable pointer in final value of constant
  --> src/main.rs:12:1
   |
12 | const UNDEFINED: &JsValue = &JsValue::Undefined;
   | ^^^^^^^^^^^^^^^^^^^^^^^^^

what happens here is that we don't do promotion because of the Drop, but the "outer scope" rule still applies. The reference is created to JsValue::Undefined and const checking concludes that this is fine since while the type has interior mutability, the value doesn't. But when we actually evaluate the initializer for UNDEFINED we just see a shared reference created to a type that has interior mutability so we mark it as potentially mutable, and then interning sees that and rejects it.

The reason this worked before was that before #119044 we did a value based (not just type based) interning traversal, so we picked up on the fact that the specific value has no interior mutability.

We have come up with two ways to get back the previous behaviour:

1. we can make const-eval do a type-driven traversal of the x in &x expressions to check whether there's any UnsafeCell anywhere.
2. special case promotion to trigger for these values, even though their type is Drop and !Freeze

The issues with those are respectively:

1. that would introduce UB into code that should not have UB (when x is not valid at the given type) and it would be (very) terrible for performance...
2. terrible extension to an already fragile part of the compiler: promotion.

So we would like to just keep rejecting this code, as it requires special cases to support instead of falling out of a set of simple rules.

[RalfJung]

A full revert will be hard since there's a bunch of follow-on work that landed since then that either further changes the interner, or requires the new interner.

I think it's possible to just turn that error ("encountered mutable pointer in final value") into a future-compat lint. Other than this bug, it can only be triggered by unsound flags/features (const_heap, miri-unleashed). Then we also get an idea of whether there is more code out there that relies on this pattern.

[RalfJung]

I was wondering why promotion doesn't trigger this error; turns out the interner simply disables it for promoteds. I guess we have that case covered in our test suite (or in the standard library itself).

I think increasingly my tendency is to fix this by changing &x semantics such that under specific conditions it traverses x to determine whether there is actually an UnsafeCell there. Ideally those conditions are "only for borrows in the tail expression of a const/static that got elevated to module scope". I don't know how easy it is to determine this; one extreme option would be "only in const/static initializers" (but not in const fn bodies).

[cuviper]

This was downgraded in #122204 and beta-backported in #122524.

[RalfJung]

We should still track this issue, it's just not a stable-to-beta regression any more.

[RalfJung]

This was downgraded in #122204 and beta-backported in #122524.

Tracking issue for that: #122153.

[RalfJung]

This was discussed in a T-lang meeting.

There was general sympathy with the t-opsem desire to not do value-based reasoning for interior mutability. But there also wasn't a clear idea for how value-based reasoning could be removed from both promotion and tail-expression checking. The one thing everyone agreed on was to at least do a crater run to see how bad it'd be. Then based on that, maybe there could be an edition-based transition plan.

For this concrete issue, the goal is to monitor the tracking issue of the future compat lint (#122153) to see if anyone speaks up -- to see if there's notable code out there not covered by crater that would regress if we just made the current check a hard error. Depending on that we could either take the hit and break old versions of boa, or do value-based reasoning in const-eval when determining the mutability of shared references (at least in the "root frame" of const-eval, i.e. the initializer expression of the const itself).

[traviscross]

@rustbot labels -I-lang-nominated

Thanks to @RalfJung and @oli-obk for joining us in the meeting to talk through this. As @RalfJung mentioned, there are some next steps here, so please renominate when we have more data and a new proposal for how to move forward based on that.

[RalfJung]

The crater experiment in #122789 which entirely removed value-based reasoning for interior mutability had almost 4000 regressions. I think we can conclude that this is not an option.^^

[RalfJung]

I'm proposing we fix this by removing the lint and just accepting the code, unfortunate as that is. See #128543.

[Diggsey]

Mutating a non-internally-mutable type is UB, but that doesn't necessarily mean that mutating an internally-mutable type is not UB.

Presumably there could just be a second rule that mutating read-only memory is also UB, even if the type being mutated is internally mutable.

As long as safe code is prevented from mutating these values, then the only danger is that of people not realising this is UB when writing unsafe code - this could be mitigated by a compiler lint, which does a "best effort" attempt at detecting writes to read-only memory.

[RalfJung]

Presumably there could just be a second rule that mutating read-only memory is also UB, even if the type being mutated is internally mutable.

Yes, we already have such a rule. I was just hoping it would not be necessary...

Also see rust-lang/unsafe-code-guidelines#493, which has the same root cause but is worse because it is not at all clear to the user that there's any "read-only memory" here.