From 099e2b51d6ee485d7a8ea5c9b460b338073b81d0 Mon Sep 17 00:00:00 2001
From: "Felix S. Klock II" <pnkfelix@pnkfx.org>
Date: Fri, 7 May 2021 15:33:32 -0400
Subject: [PATCH 01/20] Blog post explaining the fingerprint issue.

---
 posts/2021-05-07-caught-red-handed.md | 130 ++++++++++++++++++++++++++
 1 file changed, 130 insertions(+)
 create mode 100644 posts/2021-05-07-caught-red-handed.md

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
new file mode 100644
index 000000000..405eb46f0
--- /dev/null
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -0,0 +1,130 @@
+---
+layout: post
+title: "1.52.0, fingerprints and compiler error forensics"
+author: Felix Klock
+team: the compiler team <https://www.rust-lang.org/governance/teams/compiler>
+---
+
+The Rust teams are always excited to report on new features offered with each release. Sometimes, however, an important change that is not yet "fully baked" gets accidentally included in a release.
+
+There was an instance of this in yesterday's release, 1.52.0, which added a new bit of internal-consistency checking, called "incremental compilation hash verification" (abbreviated `verify-ich`). This check is also called an "unstable fingerprint" check, because the diagnostic it currently prints look [like this](https://github.com/rust-lang/rust/issues/84336):
+
+```
+thread 'rustc' panicked at 'assertion failed: `(left == right)`
+  left: `Some(Fingerprint(4565771098143344972, 7869445775526300234))`,
+   right: `Some(Fingerprint(14934403843752251060, 623484215826468126))`: found unstable fingerprints for <massive text describing rustc internals elided>
+
+error: internal compiler error: unexpected panic
+
+note: the compiler unexpectedly panicked. this is a bug.
+```
+
+This internal-consistency check, as stated in the diagnostic, yields an "Internal Compiler Error" (or ICE). In other words, it represents a bug in the internals of the Rust compiler itself. In *this* case, though, the ICE is revealing a bug that 1.) is very likely to predate the 1.52.0 release and 2.) could result in miscompilation if it had not been caught by `verify-ich`.
+
+In other words: If you are seeing the above Internal Compiler Error, you may be tempted to respond by reverting to the 1.51 release. I want to stress that a downgrade is *not* the best response to this problem.
+
+This post is going to:
+
+ 1. Explain [what the check does][part1], at a high level,
+ 2. Explain [how the check is presenting itself][part2] in the Rust 1.52.0 release,
+ 3. Tell you [what you should do][part3] if you see an unstable fingerprint on your project,
+ 4. Describe our plans for [how the Rust project will address][part4] the problems discussed here.
+
+[part1]: #what-are-fingerprints-why-are-we-checking-them
+[part2]: #how-does-this-show-up
+[part3]: #what-should-a-rust-programmer-do-in-response
+[part4]: #what-is-the-rust-project-going-to-do-to-fix-this
+
+## What are fingerprints? Why are we checking them?
+
+The Rust compiler has support for "incremental compilation", which has been described in a [2016 blog post][]. When incremental compilation is turned on, the compiler breaks the input source into pieces, and tracks how those input pieces influence the final build product. Then, when the inputs change, it detects this and reuses artifacts from previous builds, striving to expend effort solely on building the parts that need to respond to the changes to the input source code.
+
+[2016 blog post]: https://blog.rust-lang.org/2016/09/08/incremental.html
+
+Fingerprints are part of our architecture for detecting when inputs change. More specifically, a fingerprint (along with some other state to establish context) is a 128-bit value intended to uniquely identify internal values used within the compiler. Some compiler-internal results are stored on disk ("cached") between runs. Fingerprints are used to validate that a newly computed result is unchanged from the cached result. (More details about this are available in the [relevant chapter of the rustc dev guide][rustc-dev-guide-fingerprints].)
+
+[rustc-dev-guide-fingerprints]: https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation-in-detail.html#checking-query-results-for-changes-hashstable-and-fingerprints
+
+The `verify-ich` check is a safeguard asserting internal inconsistency of the fingerprints. When loading a value from the cache, it recomputes the fingerprint, and compares the resulting value against the fingerprint stored in the cache from the previous compiler run. There are multiple ways that a fingerprint mismatch could arise, but they all represent bugs within the Rust compiler itself.
+
+## History of deployment
+
+We [initially added][pr-45867] `verify-ich` as a tool to use when developing rustc itself, back in 2017; it was solely provided via an unstable `-Z` flag, only available to nightly and development builds.
+
+More recently, in March, we encountered a [miscompilation][issue-82920] that led us to [turn on `verify-ich` by default][pr-83007]. The Rust compiler team decided it was better to catch fingerprint problems and abort compilation, rather than allow for potential miscompilations (and subsequent misbehavior) to sneak into Rust programmer's binaries.
+
+[pr-45867]: https://github.com/rust-lang/rust/pull/45867
+[issue-82920]: https://github.com/rust-lang/rust/issues/82920
+[pr-83007]: https://github.com/rust-lang/rust/pull/83007
+
+When we first turned on `verify-ich` by default, we assumed we would have time to iron out all the known causes of unstable fingerprints. When we realized this assumption was false, we started [making plans][issue-84970] to improve the user-experience, so that the diagnostic issued by the check would do a better job of telling the programmer what to do in response. But we made a mistake: We thought that the switch for this check was not going to be on the Rust stable channel until version 1.53.
+
+[issue-84970]: https://github.com/rust-lang/rust/issues/84970
+
+It turns out `verify-ich` was turned on in version 1.52.0, which was [released yesterday][].
+
+[released yesterday]: /2021/05/06/Rust-1.52.0.html
+
+## How does this show up
+
+
+Essentially, for some crates, certain sequences of edit-compile cycles will cause `rustc` to hit the "unstable fingerprints" ICE. I showed one example at the start of this blog post.
+
+Another recent example looks [like this](https://github.com/rust-lang/rust/issues/85039):
+
+```
+thread 'rustc' panicked at 'found unstable fingerprints for predicates_of(core[b0ed]::convert::From): \
+GenericPredicates { parent: None, predicates: [(Binder(TraitPredicate(<Self as std::marker::Sized>)), \
+/home/coolreader18/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/cor\
+e/src/convert/mod.rs:365:20: 365:25 (#0)), (Binder(TraitPredicate(<T as std::marker::Sized>)), /home/c\
+oolreader18/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/c\
+onvert/mod.rs:365:16: 365:17 (#0)), (Binder(TraitPredicate(<Self as std::convert::From<T>>)), /home/co\
+olreader18/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/co\
+nvert/mod.rs:365:1: 365:25 (#0))] }', /rustc/88f19c6dab716c6281af7602e30f413e809c5974/compiler/rustc_q\
+uery_system/src/query/plumbing.rs:593:5
+```
+
+They all arise from inconsistencies when comparing the incremental-compilation cache stored on disk against the values computed during a current `rustc` invocation, which means they all arise from using incremental compilation.
+
+There are three ways that you may have incremental compilation turned on: You may have set the [environment variable][env-vars] `CARGO_INCREMENTAL=1`, or you may have enabled the `build.incremental` [setting in your Cargo.toml][cargo-toml], or you may be building with the `dev` or `test` [profiles][], which default to having incremental compilation enabled.
+
+[env-vars]: https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-reads
+[cargo-toml]: https://doc.rust-lang.org/cargo/reference/config.html#buildincremental
+[profiles]: (https://doc.rust-lang.org/cargo/reference/profiles.html
+
+If your project has not opted into enabling incremental compilation, then none of the fingerprint issues should affect your release builds.
+
+## What should a Rust programmer do in response
+
+The Internal Compiler Error asks you to report a bug, and if you can do so, we still want that information. We *want* to know about the cases that are failing.
+
+But regardless of whether or not you file a bug, the problem here can be resolved by either:
+
+ 1. deleting your incremental compilation cache (e.g. by running `cargo clean`), or
+ 2. force incremental compilation to be disabled, by setting `CARGO_INCREMENTAL=0` in your environment or `build.incremental` to `false` in the `config.toml`.
+
+We recommend that users of 1.52.0 disable incremental compilation, to avoid running into this problem.
+
+We do *not* recommend that users of 1.52.0 downgrade to an earlier version of Rust in response to this problem. As noted above, there is at least one instance of a silent [miscompilation][issue-82920] caused by incremental compilation that was not caught until we added the fingerprint checking.
+
+## What is the Rust project going to do to fix this
+
+### Short-term plan
+
+Based on the number of bug reports that have already come in, we know we need to do something quickly.
+
+We are going to be issuing a point release, 1.52.1. The point release will improve the diagnostic output from `verify-ich`, so that a programmer can more effectively act in response to the message (i.e., they will be told that the problem is due to the use of incremental compilation, and that they need to delete their cache, at the very least).
+
+We might also make incremental compilation opt-in for *all* Cargo [profiles][]; that is, we may turn incremental compilation off for the  `dev` and `test`, so that they match `release` and `bench`, which already have incremental compilation off. (This remains to be decided.)
+
+We do not plan to disable incremental compilation in its entirety. We believe that incremental compilation is providing value for a large number of Rust programmers, and we want to empower them to continue using it, as long as they are equipped with the tools they need to turn it off if they encounter this issue.
+
+### Long-term plan
+
+The long-term plan is to fix the bugs! Incremental compilation is the only realistic way for the Rust compiler to be able to provide a fast edit-compile-run cycle for all of its programmers, and so we need to address [all of the issues][issue-list] that have been identified thus far via `verify-ich`. (There are 32 such issues as of this writing, though many may be duplicates.)
+
+[issue-list]: https://github.com/rust-lang/rust/issues?q=is%3Aissue+is%3Aopen+unstable+fingerprints
+
+If you want to come help us do it, we would love for you to [join us][] with the effort!
+
+[join us]: https://www.rust-lang.org/community

From ba3ca920aa94e0dc4086e8ad416a6f44a8020636 Mon Sep 17 00:00:00 2001
From: "Felix S. Klock II" <pnkfelix@pnkfx.org>
Date: Fri, 7 May 2021 15:52:27 -0400
Subject: [PATCH 02/20] Aaron1011 pointed out a mistake in my previous draft.

---
 posts/2021-05-07-caught-red-handed.md | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index 405eb46f0..1154754bb 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -45,7 +45,10 @@ Fingerprints are part of our architecture for detecting when inputs change. More
 
 [rustc-dev-guide-fingerprints]: https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation-in-detail.html#checking-query-results-for-changes-hashstable-and-fingerprints
 
-The `verify-ich` check is a safeguard asserting internal inconsistency of the fingerprints. When loading a value from the cache, it recomputes the fingerprint, and compares the resulting value against the fingerprint stored in the cache from the previous compiler run. There are multiple ways that a fingerprint mismatch could arise, but they all represent bugs within the Rust compiler itself.
+The `verify-ich` check is a safeguard asserting internal inconsistency of the fingerprints.
+The compiler stores fingerprints for both cached and uncached values.
+Every time we compute an uncached value, we double-check that its newly computed fingerprint against the finger print stored in the cache.
+There are multiple ways that a fingerprint mismatch could arise, but they all represent bugs within the Rust compiler itself.
 
 ## History of deployment
 

From 5609ca69a8d2186be5e3fd397d39a3311946c006 Mon Sep 17 00:00:00 2001
From: "Felix S. Klock II" <pnkfelix@pnkfx.org>
Date: Fri, 7 May 2021 17:09:03 -0400
Subject: [PATCH 03/20] elided user name (and a bunch of other junk) from the
 second example diagnostic output.

---
 posts/2021-05-07-caught-red-handed.md | 10 +---------
 1 file changed, 1 insertion(+), 9 deletions(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index 1154754bb..d21c8e03d 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -76,15 +76,7 @@ Essentially, for some crates, certain sequences of edit-compile cycles will caus
 Another recent example looks [like this](https://github.com/rust-lang/rust/issues/85039):
 
 ```
-thread 'rustc' panicked at 'found unstable fingerprints for predicates_of(core[b0ed]::convert::From): \
-GenericPredicates { parent: None, predicates: [(Binder(TraitPredicate(<Self as std::marker::Sized>)), \
-/home/coolreader18/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/cor\
-e/src/convert/mod.rs:365:20: 365:25 (#0)), (Binder(TraitPredicate(<T as std::marker::Sized>)), /home/c\
-oolreader18/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/c\
-onvert/mod.rs:365:16: 365:17 (#0)), (Binder(TraitPredicate(<Self as std::convert::From<T>>)), /home/co\
-olreader18/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/co\
-nvert/mod.rs:365:1: 365:25 (#0))] }', /rustc/88f19c6dab716c6281af7602e30f413e809c5974/compiler/rustc_q\
-uery_system/src/query/plumbing.rs:593:5
+thread 'rustc' panicked at 'found unstable fingerprints for predicates_of(<massive text describing rustc internals elided>)', /rustc/88f19c6dab716c6281af7602e30f413e809c5974/compiler/rustc_query_system/src/query/plumbing.rs:593:5
 ```
 
 They all arise from inconsistencies when comparing the incremental-compilation cache stored on disk against the values computed during a current `rustc` invocation, which means they all arise from using incremental compilation.

From 30691e50d1a8f8225a0c68998bf22343f708037e Mon Sep 17 00:00:00 2001
From: Felix S Klock II <pnkfelix@pnkfx.org>
Date: Fri, 7 May 2021 22:51:31 -0400
Subject: [PATCH 04/20] Update posts/2021-05-07-caught-red-handed.md

Co-authored-by: Tyler Mandry <tmandry@gmail.com>
---
 posts/2021-05-07-caught-red-handed.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index d21c8e03d..fa3edd856 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -85,7 +85,7 @@ There are three ways that you may have incremental compilation turned on: You ma
 
 [env-vars]: https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-reads
 [cargo-toml]: https://doc.rust-lang.org/cargo/reference/config.html#buildincremental
-[profiles]: (https://doc.rust-lang.org/cargo/reference/profiles.html
+[profiles]: https://doc.rust-lang.org/cargo/reference/profiles.html
 
 If your project has not opted into enabling incremental compilation, then none of the fingerprint issues should affect your release builds.
 

From 36ce95f26506471eee2229f711ec447e7f40991e Mon Sep 17 00:00:00 2001
From: Felix S Klock II <pnkfelix@pnkfx.org>
Date: Fri, 7 May 2021 22:52:39 -0400
Subject: [PATCH 05/20] don't say "yesterday"

since this post isn't going out today, it should use more abstract language to refer to when 1.52.0 came out.
---
 posts/2021-05-07-caught-red-handed.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index fa3edd856..f9f41b8bb 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -7,7 +7,7 @@ team: the compiler team <https://www.rust-lang.org/governance/teams/compiler>
 
 The Rust teams are always excited to report on new features offered with each release. Sometimes, however, an important change that is not yet "fully baked" gets accidentally included in a release.
 
-There was an instance of this in yesterday's release, 1.52.0, which added a new bit of internal-consistency checking, called "incremental compilation hash verification" (abbreviated `verify-ich`). This check is also called an "unstable fingerprint" check, because the diagnostic it currently prints look [like this](https://github.com/rust-lang/rust/issues/84336):
+There was an instance of this in the most recent release, 1.52.0, which added a new bit of internal-consistency checking, called "incremental compilation hash verification" (abbreviated `verify-ich`). This check is also called an "unstable fingerprint" check, because the diagnostic it currently prints look [like this](https://github.com/rust-lang/rust/issues/84336):
 
 ```
 thread 'rustc' panicked at 'assertion failed: `(left == right)`

From 5fad60ac72b64a105753da46a86510881118a8da Mon Sep 17 00:00:00 2001
From: Felix S Klock II <pnkfelix@pnkfx.org>
Date: Fri, 7 May 2021 22:54:02 -0400
Subject: [PATCH 06/20] another "yesterday" I missed

---
 posts/2021-05-07-caught-red-handed.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index f9f41b8bb..8f4d95403 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -64,9 +64,9 @@ When we first turned on `verify-ich` by default, we assumed we would have time t
 
 [issue-84970]: https://github.com/rust-lang/rust/issues/84970
 
-It turns out `verify-ich` was turned on in version 1.52.0, which was [released yesterday][].
+It turns out `verify-ich` was turned on in version 1.52.0, which was [released recently][].
 
-[released yesterday]: /2021/05/06/Rust-1.52.0.html
+[released recently]: /2021/05/06/Rust-1.52.0.html
 
 ## How does this show up
 

From 7d67f88d7a9a8a878e7a4fd1a0b2142bbf55671b Mon Sep 17 00:00:00 2001
From: Mark Rousskov <mark.simulacrum@gmail.com>
Date: Sat, 8 May 2021 22:17:02 -0400
Subject: [PATCH 07/20] Adjustments to be more of a release blog post

Just a first draft, need to re-read and adjust the structure a little still.
---
 posts/2021-05-07-caught-red-handed.md | 96 +++++++++++++++++++++------
 1 file changed, 76 insertions(+), 20 deletions(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index 8f4d95403..a24d74527 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -1,11 +1,36 @@
 ---
 layout: post
-title: "1.52.0, fingerprints and compiler error forensics"
-author: Felix Klock
+title: "Rust 1.52.1"
+author: Felix Klock, Mark Rousskov
 team: the compiler team <https://www.rust-lang.org/governance/teams/compiler>
+release: true
 ---
 
-The Rust teams are always excited to report on new features offered with each release. Sometimes, however, an important change that is not yet "fully baked" gets accidentally included in a release.
+The Rust team has prepared a new release, 1.52.1, working around a bug
+introduced in 1.52.0. We recommend all Rust users, including those currently
+using stable versions prior to 1.52.0 upgrade to 1.52.1.
+
+If you have a previous version of Rust installed via rustup, getting Rust
+1.52.1 is as easy as:
+
+```console
+rustup update stable
+```
+
+If you don't have it already, you can [get `rustup`][install]
+from the appropriate page on our website.
+
+[install]: https://www.rust-lang.org/install.html
+
+# What's in 1.52.1 stable
+
+This point release contains a single change: it disables incremental
+compilation. Read on for more details as to why, and the next steps the Rust
+project is pursuing on this issue.
+
+# Summary
+
+The Rust teams are always excited to report on new features offered with each release. Sometimes, however, an important change that is not yet "fully baked" gets accidentally included in a release, and we need to issue a point release.
 
 There was an instance of this in the most recent release, 1.52.0, which added a new bit of internal-consistency checking, called "incremental compilation hash verification" (abbreviated `verify-ich`). This check is also called an "unstable fingerprint" check, because the diagnostic it currently prints look [like this](https://github.com/rust-lang/rust/issues/84336):
 
@@ -19,9 +44,9 @@ error: internal compiler error: unexpected panic
 note: the compiler unexpectedly panicked. this is a bug.
 ```
 
-This internal-consistency check, as stated in the diagnostic, yields an "Internal Compiler Error" (or ICE). In other words, it represents a bug in the internals of the Rust compiler itself. In *this* case, though, the ICE is revealing a bug that 1.) is very likely to predate the 1.52.0 release and 2.) could result in miscompilation if it had not been caught by `verify-ich`.
+This internal-consistency check, as stated in the diagnostic, yields an "Internal Compiler Error" (or ICE). In other words, it represents a bug in the internals of the Rust compiler itself. In *this* case, though, the ICE is revealing a bug that 1.) predates the 1.52.0 release and 2.) could result in miscompilation if it had not been caught by `verify-ich`.
 
-In other words: If you are seeing the above Internal Compiler Error, you may be tempted to respond by reverting to the 1.51 release. I want to stress that a downgrade is *not* the best response to this problem.
+In other words: If you are seeing the above Internal Compiler Error, you may be tempted to respond by reverting to the 1.51 release. It is important to note that a downgrade is *not* the best response to this problem.
 
 This post is going to:
 
@@ -60,7 +85,13 @@ More recently, in March, we encountered a [miscompilation][issue-82920] that led
 [issue-82920]: https://github.com/rust-lang/rust/issues/82920
 [pr-83007]: https://github.com/rust-lang/rust/pull/83007
 
-When we first turned on `verify-ich` by default, we assumed we would have time to iron out all the known causes of unstable fingerprints. When we realized this assumption was false, we started [making plans][issue-84970] to improve the user-experience, so that the diagnostic issued by the check would do a better job of telling the programmer what to do in response. But we made a mistake: We thought that the switch for this check was not going to be on the Rust stable channel until version 1.53.
+When we first turned on `verify-ich` by default, there was a steady stream of
+issues filed by users of the nightly (and beta) toolchains, and steady progress
+has been made on identifying fixes, a number of which have already landed. This
+last week, we noted incorrectly that the error would be shipping to stable
+next cycle in 1.53.0, and we started [making plans][issue-84970] to improve the
+user-experience, so that the diagnostic issued by the check would do a better
+job of telling the programmer what to do in response.
 
 [issue-84970]: https://github.com/rust-lang/rust/issues/84970
 
@@ -68,8 +99,11 @@ It turns out `verify-ich` was turned on in version 1.52.0, which was [released r
 
 [released recently]: /2021/05/06/Rust-1.52.0.html
 
-## How does this show up
+This release, 1.52.1, works around the breakage caused by the newly added
+verification by temporarily changing the defaults in the Rust compiler to disable
+incremental unless the user knowingly opts in.
 
+## How does this show up
 
 Essentially, for some crates, certain sequences of edit-compile cycles will cause `rustc` to hit the "unstable fingerprints" ICE. I showed one example at the start of this blog post.
 
@@ -87,7 +121,9 @@ There are three ways that you may have incremental compilation turned on: You ma
 [cargo-toml]: https://doc.rust-lang.org/cargo/reference/config.html#buildincremental
 [profiles]: https://doc.rust-lang.org/cargo/reference/profiles.html
 
-If your project has not opted into enabling incremental compilation, then none of the fingerprint issues should affect your release builds.
+Incremental is disabled by default for the release profile, which should mean
+that unless you have enabled it yourself, these errors do not affect release
+builds, as they are only present in incremental builds.
 
 ## What should a Rust programmer do in response
 
@@ -95,31 +131,51 @@ The Internal Compiler Error asks you to report a bug, and if you can do so, we s
 
 But regardless of whether or not you file a bug, the problem here can be resolved by either:
 
- 1. deleting your incremental compilation cache (e.g. by running `cargo clean`), or
- 2. force incremental compilation to be disabled, by setting `CARGO_INCREMENTAL=0` in your environment or `build.incremental` to `false` in the `config.toml`.
+ 1. upgrading to 1.52.1, if you have not yet done so (which will disable
+    incremental for you).
+ 2. deleting your incremental compilation cache (e.g. by running `cargo clean`), or
+ 3. force incremental compilation to be disabled, by setting `CARGO_INCREMENTAL=0` in your environment or `build.incremental` to `false` in the `config.toml`.
 
-We recommend that users of 1.52.0 disable incremental compilation, to avoid running into this problem.
+We recommend that users of 1.52.0 upgrade to 1.52.1, which disables incremental
+compilation.
 
 We do *not* recommend that users of 1.52.0 downgrade to an earlier version of Rust in response to this problem. As noted above, there is at least one instance of a silent [miscompilation][issue-82920] caused by incremental compilation that was not caught until we added the fingerprint checking.
 
+If a user is willing to deal with the incremental verification ICE's, and wishes
+to opt back into the 1.52.0 behavior, they may set `RUSTC_FORCE_INCREMENTAL` to
+`1` in their environment. The Rust compiler will then respect the
+`-Cincremental` option passed by Cargo, and things will work as before. Note
+that this flag does not enable incremental if it has not already been separately
+enabled (whether by Cargo or otherwise).
+
 ## What is the Rust project going to do to fix this
 
 ### Short-term plan
 
-Based on the number of bug reports that have already come in, we know we need to do something quickly.
+We have issued 1.52.1 today which:
 
-We are going to be issuing a point release, 1.52.1. The point release will improve the diagnostic output from `verify-ich`, so that a programmer can more effectively act in response to the message (i.e., they will be told that the problem is due to the use of incremental compilation, and that they need to delete their cache, at the very least).
+* Disables incremental compilation in the Rust compiler (unless asked for by a
+  new environment variable, `RUSTC_FORCE_INCREMENTAL=1`).
+* Improves diagnostic output for the new verification if incremental compilation is enabled, indicating how to work around the bugs.
 
-We might also make incremental compilation opt-in for *all* Cargo [profiles][]; that is, we may turn incremental compilation off for the  `dev` and `test`, so that they match `release` and `bench`, which already have incremental compilation off. (This remains to be decided.)
+This is intended to be a mitigation that helps the majority of Rust users have
+an upgrade path to a safe Rust compiler which does not have the risk of
+miscompiling their code, but also provide the option for users willing to deal
+with the errors to do so.
 
-We do not plan to disable incremental compilation in its entirety. We believe that incremental compilation is providing value for a large number of Rust programmers, and we want to empower them to continue using it, as long as they are equipped with the tools they need to turn it off if they encounter this issue.
+We expect to continue to actively invest in fixing the bugs, and depending on
+our confidence in the fixes, may issue a 1.52.2 point release which backports
+those fixes to the stable channel. Users wishing to help us test can use the
+nightly channel, and report bugs to rust-lang/rust with any ICEs they are
+seeing. We do not expect at this time to disable incremental by default on the
+nightly channel.
 
 ### Long-term plan
 
-The long-term plan is to fix the bugs! Incremental compilation is the only realistic way for the Rust compiler to be able to provide a fast edit-compile-run cycle for all of its programmers, and so we need to address [all of the issues][issue-list] that have been identified thus far via `verify-ich`. (There are 32 such issues as of this writing, though many may be duplicates.)
+The long-term plan is to fix the bugs! Incremental compilation is the only realistic way for the Rust compiler to be able to provide a fast edit-compile-run cycle for all of its programmers, and so we need to address [all of the issues][issue-list] that have been identified thus far via `verify-ich`. (There are 32 such issues as of this writing, though many are duplicates.)
 
-[issue-list]: https://github.com/rust-lang/rust/issues?q=is%3Aissue+is%3Aopen+unstable+fingerprints
-
-If you want to come help us do it, we would love for you to [join us][] with the effort!
+We are actively investing in this, and a number of bugs have already been
+identified and fixed. Depending on the state of the fixes, future stable
+releases (1.53 and onwards) will likely re-enable incremental compilation.
 
-[join us]: https://www.rust-lang.org/community
+[issue-list]: https://github.com/rust-lang/rust/issues?q=is%3Aissue+is%3Aopen+unstable+fingerprints

From 2190a79cc21f2ad1671a2575f69abe97157e2738 Mon Sep 17 00:00:00 2001
From: Mark Rousskov <mark.simulacrum@gmail.com>
Date: Sun, 9 May 2021 18:44:46 -0400
Subject: [PATCH 08/20] Apply more feedback

---
 posts/2021-05-07-caught-red-handed.md | 121 ++++++++++++++++----------
 1 file changed, 77 insertions(+), 44 deletions(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index a24d74527..14d78bfac 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -6,9 +6,11 @@ team: the compiler team <https://www.rust-lang.org/governance/teams/compiler>
 release: true
 ---
 
-The Rust team has prepared a new release, 1.52.1, working around a bug
-introduced in 1.52.0. We recommend all Rust users, including those currently
-using stable versions prior to 1.52.0 upgrade to 1.52.1.
+The Rust team has prepared a new release, 1.52.1, working around a bug in
+incremental compilation which was made into a compiler error in 1.52.0. We
+recommend all Rust users, including those currently using stable versions prior
+to 1.52.0, upgrade to 1.52.1 or disable incremental compilation. Guidance on how
+to do so is available below.
 
 If you have a previous version of Rust installed via rustup, getting Rust
 1.52.1 is as easy as:
@@ -22,44 +24,51 @@ from the appropriate page on our website.
 
 [install]: https://www.rust-lang.org/install.html
 
-# What's in 1.52.1 stable
-
-This point release contains a single change: it disables incremental
-compilation. Read on for more details as to why, and the next steps the Rust
-project is pursuing on this issue.
-
 # Summary
 
-The Rust teams are always excited to report on new features offered with each release. Sometimes, however, an important change that is not yet "fully baked" gets accidentally included in a release, and we need to issue a point release.
-
-There was an instance of this in the most recent release, 1.52.0, which added a new bit of internal-consistency checking, called "incremental compilation hash verification" (abbreviated `verify-ich`). This check is also called an "unstable fingerprint" check, because the diagnostic it currently prints look [like this](https://github.com/rust-lang/rust/issues/84336):
-
-```
-thread 'rustc' panicked at 'assertion failed: `(left == right)`
-  left: `Some(Fingerprint(4565771098143344972, 7869445775526300234))`,
-   right: `Some(Fingerprint(14934403843752251060, 623484215826468126))`: found unstable fingerprints for <massive text describing rustc internals elided>
-
-error: internal compiler error: unexpected panic
+This release works around broken builds on 1.52.0, which are caused by newly
+added verification. The bugs this verification detects are present in all Rust
+versions, and can trigger miscompilations in incremental builds, so downgrading
+to a prior stable version is not a fix.
 
-note: the compiler unexpectedly panicked. this is a bug.
-```
+Users are encouraged to upgrade to 1.52.1 or disable incremental in their local
+environment if on a prior versions: please see the [what you should do][part3]
+section for details on how to do so.
 
-This internal-consistency check, as stated in the diagnostic, yields an "Internal Compiler Error" (or ICE). In other words, it represents a bug in the internals of the Rust compiler itself. In *this* case, though, the ICE is revealing a bug that 1.) predates the 1.52.0 release and 2.) could result in miscompilation if it had not been caught by `verify-ich`.
-
-In other words: If you are seeing the above Internal Compiler Error, you may be tempted to respond by reverting to the 1.51 release. It is important to note that a downgrade is *not* the best response to this problem.
+Incremental compilation is off by default for release builds, so few
+production builds should be affected (only users who may have opted in).
 
 This post is going to:
 
+ 1. Explain [what the errors look like][part0],
  1. Explain [what the check does][part1], at a high level,
  2. Explain [how the check is presenting itself][part2] in the Rust 1.52.0 release,
  3. Tell you [what you should do][part3] if you see an unstable fingerprint on your project,
  4. Describe our plans for [how the Rust project will address][part4] the problems discussed here.
 
+[part0]: #what-does-the-error-look-like
 [part1]: #what-are-fingerprints-why-are-we-checking-them
 [part2]: #how-does-this-show-up
 [part3]: #what-should-a-rust-programmer-do-in-response
 [part4]: #what-is-the-rust-project-going-to-do-to-fix-this
 
+## What does the error look like?
+
+The error message looks something like this, with the key piece being the "found
+unstable fingerprints" text.
+
+```
+thread 'rustc' panicked at 'assertion failed: `(left == right)`
+  left: `Some(Fingerprint(4565771098143344972, 7869445775526300234))`,
+  right: `Some(Fingerprint(14934403843752251060, 623484215826468126))`: found unstable fingerprints for <massive text describing rustc internals elided>
+
+error: internal compiler error: unexpected panic
+
+note: the compiler unexpectedly panicked. this is a bug.
+```
+
+This is the error caused by the internal consistency check, and as stated in the diagnostic, it yields an "Internal Compiler Error" (or ICE). In other words, it represents a bug in the internals of the Rust compiler itself. In *this* case, the ICE is revealing a bug that predates the 1.52.0 release and could result in miscompilation if it had not been caught by `verify-ich`.
+
 ## What are fingerprints? Why are we checking them?
 
 The Rust compiler has support for "incremental compilation", which has been described in a [2016 blog post][]. When incremental compilation is turned on, the compiler breaks the input source into pieces, and tracks how those input pieces influence the final build product. Then, when the inputs change, it detects this and reuses artifacts from previous builds, striving to expend effort solely on building the parts that need to respond to the changes to the input source code.
@@ -70,14 +79,18 @@ Fingerprints are part of our architecture for detecting when inputs change. More
 
 [rustc-dev-guide-fingerprints]: https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation-in-detail.html#checking-query-results-for-changes-hashstable-and-fingerprints
 
-The `verify-ich` check is a safeguard asserting internal inconsistency of the fingerprints.
-The compiler stores fingerprints for both cached and uncached values.
-Every time we compute an uncached value, we double-check that its newly computed fingerprint against the finger print stored in the cache.
-There are multiple ways that a fingerprint mismatch could arise, but they all represent bugs within the Rust compiler itself.
+The `verify-ich` check is a safeguard asserting internal consistency of the
+fingerprints. Sometimes the compiler is forced to rerun a query, and expects
+that the output is the same as from a prior incremental compilation session. The
+newly enabled verification checks that the value is indeed as expected, rather
+than assuming so. In some cases, due to bugs in the compiler's implementation,
+this was not actually the case.
 
-## History of deployment
+## History
 
-We [initially added][pr-45867] `verify-ich` as a tool to use when developing rustc itself, back in 2017; it was solely provided via an unstable `-Z` flag, only available to nightly and development builds.
+We [initially added][pr-45867] `verify-ich` as a tool to use when developing
+rustc itself, back in 2017. It was solely provided via an unstable `-Z` flag,
+only available to nightly and development builds.
 
 More recently, in March, we encountered a [miscompilation][issue-82920] that led us to [turn on `verify-ich` by default][pr-83007]. The Rust compiler team decided it was better to catch fingerprint problems and abort compilation, rather than allow for potential miscompilations (and subsequent misbehavior) to sneak into Rust programmer's binaries.
 
@@ -87,11 +100,13 @@ More recently, in March, we encountered a [miscompilation][issue-82920] that led
 
 When we first turned on `verify-ich` by default, there was a steady stream of
 issues filed by users of the nightly (and beta) toolchains, and steady progress
-has been made on identifying fixes, a number of which have already landed. This
-last week, we noted incorrectly that the error would be shipping to stable
-next cycle in 1.53.0, and we started [making plans][issue-84970] to improve the
+has been made on identifying fixes, a number of which have already landed.
+
+In the past week, we had started [making plans][issue-84970] to improve the
 user-experience, so that the diagnostic issued by the check would do a better
-job of telling the programmer what to do in response.
+job of telling the programmer what to do in response. Unfortunately, this was
+done under the assumption that the new verification would ship in 1.53, not
+1.52.
 
 [issue-84970]: https://github.com/rust-lang/rust/issues/84970
 
@@ -110,20 +125,27 @@ Essentially, for some crates, certain sequences of edit-compile cycles will caus
 Another recent example looks [like this](https://github.com/rust-lang/rust/issues/85039):
 
 ```
-thread 'rustc' panicked at 'found unstable fingerprints for predicates_of(<massive text describing rustc internals elided>)', /rustc/88f19c6dab716c6281af7602e30f413e809c5974/compiler/rustc_query_system/src/query/plumbing.rs:593:5
+thread 'rustc' panicked at 'found unstable fingerprints for predicates_of(<massive text describing rustc internals elided>)', /rustc/.../compiler/rustc_query_system/src/query/plumbing.rs:593:5
 ```
 
 They all arise from inconsistencies when comparing the incremental-compilation cache stored on disk against the values computed during a current `rustc` invocation, which means they all arise from using incremental compilation.
 
-There are three ways that you may have incremental compilation turned on: You may have set the [environment variable][env-vars] `CARGO_INCREMENTAL=1`, or you may have enabled the `build.incremental` [setting in your Cargo.toml][cargo-toml], or you may be building with the `dev` or `test` [profiles][], which default to having incremental compilation enabled.
+There are several ways that you may have incremental compilation turned on:
+
+1. You may be building with the `dev` or `test` [profiles][] which default to having incremental compilation enabled.
+2. You may have set the [environment variable][env-vars] `CARGO_INCREMENTAL=1`
+3. You may have enabled the `build.incremental` [setting in your Cargo config][cargo-config]
+4. You may have enabled the `incremental` [setting in your Cargo.toml][cargo-toml] for a given profile
 
 [env-vars]: https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-reads
-[cargo-toml]: https://doc.rust-lang.org/cargo/reference/config.html#buildincremental
+[cargo-config]: https://doc.rust-lang.org/cargo/reference/config.html#buildincremental
+[cargo-toml]: https://doc.rust-lang.org/cargo/reference/profiles.html#incremental
 [profiles]: https://doc.rust-lang.org/cargo/reference/profiles.html
 
-Incremental is disabled by default for the release profile, which should mean
-that unless you have enabled it yourself, these errors do not affect release
-builds, as they are only present in incremental builds.
+If your project has not adjusted the defaults, then when running `cargo build
+--release` or otherwise in the `release` profile configuration incremental is
+disabled on all Rust versions, and these issues should not affect your release
+builds.
 
 ## What should a Rust programmer do in response
 
@@ -144,9 +166,18 @@ We do *not* recommend that users of 1.52.0 downgrade to an earlier version of Ru
 If a user is willing to deal with the incremental verification ICE's, and wishes
 to opt back into the 1.52.0 behavior, they may set `RUSTC_FORCE_INCREMENTAL` to
 `1` in their environment. The Rust compiler will then respect the
-`-Cincremental` option passed by Cargo, and things will work as before. Note
-that this flag does not enable incremental if it has not already been separately
-enabled (whether by Cargo or otherwise).
+`-Cincremental` option passed by Cargo, and things will work as before, though
+with the added verification. Note that this flag does not enable incremental if
+it has not already been separately enabled (whether by Cargo or otherwise).
+
+If you are currently using a toolchain prior to 1.52.0, and wish to continue
+doing so, we recommend that you disable incremental compilation to avoid hitting
+silent miscompilations.
+
+On all Rust builds since incremental has landed, it has been a major
+improvement to compile times for many users, and has only improved over time. We
+acknowledge that the workarounds presented here and recommendations are painful,
+and will be working hard to ensure the situation is as temporary as possible.
 
 ## What is the Rust project going to do to fix this
 
@@ -156,7 +187,9 @@ We have issued 1.52.1 today which:
 
 * Disables incremental compilation in the Rust compiler (unless asked for by a
   new environment variable, `RUSTC_FORCE_INCREMENTAL=1`).
-* Improves diagnostic output for the new verification if incremental compilation is enabled, indicating how to work around the bugs.
+* Improves diagnostic output for the new verification if incremental compilation is enabled,
+  indicating how to work around the bugs by purging incremental state or
+  disabling incremental.
 
 This is intended to be a mitigation that helps the majority of Rust users have
 an upgrade path to a safe Rust compiler which does not have the risk of

From f5369edf4ffbbd419b7d5dbbd21e04068db131f9 Mon Sep 17 00:00:00 2001
From: Mark Rousskov <mark.simulacrum@gmail.com>
Date: Sun, 9 May 2021 19:34:58 -0400
Subject: [PATCH 09/20] Update posts/2021-05-07-caught-red-handed.md

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
---
 posts/2021-05-07-caught-red-handed.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index 14d78bfac..73cf0e688 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -67,7 +67,7 @@ error: internal compiler error: unexpected panic
 note: the compiler unexpectedly panicked. this is a bug.
 ```
 
-This is the error caused by the internal consistency check, and as stated in the diagnostic, it yields an "Internal Compiler Error" (or ICE). In other words, it represents a bug in the internals of the Rust compiler itself. In *this* case, the ICE is revealing a bug that predates the 1.52.0 release and could result in miscompilation if it had not been caught by `verify-ich`.
+This is the error caused by the internal consistency check, and as stated in the diagnostic, it yields an "Internal Compiler Error" (or ICE). In other words, it represents a bug in the internals of the Rust compiler itself. In *this* case, the ICE is revealing a bug in incremental compilation that predates the 1.52.0 release and could result in miscompilation if it had not been caught by `verify-ich`.
 
 ## What are fingerprints? Why are we checking them?
 

From e1c66aefe12fac5d7c28e291e5586dbbd551ea35 Mon Sep 17 00:00:00 2001
From: Mark Rousskov <mark.simulacrum@gmail.com>
Date: Sun, 9 May 2021 19:35:09 -0400
Subject: [PATCH 10/20] Update posts/2021-05-07-caught-red-handed.md

Co-authored-by: Carol (Nichols || Goulding) <193874+carols10cents@users.noreply.github.com>
---
 posts/2021-05-07-caught-red-handed.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index 73cf0e688..9c6a5b3c5 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -32,7 +32,7 @@ versions, and can trigger miscompilations in incremental builds, so downgrading
 to a prior stable version is not a fix.
 
 Users are encouraged to upgrade to 1.52.1 or disable incremental in their local
-environment if on a prior versions: please see the [what you should do][part3]
+environment if on a prior version: please see the [what you should do][part3]
 section for details on how to do so.
 
 Incremental compilation is off by default for release builds, so few

From e304ef0e085a6cb2378b736817e73218172b76d9 Mon Sep 17 00:00:00 2001
From: Mark Rousskov <mark.simulacrum@gmail.com>
Date: Sun, 9 May 2021 19:47:50 -0400
Subject: [PATCH 11/20] Add some language around what miscompilations are and
 what releases they are in

---
 posts/2021-05-07-caught-red-handed.md | 42 ++++++++++++++++++---------
 1 file changed, 29 insertions(+), 13 deletions(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index 9c6a5b3c5..be6586ed3 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -1,6 +1,6 @@
 ---
 layout: post
-title: "Rust 1.52.1"
+title: "Announcing Rust 1.52.1"
 author: Felix Klock, Mark Rousskov
 team: the compiler team <https://www.rust-lang.org/governance/teams/compiler>
 release: true
@@ -28,7 +28,7 @@ from the appropriate page on our website.
 
 This release works around broken builds on 1.52.0, which are caused by newly
 added verification. The bugs this verification detects are present in all Rust
-versions, and can trigger miscompilations in incremental builds, so downgrading
+versions[^1], and can trigger miscompilations in incremental builds, so downgrading
 to a prior stable version is not a fix.
 
 Users are encouraged to upgrade to 1.52.1 or disable incremental in their local
@@ -38,6 +38,14 @@ section for details on how to do so.
 Incremental compilation is off by default for release builds, so few
 production builds should be affected (only users who may have opted in).
 
+Miscompilations that can arise from the bugs in incremental compilation generate incorrect code in final
+artifacts, essentially producing malformed binaries, which means that in theory
+any behavior is possible. In practice we are currently only aware of one
+particular known miscompilation, but bugs due to incremental are notoriously
+hard to track down: users frequently simply rebuild after some light editing if
+they see unexpected results from their binaries, and this often causes
+sufficient recompilation to fix the bug(s).
+
 This post is going to:
 
  1. Explain [what the errors look like][part0],
@@ -57,7 +65,7 @@ This post is going to:
 The error message looks something like this, with the key piece being the "found
 unstable fingerprints" text.
 
-```
+```text
 thread 'rustc' panicked at 'assertion failed: `(left == right)`
   left: `Some(Fingerprint(4565771098143344972, 7869445775526300234))`,
   right: `Some(Fingerprint(14934403843752251060, 623484215826468126))`: found unstable fingerprints for <massive text describing rustc internals elided>
@@ -67,7 +75,7 @@ error: internal compiler error: unexpected panic
 note: the compiler unexpectedly panicked. this is a bug.
 ```
 
-This is the error caused by the internal consistency check, and as stated in the diagnostic, it yields an "Internal Compiler Error" (or ICE). In other words, it represents a bug in the internals of the Rust compiler itself. In *this* case, the ICE is revealing a bug in incremental compilation that predates the 1.52.0 release and could result in miscompilation if it had not been caught by `verify-ich`.
+This is the error caused by the internal consistency check, and as stated in the diagnostic, it yields an "Internal Compiler Error" (or ICE). In other words, it represents a bug in the internals of the Rust compiler itself. In *this* case, the ICE is revealing a bug in incremental compilation that predates the 1.52.0 release and could result in miscompilation if it had not been caught.
 
 ## What are fingerprints? Why are we checking them?
 
@@ -79,8 +87,8 @@ Fingerprints are part of our architecture for detecting when inputs change. More
 
 [rustc-dev-guide-fingerprints]: https://rustc-dev-guide.rust-lang.org/queries/incremental-compilation-in-detail.html#checking-query-results-for-changes-hashstable-and-fingerprints
 
-The `verify-ich` check is a safeguard asserting internal consistency of the
-fingerprints. Sometimes the compiler is forced to rerun a query, and expects
+The fingerprint stability check is a safeguard asserting internal consistency of
+the fingerprints. Sometimes the compiler is forced to rerun a query, and expects
 that the output is the same as from a prior incremental compilation session. The
 newly enabled verification checks that the value is indeed as expected, rather
 than assuming so. In some cases, due to bugs in the compiler's implementation,
@@ -88,9 +96,9 @@ this was not actually the case.
 
 ## History
 
-We [initially added][pr-45867] `verify-ich` as a tool to use when developing
-rustc itself, back in 2017. It was solely provided via an unstable `-Z` flag,
-only available to nightly and development builds.
+We [initially added][pr-45867] these fingerprint checks as a tool to use when
+developing rustc itself, back in 2017. It was solely provided via an unstable
+`-Z` flag, only available to nightly and development builds.
 
 More recently, in March, we encountered a [miscompilation][issue-82920] that led us to [turn on `verify-ich` by default][pr-83007]. The Rust compiler team decided it was better to catch fingerprint problems and abort compilation, rather than allow for potential miscompilations (and subsequent misbehavior) to sneak into Rust programmer's binaries.
 
@@ -98,9 +106,10 @@ More recently, in March, we encountered a [miscompilation][issue-82920] that led
 [issue-82920]: https://github.com/rust-lang/rust/issues/82920
 [pr-83007]: https://github.com/rust-lang/rust/pull/83007
 
-When we first turned on `verify-ich` by default, there was a steady stream of
-issues filed by users of the nightly (and beta) toolchains, and steady progress
-has been made on identifying fixes, a number of which have already landed.
+When we first turned on the fingerprint checks by default, there was a steady
+stream of issues filed by users of the nightly (and beta) toolchains, and steady
+progress has been made on identifying fixes, a number of which have already
+landed.
 
 In the past week, we had started [making plans][issue-84970] to improve the
 user-experience, so that the diagnostic issued by the check would do a better
@@ -144,7 +153,7 @@ There are several ways that you may have incremental compilation turned on:
 
 If your project has not adjusted the defaults, then when running `cargo build
 --release` or otherwise in the `release` profile configuration incremental is
-disabled on all Rust versions, and these issues should not affect your release
+disabled on all Rust versions[^1], and these issues should not affect your release
 builds.
 
 ## What should a Rust programmer do in response
@@ -212,3 +221,10 @@ identified and fixed. Depending on the state of the fixes, future stable
 releases (1.53 and onwards) will likely re-enable incremental compilation.
 
 [issue-list]: https://github.com/rust-lang/rust/issues?q=is%3Aissue+is%3Aopen+unstable+fingerprints
+
+The Rust teams will also be developing plans to ensure we have better tracking
+systems in place in the future for bugs, both to prevent situations like this
+from arising again, but also to further increase the stability of our releases
+by tracking bugs more accurately as they propagate across channels.
+
+[^1]: Since incremental was first enabled, which was in Rust 1.24.

From c250c4e5cb50f8edfda8580f90f54705ff6a94e9 Mon Sep 17 00:00:00 2001
From: Mark Rousskov <mark.simulacrum@gmail.com>
Date: Mon, 10 May 2021 08:48:10 -0400
Subject: [PATCH 12/20] Update posts/2021-05-07-caught-red-handed.md

Co-authored-by: est31 <est31@users.noreply.github.com>
---
 posts/2021-05-07-caught-red-handed.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index be6586ed3..a84f52c1b 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -160,7 +160,7 @@ builds.
 
 The Internal Compiler Error asks you to report a bug, and if you can do so, we still want that information. We *want* to know about the cases that are failing.
 
-But regardless of whether or not you file a bug, the problem here can be resolved by either:
+But regardless of whether or not you file a bug, the problem can be worked around on your end by either:
 
  1. upgrading to 1.52.1, if you have not yet done so (which will disable
     incremental for you).

From 7ac6272c344e9e12c7be59b8df9af2e775f763bf Mon Sep 17 00:00:00 2001
From: Mark Rousskov <mark.simulacrum@gmail.com>
Date: Mon, 10 May 2021 08:51:21 -0400
Subject: [PATCH 13/20] Update posts/2021-05-07-caught-red-handed.md

Co-authored-by: est31 <est31@users.noreply.github.com>
---
 posts/2021-05-07-caught-red-handed.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index a84f52c1b..159fd7578 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -36,7 +36,7 @@ environment if on a prior version: please see the [what you should do][part3]
 section for details on how to do so.
 
 Incremental compilation is off by default for release builds, so few
-production builds should be affected (only users who may have opted in).
+production builds should be affected (only for users who have opted in).
 
 Miscompilations that can arise from the bugs in incremental compilation generate incorrect code in final
 artifacts, essentially producing malformed binaries, which means that in theory

From cda2b34ab1f7d46c936147606a828b8d843f93e4 Mon Sep 17 00:00:00 2001
From: Mark Rousskov <mark.simulacrum@gmail.com>
Date: Mon, 10 May 2021 08:51:38 -0400
Subject: [PATCH 14/20] Update posts/2021-05-07-caught-red-handed.md

Co-authored-by: est31 <est31@users.noreply.github.com>
---
 posts/2021-05-07-caught-red-handed.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index 159fd7578..269f1689c 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -165,7 +165,7 @@ But regardless of whether or not you file a bug, the problem can be worked aroun
  1. upgrading to 1.52.1, if you have not yet done so (which will disable
     incremental for you).
  2. deleting your incremental compilation cache (e.g. by running `cargo clean`), or
- 3. force incremental compilation to be disabled, by setting `CARGO_INCREMENTAL=0` in your environment or `build.incremental` to `false` in the `config.toml`.
+ 3. forcing incremental compilation to be disabled, by setting `CARGO_INCREMENTAL=0` in your environment or `build.incremental` to `false` in the `config.toml`.
 
 We recommend that users of 1.52.0 upgrade to 1.52.1, which disables incremental
 compilation.

From 2c47d8a0142a4b92d2d2232515c483b0a2210f9b Mon Sep 17 00:00:00 2001
From: Mark Rousskov <mark.simulacrum@gmail.com>
Date: Mon, 10 May 2021 08:55:38 -0400
Subject: [PATCH 15/20] Add note about plans for 1.53 beta channel

---
 posts/2021-05-07-caught-red-handed.md | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index 269f1689c..3dc155af8 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -208,9 +208,14 @@ with the errors to do so.
 We expect to continue to actively invest in fixing the bugs, and depending on
 our confidence in the fixes, may issue a 1.52.2 point release which backports
 those fixes to the stable channel. Users wishing to help us test can use the
-nightly channel, and report bugs to rust-lang/rust with any ICEs they are
-seeing. We do not expect at this time to disable incremental by default on the
-nightly channel.
+nightly channel, and report bugs to rust-lang/rust with any ICEs they
+are seeing.
+
+We are also currently not planning to disable incremental on the beta channel,
+but this decision has not been firmly committed to. A number of fixes are
+available on 1.53 beta today, so users who wish to continue using incremental
+may want to switch to that. Nightly will always have the latest in fixes, of
+course.
 
 ### Long-term plan
 

From 227384b5926ddfc7ee499069a519baa74d2fedbd Mon Sep 17 00:00:00 2001
From: Mark Rousskov <mark.simulacrum@gmail.com>
Date: Mon, 10 May 2021 09:11:01 -0400
Subject: [PATCH 16/20] Update posts/2021-05-07-caught-red-handed.md

Co-authored-by: Felix S Klock II <pnkfelix@pnkfx.org>
---
 posts/2021-05-07-caught-red-handed.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index 3dc155af8..8b69e3e9b 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -123,7 +123,7 @@ It turns out `verify-ich` was turned on in version 1.52.0, which was [released r
 
 [released recently]: /2021/05/06/Rust-1.52.0.html
 
-This release, 1.52.1, works around the breakage caused by the newly added
+Today's new release, 1.52.1, works around the breakage caused by the newly added
 verification by temporarily changing the defaults in the Rust compiler to disable
 incremental unless the user knowingly opts in.
 

From de97a34c736b2daf35e2b8801e964e4b8dd2de76 Mon Sep 17 00:00:00 2001
From: Mark Rousskov <mark.simulacrum@gmail.com>
Date: Mon, 10 May 2021 09:37:30 -0400
Subject: [PATCH 17/20] Update posts/2021-05-07-caught-red-handed.md

Co-authored-by: Felix S Klock II <pnkfelix@pnkfx.org>
---
 posts/2021-05-07-caught-red-handed.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-07-caught-red-handed.md
index 8b69e3e9b..72d5c66ca 100644
--- a/posts/2021-05-07-caught-red-handed.md
+++ b/posts/2021-05-07-caught-red-handed.md
@@ -163,7 +163,7 @@ The Internal Compiler Error asks you to report a bug, and if you can do so, we s
 But regardless of whether or not you file a bug, the problem can be worked around on your end by either:
 
  1. upgrading to 1.52.1, if you have not yet done so (which will disable
-    incremental for you).
+    incremental for you), or
  2. deleting your incremental compilation cache (e.g. by running `cargo clean`), or
  3. forcing incremental compilation to be disabled, by setting `CARGO_INCREMENTAL=0` in your environment or `build.incremental` to `false` in the `config.toml`.
 

From 4b940c31d5bd0f01a94056ff187e296269e2def6 Mon Sep 17 00:00:00 2001
From: Mark Rousskov <mark.simulacrum@gmail.com>
Date: Mon, 10 May 2021 09:53:28 -0400
Subject: [PATCH 18/20] move blog post to today and rename

---
 ...{2021-05-07-caught-red-handed.md => 2021-05-10-Rust-1.52.1.md} | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename posts/{2021-05-07-caught-red-handed.md => 2021-05-10-Rust-1.52.1.md} (100%)

diff --git a/posts/2021-05-07-caught-red-handed.md b/posts/2021-05-10-Rust-1.52.1.md
similarity index 100%
rename from posts/2021-05-07-caught-red-handed.md
rename to posts/2021-05-10-Rust-1.52.1.md

From 410c4d05324a0b8989ff32f40f4d9f97bac8be8e Mon Sep 17 00:00:00 2001
From: Mark Rousskov <mark.simulacrum@gmail.com>
Date: Mon, 10 May 2021 09:57:36 -0400
Subject: [PATCH 19/20] code block is text

---
 posts/2021-05-10-Rust-1.52.1.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/posts/2021-05-10-Rust-1.52.1.md b/posts/2021-05-10-Rust-1.52.1.md
index 72d5c66ca..ae24e2f08 100644
--- a/posts/2021-05-10-Rust-1.52.1.md
+++ b/posts/2021-05-10-Rust-1.52.1.md
@@ -2,7 +2,7 @@
 layout: post
 title: "Announcing Rust 1.52.1"
 author: Felix Klock, Mark Rousskov
-team: the compiler team <https://www.rust-lang.org/governance/teams/compiler>
+team: the release team <https://www.rust-lang.org/governance/teams/release>, the compiler team <https://www.rust-lang.org/governance/teams/compiler>
 release: true
 ---
 
@@ -133,7 +133,7 @@ Essentially, for some crates, certain sequences of edit-compile cycles will caus
 
 Another recent example looks [like this](https://github.com/rust-lang/rust/issues/85039):
 
-```
+```text
 thread 'rustc' panicked at 'found unstable fingerprints for predicates_of(<massive text describing rustc internals elided>)', /rustc/.../compiler/rustc_query_system/src/query/plumbing.rs:593:5
 ```
 

From 1dcc8bef551973e9d37000cfc60409d8a631320b Mon Sep 17 00:00:00 2001
From: Mark Rousskov <mark.simulacrum@gmail.com>
Date: Mon, 10 May 2021 10:06:01 -0400
Subject: [PATCH 20/20] Don't actually modify to two teams - not currently
 supported

---
 posts/2021-05-10-Rust-1.52.1.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/posts/2021-05-10-Rust-1.52.1.md b/posts/2021-05-10-Rust-1.52.1.md
index ae24e2f08..e6568bf4b 100644
--- a/posts/2021-05-10-Rust-1.52.1.md
+++ b/posts/2021-05-10-Rust-1.52.1.md
@@ -2,7 +2,7 @@
 layout: post
 title: "Announcing Rust 1.52.1"
 author: Felix Klock, Mark Rousskov
-team: the release team <https://www.rust-lang.org/governance/teams/release>, the compiler team <https://www.rust-lang.org/governance/teams/compiler>
+team: the compiler team <https://www.rust-lang.org/governance/teams/compiler>
 release: true
 ---