non-lexical lifetimes (NLL) tracking issue

[nikomatsakis]

This issue tracks the status of the transition to non-lexical lifetimes and a MIR-based borrow-checker. Both of these are jargon-y terms for compiler authors: the effect of these features on end-users is, simply put, that the compiler accepts a wider range of code with fewer bugs (and, hopefully, better error messages). We will refer to the combination of the above as NLL.

Current status

Most of NLL is already enabled in current versions of Rust. However, 'migration mode' is still enabled (#58781). This causes us to reject some code that we would eventually like to accept, and to use different code paths for emitting some diagnostics. The progress towards disabling 'migration mode' is tracked in #58781

Key facts for getting involved

• Weekly triage meeting Wednesdays at 3:30pm Boston time (see in your timezone)
  • Join the chat on Zulip
  • Notes in this dropbox paper document
  • If you are unable to attend, minutes and further discussion takes place on this internals thread.
  • Ask @nikomatsakis (on Zulip, ideally) to be added to the calendar invite.
• You can also find a YouTube playlist of pair programming videos; these are sessions where we are working through how to solve a particular problem, often explaining big parts of the system on the way.

Implementation plan

• Initial prototype: features available on nightly builds with #![feature(nll)] for experimentation.
• Valid code works: (issues labeled with NLL-complete)
  • all run-pass tests pass
  • bootstrap builds with nll enabled: see [nll] bootstrap rustc with NLL enabled #51823.
    • summary: We have bootstrapped at least once. We need to integrate such bootstrapping into the CI to ensure it stops breaking.
  • crater run with nll enabled passes without regressions
    • see NLL ICEs from nll-3 crater #52217 for the set of ICEs from recent crater run.
• Invalid code gets errors: (issues labeled with NLL-sound)
  • compile-fail/ui tests generate errors in the right places (possibly with bad diagnostics)
    • see NLL: need a (new) round of review of deltas between .stderr and .nll.stderr output #52663 as evidence that we generate errors in all cases where expected for the ui test suite. (Current plan for compile-fail suite is to port majority of those tests to ui)
• Performance is good (issues labeled with NLL-performant)
  • see this dropbox paper which tracks profiling information
  • see also the NLL perf dashboard which compares running cargo check without and with NLL turned on, so that we can see a summary of the current performance impact on the compiler's static analyses in isolation.
• Future compatibility warnings on stable: Once crater runs are clean, we can enable the future-compatibility warnings on stable: see Want -Z borrowck=migrate to test planning migration to non-lexical lifetimes #46908 and Add -Z borrowck=migrate #52681
  • currently, due to Add -Z borrowck=migrate #52681, we are issuing compatibility warnings on the 2018 edition.
  • next step is to migrate the 2015 edition in the same manner (NLL: turn on borrowck=migrate by default on 2015 edition #57804)
• Diagnostic parity: (issues labeled with NLL-diagnostics) Ensure that all diagnostics are "as good or better" than before.
  • We will primarily validate this from our test base, but also consider errors that are reported "from the wild".
  • round one: NLL: must review all .nll.stderr files #49862
  • round two: NLL: need a (new) round of review of deltas between .stderr and .nll.stderr output #52663
  • round three: NLL: need 3rd round of review comparing .stderr and .nll.stderr output #54528; remaining work items are cards on https://github.com/rust-lang/rust/projects/10
• transition from migration mode to full NLL
  • NLL warnings (downgraded from errors) become errors again
  • as an intermediate step, potentially use a deny-by-default lint before making these hard errors
• Feature stabilized: NLL and MIR-based borrow checker enabled by default,
• AST-based borrow checker removed: Rest In Peace, AST borrowck (2012-2019) #64790.

Many of the work items for unchecked bullets above are gathered into one place on #57895

How to join the working group

If you'd like to help, please join the NLL working group -- just leave a comment below or ping @nikomatsakis on gitter and you will be added to the @rust-lang/wg-compiler-nll team. You will then get occasional pings (e.g., when new mentoring instructions are available or when looking for help), as well as being eligible to be assigned to issues and so forth. We discuss things on the WG-compiler-nll channel on Gitter.

How to find an issue to work on

To find important issues, use one of the following queries:

• Top priority right now are issues tagged with NLL-Complete, corresponding to code that should be accepted but isn't right now.
• Next priority are issues tagged with NLL-sound, corresponding to code that should get errors, but isn't right now.
• Next priority are issues tagged with NLL-diagnostics, corresponding to low quality error messages.
• Issues tagged with NLL-performant, corresponding to cases where NLL analysis takes too long.

You may also wish to look for E-mentor issues, which means that they have mentoring instructions. Also, if an issue is assigned, then someone is supposed to be working on it, but it's worth checking if they have made progress and pinging them -- people often get busy with other things.

Issues tagged with NLL-deferred are low priority right now, but if one strikes your fancy, feel free to tackle it!

How are issues are organized

All issues related to NLL are tagged with WG-compiler-nll. They are further tagged with a NLL-foo label to indicate a subcategory. Issues that have no NLL-label are considered "untriaged" and need to be sorted. Issues tagged with NLL-deferred are low priority right now.

Finally, you can always take a look at the full list of NLL-related issues.

In particular, issues tagged with E-mentor are those that contain mentoring instructions that can help you get started.

Issues (and pull requests) tagged with I-nominated are meant to be reviewed by the WG-compiler-nll at each weekly meeting. Here's the current nominated list.

If you can't find anything, reach out to @nikomatsakis on gitter.

Other issues

This section tracks related issues and notes.

Bugs in AST borrow check fixed in MIR borrowck

#47366 tracks known bugs from the AST borrow checker that were fixed in the MIR borrow checker. For each such bug, we have added a test to the repo with #![feature(nll)] and then closed the relevant issue (obviously, though, the bug will still be reproducable until the MIR-based borrow checker is stabilized, presuming one uses the AST-based borrow checker). You can also search for things tagged with NLL-fixed-by-NLL.

Questions to be resolved before stabilization

• [NLL] false edges on infinite loops #46036: infinite loop false edges
• NLL should identify and respect the lifetime annotations that the user wrote NLL should identify and respect the lifetime annotations that the user wrote #47184

Possible extensions

• refine liveness with maybe-initialized at greater resolution than a single variable

[nikomatsakis]

I've written up a gist with the initial draft of a plan. It lays out the first few steps anyway and some of the initial vision. I'll just copy-and-paste the end here, which lays out some of the PRs I think would make sense:

(list moved to header)

[nikomatsakis]

@Nashenas88 has expressed some interest in doing this! It occurs to me that this project may be big enough for multiple people, too.

cc @rust-lang/compiler

[spastorino]

have talked with @nikomatsakis and I'm joining this group 😄

[chrisvittal]

With #45013 pretty much set. I'm willing to work on testing infrastructure. I just need to know what we want and what parts of the code I need to look at.

[nikomatsakis]

For those who will hack on NLL: right now, the "tip" of NLL development has not yet landed in master. In order to keep things moving, we're adopting the following strategy.

There is a nll-master branch on my fork of Rust. This contains the NLL work that is considered "done". You can open PRs against that branch on my repository.

In terms of rebasing etc, this branch works like Rust master: that is, until your PR lands, you have to rebase it against nll-master. But once a PR lands there, you are done -- I will take care of rebasing it and keeping it up to date.

This branch is not intended to live permanently far from master -- I will rebase it as needed, and I will always be trying to land the next logical chunk from it. However, this removes main Rust master as the central "gate" for Rust development.

EDIT: This is no longer true.

[camelid]

Should this be closed? The AST borrow-checker was removed over a year ago, so haven't we fully transitioned to the MIR borrow-checker now?

Also, I think the "round three" of diagnostics to-do item can be checked; the issue for it has been closed, as has the project board.

[felix91gr]

If this is closed, then it should also be removed from the Unstable Book, right? https://doc.rust-lang.org/beta/unstable-book/language-features/nll.html

[rise0chen]

#![allow(unused)]
#![feature(nll)]

use std::sync::Mutex;
fn main() {
    let mutex: Mutex<Option<u8>> = Mutex::new(None);
    if let Some(val) = &*mutex.lock().unwrap() {
        println!("got {}.", val);
    }else{
        *mutex.lock().unwrap() = Some(0);
        println!("init.");
    };
}

Should the MutexGuard be dropped in else block ?

[shepmaster]

The lock should be dropped at the end of the statement it’s created on because there is no owner of the lock.

Will the non-lexical lifetime borrow checker release locks prematurely?

Temporary scopes

[nikomatsakis]

I think the main reason this issue is still open is that we haven't fully removed the lexical region checker.

[tlyu]

Could we please have a brief status summary near the top of the issue description? I think this is great work and I appreciate the large amount of effort that went into it. As a relative newcomer, it is frustrating to not be able to know the current details about how the borrow checker works. It looks like the Reference might be out of date with respect to the NLL changes? Also, there might some language in the Book that could use some slight improvement.

When I tried to find more details about how the borrow checker works, after running into errors and behavior that I couldn't find explanations for, I repeatedly heard the term "NLL". I found this tracking issue, and it appears to still be open, which suggests that it hasn't been fully stabilized? However, I have also heard that almost all of the NLL features have already been stabilized, which was also confusing. The mention of -Z borrowck=migrate being the current default does help a little, but that's in the middle of a fairly long checklist.

It's not obvious from a quick glance that many (almost all?) of the NLL features have been stabilized around three years ago. It was certainly a surprise to me when I discovered that! I also think it's interesting that there aren't any documentation tasks in the checklist. I guess they could be out of scope for this repository, but then maybe there should be corresponding tracking issues in the documentation repositories that are linked to from this issue?

I would also be happy to work on documentation improvements, assuming I can find authoritative sources of information to draw from.

[Aaron1011]

I've started updating this issue with a summary of the current status.

[m-ou-se]

Here's an example that only compiles with #![feature(nll)] enabled: #95527

What work is left before that compiles on stable?

[m-ou-se]

I've nominated this because this is a blocker* for scoped threads, and would like to know if we can stabilize nll in Rust 1.63.

(* We could stabilize scoped threads without waiting for nll, but some use cases will then result in confusing errors.)

[m-ou-se]

Looks like tracking of this issue is happening here now: #95565 (:

[jackh726]

Also in #58781

[m-ou-se]

And also this one, it seems? #57895

That's a lot of tracking issues for the same thing (:

[jackh726]

Closed that one in favor of this and #58781