incremental compilation: could not create session directory lock file: No locks available (os error 37) #49773

szignal · 2018-04-08T02:56:44Z

I was unable to incrementally compile using cargo build.

I had added the rand dependency to the toml file, per the advice of the Guessing Game exercise in Rust Programming Language (Second Edition).

Before this particular error, I had just finished setting up a synced folder with NFS on my vagrant box (ubuntu 16.04) where rust was installed. There is some issue using the default vagrant shared folder setup with rust.

I tried this code:

use std::io;

fn main() {
    println!("Guess the number");

    println!("Please input your guess.");

    let mut guess = String::new();

    io::stdin().read_line(&mut guess)
        .expect("Failed to read line");

    print!("You guessed: {}", guess);
}

I expected to see this happen:

compile

Instead, this happened:

    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading rand v0.3.22                                                       
 Downloading libc v0.2.40                                                       
 Downloading rand v0.4.2                                                        
   Compiling libc v0.2.40                                                       
   Compiling rand v0.4.2
   Compiling rand v0.3.22
   Compiling guessing_game v0.1.0 (file:///Rust/projects/guessing_game)
error: incremental compilation: could not create session directory lock file: No locks available (os error 37)

thread 'rustc' panicked at 'librustc/session/mod.rs:727: Trying to get session directory from IncrCompSession `NotInitialized`', librustc/session/mod.rs:1180:26
note: Run with `RUST_BACKTRACE=1` for a backtrace.

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.25.0 (84203cac6 2018-03-25) running on x86_64-unknown-linux-gnu

error: Could not compile `guessing_game`.

rustc --version --verbose:

rustc 1.25.0 (84203cac6 2018-03-25)
binary: rustc
commit-hash: 84203cac67e65ca8640b8392348411098c856985
commit-date: 2018-03-25
host: x86_64-unknown-linux-gnu
release: 1.25.0
LLVM version: 6.0

Backtrace:

thread 'rustc' panicked at 'librustc/session/mod.rs:727: Trying to get session directory from IncrCompSession `NotInitialized`', librustc/session/mod.rs:1180:26
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
stack backtrace:
   0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
             at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
   1: std::sys_common::backtrace::_print
             at libstd/sys_common/backtrace.rs:71
   2: std::panicking::default_hook::{{closure}}
             at libstd/sys_common/backtrace.rs:59
             at libstd/panicking.rs:380
   3: std::panicking::default_hook
             at libstd/panicking.rs:396
   4: std::panicking::rust_panic_with_hook
             at libstd/panicking.rs:576
   5: std::panicking::begin_panic
   6: rustc::session::opt_span_bug_fmt::{{closure}}
   7: rustc::ty::context::tls::with_opt
   8: rustc::session::opt_span_bug_fmt
   9: rustc::session::bug_fmt
  10: rustc::session::Session::incr_comp_session_dir
  11: rustc_incremental::persist::fs::garbage_collect_session_directories
  12: rustc_driver::driver::phase_2_configure_and_expand_inner::{{closure}}
  13: rustc_driver::driver::phase_2_configure_and_expand_inner
  14: rustc_driver::driver::compile_input
  15: rustc_driver::run_compiler

The text was updated successfully, but these errors were encountered:

glaubitz · 2018-04-09T14:02:30Z

This is an issue we are currently seeing on linux-sparc64 as well [1]:

error: incremental compilation: could not create session directory lock file: No such process (os error 3)

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.24.1 running on sparc64-unknown-linux-gnu

thread 'rustc' panicked at 'src/librustc/session/mod.rs:665: Trying to get session directory from IncrCompSession `NotInitialized`', src/librustc/session/mod.rs:1141:26
note: Run with `RUST_BACKTRACE=1` for a backtrace.

error: Could not compile `build_helper`.
warning: build failed, waiting for other jobs to finish...

I have not yet figured out where this comes from. The same rust version worked fine in the past.

[1] https://buildd.debian.org/status/fetch.php?pkg=rustc&arch=sparc64&ver=1.25.0%2Bdfsg1-1%7Eexp2&stamp=1523052949&raw=0

CC @jrtc27 @psumbera @mkarcher

michaelwoerister · 2018-05-03T13:40:36Z

NFS does not support locking files very well, I guess. It would be good if there was some way to make Cargo put the incremental folder somewhere else (into /tmp for example).

cc @rust-lang/cargo

jrtc27 · 2018-05-03T13:45:35Z

Although the original report mentions NFS, we're not using NFS for the sparc64 buildds; the two issues have similar but different error messages ("No locks available" vs "No such process") so they could well be different problems.

michaelwoerister · 2018-05-03T14:02:31Z

@jrtc27, yes, sounds like a different root cause.

alexcrichton · 2018-05-03T14:41:30Z

In Cargo we disable file locking whenever NFS is detected, although that may not work well for rustc as it's semi-reasonable to expect Cargo to not be run in parallel much on one system but rustc probably is being executed in parallel by Cargo at least.

An alternative to moving the incremental directory is moving the locks for the incremental directory. I've considered this for Cargo where we actually want to enable file locking on NFS directories. That'd require more intrusive support in rustc, though.

glaubitz · 2018-09-22T23:36:58Z

If anyone wants to reproduce this problem, there are sparc64 porterboxes available in the gcc compile farm, both running Solaris and Linux. Any open source developer can apply for an account:

https://cfarm.tetaneutral.net/machines/list/

michaelwoerister · 2018-09-24T09:31:50Z

Is this on a file system that should support locks in general?

An alternative to moving the incremental directory is moving the locks for the incremental directory. I've considered this for Cargo where we actually want to enable file locking on NFS directories. That'd require more intrusive support in rustc, though.

As long as we don't want to support two different machines working on the same NFS share simultaneously (which we don't really do anyway), we could indeed just allocate that lock files in /tmp. The only real restriction for lock files is that they mustn't be contained in the session directory that they protect.

glaubitz · 2018-09-24T09:41:41Z

Is this on a file system that should support locks in general?

It's a standard off-the-shelf installation of Debian on ext4. Nothing special. There is no NFS or anything involved and it affects sparc64 machines with the same configuration as comparable x86_64 machines.

michaelwoerister · 2018-09-24T11:37:26Z

Interesting. That's unexpected.

glaubitz · 2018-09-24T11:39:29Z

You can test it yourself on the Linux/sparc64 porterbox available in the gcc compile farm. It's a rather fast machine.

michaelwoerister · 2018-09-24T11:48:05Z

I've requested an account. Thanks for the tip!

glaubitz · 2018-09-24T11:58:44Z

I have root privileges on the sparc64 machine (but not the others), so I can install additional packages if required.

glaubitz · 2018-10-14T14:41:38Z

@michaelwoerister Did you have any success getting an account in the gcc compile farm?

michaelwoerister · 2018-10-23T15:17:45Z

@glaubitz Yes, I have an account. I haven't gotten around to actually using it though.

glaubitz · 2018-12-20T13:53:49Z

@michaelwoerister @alexcrichton

@karcherm has had a look at the Rust sources and he found the problem rather quickly. Rust is hard-wiring the constants for the fnctls from glibc assuming they are constant across which they are not.

Compare:

https://github.molgen.mpg.de/git-mirror/glibc/blob/master/sysdeps/unix/sysv/linux/sparc/bits/fcntl.h#L53

and:

https://doc.rust-lang.org/nightly/nightly-rustc/src/rustc_data_structures/flock.rs.html#45

Rust is trying to call "Set Lock" and uses function control number 6. But that's "F_SETOWN" on sparc64 which expects a PID as a parameter which is why the error message is "No such process".

michaelwoerister · 2018-12-20T14:05:01Z

Awesome find, @karcherm & @glaubitz!

jonathan-sha · 2020-03-23T19:24:33Z

I get this when compiling files stored on WSL2, but running cargo on windows. So running on windows and compiling on this path:

\\wsl$\Ubuntu\home\...

nfwatsoncom · 2020-05-10T05:52:33Z

Ditto from @tpn-station ... I likewise have source on WSL2 (Ubuntu-20.04 w/ native Kernel) filesystem with Ubuntu's root FS mounted as drive "N:" in Windows. To Windows it looks like a network drive. My Rust project is configured for Windows build (as opposed to running CLion in Ubuntu).

I also tried avoiding the "N:" drive-letter mapping and specifying source files directly as "'\\wsl$\Ubuntu-20.04\home..." and hit the same error.

DrDinosaur · 2020-05-26T00:18:55Z

Same issue on WSL 2 as above.

daswafford · 2020-06-06T03:39:07Z

Hi, I think I am seeing a related error. I was trying to build the hello world example from https://rust-cli.github.io/book/tutorial/setup.html and when issuing cargo run, get:

dswafford@Davids-Mac-mini.local:/Volumes/dswafford/CODE/rust/hello/grrs$cargo run --verbose
   Compiling grrs v0.1.0 (/Volumes/dswafford/CODE/rust/hello/grrs)
     Running `rustc --crate-name grrs --edition=2018 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=a4d4421a81652fc4 -C extra-filename=-a4d4421a81652fc4 --out-dir /Volumes/dswafford/CODE/rust/hello/grrs/target/debug/deps -C incremental=/Volumes/dswafford/CODE/rust/hello/grrs/target/debug/incremental -L dependency=/Volumes/dswafford/CODE/rust/hello/grrs/target/debug/deps`
error: incremental compilation: could not create session directory lock file: Operation not supported (os error 45)

thread 'rustc' panicked at 'trying to get session directory from `IncrCompSession`: NotInitialized', src/librustc_session/session.rs:690:48
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

error: internal compiler error: unexpected panic

note: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports

note: rustc 1.44.0 (49cae5576 2020-06-01) running on x86_64-apple-darwin

note: compiler flags: -C debuginfo=2 -C incremental --crate-type bin

note: some of the compiler flags provided by cargo are hidden

error: aborting due to previous error

error: could not compile `grrs`.

Caused by:
  process didn't exit successfully: `rustc --crate-name grrs --edition=2018 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=a4d4421a81652fc4 -C extra-filename=-a4d4421a81652fc4 --out-dir /Volumes/dswafford/CODE/rust/hello/grrs/target/debug/deps -C incremental=/Volumes/dswafford/CODE/rust/hello/grrs/target/debug/incremental -L dependency=/Volumes/dswafford/CODE/rust/hello/grrs/target/debug/deps` (exit code: 101)

I am on a MacOS computer running 10.15.4 and the current working directory where I ran "cargo" run" is an smb mount to a network-based file server (Ubuntu):

dswafford@Davids-Mac-mini.local:/Volumes/dswafford/CODE/rust/hello/grrs$mount
...
//dswafford@nas.ntwk.cc/dswafford on /Volumes/dswafford (smbfs, nodev, nosuid, mounted by dswafford)

Tarnadas · 2020-11-10T09:05:58Z

Current workaround for Windows compiling a project in WSL2:

# powershell
$env:CARGO_INCREMENTAL=0; cargo build

see #66513 (comment)

ms140569 · 2020-12-29T20:30:28Z

I'm facing the very same issue (Rust 1.48.0, Mac -> SMB), documented here:

#76251 (comment)

Both of these two workarounds are valid on Mac OSX:

CARGO_INCREMENTAL=0 cargo build
CARGO_TARGET_DIR=/tmp cargo build

Sorry, if i commented/documented on the wrong issue ...

…tebank Don't panic when failing to initialize incremental directory. This removes a panic when rustc fails to initialize the incremental directory. This can commonly happen on various filesystems that don't support locking (often various network filesystems). Panics can be confusing and scary, and there are already plenty of issues reporting this. This has been panicking since 1.22 due to I think rust-lang#44502 which was a major rework of how things work. Previously, things were simpler and the [`load_dep_graph`](https://github.com/rust-lang/rust/blob/1.21.0/src/librustc_incremental/persist/load.rs#L43-L65) function would emit an error and then continue on without panicking. With 1.22, [`load_dep_graph`](https://github.com/rust-lang/rust/blob/1.22.0/src/librustc_incremental/persist/load.rs#L44) was changed so that it assumes it can load the data without errors. Today, the problem is that it calls [`prepare_session_directory`](https://github.com/rust-lang/rust/blob/fbf1b1a7193cda17008ab590e06ad28d9924023b/compiler/rustc_interface/src/passes.rs#L175-L179) and then immediately calls `garbage_collect_session_directories` which will panic since the session is `IncrCompSession::NotInitialized`. The solution here is to have `prepare_session_directory` return an error that must be handled so that compilation stops if it fails. Some other options: * Ignore directory lock failures. * Print a warning on directory lock failure, but otherwise continue with incremental enabled. * Print a warning on directory lock failure, and disable incremental. * Provide a different locking mechanism. Cargo ignores lock errors if locking is not supported, so that would be a precedent for the first option. These options would require quite a bit more changes, but I'm happy to entertain any of them, as I think they all have valid justifications. There is more discussion on the many issues where this is reported: rust-lang#49773, rust-lang#59224, rust-lang#66513, rust-lang#76251. I'm not sure if this can be considered closing any of those, though, since I think there is some value in discussing if there is a way to avoid the error altogether. But I think it would make sense to at least close all but one to consolidate them.

ehuss · 2021-05-29T22:36:06Z

#85698 has changed it so that rustc does not panic in this situation, but it will still generate an error.

Leaving this open as I think it is still worthwhile to consider what other options there are for locking, or if certain situations can be fixed.

A summary of the things people have reported:

(this issue) NFS synced folder with vagrant (ubuntu 16.04) locking fails with ENOLCK.
incremental compilation: could not create session directory lock file: No locks available (os error 37) #49773 (comment) linux-sparc64 failed with ESRCH (which was probably resolved by flock: Use fcntl constants directly from libc crate on Unix targets #57121, but uncertain)
incremental compilation: could not create session directory lock file: No locks available (os error 37) #49773 (comment) and Incremental compilation ICEs on Windows in file systems that don’t support file locking #66513 WSL2 (Windows to WSL paths) fails with ERROR_INVALID_FUNCTION
incremental compilation: could not create session directory lock file: No locks available (os error 37) #49773 (comment) and error: incremental compilation: could not create session directory lock file: Input/output error (os error 5) #76251 (comment) and thread 'rustc' panicked at 'src/librustc/session/mod.rs:809 with clean cargo new project #59224 SMB mounts on macOS fails with ENOTSUP
error: incremental compilation: could not create session directory lock file: Input/output error (os error 5) #76251 NFS mount failed with EIO

kngrektor · 2021-09-17T13:23:29Z

regarding SMB mounts on macOS: I checked ages ago and flock seems to work fine with SMB mounts it's specifically F_SETLK with fcntl that doesn't work. I believe this means that the Linux code path would work fine for macOS but either way a fallback would be nice.

tommythorn · 2022-01-21T19:11:27Z

How is rustc deciding that locks aren't working? Is it just a heuristic? Because my NFS4 mount on macOS does support locks and they work. If it's a heuristic can it be overruled?

Xymist · 2022-04-20T17:17:42Z

This may or may not be useful, but for possible future Googlers....

I keep my dev folder on a TrueNAS server and access via an NFS share. Ended up with this issue after moving from Core to Scale. The fix was to turn on both "Enable NFSv4" and "NFSv3 ownership model for NFSv4" in the NFS service settings.

michaelwoerister added A-incr-comp Area: Incremental compilation I-ICE Issue: The compiler panicked, giving an Internal Compilation Error (ICE) ❄️ labels May 3, 2018

XAMPPRocky added T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. C-bug Category: This is a bug. labels Jul 7, 2018

glaubitz mentioned this issue Aug 16, 2018

powerpc-unknown-linux-gnu fails to compile simple crates #50960

Closed

glaubitz mentioned this issue Dec 20, 2018

File-locking API uses incorrect fnctl constants on at least sparc64 #57007

Closed

ehuss mentioned this issue Apr 27, 2021

remote file lock issue causes a panic #84417

Closed

ehuss mentioned this issue May 25, 2021

Don't panic when failing to initialize incremental directory. #85698

Merged

This was referenced Aug 22, 2023

Panic compiling in wsl (otherwise link error LNK1181) #81999

Closed

cargo run encounters ICE on filesystems without flock #56976

Closed

os error 45 on macOS 10.13.4 #55733

Closed

ehuss mentioned this issue Sep 9, 2023

Incremental compilation does not work through mounted WSL folder on Windows rust-lang/cargo#12650

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

incremental compilation: could not create session directory lock file: No locks available (os error 37) #49773

incremental compilation: could not create session directory lock file: No locks available (os error 37) #49773

szignal commented Apr 8, 2018 •

edited by JohnTitor

Loading

glaubitz commented Apr 9, 2018 •

edited

Loading

michaelwoerister commented May 3, 2018

jrtc27 commented May 3, 2018

michaelwoerister commented May 3, 2018 •

edited

Loading

alexcrichton commented May 3, 2018

glaubitz commented Sep 22, 2018

michaelwoerister commented Sep 24, 2018

glaubitz commented Sep 24, 2018

michaelwoerister commented Sep 24, 2018

glaubitz commented Sep 24, 2018

michaelwoerister commented Sep 24, 2018

glaubitz commented Sep 24, 2018

glaubitz commented Oct 14, 2018

michaelwoerister commented Oct 23, 2018

glaubitz commented Dec 20, 2018

michaelwoerister commented Dec 20, 2018

jonathan-sha commented Mar 23, 2020

nfwatsoncom commented May 10, 2020 •

edited

Loading

DrDinosaur commented May 26, 2020

daswafford commented Jun 6, 2020 •

edited

Loading

Tarnadas commented Nov 10, 2020

ms140569 commented Dec 29, 2020 •

edited

Loading

ehuss commented May 29, 2021

kngrektor commented Sep 17, 2021 •

edited

Loading

tommythorn commented Jan 21, 2022

Xymist commented Apr 20, 2022

incremental compilation: could not create session directory lock file: No locks available (os error 37) #49773

incremental compilation: could not create session directory lock file: No locks available (os error 37) #49773

Comments

szignal commented Apr 8, 2018 • edited by JohnTitor Loading

glaubitz commented Apr 9, 2018 • edited Loading

michaelwoerister commented May 3, 2018

jrtc27 commented May 3, 2018

michaelwoerister commented May 3, 2018 • edited Loading

alexcrichton commented May 3, 2018

glaubitz commented Sep 22, 2018

michaelwoerister commented Sep 24, 2018

glaubitz commented Sep 24, 2018

michaelwoerister commented Sep 24, 2018

glaubitz commented Sep 24, 2018

michaelwoerister commented Sep 24, 2018

glaubitz commented Sep 24, 2018

glaubitz commented Oct 14, 2018

michaelwoerister commented Oct 23, 2018

glaubitz commented Dec 20, 2018

michaelwoerister commented Dec 20, 2018

jonathan-sha commented Mar 23, 2020

nfwatsoncom commented May 10, 2020 • edited Loading

DrDinosaur commented May 26, 2020

daswafford commented Jun 6, 2020 • edited Loading

Tarnadas commented Nov 10, 2020

ms140569 commented Dec 29, 2020 • edited Loading

ehuss commented May 29, 2021

kngrektor commented Sep 17, 2021 • edited Loading

tommythorn commented Jan 21, 2022

Xymist commented Apr 20, 2022

szignal commented Apr 8, 2018 •

edited by JohnTitor

Loading

glaubitz commented Apr 9, 2018 •

edited

Loading

michaelwoerister commented May 3, 2018 •

edited

Loading

nfwatsoncom commented May 10, 2020 •

edited

Loading

daswafford commented Jun 6, 2020 •

edited

Loading

ms140569 commented Dec 29, 2020 •

edited

Loading

kngrektor commented Sep 17, 2021 •

edited

Loading