build --release takes forever on latest rust release

[Qumeric]

I have a project of around 10k lines and 1000 dependency crates. I cannot share it because it's closed source.

I was working on it for a few months and previously compiled it with stable, beta and nightly rust.

I decided to update rust to latest version. I run cargo build --release and it stuck at

   Compiling reth-block-validator v0.1.0 (/path/...)
   Building [=====================> ] 1157/1161: my-crate-name

I tried to build my-crate-name solely (full project is workspace) but it stuck in a same way.

I tried to do it with macOS + lld, linux + mold, linux + ld. Linux were 3 different machines with 5x00X Ryzen cpus.

It always stuck (I waited at least 10 and up to 60 minutes). Both in latest (installed with rustup) stable, beta and nightly.

I tried different standard things like cargo clean but nothing helped.

The project builds without --release just fine and builds with --release in rust 1.74.1 (and some previous versions of nightly and beta). It takes ~4 minutes for clean --release build and ~1.5 minutes for clean debug build.

I may try git bisect on compiler versions later if I will have time for that. Although I am not completely sure if it's compiler issue, could be cargo issue as well?

[ehuss]

Thanks for the report!

I may try git bisect on compiler versions later if I will have time for that. Although I am not completely sure if it's compiler issue, could be cargo issue as well?

We have a tool called cargo-bisect-rustc specifically for bisecting these issues. There is a --timeout option for diagnosing slow/hung builds (documented here, though it currently does not work on macOS).

[Qumeric]

I finally run bisect tool, thanks for pointing on it.

Report:

********************************************************************************
Regression in 2e5a9dd6c9eaa42f0684b4b760bd68fc27cbe51b
********************************************************************************

Attempting to search unrolled perf builds
ERROR: couldn't find perf build comment
==================================================================================
= Please file this regression report on the rust-lang/rust GitHub repository     =
=        New issue: https://github.com/rust-lang/rust/issues/new                 =
=     Known issues: https://github.com/rust-lang/rust/issues                     =
= Copy and paste the text below into the issue report thread.  Thanks!           =
==================================================================================

searched nightlies: from nightly-2023-09-01 to nightly-2024-01-01
regressed nightly: nightly-2023-10-03
searched commit range: e0d7ed1...2e5a9dd
regressed commit: 2e5a9dd

bisected with cargo-bisect-rustc v0.6.8

Host triple: x86_64-unknown-linux-gnu
Reproduce with:
bash
cargo bisect-rustc --end=2024-01-01 --timeout 300 -- build --release

Also, A bit of feedback on this tool:
RESULT: nightly-2023-09-01, ===> No is a bit confusing, I am not sure what "No" means? Does it mean success or fail or it depends on what it's trying to check? I suggest changing it to something like "Timeout", "ICE", "Success" etc. (perhaps in addition to Yes/No).

I guess that "No" means "No failure" and "Yes" means "Failure" but I only understood it after the run.

[Qumeric]

Tried to build manually with nightly-2023-10-02 and can confirm it works.

Also tried the latest nightly and beta and it is still broken. Pretty sure the latest stable is also still broken but I did not check it.

[ehuss]

Thanks for investigating.

cc @nikic @InnovativeInventor, this seems to be caused by #102099. Is this kind of regression expected? Are there additional tasks for investigating why it appears to be getting stuck?

One possibility that comes to mind in this situation is that maybe it is doing significantly more inlining, and possibly running out of memory and hitting swap.

[nikic]

Yes, this is "expected", in the sense that it can result in pathological compile-times for pathological inputs -- most likely, you have a proc macro somewhere creating a huge function (say, hundreds of thousands instructions), and this change perturbs things enough for it to trigger some super-linear optimization behavior.

Beyond that, there's not much I can say here without a reproducer or even a profile.

[Qumeric]

I don't think it's OOM, I had 256GB of RAM on some machines I tried this.

In terms of macro I can think of only async_trait, everything else doesn't look heavy. And in general I don't have an idea what can be pathological in my project, there is nothing unusual to my eye.

I will try to remove async_trait and see if it helps, also I will try to run compilation overnight, maybe it will finish.

Do you have any advice on what I can do to figure out what's going on?

[nikic]

I'd start by adding -v to the cargo command and finding the rustc command that hangs.

Then run it under perf record (it's okay to just ctrl+c after a while) and then check perf report -- I'd expect to see most time being taken up somewhere in LLVM. If we're very lucky it may be possible to identify what in LLVM causes this just from that.

The other thing I'd try is adding --emit=llvm-ir -C no-prepopulate-passes and seeing whether there are any large functions in the .ll file.

[Qumeric]

I think this issue was fixed, at least my project (with library updates) compiles at latest stable with --release

[apiraino]

thanks @Qumeric for the feedback. I don't see other comments (or comment from other issues) so feel free to close this issue if everything's fine on your end