Tracking peak total storage use

[workingjubilee]

Related August 29th CI event

On August 29th, around 3:59 Pacific Daylight Time, our CI started to fail due to not having enough storage available. It merged a few PRs, but then about 10 hours later it merged the final PR that it would merge that day: 0d63418

It continued to fail for hours. The spotty CI passes were probably due to GitHub initiating a rollout to their fleet that took 12 hours to reach complete global saturation. At that moment, GitHub reduced the actual levels of storage offered to runners to levels that closely reflect their service agreement. See actions/runner-images#10511 for more on that.

Eventually, I landed #129797 which seemed to get CI going again.

Do we take up too much space?

We have had our storage usage grow, arguably to concerning levels, over time. Yes, a lot compresses for transfer, but I'm talking about peak storage occupancy here. And tarballs are not a format that are conducive to accessing individual files, so in practice, the relevant data occupies hosts in its full, uncompressed glory nonetheless. We also generate quite a lot of build intermediates. Big ones. Some of this is unavoidable, but we should consider investigating ways to reduce storage occupancy of the toolchain and its build intermediates.

Besides, we are having issues keeping our storage usage under the amount available to CI, even if there are other aggravating events. Obviously, clearing CI storage space can be done as a dirty hack to get things running again, but changes that benefit the entire ecosystem are more desirable. However, note that a solution that reduces storage but significantly increases the number of filesystem accesses, especially during compiler or tool builds, is likely to make CI problems worse due to this fun little issue:

• Tracking Issue for high failure rates on Windows MSVC CI with filesystem errors #127883

I'm opening this issue as a question, effectively: We track how much time the compiler costs, but what about space? Where are we tracking things like e.g. total doc size (possibly divided between libstd doc size and so on)? Are we aware of things like how much space is used by incremental compilation or other intermediates, and how it changes between versions? How about things like e.g. how many crater subjobs run out of space in each beta run? Where would someone find this information?

[VorpalBlade]

While this issue is about improving this for rustc itself, is there any issue about reducing the size of "normal" rust programs built with cargo? That is also a problem. For example:

• I cannot use the rust-cache github action effectively because the total cache from all the different configurations (lots of architectures x stable, nightly, MSRV) ends up above the limit of 10 GB.
• Local build directories on my laptop are also massive (thankfully my laptop has upgradable storage, but still it would be nice to use less) and cargo doesn't seem to clean out old builds with different hashes (meaning I have to regularly delete target and rebuild from scratch, which increases wear on the SSD and wastes electricity).
• The build size is a common complaint on reddit and similar places too, especially from people without upgradable storage (Mac... But also some x86-64 laptops)

[jieyouxu]

cc Metrics Initiative #128914

[bjorn3]

I cannot use the rust-cache github action effectively because the total cache from all the different configurations (lots of architectures x stable, nightly, MSRV) ends up above the limit of 10 GB.

Have you disabled incr comp already? CI generally benefits less from incr comp, but it does take a lot of space. (To be precise restoring the incr comp cache from the network can take longer than the amount of time saved by incr comp) CARGO_INCREMENTAL=false can be used to disable it.

[VorpalBlade]

Have you disabled incr comp already? CI generally benefits less from incr comp, but it does take a lot of space. (To be precise restoring the incr comp cache from the network can take longer than the amount of time saved by incr comp) CARGO_INCREMENTAL=false can be used to disable it.

I did that yes. And disable debug info too. But I target 32 and 64 bit. Across x86, ARM, RISCV and PPC. And all three major operating systems (obviously not all combinations are possible). That quickly adds up due to combinatorial explosion. The total cache size I would need currently is around 18 GB for that project.

As such it would be nice to be able to reduce the size of things that rustc produces. From my experience C++ doesn't expand (based on LOC) quite so voluminously. Of course, they are different languages (and Rust has more advanced abstractions) but perhaps there are things that could be done to the intermediate file formats.

[workingjubilee]

This is not actually about reducing the size per se.
It is about being aware of whether or not we have reduced the size, as a first step.
If we just reduce the size and then do not know the history before or after, it is a bit like merging code without regression tests. You should avoid it if you are trying to actually make an improvement, especially if you want it to stick.

[the8472]

There's cpu-usage-over-time.py, perhaps that could be extended to also gather disk usage. Though it currently emits CSV, json-lines would be more appropriate for more structured data.

[workingjubilee]

( I waffled back and forth on whether libs cares, and then I remembered all the libs PRs about compiled binary size, so... )

Anyways, nominating this for uh Basically Everyone I guess. I asked some more specific questions but also feel free to answer the question in a broad way, basically on two parts:

1. What information is this team currently tracking re: size?
2. What information would this team like to have available re: size?

@rustbot label: +I-compiler-nominated +I-libs-nominated +I-release-nominated

[apiraino]

Would T-infra (the only team not nominated 😅 ) have more historical context? Maybe I didn't search deep enough but I couldn't find on Zulip a thread with them discussing this issue

[Amanieu]

T-libs is not really concerned with the size of intermediate build products, that's mostly a T-compiler/T-cargo issue. We do track code size for the final binary, and keep an eye out for issues related to the size of rlibs shipped in rustup or the size of docs but that's it.

[workingjubilee]

Would T-infra (the only team not nominated 😅 ) have more historical context? Maybe I didn't search deep enough but I couldn't find on Zulip a thread with them discussing this issue

probably but there's no I-infra-nominated so in the absence of confirmation that they are attending to it, it seemed easiest to volley the question at a relevant subteam. :^) (GitHub notifications seemed more disruptive since they would continue to pester people in a stochastic way even after someone took care of responding.)

[apiraino]

Discussed by T-compiler last week on Zulip.

T-compiler is not aware of anything obvious that contributes to disk usage exploding but happy to try and make savings. We would need more information on what's taking up space during the CI run, though.

@rustbot label -I-compiler-nominated

[Noratrieb]

this is release nominated, but I don't know if release looks at nominations (given there's been no response I suspect no). @rust-lang/release in case you don't