Stop keeping metadata in memory before writing it to disk #96544

m-ysk · 2022-04-29T07:28:31Z

I created this PR according with the instruction given in the issue except for the following points:

While the issue says "Write metadata into the temporary file in encode_and_write_metadata even if !need_metadata_file", I could not do that. That is because though I tried to do that and run x.py test, I got a lot of test failures as follows.

List of failed tests


failures:
    [ui] src/test/ui/json-multiple.rs
    [ui] src/test/ui/json-options.rs
    [ui] src/test/ui/rmeta/rmeta-rpass.rs
    [ui] src/test/ui/save-analysis/emit-notifications.rs
    [ui] src/test/ui/svh/changing-crates.rs
    [ui] src/test/ui/svh/svh-change-lit.rs
    [ui] src/test/ui/svh/svh-change-significant-cfg.rs
    [ui] src/test/ui/svh/svh-change-trait-bound.rs
    [ui] src/test/ui/svh/svh-change-type-arg.rs
    [ui] src/test/ui/svh/svh-change-type-ret.rs
    [ui] src/test/ui/svh/svh-change-type-static.rs
    [ui] src/test/ui/svh/svh-use-trait.rs
test result: FAILED. 12915 passed; 12 failed; 100 ignored; 0 measured; 0 filtered out; finished in 71.41s
Some tests failed in compiletest suite=ui mode=ui host=x86_64-unknown-linux-gnu target=x86_64-unknown-linux-gnu

Build completed unsuccessfully in 0:01:58

I could not resolve the extra tasks about create_rmeta_file and create_compressed_metadata_file for my lack of ability.

rust-highfive · 2022-04-29T07:28:34Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @oli-obk (or someone else) soon.

Please see the contribution instructions for more information.

compiler/rustc_codegen_ssa/src/back/link.rs

cjgillot

Thanks @m-ysk for the PR. I left a few comments.
This creates some back-and-forth with re-opening the just-closed file. Can some of these be avoided?

compiler/rustc_metadata/src/fs.rs

cjgillot · 2022-04-29T18:00:53Z

compiler/rustc_metadata/src/rmeta/encoder.rs

-    #[inline]
-    pub fn raw_data(&self) -> &[u8] {
-        &self.raw_data
+impl<S: Encoder> Encodable<S> for EncodedMetadata {


Are these Encodable/Decodable impls used anywhere?

Yes, the metadata is included in CodegenResults which can be serialized into a .rlink file when using -Zlink-only. Might be nice to move the writing of both the rlib metadata object and dylib compressed metadata object to before the linking phase so the .rlink file wouldn't need to contain the full metadata and instead just reference the written object files for use during linking.

cjgillot · 2022-04-29T18:05:45Z

compiler/rustc_metadata/src/rmeta/encoder.rs

+    let mut file = std::fs::OpenOptions::new()
+        .write(true)
+        .open(path.as_ref())
+        .unwrap_or_else(|err| tcx.sess.fatal(&format!("failed to open the file: {}", err)));


Can we avoid re-opening the file by writing the root position at the end instead of the beginning?
For instance, DepGraph and on-disk-cache serialization do this.

That would require a breaking change to the rustc version independent part of the metadata, which would mean that errors about incompatible rustc versions won't list the rustc version during the transition period. Also I'm not sure if the .rustc section will never have padding bytes. Padding bytes would break finding the place where the root position is written if it is at the end.

@cjgillot

Can we avoid re-opening the file by writing the root position at the end instead of the beginning?

I did that in b339176 and confirmed that all tests have passed locally. But I'm afraid that I cannot evaluate the breaking change bjorn3 pointed out because I'm new to this project. So could you review the above commit and tell me what I should do?

Back to this issue. We will have two writing open handles to the file. Can this cause issues on some platforms?
Could we rewind, like you did to count zero bytes?

compiler/rustc_metadata/src/fs.rs

cjgillot · 2022-04-29T18:10:54Z

@bors try @rust-timer queue

rust-timer · 2022-04-29T18:10:56Z

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

bors · 2022-04-29T18:11:03Z

⌛ Trying commit 05a86dcd9b570e250698853b9930a34b73c88b1c with merge c73f0fb2d935673127b13925a037cfd431be2b67...

compiler/rustc_metadata/src/fs.rs

bors · 2022-04-29T19:42:45Z

☀️ Try build successful - checks-actions
Build commit: c73f0fb2d935673127b13925a037cfd431be2b67 (c73f0fb2d935673127b13925a037cfd431be2b67)

rust-timer · 2022-04-29T19:42:47Z

Queued c73f0fb2d935673127b13925a037cfd431be2b67 with parent f1d8a7d, future comparison URL.

rust-timer · 2022-04-29T21:00:04Z

Finished benchmarking commit (c73f0fb2d935673127b13925a037cfd431be2b67): comparison url.

Summary:

Primary benchmarks: 🎉 relevant improvements found
Secondary benchmarks: 🎉 relevant improvements found

	Regressions 😿 (primary)	Regressions 😿 (secondary)	Improvements 🎉 (primary)	Improvements 🎉 (secondary)	All 😿 🎉 (primary)
count¹	0	1	5	19	5
mean²	N/A	0.2%	-0.4%	-0.5%	-0.4%
max	N/A	0.2%	-0.6%	-0.6%	-0.6%

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. While you can manually mark this PR as fit for rollup, we strongly recommend not doing so since this PR may lead to changes in compiler perf.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf -perf-regression

number of relevant changes ↩
the arithmetic mean of the percent change ↩

cjgillot · 2022-04-29T21:22:41Z

Perf shows max-rss improvement up to 4% on primary benchmarks. Well done!

m-ysk · 2022-05-02T08:15:11Z

compiler/rustc_metadata/src/rmeta/encoder.rs

@@ -750,8 +754,10 @@ impl<'a, 'tcx> EncodeContext<'a, 'tcx> {

        if tcx.sess.meta_stats() {
            let mut zero_bytes = 0;
-            for e in self.opaque.data.iter() {
-                if *e == 0 {
+            self.opaque.file().seek(std::io::SeekFrom::Start(0)).unwrap();


I added this seek by dd15cce to read the file from the start.

There are writes to this file after this stat is computed.
Could you add an assertion below that we get back to the position we left?
Should we flush the encoder buffer before computing this stat?

Fixed by 5ed4574 .

I add flush before rewinding the file. In addition, I store the position before rewinding and asserts after counting zero bytes.

cjgillot · 2022-07-03T12:59:54Z

src/test/run-make-fulldeps/issue-26092/Makefile

-include ../tools.mk
-
-all:
-	$(RUSTC) -o "" blank.rs 2>&1 | $(CGREP) -i 'No such file or directory'


Could you keep the test, but just check that rustc does not panic instead of checking for this very message?

I tried to do that by 1147d50 . I have confirmed that this test works well by intentionally invoking a panic. However, is there any smarter way to ensure the same thing?

m-ysk · 2022-07-03T15:43:34Z

@cjgillot
I have fixed the test by 1147d50 . Could you re-review this?

cjgillot · 2022-07-14T09:14:25Z

@bors r+

bors · 2022-07-14T09:14:27Z

📌 Commit 1147d50 has been approved by cjgillot

It is now in the queue for this repository.

bors · 2022-07-14T21:50:17Z

⌛ Testing commit 1147d50 with merge 1ba1fec...

bors · 2022-07-15T00:30:58Z

☀️ Test successful - checks-actions
Approved by: cjgillot
Pushing 1ba1fec to master...

rust-timer · 2022-07-15T01:48:22Z

Finished benchmarking commit (1ba1fec): comparison url.

Instruction count

Primary benchmarks: 🎉 relevant improvement found
Secondary benchmarks: mixed results

	mean¹	max	count²
Regressions 😿 (primary)	N/A	N/A	0
Regressions 😿 (secondary)	0.3%	0.4%	3
Improvements 🎉 (primary)	-0.3%	-0.3%	1
Improvements 🎉 (secondary)	-4.1%	-4.1%	2
All 😿🎉 (primary)	-0.3%	-0.3%	1

Max RSS (memory usage)

Results

Primary benchmarks: 🎉 relevant improvements found
Secondary benchmarks: mixed results

	mean¹	max	count²
Regressions 😿 (primary)	1.7%	2.9%	2
Regressions 😿 (secondary)	4.1%	4.1%	1
Improvements 🎉 (primary)	-2.8%	-5.9%	63
Improvements 🎉 (secondary)	-3.6%	-7.4%	23
All 😿🎉 (primary)	-2.6%	-5.9%	65

Cycles

Results

Primary benchmarks: no relevant changes found
Secondary benchmarks: 🎉 relevant improvements found

	mean¹	max	count²
Regressions 😿 (primary)	N/A	N/A	0
Regressions 😿 (secondary)	N/A	N/A	0
Improvements 🎉 (primary)	N/A	N/A	0
Improvements 🎉 (secondary)	-7.8%	-9.6%	5
All 😿🎉 (primary)	N/A	N/A	0

If you disagree with this performance assessment, please file an issue in rust-lang/rustc-perf.

@rustbot label: -perf-regression

the arithmetic mean of the percent change ↩ ↩² ↩³
number of relevant changes ↩ ↩² ↩³

rylev · 2022-07-19T17:18:47Z

Perf here is a wash. Going to mark as triaged.

@rustbot triage: +perf-regression-triaged

lqd · 2022-07-19T17:55:07Z

@rylev did it show up in triage ? That’s interesting, as the bot did not mark this PR as a regression needing investigation.

rylev · 2022-07-20T16:39:09Z

Yes, I think there's a tiny diff between what shows up on the triage report and what gets marked as a regression by the bot. I missed that the final run removed the regression label so my comment above wasn't really necessary. Sorry for the noise!

Pkgsrc changes: * Add patch to fix vendor/kqueue issue (on 32-bit hosts) * Adjust other patches & line numbers * Version bumps & checksum changes. Upstream changes: Version 1.64.0 (2022-09-22) =========================== Language -------- - [Unions with mutable references or tuples of allowed types are now allowed](rust-lang/rust#97995) - It is now considered valid to deallocate memory pointed to by a shared reference `&T` [if every byte in `T` is inside an `UnsafeCell`](rust-lang/rust#98017) - Unused tuple struct fields are now warned against in an allow-by-default lint, [`unused_tuple_struct_fields`] (rust-lang/rust#95977), similar to the existing warning for unused struct fields. This lint will become warn-by-default in the future. Compiler -------- - [Add Nintendo Switch as tier 3 target] (rust-lang/rust#88991) - Refer to Rust's [platform support page][platform-support-doc] for more information on Rust's tiered platform support. - [Only compile `#[used]` as llvm.compiler.used for ELF targets] (rust-lang/rust#93718) - [Add the `--diagnostic-width` compiler flag to define the terminal width.] (rust-lang/rust#95635) - [Add support for link-flavor `rust-lld` for iOS, tvOS and watchOS] (rust-lang/rust#98771) Libraries --------- - [Remove restrictions on compare-exchange memory ordering.] (rust-lang/rust#98383) - You can now `write!` or `writeln!` into an `OsString`: [Implement `fmt::Write` for `OsString`](rust-lang/rust#97915) - [Make RwLockReadGuard covariant] (rust-lang/rust#96820) - [Implement `FusedIterator` for `std::net::[Into]Incoming`] (rust-lang/rust#97300) - [`impl<T: AsRawFd> AsRawFd for {Arc,Box}<T>`] (rust-lang/rust#97437) - [`ptr::copy` and `ptr::swap` are doing untyped copies] (rust-lang/rust#97712) - [Add cgroupv1 support to `available_parallelism`] (rust-lang/rust#97925) - [Mitigate many incorrect uses of `mem::uninitialized`] (rust-lang/rust#99182) Stabilized APIs --------------- - [`future::IntoFuture`] (https://doc.rust-lang.org/stable/std/future/trait.IntoFuture.html) - [`future::poll_fn`] (https://doc.rust-lang.org/stable/std/future/fn.poll_fn.html) - [`task::ready!`] (https://doc.rust-lang.org/stable/std/task/macro.ready.html) - [`num::NonZero*::checked_mul`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroUsize.html#method.checked_mul) - [`num::NonZero*::checked_pow`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroUsize.html#method.checked_pow) - [`num::NonZero*::saturating_mul`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroUsize.html#method.saturating_mul) - [`num::NonZero*::saturating_pow`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroUsize.html#method.saturating_pow) - [`num::NonZeroI*::abs`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroIsize.html#method.abs) - [`num::NonZeroI*::checked_abs`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroIsize.html#method.checked_abs) - [`num::NonZeroI*::overflowing_abs`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroIsize.html#method.overflowing_abs) - [`num::NonZeroI*::saturating_abs`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroIsize.html#method.saturating_abs) - [`num::NonZeroI*::unsigned_abs`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroIsize.html#method.unsigned_abs) - [`num::NonZeroI*::wrapping_abs`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroIsize.html#method.wrapping_abs) - [`num::NonZeroU*::checked_add`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroUsize.html#method.checked_add) - [`num::NonZeroU*::checked_next_power_of_two`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroUsize.html#method.checked_next_power_of_two) - [`num::NonZeroU*::saturating_add`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroUsize.html#method.saturating_add) - [`os::unix::process::CommandExt::process_group`] (https://doc.rust-lang.org/stable/std/os/unix/process/trait.CommandExt.html#tymethod.process_group) - [`os::windows::fs::FileTypeExt::is_symlink_dir`] (https://doc.rust-lang.org/stable/std/os/windows/fs/trait.FileTypeExt.html#tymethod.is_symlink_dir) - [`os::windows::fs::FileTypeExt::is_symlink_file`] (https://doc.rust-lang.org/stable/std/os/windows/fs/trait.FileTypeExt.html#tymethod.is_symlink_file) These types were previously stable in `std::ffi`, but are now also available in `core` and `alloc`: - [`core::ffi::CStr`] (https://doc.rust-lang.org/stable/core/ffi/struct.CStr.html) - [`core::ffi::FromBytesWithNulError`] (https://doc.rust-lang.org/stable/core/ffi/struct.FromBytesWithNulError.html) - [`alloc::ffi::CString`] (https://doc.rust-lang.org/stable/alloc/ffi/struct.CString.html) - [`alloc::ffi::FromVecWithNulError`] (https://doc.rust-lang.org/stable/alloc/ffi/struct.FromVecWithNulError.html) - [`alloc::ffi::IntoStringError`] (https://doc.rust-lang.org/stable/alloc/ffi/struct.IntoStringError.html) - [`alloc::ffi::NulError`] (https://doc.rust-lang.org/stable/alloc/ffi/struct.NulError.html) These types were previously stable in `std::os::raw`, but are now also available in `core::ffi` and `std::ffi`: - [`ffi::c_char`] (https://doc.rust-lang.org/stable/std/ffi/type.c_char.html) - [`ffi::c_double`] (https://doc.rust-lang.org/stable/std/ffi/type.c_double.html) - [`ffi::c_float`] (https://doc.rust-lang.org/stable/std/ffi/type.c_float.html) - [`ffi::c_int`] (https://doc.rust-lang.org/stable/std/ffi/type.c_int.html) - [`ffi::c_long`] (https://doc.rust-lang.org/stable/std/ffi/type.c_long.html) - [`ffi::c_longlong`] (https://doc.rust-lang.org/stable/std/ffi/type.c_longlong.html) - [`ffi::c_schar`] (https://doc.rust-lang.org/stable/std/ffi/type.c_schar.html) - [`ffi::c_short`] (https://doc.rust-lang.org/stable/std/ffi/type.c_short.html) - [`ffi::c_uchar`] (https://doc.rust-lang.org/stable/std/ffi/type.c_uchar.html) - [`ffi::c_uint`] (https://doc.rust-lang.org/stable/std/ffi/type.c_uint.html) - [`ffi::c_ulong`] (https://doc.rust-lang.org/stable/std/ffi/type.c_ulong.html) - [`ffi::c_ulonglong`] (https://doc.rust-lang.org/stable/std/ffi/type.c_ulonglong.html) - [`ffi::c_ushort`] (https://doc.rust-lang.org/stable/std/ffi/type.c_ushort.html) These APIs are now usable in const contexts: - [`slice::from_raw_parts`] (https://doc.rust-lang.org/stable/core/slice/fn.from_raw_parts.html) Cargo ----- - [Packages can now inherit settings from the workspace so that the settings can be centralized in one place.] (rust-lang/cargo#10859) See [`workspace.package`](https://doc.rust-lang.org/nightly/cargo/reference/workspaces.html#the-workspacepackage-table) and [`workspace.dependencies`](https://doc.rust-lang.org/nightly/cargo/reference/workspaces.html#the-workspacedependencies-table) for more details on how to define these common settings. - [Cargo commands can now accept multiple `--target` flags to build for multiple targets at once] (rust-lang/cargo#10766), and the [`build.target`](https://doc.rust-lang.org/nightly/cargo/reference/config.html#buildtarget) config option may now take an array of multiple targets. - [The `--jobs` argument can now take a negative number to count backwards from the max CPUs.] (rust-lang/cargo#10844) - [`cargo add` will now update `Cargo.lock`.] (rust-lang/cargo#10902) - [Added](rust-lang/cargo#10838) the [`--crate-type`](https://doc.rust-lang.org/nightly/cargo/commands/cargo-rustc.html#option-cargo-rustc---crate-type) flag to `cargo rustc` to override the crate type. - [Significantly improved the performance fetching git dependencies from GitHub when using a hash in the `rev` field.] (rust-lang/cargo#10079) Misc ---- - [The `rust-analyzer` rustup component is now available on the stable channel.] (rust-lang/rust#98640) Compatibility Notes ------------------- - The minimum required versions for all `-linux-gnu` targets are now at least kernel 3.2 and glibc 2.17, for targets that previously supported older versions: [Increase the minimum linux-gnu versions](rust-lang/rust#95026) - [Network primitives are now implemented with the ideal Rust layout, not the C system layout] (rust-lang/rust#78802). This can cause problems when transmuting the types. - [Add assertion that `transmute_copy`'s `U` is not larger than `T`] (rust-lang/rust#98839) - [A soundness bug in `BTreeMap` was fixed] (rust-lang/rust#99413) that allowed data it was borrowing to be dropped before the container. - [The Drop behavior of C-like enums cast to ints has changed] (rust-lang/rust#96862). These are already discouraged by a compiler warning. - [Relate late-bound closure lifetimes to parent fn in NLL] (rust-lang/rust#98835) - [Errors at const-eval time are now in future incompatibility reports] (rust-lang/rust#97743) - On the `thumbv6m-none-eabi` target, some incorrect `asm!` statements were erroneously accepted if they used the high registers (r8 to r14) as an input/output operand. [This is no longer accepted] (rust-lang/rust#99155). - [`impl Trait` was accidentally accepted as the associated type value of return-position `impl Trait`] (rust-lang/rust#97346), without fulfilling all the trait bounds of that associated type, as long as the hidden type satisfies said bounds. This has been fixed. Internal Changes ---------------- These changes do not affect any public interfaces of Rust, but they represent significant improvements to the performance or internals of rustc and related tools. - Windows builds now use profile-guided optimization, providing 10-20% improvements to compiler performance: [Utilize PGO for windows x64 rustc dist builds] (rust-lang/rust#96978) - [Stop keeping metadata in memory before writing it to disk] (rust-lang/rust#96544) - [compiletest: strip debuginfo by default for mode=ui] (rust-lang/rust#98140) - Many improvements to generated code for derives, including performance improvements: - [Don't use match-destructuring for derived ops on structs.] (rust-lang/rust#98446) - [Many small deriving cleanups] (rust-lang/rust#98741) - [More derive output improvements] (rust-lang/rust#98758) - [Clarify deriving code](rust-lang/rust#98915) - [Final derive output improvements] (rust-lang/rust#99046) - [Stop injecting `#[allow(unused_qualifications)]` in generated `derive` implementations](rust-lang/rust#99485) - [Improve `derive(Debug)`](rust-lang/rust#98190) - [Bump to clap 3](rust-lang/rust#98213) - [fully move dropck to mir](rust-lang/rust#98641) - [Optimize `Vec::insert` for the case where `index == len`.] (rust-lang/rust#98755) - [Convert rust-analyzer to an in-tree tool] (rust-lang/rust#99603)

Pkgsrc changes: * This package now contains rust-analyzer, so implicitly conflicts with that pkgsrc package. The same goes for the rust-src package. * Add NetBSD/arm6 port * Add unfinished NetBSD/mipsel port * Revert the use of the internal LLVM, should now build with the new pkgsrc LLVM (15). * Add depndence on compat80 for sparc64 to fix the build * Adapt patches * Add CHECK_INTERPRETER_SKIP for a few (mostly unused) files. (A proper fix may come later.) Upstream changes: Version 1.64.0 (2022-09-22) =========================== Language -------- - [Unions with mutable references or tuples of allowed types are now allowed](rust-lang/rust#97995) - It is now considered valid to deallocate memory pointed to by a shared reference `&T` [if every byte in `T` is inside an `UnsafeCell`](rust-lang/rust#98017) - Unused tuple struct fields are now warned against in an allow-by-default lint, [`unused_tuple_struct_fields`] (rust-lang/rust#95977), similar to the existing warning for unused struct fields. This lint will become warn-by-default in the future. Compiler -------- - [Add Nintendo Switch as tier 3 target] (rust-lang/rust#88991) - Refer to Rust's [platform support page][platform-support-doc] for more information on Rust's tiered platform support. - [Only compile `#[used]` as llvm.compiler.used for ELF targets] (rust-lang/rust#93718) - [Add the `--diagnostic-width` compiler flag to define the terminal width.] (rust-lang/rust#95635) - [Add support for link-flavor `rust-lld` for iOS, tvOS and watchOS] (rust-lang/rust#98771) Libraries --------- - [Remove restrictions on compare-exchange memory ordering.] (rust-lang/rust#98383) - You can now `write!` or `writeln!` into an `OsString`: [Implement `fmt::Write` for `OsString`](rust-lang/rust#97915) - [Make RwLockReadGuard covariant] (rust-lang/rust#96820) - [Implement `FusedIterator` for `std::net::[Into]Incoming`] (rust-lang/rust#97300) - [`impl<T: AsRawFd> AsRawFd for {Arc,Box}<T>`] (rust-lang/rust#97437) - [`ptr::copy` and `ptr::swap` are doing untyped copies] (rust-lang/rust#97712) - [Add cgroupv1 support to `available_parallelism`] (rust-lang/rust#97925) - [Mitigate many incorrect uses of `mem::uninitialized`] (rust-lang/rust#99182) Stabilized APIs --------------- - [`future::IntoFuture`] (https://doc.rust-lang.org/stable/std/future/trait.IntoFuture.html) - [`future::poll_fn`] (https://doc.rust-lang.org/stable/std/future/fn.poll_fn.html) - [`task::ready!`] (https://doc.rust-lang.org/stable/std/task/macro.ready.html) - [`num::NonZero*::checked_mul`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroUsize.html#method.checked_mul) - [`num::NonZero*::checked_pow`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroUsize.html#method.checked_pow) - [`num::NonZero*::saturating_mul`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroUsize.html#method.saturating_mul) - [`num::NonZero*::saturating_pow`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroUsize.html#method.saturating_pow) - [`num::NonZeroI*::abs`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroIsize.html#method.abs) - [`num::NonZeroI*::checked_abs`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroIsize.html#method.checked_abs) - [`num::NonZeroI*::overflowing_abs`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroIsize.html#method.overflowing_abs) - [`num::NonZeroI*::saturating_abs`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroIsize.html#method.saturating_abs) - [`num::NonZeroI*::unsigned_abs`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroIsize.html#method.unsigned_abs) - [`num::NonZeroI*::wrapping_abs`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroIsize.html#method.wrapping_abs) - [`num::NonZeroU*::checked_add`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroUsize.html#method.checked_add) - [`num::NonZeroU*::checked_next_power_of_two`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroUsize.html#method.checked_next_power_of_two) - [`num::NonZeroU*::saturating_add`] (https://doc.rust-lang.org/stable/std/num/struct.NonZeroUsize.html#method.saturating_add) - [`os::unix::process::CommandExt::process_group`] (https://doc.rust-lang.org/stable/std/os/unix/process/trait.CommandExt.html#tymethod.process_group) - [`os::windows::fs::FileTypeExt::is_symlink_dir`] (https://doc.rust-lang.org/stable/std/os/windows/fs/trait.FileTypeExt.html#tymethod.is_symlink_dir) - [`os::windows::fs::FileTypeExt::is_symlink_file`] (https://doc.rust-lang.org/stable/std/os/windows/fs/trait.FileTypeExt.html#tymethod.is_symlink_file) These types were previously stable in `std::ffi`, but are now also available in `core` and `alloc`: - [`core::ffi::CStr`] (https://doc.rust-lang.org/stable/core/ffi/struct.CStr.html) - [`core::ffi::FromBytesWithNulError`] (https://doc.rust-lang.org/stable/core/ffi/struct.FromBytesWithNulError.html) - [`alloc::ffi::CString`] (https://doc.rust-lang.org/stable/alloc/ffi/struct.CString.html) - [`alloc::ffi::FromVecWithNulError`] (https://doc.rust-lang.org/stable/alloc/ffi/struct.FromVecWithNulError.html) - [`alloc::ffi::IntoStringError`] (https://doc.rust-lang.org/stable/alloc/ffi/struct.IntoStringError.html) - [`alloc::ffi::NulError`] (https://doc.rust-lang.org/stable/alloc/ffi/struct.NulError.html) These types were previously stable in `std::os::raw`, but are now also available in `core::ffi` and `std::ffi`: - [`ffi::c_char`] (https://doc.rust-lang.org/stable/std/ffi/type.c_char.html) - [`ffi::c_double`] (https://doc.rust-lang.org/stable/std/ffi/type.c_double.html) - [`ffi::c_float`] (https://doc.rust-lang.org/stable/std/ffi/type.c_float.html) - [`ffi::c_int`] (https://doc.rust-lang.org/stable/std/ffi/type.c_int.html) - [`ffi::c_long`] (https://doc.rust-lang.org/stable/std/ffi/type.c_long.html) - [`ffi::c_longlong`] (https://doc.rust-lang.org/stable/std/ffi/type.c_longlong.html) - [`ffi::c_schar`] (https://doc.rust-lang.org/stable/std/ffi/type.c_schar.html) - [`ffi::c_short`] (https://doc.rust-lang.org/stable/std/ffi/type.c_short.html) - [`ffi::c_uchar`] (https://doc.rust-lang.org/stable/std/ffi/type.c_uchar.html) - [`ffi::c_uint`] (https://doc.rust-lang.org/stable/std/ffi/type.c_uint.html) - [`ffi::c_ulong`] (https://doc.rust-lang.org/stable/std/ffi/type.c_ulong.html) - [`ffi::c_ulonglong`] (https://doc.rust-lang.org/stable/std/ffi/type.c_ulonglong.html) - [`ffi::c_ushort`] (https://doc.rust-lang.org/stable/std/ffi/type.c_ushort.html) These APIs are now usable in const contexts: - [`slice::from_raw_parts`] (https://doc.rust-lang.org/stable/core/slice/fn.from_raw_parts.html) Cargo ----- - [Packages can now inherit settings from the workspace so that the settings can be centralized in one place.] (rust-lang/cargo#10859) See [`workspace.package`](https://doc.rust-lang.org/nightly/cargo/reference/workspaces.html#the-workspacepackage-table) and [`workspace.dependencies`](https://doc.rust-lang.org/nightly/cargo/reference/workspaces.html#the-workspacedependencies-table) for more details on how to define these common settings. - [Cargo commands can now accept multiple `--target` flags to build for multiple targets at once] (rust-lang/cargo#10766), and the [`build.target`](https://doc.rust-lang.org/nightly/cargo/reference/config.html#buildtarget) config option may now take an array of multiple targets. - [The `--jobs` argument can now take a negative number to count backwards from the max CPUs.] (rust-lang/cargo#10844) - [`cargo add` will now update `Cargo.lock`.] (rust-lang/cargo#10902) - [Added](rust-lang/cargo#10838) the [`--crate-type`](https://doc.rust-lang.org/nightly/cargo/commands/cargo-rustc.html#option-cargo-rustc---crate-type) flag to `cargo rustc` to override the crate type. - [Significantly improved the performance fetching git dependencies from GitHub when using a hash in the `rev` field.] (rust-lang/cargo#10079) Misc ---- - [The `rust-analyzer` rustup component is now available on the stable channel.] (rust-lang/rust#98640) Compatibility Notes ------------------- - The minimum required versions for all `-linux-gnu` targets are now at least kernel 3.2 and glibc 2.17, for targets that previously supported older versions: [Increase the minimum linux-gnu versions](rust-lang/rust#95026) - [Network primitives are now implemented with the ideal Rust layout, not the C system layout] (rust-lang/rust#78802). This can cause problems when transmuting the types. - [Add assertion that `transmute_copy`'s `U` is not larger than `T`] (rust-lang/rust#98839) - [A soundness bug in `BTreeMap` was fixed] (rust-lang/rust#99413) that allowed data it was borrowing to be dropped before the container. - [The Drop behavior of C-like enums cast to ints has changed] (rust-lang/rust#96862). These are already discouraged by a compiler warning. - [Relate late-bound closure lifetimes to parent fn in NLL] (rust-lang/rust#98835) - [Errors at const-eval time are now in future incompatibility reports] (rust-lang/rust#97743) - On the `thumbv6m-none-eabi` target, some incorrect `asm!` statements were erroneously accepted if they used the high registers (r8 to r14) as an input/output operand. [This is no longer accepted] (rust-lang/rust#99155). - [`impl Trait` was accidentally accepted as the associated type value of return-position `impl Trait`] (rust-lang/rust#97346), without fulfilling all the trait bounds of that associated type, as long as the hidden type satisfies said bounds. This has been fixed. Internal Changes ---------------- These changes do not affect any public interfaces of Rust, but they represent significant improvements to the performance or internals of rustc and related tools. - Windows builds now use profile-guided optimization, providing 10-20% improvements to compiler performance: [Utilize PGO for windows x64 rustc dist builds] (rust-lang/rust#96978) - [Stop keeping metadata in memory before writing it to disk] (rust-lang/rust#96544) - [compiletest: strip debuginfo by default for mode=ui] (rust-lang/rust#98140) - Many improvements to generated code for derives, including performance improvements: - [Don't use match-destructuring for derived ops on structs.] (rust-lang/rust#98446) - [Many small deriving cleanups] (rust-lang/rust#98741) - [More derive output improvements] (rust-lang/rust#98758) - [Clarify deriving code](rust-lang/rust#98915) - [Final derive output improvements] (rust-lang/rust#99046) - [Stop injecting `#[allow(unused_qualifications)]` in generated `derive` implementations](rust-lang/rust#99485) - [Improve `derive(Debug)`](rust-lang/rust#98190) - [Bump to clap 3](rust-lang/rust#98213) - [fully move dropck to mir](rust-lang/rust#98641) - [Optimize `Vec::insert` for the case where `index == len`.] (rust-lang/rust#98755) - [Convert rust-analyzer to an in-tree tool] (rust-lang/rust#99603)

rust-highfive assigned oli-obk Apr 29, 2022

rustbot added the T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. label Apr 29, 2022

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 29, 2022

This comment has been minimized.

Sign in to view

m-ysk marked this pull request as ready for review April 29, 2022 08:52

cjgillot self-assigned this Apr 29, 2022

bjorn3 reviewed Apr 29, 2022

View reviewed changes

compiler/rustc_codegen_ssa/src/back/link.rs Outdated Show resolved Hide resolved

bjorn3 reviewed Apr 29, 2022

View reviewed changes

compiler/rustc_codegen_ssa/src/back/link.rs Outdated Show resolved Hide resolved

cjgillot reviewed Apr 29, 2022

View reviewed changes

rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 29, 2022

bjorn3 reviewed Apr 29, 2022

View reviewed changes

compiler/rustc_metadata/src/fs.rs Outdated Show resolved Hide resolved

rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Apr 29, 2022

This comment has been minimized.

Sign in to view

m-ysk commented May 2, 2022

View reviewed changes

oli-obk removed their assignment May 2, 2022

m-ysk requested review from cjgillot and bjorn3 May 2, 2022 08:45

This comment has been minimized.

Sign in to view

m-ysk force-pushed the feature/issue-96358 branch from a3240f9 to f9057ae Compare May 5, 2022 15:54

m-ysk added 5 commits July 2, 2022 22:54

add some comments for encode_and_write_metadata

ad55481

add a comment about counting zero bytes

3e30935

avoid holding the temp_dir for empty metadata file

ea0e0f4

seek instead of opening a new file handle

a206121

fall back on the blank path if out_filename is blank

7a5e773

m-ysk force-pushed the feature/issue-96358 branch from c7cdf6e to 0507e02 Compare July 2, 2022 14:35

cjgillot reviewed Jul 3, 2022

View reviewed changes

ensure rustc does not panic by the test for issue-26092

1147d50

m-ysk force-pushed the feature/issue-96358 branch from 0507e02 to 1147d50 Compare July 3, 2022 14:05

m-ysk requested a review from cjgillot July 3, 2022 14:10

cjgillot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Jul 11, 2022

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jul 14, 2022

bors added the merged-by-bors This PR was explicitly merged by bors. label Jul 15, 2022

bors merged commit 1ba1fec into rust-lang:master Jul 15, 2022

rustbot added this to the 1.64.0 milestone Jul 15, 2022

rustbot removed the perf-regression Performance regression. label Jul 15, 2022

klensy mentioned this pull request Aug 25, 2022

-Zmeta-stats ICE: with thread 'rustc' panicked at 'called Result::unwrap() on an Err value: Os { code: 5, kind: PermissionDenied, message: "Access is denied." } #101001

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stop keeping metadata in memory before writing it to disk #96544

Stop keeping metadata in memory before writing it to disk #96544

m-ysk commented Apr 29, 2022 •

edited

Loading

rust-highfive commented Apr 29, 2022

This comment has been minimized.

This comment has been minimized.

cjgillot left a comment

cjgillot Apr 29, 2022

bjorn3 Apr 29, 2022

cjgillot Apr 29, 2022

bjorn3 Apr 29, 2022

m-ysk May 2, 2022

cjgillot Jun 5, 2022

cjgillot commented Apr 29, 2022

rust-timer commented Apr 29, 2022

bors commented Apr 29, 2022

bors commented Apr 29, 2022

rust-timer commented Apr 29, 2022

rust-timer commented Apr 29, 2022

cjgillot commented Apr 29, 2022

This comment has been minimized.

This comment has been minimized.

m-ysk May 2, 2022 •

edited

Loading

cjgillot May 7, 2022

m-ysk May 8, 2022

This comment has been minimized.

cjgillot Jul 3, 2022

m-ysk Jul 3, 2022 •

edited

Loading

m-ysk commented Jul 3, 2022

cjgillot commented Jul 14, 2022

bors commented Jul 14, 2022

bors commented Jul 14, 2022

bors commented Jul 15, 2022

rust-timer commented Jul 15, 2022

rylev commented Jul 19, 2022

lqd commented Jul 19, 2022

rylev commented Jul 20, 2022

Stop keeping metadata in memory before writing it to disk #96544

Stop keeping metadata in memory before writing it to disk #96544

Conversation

m-ysk commented Apr 29, 2022 • edited Loading

rust-highfive commented Apr 29, 2022

This comment has been minimized.

This comment has been minimized.

cjgillot left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjgillot commented Apr 29, 2022

rust-timer commented Apr 29, 2022

bors commented Apr 29, 2022

bors commented Apr 29, 2022

rust-timer commented Apr 29, 2022

rust-timer commented Apr 29, 2022

Footnotes

cjgillot commented Apr 29, 2022

This comment has been minimized.

This comment has been minimized.

m-ysk May 2, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

Choose a reason for hiding this comment

m-ysk Jul 3, 2022 • edited Loading

Choose a reason for hiding this comment

m-ysk commented Jul 3, 2022

cjgillot commented Jul 14, 2022

bors commented Jul 14, 2022

bors commented Jul 14, 2022

bors commented Jul 15, 2022

rust-timer commented Jul 15, 2022

Footnotes

rylev commented Jul 19, 2022

lqd commented Jul 19, 2022

rylev commented Jul 20, 2022

m-ysk commented Apr 29, 2022 •

edited

Loading

m-ysk May 2, 2022 •

edited

Loading

m-ysk Jul 3, 2022 •

edited

Loading