Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix excessive initialization and reads beyond EOF in io::copy(_, Vec<u8>) specialization #117576

Merged
merged 2 commits into from
Nov 6, 2023
Merged

Fix excessive initialization and reads beyond EOF in io::copy(_, Vec<u8>) specialization #117576

merged 2 commits into from
Nov 6, 2023

Conversation

the8472
Copy link
Member

@the8472 the8472 commented Nov 4, 2023

@rustbot
Copy link
Collaborator

rustbot commented Nov 4, 2023

r? @Mark-Simulacrum

(rustbot has picked a reviewer for you, use r? to override)

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-libs Relevant to the library team, which will review and decide on the PR/issue. labels Nov 4, 2023
@the8472 the8472 mentioned this pull request Nov 4, 2023
Merged
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@the8472
avoid excessive initialization when copying to a Vec
8d8f06b 
It now keeps track of initialized bytes to avoid reinitialization.
It also keeps track of read sizes to avoid initializing more bytes
than the reader needs. This is important when passing a huge vector to a
Read that only has a few bytes to offer and doesn't implement read_buf().
@the8472
detect EOF earlier
78aa5e5 
The initial probe-for-empty-source by stack_buffer_copy only detected EOF
if the source was empty but not when it was merely small which lead to
additional calls to read() after Ok(0) had already been returned
in the stack copy routine
@Mark-Simulacrum
Copy link
Member

@bors r+

I wonder if the detection here should be publicly available on BorrowedBuf or similar - seems like it might be useful elsewhere too. But shouldn't block this.

@bors
Copy link
Contributor

bors commented Nov 4, 2023

📌 Commit 78aa5e5 has been approved by Mark-Simulacrum

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 4, 2023
@bors
Copy link
Contributor

bors commented Nov 6, 2023

⌛ Testing commit 78aa5e5 with merge 7a892ab...

@bors
Copy link
Contributor

bors commented Nov 6, 2023

☀️ Test successful - checks-actions
Approved by: Mark-Simulacrum
Pushing 7a892ab to master...

@bors bors added the merged-by-bors This PR was explicitly merged by bors. label Nov 6, 2023
@bors bors merged commit 7a892ab into rust-lang:master Nov 6, 2023
12 checks passed
@rustbot rustbot added this to the 1.75.0 milestone Nov 6, 2023
@github-actions github-actions bot mentioned this pull request Nov 6, 2023
Merged
bors-ferrocene bot added a commit to ferrocene/ferrocene that referenced this pull request Nov 6, 2023
@bors-ferrocene @BenWiederhake
@SabrinaJewson @J-ZhengLi @koka831 @bjorn3 @joshlf @lengyijun @Zalathar @oli-obk @flip1995 @y21 @bors @bvanjoi
Merge #84
a01e81f 
84: Automated pull from upstream `master` r=Dajamante a=github-actions[bot]


This PR pulls the following changes from the upstream repository:

* rust-lang/rust#117585
* rust-lang/rust#117576
* rust-lang/rust#96979
* rust-lang/rust#117191
* rust-lang/rust#117179
* rust-lang/rust#117574
* rust-lang/rust#117537
* rust-lang/rust#117608
  * rust-lang/rust#117596
  * rust-lang/rust#117588
  * rust-lang/rust#117524
  * rust-lang/rust#116017
* rust-lang/rust#117504
* rust-lang/rust#117469
* rust-lang/rust#116218
* rust-lang/rust#117589
* rust-lang/rust#117581
* rust-lang/rust#117503
* rust-lang/rust#117590
  * rust-lang/rust#117583
  * rust-lang/rust#117570
  * rust-lang/rust#117562
  * rust-lang/rust#117534
  * rust-lang/rust#116894
  * rust-lang/rust#110340
* rust-lang/rust#113343
* rust-lang/rust#117579
* rust-lang/rust#117094
* rust-lang/rust#117566
* rust-lang/rust#117564
  * rust-lang/rust#117554
  * rust-lang/rust#117550
  * rust-lang/rust#117343
* rust-lang/rust#115274
* rust-lang/rust#117540
* rust-lang/rust#116412
* rust-lang/rust#115333
* rust-lang/rust#117507
* rust-lang/rust#117538
  * rust-lang/rust#117533
  * rust-lang/rust#117523
  * rust-lang/rust#117520
  * rust-lang/rust#117505
  * rust-lang/rust#117434
* rust-lang/rust#117535
* rust-lang/rust#117510
* rust-lang/rust#116439
* rust-lang/rust#117508



Co-authored-by: Ben Wiederhake <BenWiederhake.GitHub@gmx.de>
Co-authored-by: SabrinaJewson <sejewson@gmail.com>
Co-authored-by: J-ZhengLi <lizheng135@huawei.com>
Co-authored-by: koka <koka.code@gmail.com>
Co-authored-by: bjorn3 <17426603+bjorn3@users.noreply.github.com>
Co-authored-by: Joshua Liebow-Feeser <joshlf@users.noreply.github.com>
Co-authored-by: lengyijun <sjtu5140809011@gmail.com>
Co-authored-by: Zalathar <Zalathar@users.noreply.github.com>
Co-authored-by: Oli Scherer <git-spam-no-reply9815368754983@oli-obk.de>
Co-authored-by: Philipp Krones <hello@philkrones.com>
Co-authored-by: y21 <30553356+y21@users.noreply.github.com>
Co-authored-by: bors <bors@rust-lang.org>
Co-authored-by: bohan <bohan-zhang@foxmail.com>
@rust-timer
Copy link
Collaborator

Finished benchmarking commit (7a892ab): comparison URL.

Overall result: ✅ improvements - no action needed

@rustbot label: -perf-regression

Instruction count

This is a highly reliable metric that was used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)		 - - 0
Regressions ❌
(secondary)		 - - 0
Improvements ✅
(primary)		 -1.0% [-1.0%, -1.0%] 1
Improvements ✅
(secondary)		 - - 0
All ❌✅ (primary) -1.0% [-1.0%, -1.0%] 1

Max RSS (memory usage)

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)		 1.0% [0.7%, 1.4%] 2
Regressions ❌
(secondary)		 2.5% [2.5%, 2.5%] 1
Improvements ✅
(primary)		 -0.6% [-0.8%, -0.4%] 3
Improvements ✅
(secondary)		 - - 0
All ❌✅ (primary) 0.1% [-0.8%, 1.4%] 5

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

mean range count
Regressions ❌
(primary)		 0.6% [0.5%, 0.7%] 3
Regressions ❌
(secondary)		 0.4% [0.4%, 0.4%] 1
Improvements ✅
(primary)		 -0.5% [-0.5%, -0.5%] 1
Improvements ✅
(secondary)		 -2.6% [-2.6%, -2.6%] 1
All ❌✅ (primary) 0.3% [-0.5%, 0.7%] 4

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 636.185s -> 637.484s (0.20%)
Artifact size: 304.50 MiB -> 304.49 MiB (-0.00%)

@bonsairobo bonsairobo mentioned this pull request Nov 8, 2023
Closed
@a1phyr
Copy link
Contributor

a1phyr commented Nov 9, 2023

Side question: is there a reason why this is not implemented as reader.read_to_end(self) ?

@the8472
Copy link
Member Author

the8472 commented Nov 9, 2023

Good point, they're quite similar though they contain some different optimizations. It probably make sense to unify them.

@the8472 the8472 added the beta-nominated Nominated for backporting to the compiler in the beta channel. label Nov 9, 2023
@the8472
Copy link
Member Author

the8472 commented Nov 9, 2023

beta-nominated on behalf of this comment

@Mark-Simulacrum Mark-Simulacrum mentioned this pull request Nov 13, 2023
Merged
bors added a commit to rust-lang-ci/rust that referenced this pull request Nov 13, 2023
@bors
Auto merge of rust-lang#117843 - Mark-Simulacrum:stable-next, r=Mark-…
79e9716 
…Simulacrum

[stable] Prepare 1.74.0 release

https://forge.rust-lang.org/release/process.html#promote-branches-t-3-days-monday

Also backports:

* Disabling specialization as an alternative backport of "Fix excessive initialization and reads beyond EOF in io::copy(_, Vec<u8>) specialization rust-lang#117576"
*  coverage: Avoid creating malformed macro name spans rust-lang#117827

r? `@Mark-Simulacrum`
@Amanieu Amanieu added the beta-accepted Accepted for backporting to the compiler in the beta channel. label Nov 15, 2023
@cuviper
Copy link
Member

cuviper commented Nov 17, 2023

This was backported during the 1.74.0 stable promotion in #117843, and it already landed in what is now 1.75-beta, so there's nothing more to do here.

@rustbot label -beta-nominated

@rustbot rustbot removed the beta-nominated Nominated for backporting to the compiler in the beta channel. label Nov 17, 2023
@cuviper cuviper modified the milestones: 1.75.0, 1.74.0 Nov 17, 2023
@the8472 the8472 mentioned this pull request Nov 24, 2023
Merged
matthiaskrgr added a commit to matthiaskrgr/rust that referenced this pull request Nov 28, 2023
@matthiaskrgr
Rollup merge of rust-lang#118222 - the8472:copy-use-vec-write, r=m-ou-se
97ef5a3 
unify read_to_end and io::copy impls for reading into a Vec

This ports over the initial probe (to avoid allocation) and the dynamic read sizing from the io::copy specialization to the `default_read_to_end` implementation which already had its own optimizations for different cases.

I think it should be a best-of-both now.

suggested by `@a1phyr` in rust-lang#117576 (comment)
rust-timer added a commit to rust-lang-ci/rust that referenced this pull request Nov 29, 2023
@rust-timer
Unrolled build for rust-lang#118222
98f60b3 
Rollup merge of rust-lang#118222 - the8472:copy-use-vec-write, r=m-ou-se

unify read_to_end and io::copy impls for reading into a Vec

This ports over the initial probe (to avoid allocation) and the dynamic read sizing from the io::copy specialization to the `default_read_to_end` implementation which already had its own optimizations for different cases.

I think it should be a best-of-both now.

suggested by `@a1phyr` in rust-lang#117576 (comment)
bors pushed a commit to rust-lang/miri that referenced this pull request Nov 30, 2023
@matthiaskrgr
Rollup merge of #118222 - the8472:copy-use-vec-write, r=m-ou-se
8ea3070 
unify read_to_end and io::copy impls for reading into a Vec

This ports over the initial probe (to avoid allocation) and the dynamic read sizing from the io::copy specialization to the `default_read_to_end` implementation which already had its own optimizations for different cases.

I think it should be a best-of-both now.

suggested by `@a1phyr` in rust-lang/rust#117576 (comment)
celinval added a commit to celinval/rust-dev that referenced this pull request Jun 4, 2024
@github-actions @celinval @zhassan-aws
Automatic toolchain upgrade to nightly-2023-11-07 (rust-lang#2862)
b291e73 
Update Rust toolchain from nightly-2023-11-06 to nightly-2023-11-07
without any other source changes.
This is an automatically generated pull request. If any of the CI checks
fail, manual intervention is required. In such a case, review the
changes at https://github.com/rust-lang/rust from
rust-lang@fee5518
up to
rust-lang@189d6c7.
The log for this commit range is:
rust-lang@189d6c71f3 Auto merge of
rust-lang#117641 - matthiaskrgr:rollup-f9c12td, r=matthiaskrgr
rust-lang@9efe60b1eb Rollup merge of
rust-lang#117637 - lqd:trivial-bounds-with-binder-vars, r=compiler-errors
rust-lang@2a1f8bccee Rollup merge of
rust-lang#117631 - smarnach:error-request-doc-fix, r=ChrisDenton
rust-lang@8a34fea835 Rollup merge of
rust-lang#117516 - matthiaskrgr:test_effects_113375_oob, r=fee1-dead
rust-lang@1683df7d3b Rollup merge of
rust-lang#117190 - matthiaskrgr:test_effects_113381, r=fee1-dead
rust-lang@fb61292105 Auto merge of
rust-lang#117292 - estebank:issue-80446, r=davidtwco
rust-lang@4b7aacaa4f Silence redundant
error on typo resulting on binop
rust-lang@aea82b268a Auto merge of
rust-lang#117603 - HKalbasi:make-feature-additive, r=Nilstrieb
rust-lang@58351ae03f add test for trivial
bound not holding in `soa-derive`
rust-lang@2beca157c9 check binders with
bound vars for global bounds that don't hold
rust-lang@b049093560 Auto merge of
rust-lang#116988 - RalfJung:null, r=WaffleLapkin
rust-lang@e1fcecb1b9 Auto merge of
rust-lang#117630 - matthiaskrgr:rollup-v0d5p3f, r=matthiaskrgr
rust-lang@3a096e96fa Documentation
cleanup for core::error::Request.
rust-lang@b88d62e222 Rollup merge of
rust-lang#117615 - bjorn3:misc_changes, r=davidtwco
rust-lang@ec35c03143 Rollup merge of
rust-lang#117613 - jackh726:vacation, r=compiler-errors
rust-lang@60dca87cff Rollup merge of
rust-lang#117592 - thomcc:env-span-wrong, r=davidtwco
rust-lang@6bf2fb3679 Auto merge of
rust-lang#117578 - compiler-errors:derive-encode-in-rustc_type_ir, r=davidtwco
rust-lang@f9b644636f Auto merge of
rust-lang#117435 - SparrowLii:nightly_parallel, r=oli-obk,davidtwco
rust-lang@fcca978aa0 Auto merge of
rust-lang#117607 - klensy:clang-17.0.4, r=Mark-Simulacrum
rust-lang@152a4e90d1 Auto merge of
rust-lang#117585 - dnbln:feat/move-kw-span, r=cjgillot
rust-lang@f2a40e99ff use portable
AtomicU64 for powerPC and MIPS
rust-lang@7a892ab8d8 Auto merge of
rust-lang#117576 - the8472:fix-io-copy-vec, r=Mark-Simulacrum
rust-lang@c8a25eddfe Make the randomize
feature of rustc_abi additive
rust-lang@ec29a02071 Update doc comment
for CodegenBackend::link
rust-lang@ba82056a14 Use the actual
computed crate name for -Zprint-vtable-sizes
rust-lang@1a1b10fa63 Don't steal the
parse query when using --pretty
rust-lang@6e33e89337 Remove from vacation
and compiler review group
rust-lang@38ff91c453 bump clang version
for dist-x86_64-linux from 17.0.2 to 17.0.4
rust-lang@86fca873ba Use the correct span
when emitting the `env!` result
rust-lang@c077147200 fix clippy author
and failing test
rust-lang@876f698790 Add the
vis.visit_capture_by() in noop_visit_expr
rust-lang@54ce0346c0 add `fn
visit_capture_by` to MutVisitor and fix pprust-expr-roundtrip.rs
rust-lang@df85b28b72 fixes for rustfmt +
ast visitor
rust-lang@a6b41aa6ba fmt
rust-lang@241a654c07 Fix remaining uses
of `CaptureBy::Value`
rust-lang@8de489918b feat(hir): Store the
`Span` of the `move` keyword
rust-lang@bee7b5889e Derive
TyEncodable/TyDecodable implementations that are parameterized over
interner
rust-lang@78aa5e511c detect EOF earlier
rust-lang@8d8f06b277 avoid excessive
initialization when copying to a Vec
rust-lang@666cad9a5f add test for rust-lang#113375
rust-lang@384f7382ba add test for rust-lang#113381
rust-lang@248dd14fa5 update
config.example.toml
rust-lang@bf5fb7614b update bootstrap
change history
rust-lang@ab8101d019 enable parallel
rustc in nightly builds
rust-lang@231f93524d Detect misparsed
binop caused by missing semi
rust-lang@98d54da1ee document that the
null pointer has the 0 address

---------

Co-authored-by: celinval <celinval@users.noreply.github.com>
Co-authored-by: Zyad Hassan <zyadh@amazon.com>
Co-authored-by: Zyad Hassan <88045115+zhassan-aws@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@Mark-Simulacrum Mark-Simulacrum

Labels
beta-accepted Accepted for backporting to the compiler in the beta channel. merged-by-bors This PR was explicitly merged by bors. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-libs Relevant to the library team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
1.74.0
Development

Successfully merging this pull request may close these issues.

Exponential Performance Regression in Rust 1.73 Read API from BorrowedCursor::ensure_init
9 participants
@the8472 @rustbot @rust-log-analyzer @Mark-Simulacrum @bors @rust-timer @a1phyr @cuviper @Amanieu