Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Excessive integration test sizes causing builds to fail #1550

Closed
rtyler opened this issue Jul 20, 2023 · 1 comment · Fixed by #1552
Closed

Excessive integration test sizes causing builds to fail #1550

rtyler opened this issue Jul 20, 2023 · 1 comment · Fixed by #1552
Assignees
@rtyler
Labels
binding/rust Issues for the Rust crate bug Something isn't working

Comments

@rtyler
Copy link
Member

rtyler commented Jul 20, 2023

Environment

Delta-rs version: v0.13.1

Binding: rust

Bug

What happened:

When running cargo test -p deltalake --no-default-features --features integration_test,s3-native-tls,datafusion an excessive amount of disk space is consumed When running with RUSTFLAGS="-C debuginfo=1" CARGO_INCREMENTAL=0 as CI does, the local target/ directory generates about 3.3GB of junk for me, with each binary linked for the tests/ coming in between 400-500MB.

What you expected to happen:

cargo not to shred my disks that much.

How to reproduce it:

Run zee tests

More details:

😬
@rtyler rtyler added bug Something isn't working binding/rust Issues for the Rust crate labels Jul 20, 2023
@rtyler rtyler self-assigned this Jul 20, 2023
@rtyler
Copy link
Member Author

rtyler commented Jul 21, 2023

Some interesting details I encountered while testing, observing a build:

  • Stock command_optimize binary with HEAD on main: 511M
  • with strip = "debuginfo" in `Cargo.toml: 116M
  • with strip = "debuginfo" and opt-level = "z"inCargo.toml`: 43M

The build is quite slow with opt-level set to z, but the strip configuration is reasonably fast to me. Still doing some experimentation to understand how these binary sizes are becoming so large before I spend time consolidating some things which behave more like unit tests out of these files

rtyler added a commit to rtyler/delta-rs that referenced this issue Jul 21, 2023
@rtyler
Introduce debuginfo stripping for the test profile
b1f8477 
The integration tests and examples binary sizes are absolutely massive without
this change, regularly they _each_ weigh over 500MB which results in immense
disk I/O usage during the build but also on disk. The excessive size is
exhausting space in our GitHub Action runners as well.

Part of addressing delta-io#1550

Sponsored-by: Databricks Inc
rtyler added a commit to rtyler/delta-rs that referenced this issue Jul 21, 2023
@rtyler
Remove unused crates in the tests/ and examples/ code
d76fc7f 
Unfortunately `cargo +nightly udeps --tests --no-default-features --features
integration_test,datafusion,s3-native-tls` did not produce more unused
dependencies which means there is fewer cruft to chop out of these dependency
trees at the moment.

Related to delta-io#1550

Sponsored-by: Databricks In
rtyler added a commit to rtyler/delta-rs that referenced this issue Jul 21, 2023
@rtyler
Remove unused crates in the tests/ and examples/ code
9923f26 
Unfortunately `cargo +nightly udeps --tests --no-default-features --features
integration_test,datafusion,s3-native-tls` did not produce more unused
dependencies which means there is fewer cruft to chop out of these dependency
trees at the moment.

Related to delta-io#1550

Sponsored-by: Databricks In
@rtyler rtyler mentioned this issue Jul 21, 2023
Merged
rtyler added a commit that referenced this issue Jul 22, 2023
@rtyler
Remove unused crates in the tests/ and examples/ code
a7257f7 
Unfortunately `cargo +nightly udeps --tests --no-default-features --features
integration_test,datafusion,s3-native-tls` did not produce more unused
dependencies which means there is fewer cruft to chop out of these dependency
trees at the moment.

Related to #1550

Sponsored-by: Databricks In
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rtyler rtyler

Labels
binding/rust Issues for the Rust crate bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

chore: address some integration test bloat of disk usage for development rtyler/delta-rs
1 participant
@rtyler