Run cargo clean on CI when the version of rustc changes
#1578
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I plan to test this on CI by forcing a rebuild of this PR after a new nightly release. |
Recently I've been monitoring where our CI time is spent. Currently, the build and test stages take about 14 minutes without a cache and 1.7 to 3.6 minutes when a cache is present. The variation in the latter comes from the `cargo install diesel_cli` step which is partially rebuilt any time a new compatible version of a dependency is published. For each channel, we cache about 1GB of build artifacts. This results in a total of 4 minutes that are spent in caching related activities. However, extracting the cache on Travis appears to be O(n^3) because once a new version of rustc is released the size of the `target/` directory doubles but cache extraction take 8 times as long and total cache related time nearly quadruples to 13.5 minutes. This effectively wipes out any gains of caching build artifacts for our codebase, especially on the beta and nightly channels. This commit adds a shell script that is run before any build steps on CI. The script works by recording the version of rustc in a file under `target/`. Upon the next build, if the cache was created by the same version of rustc then no action is taken. However, if the version of rustc has changed then `cargo clean` is run to clear out the stale cached files. This commit also sets the cache timeout (for building and uploading the cache) to 6 minutes. The default is 3 minutes and this portion is currently taking about 2.6 minutes. The rationale here is to provide some additional headroom on the stable channel. As dependencies are updated, the cache will gradually grow until a new release causes the cache to be cleared. The increased timeout should allow for this natural growth while also enforcing a reasonable limit so that caching overhead doesn’t grow too large.
jtgeibel
force-pushed
the
clear-cache-on-ci-when-rustc-version-changes
branch
from
453b057 to
216f726
Compare
|
I'm seeing the expected results on CI with the new nightly. I've cleared the caches and I'm merging directly to master to kick off that build. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Recently I've been monitoring where our CI time is spent. Currently,
the build and test stages take about 14 minutes without a cache and
1.7 to 3.6 minutes when a cache is present. The variation in the
latter comes from the
cargo install diesel_clistep which ispartially rebuilt any time a new compatible version of a dependency
is published.
For each channel, we cache about 1GB of build artifacts. This results
in a total of 4 minutes that are spent in caching related activities.
However, extracting the cache on Travis appears to be O(n^3) because
once a new version of rustc is released the size of the
target/directory doubles but cache extraction take 8 times as long and total
cache related time nearly quadruples to 13.5 minutes. This effectively
wipes out any gains of caching build artifacts for our codebase,
especially on the beta and nightly channels.
This commit adds a shell script that is run before any build steps on
CI. The script works by recording the version of rustc in a file under
target/. Upon the next build, if the cache was created by the sameversion of rustc then no action is taken. However, if the version of
rustc has changed then
cargo cleanis run to clear out the stalecached files.
This commit also sets the cache timeout (for building and uploading
the cache) to 6 minutes. The default is 3 minutes and this portion is
currently taking about 2.6 minutes. The rationale here is to provide
some additional headroom on the stable channel. As dependencies are
updated, the cache will gradually grow until a new release causes the
cache to be cleared. The increased timeout should allow for this
natural growth while also enforcing a reasonable limit so that caching
overhead doesn’t grow too large.