[CI] Optimise stash/unstash performance #18473
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
botelastic
bot
added
the
needs_team
v1v
added
automation
and removed
needs_team
botelastic
bot
added
the
needs_team
v1v
added
ci
enhancement
Team:Automation
v1v
commented
May 13, 2020
💔 Tests Failed
Expand to view the summary
Build stats
Test stats 🧪
Test errorsExpand to view the tests failures
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
v1v
force-pushed
the
feature/ci-stash-optimisation
branch
from
f671050
to
842fe7f
Compare
v1v
force-pushed
the
feature/ci-stash-optimisation
branch
2 times, most recently
from
1a01539
to
161fc4f
Compare
v1v
force-pushed
the
feature/ci-stash-optimisation
branch
from
161fc4f
to
df947c2
Compare
2 tasks
v1v
commented
May 14, 2020
v1v
commented
May 14, 2020
v1v
commented
May 14, 2020
| @@ -1027,7 +1029,7 @@ def terraformCleanup(String stashName, String directory) { | |||
| stage("Remove cloud scenarios in ${directory}"){ | |||
| withCloudTestEnv() { | |||
| withBeatsEnv(false) { | |||
| unstash "terraform-${stashName}" | |||
| unstash("terraform-${stashName}") | |||
v1v
commented
May 14, 2020
| @@ -1176,7 +1178,7 @@ def runbld() { | |||
| // Unstash the test reports | |||
| stashedTestReports.each { k, v -> | |||
| dir(k) { | |||
| unstash v | |||
| unstash(v) | |||
v1v
requested review from
andrewkroh,
kuisathaverat,
jsoriano,
urso,
cachedout and
mikemadden42
cachedout
approved these changes
May 14, 2020
kuisathaverat
approved these changes
May 14, 2020
jsoriano
approved these changes
May 14, 2020
v1v
added a commit
to v1v/beats
that referenced
this pull request
May 14, 2020
v1v
added a commit
to v1v/beats
that referenced
this pull request
May 14, 2020
v1v
added a commit
that referenced
this pull request
May 15, 2020
v1v
added a commit
that referenced
this pull request
May 15, 2020
v1v
added a commit
to v1v/beats
that referenced
this pull request
May 15, 2020
…w-oss * upstream/master: (27 commits) Disable host fields for "cloud", panw, cef modules (elastic#18223) [docs] Rename monitoring collection from legacy internal collection to legacy collection (elastic#18504) Introduce auto detection of format (elastic#18095) Add additional fields to address issue elastic#18465 for googlecloud audit log (elastic#18472) Fix libbeat import path in seccomp policy template (elastic#18418) Address Okta input issue elastic#18530 (elastic#18534) [Ingest Manager] Avoid Chown on windows (elastic#18512) Fix Cisco ASA/FTD msgs that use a host name as NAT address (elastic#18376) [CI] Optimise stash/unstash performance (elastic#18473) Libbeat: Remove global loggers from libbeat/metric and libbeat/cloudid (elastic#18500) Fix PANW bad mapping of client/source and server/dest packets and bytes (elastic#18525) Add a file lock to the data directory on startup to prevent multiple agents. (elastic#18483) Followup to 12606 (elastic#18316) changed input from syslog to tcp/udp due to unsupported RFC (elastic#18447) Improve ECS field mappings in Sysmon module. (elastic#18381) [Elastic Agent] Cleaner output of inspect command (elastic#18405) [Elastic Agent] Pick up version from libbeat (elastic#18350) Update communitybeats.asciidoc (elastic#18470) [Metricbeat] Change visualization interval from 15m to >=15m (elastic#18466) docs: Fix typo in kerberos docs (elastic#18503) ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Stash and unstash with the Google storage plugin, this could help to have a more predictable time, since it does run in the workers rather than in the master.
Compress/uncompress is required as long as the Google storage plugin does not support multiple wildcards. Although I'd suggest to use a zip/tarball rather than uploading each of the 40k files....
Why is it important?
Speed up the overall build time when there are build peaks. There was an initial approach to tackle the performance issues in #17939.
Tasks
Performance
Linux
(similar)(9x faster)(~9x faster)(~50x faster)Windows
(~3x faster)(~30x faster)Issues
zipstep does not compress the.gitfolderjarcommand does not preserve permissions.