Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: stage execution cache #24780

Merged
merged 11 commits into from
Apr 5, 2021
Merged

feat: stage execution cache #24780

merged 11 commits into from
Apr 5, 2021

Conversation

kuisathaverat
Copy link
Contributor

What does this PR do?

It implements a persistent status between stages executions for the same commit SHA.

Why is it important?

This tries to resolve the use of resources and time when we run the same build for the same commit. To do that after the execution of each stage its status is saved, the next time the stage is executed for the same commit will decide if execute the stage based on its last result, so we will execute only once the stage for the same commit SHA. This will speed up the process to retry CI build in case of environmental issues, test failures, or any other flakiness.

Checklist

  • [ ] My code follows the style guidelines of this project
  • [ ] I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • [ ] I have added tests that prove my fix is effective or that my feature works
  • [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@kuisathaverat kuisathaverat added enhancement automation Team:Automation Label for the Observability productivity team labels Mar 26, 2021
@kuisathaverat kuisathaverat requested review from a team March 26, 2021 11:06
@kuisathaverat kuisathaverat self-assigned this Mar 26, 2021
@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Mar 26, 2021
@kuisathaverat
Copy link
Contributor Author

In this first implementation, we will see a step failing on all stages with a warning after, we have to find a way to avoid this error.

Screenshot 2021-03-26 at 12 14 38

@elasticmachine
Copy link
Collaborator

elasticmachine commented Mar 26, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: kuisathaverat commented: /test

  • Start Time: 2021-03-31T15:38:25.064+0000

  • Duration: 47 min 40 sec

  • Commit: 06bde94

Test stats 🧪

Test Results
Failed 0
Passed 3698
Skipped 309
Total 4007

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 3698
Skipped 309
Total 4007

@v1v
Copy link
Member

v1v commented Mar 26, 2021

In this first implementation, we will see a step failing on all stages with a warning after, we have to find a way to avoid this error.

maybe if BUILD_ID == 1

v1v
v1v approved these changes Mar 26, 2021
View reviewed changes
Copy link
Member

@v1v v1v left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯

@kuisathaverat
Copy link
Contributor Author

/test

@kuisathaverat
Copy link
Contributor Author

/test

@kuisathaverat
Copy link
Contributor Author

/test

1 similar comment
@kuisathaverat
Copy link
Contributor Author

/test

@kuisathaverat
Copy link
Contributor Author

/test

@kuisathaverat
Copy link
Contributor Author

/test

1 similar comment
@kuisathaverat
Copy link
Contributor Author

/test

@kuisathaverat
Copy link
Contributor Author

/test

@kuisathaverat
Copy link
Contributor Author

/test

@kuisathaverat
Copy link
Contributor Author

waiting for elastic/apm-pipeline-library#1061

@kuisathaverat
Copy link
Contributor Author

/test

1 similar comment
@kuisathaverat
Copy link
Contributor Author

/test

@kuisathaverat kuisathaverat merged commit 623e4ab into elastic:master Apr 5, 2021
@kuisathaverat kuisathaverat mentioned this pull request Apr 5, 2021
Merged
kuisathaverat added a commit to kuisathaverat/beats that referenced this pull request Apr 5, 2021
@kuisathaverat
feat: stage execution cache (elastic#24780)
aa0b97c 
* feat: stage execution cache

* fix: use correct context

* fix: do not check stage status on the first run

* fix: proper URL

* chore: show message when the stache is skip

* fix: correct path

* fix: add final /

* test: is the path needed?

* fix: remove prefix

* chore: refactor to use curl to download

* chore: use pipeline step
kuisathaverat added a commit to kuisathaverat/beats that referenced this pull request Apr 5, 2021
@kuisathaverat
feat: stage execution cache (elastic#24780)
0eba7cd 
* feat: stage execution cache

* fix: use correct context

* fix: do not check stage status on the first run

* fix: proper URL

* chore: show message when the stache is skip

* fix: correct path

* fix: add final /

* test: is the path needed?

* fix: remove prefix

* chore: refactor to use curl to download

* chore: use pipeline step
@kuisathaverat kuisathaverat mentioned this pull request Apr 5, 2021
Merged
kuisathaverat added a commit to kuisathaverat/beats that referenced this pull request Apr 5, 2021
@kuisathaverat
feat: stage execution cache (elastic#24780)
76118f9 
* feat: stage execution cache

* fix: use correct context

* fix: do not check stage status on the first run

* fix: proper URL

* chore: show message when the stache is skip

* fix: correct path

* fix: add final /

* test: is the path needed?

* fix: remove prefix

* chore: refactor to use curl to download

* chore: use pipeline step
# Conflicts:
#	Jenkinsfile
@kuisathaverat kuisathaverat mentioned this pull request Apr 5, 2021
Merged
kuisathaverat added a commit that referenced this pull request Apr 6, 2021
@kuisathaverat
feat: stage execution cache (#24780) (#24933)
8d2d4b2 
* feat: stage execution cache

* fix: use correct context

* fix: do not check stage status on the first run

* fix: proper URL

* chore: show message when the stache is skip

* fix: correct path

* fix: add final /

* test: is the path needed?

* fix: remove prefix

* chore: refactor to use curl to download

* chore: use pipeline step
# Conflicts:
#	Jenkinsfile
kuisathaverat added a commit that referenced this pull request Apr 6, 2021
@kuisathaverat
feat: stage execution cache (#24780) (#24931)
c3a00b2 
* feat: stage execution cache

* fix: use correct context

* fix: do not check stage status on the first run

* fix: proper URL

* chore: show message when the stache is skip

* fix: correct path

* fix: add final /

* test: is the path needed?

* fix: remove prefix

* chore: refactor to use curl to download

* chore: use pipeline step
kuisathaverat added a commit that referenced this pull request Apr 6, 2021
@kuisathaverat
feat: stage execution cache (#24780) (#24930)
dd616c6 
* feat: stage execution cache

* fix: use correct context

* fix: do not check stage status on the first run

* fix: proper URL

* chore: show message when the stache is skip

* fix: correct path

* fix: add final /

* test: is the path needed?

* fix: remove prefix

* chore: refactor to use curl to download

* chore: use pipeline step
michalpristas pushed a commit to michalpristas/beats that referenced this pull request Apr 7, 2021
@kuisathaverat @michalpristas
feat: stage execution cache (elastic#24780)
31156c8 
* feat: stage execution cache

* fix: use correct context

* fix: do not check stage status on the first run

* fix: proper URL

* chore: show message when the stache is skip

* fix: correct path

* fix: add final /

* test: is the path needed?

* fix: remove prefix

* chore: refactor to use curl to download

* chore: use pipeline step
v1v added a commit to v1v/beats that referenced this pull request Apr 7, 2021
@v1v
Merge remote-tracking branch 'upstream/master' into feature/mergify
c1238b8 
* upstream/master: (91 commits)
  [Filebeat] Change okta.target to nested field (elastic#24636)
  Add RFC5424 format support for syslog input  (elastic#23954)
  Fix links to Beats product pages (elastic#24821)
  [DOCS] Fix 'make setup' instructions for a new beat (elastic#24944)
  Remove duplicate decode_xml entry (elastic#24941)
  [libbeat] Add wineventlog schema to decode_xml processor (elastic#24726)
  [Elastic Agent] Add check for URL set when cert and cert key. (elastic#24904)
  feat: stage execution cache (elastic#24780)
  Fix error in Journalbeat commands (elastic#24880)
  Add baseline ECS 1.9.0 upgrade (elastic#24909)
  [Elastic Agent] Cloud container legacy apm files. (elastic#24896)
  [Elastic Agent]: Reduce allowed socket path length (elastic#24914)
  Add ability to destroy indices with wildcards in testing (elastic#24915)
  Add status subcommand to report status of running daemon. (elastic#24856)
  Fix types of fields GetHits and Ops in Metricbeat module for Couchbase (elastic#23287)
  Add support for Filestream input in elastic agent. (elastic#24820)
  Implement k8s secrets provider for Agent (elastic#24789)
  Sort processor list in docs (elastic#24874)
  Add support for SCRAM authentication in kafka metricbeat module (elastic#24810)
  Properly update offset in case of unparasable line (elastic#22685)
  ...
@v1v v1v mentioned this pull request Jul 1, 2021
Merged
@zube zube bot removed the [zube]: Done label Jul 5, 2021
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
@kuisathaverat
feat: stage execution cache (elastic#24780) (elastic#24933)
2520323 
* feat: stage execution cache

* fix: use correct context

* fix: do not check stage status on the first run

* fix: proper URL

* chore: show message when the stache is skip

* fix: correct path

* fix: add final /

* test: is the path needed?

* fix: remove prefix

* chore: refactor to use curl to download

* chore: use pipeline step
# Conflicts:
#	Jenkinsfile
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
@kuisathaverat
feat: stage execution cache (elastic#24780) (elastic#24931)
87b18fb 
* feat: stage execution cache

* fix: use correct context

* fix: do not check stage status on the first run

* fix: proper URL

* chore: show message when the stache is skip

* fix: correct path

* fix: add final /

* test: is the path needed?

* fix: remove prefix

* chore: refactor to use curl to download

* chore: use pipeline step
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@urso urso urso approved these changes

@cachedout cachedout cachedout approved these changes

@v1v v1v v1v approved these changes

Assignees

@kuisathaverat kuisathaverat

Labels
automation enhancement Team:Automation Label for the Observability productivity team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@kuisathaverat @elasticmachine @v1v @urso @cachedout