Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

packer cache support for the 7.x and 7.latestMinor branches #25091

Merged
merged 5 commits into from
Apr 15, 2021

Conversation

v1v
Copy link
Member

@v1v v1v commented Apr 14, 2021

What does this PR do?

Refactor the packer-cache to support:

  • master branch
  • 7.x branch
  • 7.. <--- this might require some tuning in the future, there is a github api limitation with the number of results/

For such, it split the docker images in two different categories:

  • common ones
  • specific per branch (such as elasticsearch for 7.x differs from the one in master and so on)

Why is it important?

Cache as much as possible.

Test

I validated the packer-cache.sh was triggered correctly on a build with the commit f1f5c27

https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats%2FPR-25091/detail/PR-25091/3/pipeline/ is the one with the packer-cache.sh call, and the analysis was done in the comment -> #25091 (comment)

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Apr 14, 2021
Comment on lines +15 to +17
function getBeatsVersion() {
grep 'defaultBeatVersion' libbeat/version/version.go | cut -d= -f2 | sed 's#"##g' | tr -d " "
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a make get-version goal but it requires go and mage, so I didn't want to get crazy to prepare the environment and decided to use some bash commands instead

Jenkinsfile Outdated Show resolved Hide resolved
@elasticmachine
Copy link
Collaborator

elasticmachine commented Apr 14, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #25091 updated

  • Start Time: 2021-04-14T21:07:29.650+0000

  • Duration: 19 min 19 sec

  • Commit: fa37937

Trends 🧪

Image of Build Times

❕ Flaky test report

No test was executed to be analysed.

.ci/packer_cache.sh Outdated Show resolved Hide resolved
.ci/packer_cache.sh Outdated Show resolved Hide resolved
@v1v
Copy link
Member Author

v1v commented Apr 14, 2021

Tests

 curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/pipelines/beats/pipelines/PR-25091/runs/3/nodes/27/log/\?start\=0 -s | grep 'Status: Downloaded newer image for docker.*SNAPSHOT'
[2021-04-14T20:43:05.962Z] Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT
[2021-04-14T20:43:31.359Z] Status: Downloaded newer image for docker.elastic.co/kibana/kibana:8.0.0-SNAPSHOT
[2021-04-14T20:43:55.991Z] Status: Downloaded newer image for docker.elastic.co/logstash/logstash:8.0.0-SNAPSHOT
[2021-04-14T20:44:12.891Z] Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.13.0-SNAPSHOT
[2021-04-14T20:44:34.636Z] Status: Downloaded newer image for docker.elastic.co/kibana/kibana:7.13.0-SNAPSHOT
[2021-04-14T20:45:02.900Z] Status: Downloaded newer image for docker.elastic.co/logstash/logstash:7.13.0-SNAPSHOT
[2021-04-14T20:45:17.539Z] Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.12.1-SNAPSHOT
[2021-04-14T20:45:40.348Z] Status: Downloaded newer image for docker.elastic.co/kibana/kibana:7.12.1-SNAPSHOT
[2021-04-14T20:45:52.815Z] Status: Downloaded newer image for docker.elastic.co/logstash/logstash:7.12.1-SNAPSHOT
curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/pipelines/beats/pipelines/PR-25091/runs/3/nodes/27/log/\?start\=0 -s | grep 'git checkout' 
[2021-04-14T20:37:06.348Z]  > git checkout -f ad7971818d360296e3af14564a1b149c28ad55ab # timeout=15
[2021-04-14T20:37:09.435Z]  > git checkout -f a6f836a1227404b78a4e06f9cec184a9ee09aa12 # timeout=15
[2021-04-14T20:37:51.154Z]  > git checkout -f ad7971818d360296e3af14564a1b149c28ad55ab # timeout=15
[2021-04-14T20:37:53.948Z]  > git checkout -f a76e93af197d11e3f5cbe0df692160498a9fc927 # timeout=15
[2021-04-14T20:43:58.428Z] + git checkout 7.x
[2021-04-14T20:45:05.695Z] + git checkout 7.12

Jenkinsfile Outdated Show resolved Hide resolved
@v1v v1v marked this pull request as ready for review April 15, 2021 08:22
@v1v v1v requested a review from a team as a code owner April 15, 2021 08:22
@v1v v1v self-assigned this Apr 15, 2021
@v1v v1v requested a review from urso April 15, 2021 08:23
@botelastic botelastic bot added Team:Automation Label for the Observability productivity team and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 15, 2021
@v1v v1v merged commit ae02c5f into elastic:master Apr 15, 2021
v1v added a commit to v1v/beats that referenced this pull request Apr 15, 2021
* upstream/master:
  packer cache support for the 7.x and 7.latestMinor branches (elastic#25091)
  Remove EventFetcher and EventsFetcher interface (elastic#25093)
  Update go-structform to 0.0.8 (elastic#25051)
  Update copy_fields.asciidoc (elastic#25053)
  [elastic-agent] ensure container is backwards compatible (elastic#25092)
  Add --fleet-server-service-token. Rename --fleet-server to --fleet-server-es. (elastic#25083)
  Add cgroup.cpuacct percentages (elastic#25057)
  Add tests for truncated and symlinked files in filestream input (elastic#24425)
  Fix panic when Hearbeat monitor initialization fails twice (elastic#25073)
  [Filebeat][httpjson] Change append transform to initiate new fields as a slice (elastic#25074)
  Osquerybeat: Result values type translation (elastic#25012)
  Update Osquerybeat spec to get it downloading from the correct artifactory path (elastic#25076)
  Fix changelog (elastic#25079)
  Strip Azure EventHub connection string in debug logs (elastic#25066)
  Change googlecloud to gcp in field names (elastic#25038)
  Bump stack version to 7.12.0 for testing (elastic#24957)
  packer-cache: cache the existing docker images on ARM and some more (elastic#25068)
  Disable logstash TestFetch flaky test (elastic#25044)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Automation Label for the Observability productivity team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants