Skip to content
This repository has been archived by the owner on Sep 17, 2024. It is now read-only.

fix: enhance GCP queries for PRs #504

Merged
merged 5 commits into from
Dec 2, 2020

Conversation

mdelapenya
Copy link
Contributor

@mdelapenya mdelapenya commented Dec 1, 2020

What does this PR do?

It enriches the logs when querying the GCP storage bucket, also reducing the amount of entries to look up adding a filter in the form of a prefix=PREFIX query string parameter to limit results to objects that have the specific prefix (See https://cloud.google.com/storage/docs/listing-objects#rest-list-objects).

As a consequence, we noticed that the filename used for the TAR.GZ files for snapshots was incorrect, as it needs the OS after version and before architecture.

Why is it important?

Beats packaging jobs were using wrong artifacts, because of several reasons:

  1. Beats packaging job was setting the wrong parameter when triggering this job. It caused that the snapshot in the GCP bucket was never used, using the released artifacts from artifactory.
  2. Once fixed, the file name for TAR.GZ artifacts was not built properly.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have run the Unit tests for the CLI, and they are passing locally
  • I have run the End-2-End tests for the suite I'm working on, and they are passing locally
  • I have noticed new Go dependencies (run make notice in the proper directory)

Author's Checklist

How to test this PR locally

SUITE="fleet" \
    TAGS="fleet_mode && enroll" DEVELOPER_MODE=true \
    TIMEOUT_FACTOR=3 LOG_LEVEL=TRACE \
    ELASTIC_AGENT_VERSION="pr-22790" ELASTIC_AGENT_USE_CI_SNAPSHOTS=true \
    make -C e2e functional-test

Related issues

Use cases

  • Beats PRs triggering the packaging job will trigger the E2E tests. They need to run the tests using the snapshots produced by the CI
  • Local execution of the tests reproducing CI steps, which is useful for manual testing already pushed artifacts/snapshots.

@mdelapenya mdelapenya self-assigned this Dec 1, 2020
@mdelapenya mdelapenya requested review from a team and EricDavisX December 1, 2020 21:23
@mdelapenya mdelapenya added automation bug Something isn't working labels Dec 1, 2020
@elasticmachine
Copy link
Contributor

elasticmachine commented Dec 1, 2020

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Started by user Manuel de la Peña

  • Start Time: 2020-12-02T06:15:14.585+0000

  • Duration: 29 min 44 sec

Test stats 🧪

Test Results
Failed 0
Passed 45
Skipped 0
Total 45

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 45
Skipped 0
Total 45

@mdelapenya mdelapenya marked this pull request as ready for review December 1, 2020 22:29
@mdelapenya
Copy link
Contributor Author

I verified that running this PR against an existing snapshot (elastic/beats#22790) uses it during tests: https://beats-ci.elastic.co/job/e2e-tests/job/e2e-testing-mbp/job/PR-504/3/parameters/

@mdelapenya mdelapenya merged commit 1bb0368 into elastic:master Dec 2, 2020
@mdelapenya mdelapenya deleted the 503-fix-snapshots branch December 2, 2020 15:41
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
automation bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CI Snapshots is not finding pull request objects because of the huge amount of objects
3 participants