ci: beats PR testing seems to pull in incorrect docker images during test

[adam-stokes]

We had a recent failure on a beats PR: elastic/beats#26749, however when we investigated further it was determined that the actual code that was altered (basically some output being printed to screen) was not being seen. This led us to look into the test logs to verify that the code submitted in the PR was not the actual code being tested.

Next steps:

• Determine if the gcp bucket contains a built image for that particular PR
• Verify that the contents within that image match the code that was submitted in the PR
• Verify that e2e-testing references the correct image when executing the test suite

[adam-stokes]

@mdelapenya I will need your help with this when you get back to verify this particular workflow

[mdelapenya]

We manually triggered a build for the elastic/beats#26749 PR: https://beats-ci.elastic.co/blue/organizations/jenkins/e2e-tests%2Fe2e-testing-mbp/detail/master/1147/pipeline/191/

This build has been launched with GITHUB_CHECK_SHA1=208b8bc7af27e07f6a45b4439814688f0c8f0b95 and BEATS_USE_CI_SNAPSHOTS=true, so that the tests are fetching the binaries for that hash from the GCP bucket.

It's possible to reproduce it locally with:

TAGS="fleet_mode_agent" \
   GITHUB_CHECK_SHA1=208b8bc7af27e07f6a45b4439814688f0c8f0b95 \
   BEATS_USE_CI_SNAPSHOTS=true \
   TIMEOUT_FACTOR=5 \
   LOG_LEVEL=TRACE \
   DEVELOPER_MODE=true \
   ELASTIC_APM_ACTIVE=false \
make -C e2e/_suites/fleet functional-test

[cachedout]

@mdelapenya or @adam-stokes Could you please update the status here? I would like to know whether or not #1336 resolved this issue and if not, specifically which work remains to resolve this. Thanks!

[adam-stokes]

@cachedout This issue still needs to be addressed, reading over #1405 makes me wonder if the problem started after we included the automated beats version bump. Still investigating this week

[cachedout]

@adam-stokes Sounds good. Please provide another status by the end of this work-week (Aug 5) if a resolution hasn't been found before then. Thanks!

[mdelapenya]

I think elastic/apm-pipeline-library#1227 fixed it, but let's monitor current builds for Beats PRs since that PR was merged

[mdelapenya]

I'm going to close this one, as it seems the images are now correct. Let's reopen it if we see it again. Thanks!