Skip to content
This repository has been archived by the owner on Sep 17, 2024. It is now read-only.

feat: add support for running the tests in ARM #707

Merged
merged 86 commits into from
Jun 1, 2021

Conversation

mdelapenya
Copy link
Contributor

@mdelapenya mdelapenya commented Feb 8, 2021

What does this PR do?

It enables the ARM platform for metricbeat and fleet test suites, grabbing ARM workers, and for that reason we are refactoring the way we define the architecture in the e2e code. Instead of hardcoding the amd64 architecture, we are going to read it from the environment. Golang provides a GOARCH environment variable that is automatically set by Go's runtime. With that variable we are going to calculate the artifact name based on the underlying architecture:

  • for CI workers running amd64, the code will generate an artifact for amd64.
  • for CI workers running arm64, the code will generate an artifact for arm64.

Besides that, we had to rewrite the Jenkins pipeline to properly assign a worker in the right platform, to retrieve a non-immutable worker for arm.

Finally, we are adding the support to exclude scenarios per platform, leveraging the .e2e-testing.yaml file, being possiblee for a scenario to override its parent suite's platforms. To add awareness at this level, we are always removing the scenarios tagged with @skip:PLATFORM (amd64, arm64), including those tags in the feature file when needed, so that developers now what is going to be run directly in the source code, not in the CI descriptor.

Why is it important?

Add support to ARM platform

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have run the Unit tests for the CLI, and they are passing locally
  • I have run the Unit tests for the E2E, and they are passing locally
  • I have run the End-2-End tests for the suite I'm working on, and they are passing locally
  • I have noticed new Go dependencies (run make notice in the proper directory)

Author's Checklist

  • Beats merged the compilation in arm.
  • Official Ubi8 Docker images for arm are not in place. See [ARM] Ubi8 Docker images are not included in the unified release beats#24186. This will cause that any job for the E2E consuming an official release will fail the ubi8 tests for arm. The jobs triggered by Beats project (PRs and branches) won't be affected, as they consume the CI artifacts.
  • Beats-CI Ubi8 Docker images for arm are generated and pushed in our GCP bucket

How to test this PR locally

Related issues

Screenshots

Thanks to the refactors we did in the past, it's easy to add parallel branches for new platforms, simply adding the Jenkins labels used by infra to grab the right CI workers.

Screenshot 2021-02-08 at 17 34 56

Follow-ups

This PR is incomplete: let's see where it fails, as we need to update how the arch is calculated when creating the installers.

@mdelapenya mdelapenya self-assigned this Feb 8, 2021
@mdelapenya mdelapenya requested a review from a team February 8, 2021 16:28
@elasticmachine
Copy link
Contributor

elasticmachine commented Feb 8, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Started by user Manuel de la Peña

  • Start Time: 2021-06-01T09:51:15.984+0000

  • Duration: 30 min 28 sec

  • Commit: aa09c0e

Test stats 🧪

Test Results
Failed 0
Passed 202
Skipped 0
Total 202

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 202
Skipped 0
Total 202

.ci/Jenkinsfile Outdated
@@ -316,8 +316,15 @@ def generateFunctionalTestStep(Map args = [:]){
}
tags += excludeNightlyTag

workerLabels = "${platform}"
if (platform != "arm") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you describe the intent of the branching? I see it has to do with immutable workers, but don't have context - maybe that is what you could comment better on?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly, this is how Infra's workers are labelled: arm are not immutable yet

@EricDavisX
Copy link
Contributor

@kvch fyi - Engg Productivity team is getting ready to hook our e2e-testing scenarios up for linux ARM testing. awesome!

Thanks for putting the quick change in, Manu. Question, I saw a note that we would need to wait until at least 7.12 BC2 (and after Noemi merges changes on Beats side, too) before we can hope to automate the scenario, let us all keep in touch. :)

.ci/Jenkinsfile Outdated
// ARM is not ready as immutable workers
workerLabels += " && immutable"
}
workerLabels += " && docker"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will add this!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mmm... does it mean that we cannot use immutable workers on ARM? If that's the case, the logic here is correct, right? we are only adding the immutable label for !arm workers.

Comment on lines 78 to 79
arch := e2e.GetArchitecture()
dockerImageTag += "-" + arch
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docker images are pushed to the docker registry, and their tag name contains the architecture, for further details refers to elastic/beats#24121

@@ -408,7 +410,7 @@ func (mts *MetricbeatTestSuite) runMetricbeatService() error {

err = docker.TagImage(
"docker.elastic.co/beats/metricbeat:"+metricbeatVersionBase,
"docker.elastic.co/observability-ci/metricbeat:"+mts.Version+"-amd64",
"docker.elastic.co/observability-ci/metricbeat:"+mts.Version+"-"+arch,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docker images are pushed to the docker registry, and their tag name contains the architecture, for further details refers to elastic/beats#24121

@@ -99,7 +100,7 @@ func (sats *StandAloneTestSuite) aStandaloneAgentIsDeployed(image string) error

profileEnv["elasticAgentContainerName"] = containerName
profileEnv["elasticAgentConfigFile"] = sats.AgentConfigFilePath
profileEnv["elasticAgentPlatform"] = "linux/amd64"
profileEnv["elasticAgentPlatform"] = "linux/" + e2e.GetArchitecture()
Copy link
Contributor Author

@mdelapenya mdelapenya Mar 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reuse the variable

Suggested change
profileEnv["elasticAgentPlatform"] = "linux/" + e2e.GetArchitecture()
profileEnv["elasticAgentPlatform"] = "linux/" + arch

@mdelapenya mdelapenya added the triaged Triaged issues will end up in Backlog column in Robots GH Project label May 27, 2021
@mdelapenya
Copy link
Contributor Author

@adam-stokes waiting for your final review, and then this is ready to go!

@adam-stokes
Copy link
Contributor

@mdelapenya this should have minimal impact on my PR #1208 as I would only need to update the internal/installer code to make use of the architecture specific items. We can go ahead and merge this first and I'll update my PR on top of it.

@mdelapenya
Copy link
Contributor Author

Let's merge this one once the regression with the integrations is resolved

@mergify
Copy link
Contributor

mergify bot commented May 30, 2021

This pull request is now in conflict. Could you fix it @mdelapenya? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b arm-suppport upstream/arm-suppport
git merge upstream/master
git push upstream arm-suppport

@adam-stokes
Copy link
Contributor

@mdelapenya this LGTM, just want to make sure you're ready for this to merge?

@mdelapenya
Copy link
Contributor Author

mdelapenya commented Jun 1, 2021

@mdelapenya this LGTM, just want to make sure you're ready for this to merge?

Yes! I did not mention it online, but I'd like to merge this today (cc/ @EricDavisX). We have been running this for days, twice, three, four times a day, and the results are consistent.

What I'd like to discuss with the @elastic/observablt-robots is about the schedule:

  • should we run ARM tests with every PR (here or/and Beats/Kibana workflows)?

If we want to move the ARM tests to a less frequent schedule (because workers are more expensive), we could listen to commands in the upstream repos to run a test suite in a platform, i.e. Fleet tests in ARM (plus the basic AMD tests). But it could be done in a follow-up issue.

@mdelapenya mdelapenya merged commit e1a97f3 into elastic:master Jun 1, 2021
@mdelapenya mdelapenya deleted the arm-suppport branch June 1, 2021 14:50
mergify bot pushed a commit that referenced this pull request Jun 1, 2021
* feat: add support for running the tests in ARM

* fix: arm workers are not immutable yet

* chore: calculate artifact from the GOARCH variable

At this moment we support amd64 and arm64

* chore: run elasticsearch in the right platform

* chore: add log for architecture

* chore: print env variable for GOARCH

* chore: set GOARCH based in the worker

* fix: wrong copy&paste

* chore: install k8s dependencies on ARM

* chore: install godog using GOARCH

* fix: proper values for GO env in Makefile

* chore: add log when requesting a node

* fix: define Groovy variable

* chore: use proper platform for kibana

* chore: simplify platforms selection logic

* fix: use proper CI label for node selector

* chore: remove running containers in clean-docker script

* chore: clean up docker state on mutable workers

* chore: use parallel branch name for filebeat logs

* fix: add forgotten double quotes

* chore: print Docker compose version On ARM

* chore: use aarch64 architecture for RPM packages

* chore: support ARM in tar installer

* fix: proper arch for TAR installer

* chore: log when the artifact is found

* chore: be defensive about no running containers

* chore: add tests for RPM artifact name

* chore: add tests for x86_64 TAR artifact

* chore: remove docker label from CI workers

* chore: separate ARM tests in a second parallel stage

This pretends to avoid using too many workers, adding a second layer for ARM.

* chore: update stage name

* fix: update arm target platform

* chore: use GOARCH in the junit file

* chore: use GOARCH in filebeat logs

* chore: run all stages in parallel

* chore: add platform to log when skipping tests

* fix: use a filebeat version that supports multiplatform

* chore: do not run integrations on ARM

See elastic/beats#25195

* chore: move test resources for ARM

* chore: use staging environment for package-registry

See https://github.com/elastic/package-storage

* fix: remove clean because arm workers are now immutable

* chore: use new multiplatform images

* chore: use new multiplatform base images for fleet-server

* chore: add platform to elastic-agent cloud flavour

* fix: use proper variable for platform

* chore: do not clean up installer's workspace

Because we are using ephemeral workers, we do not want to remove previous files

* Revert "chore: do not clean up installer's workspace"

This reverts commit 9104501.

* chore(ci): skip pull of images from code

* Revert "chore(ci): skip pull of images from code"

This reverts commit 01cc441.

* chore: set platform when pulling images

* chore: update packer cache with our systemd images

* fix: request proper platform image for the agent

* feat: exclude scenarios per platform

* chore: remove tags for endpoint, as they are not used at all

They are also blocking the addition of a parent tag at the feature level, as maximun number of tags is 3

* fix: skip endpoint integration on ARM64

* feat: provide supported platforms at the scenario level on CI

The scenario will be able to override its parent suite's platforms, making
possible to select different platforms at the scenario level.

* fix: use proper length method for groovy arrays

* fix: groovy arrays use size

* fix: override suite platforms per scenario

* chore: update Jenkins label for arm workers

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: simplify platforms, reusing stack

* chore: explicitly pass GOARCH to CI script

* Revert "chore: update Jenkins label for arm workers"

This reverts commit 9ef81ca.

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
(cherry picked from commit e1a97f3)

# Conflicts:
#	.ci/packer_cache.sh
#	cli/config/compose/profiles/fleet/docker-compose.yml
#	cli/config/compose/profiles/metricbeat/docker-compose.yml
#	e2e/_suites/fleet/stand-alone.go
#	internal/deploy/docker_client.go
#	internal/installer/elasticagent_deb.go
#	internal/installer/elasticagent_docker.go
#	internal/installer/elasticagent_rpm.go
#	internal/installer/elasticagent_tar.go
#	internal/utils/utils_test.go
mergify bot pushed a commit that referenced this pull request Jun 1, 2021
* feat: add support for running the tests in ARM

* fix: arm workers are not immutable yet

* chore: calculate artifact from the GOARCH variable

At this moment we support amd64 and arm64

* chore: run elasticsearch in the right platform

* chore: add log for architecture

* chore: print env variable for GOARCH

* chore: set GOARCH based in the worker

* fix: wrong copy&paste

* chore: install k8s dependencies on ARM

* chore: install godog using GOARCH

* fix: proper values for GO env in Makefile

* chore: add log when requesting a node

* fix: define Groovy variable

* chore: use proper platform for kibana

* chore: simplify platforms selection logic

* fix: use proper CI label for node selector

* chore: remove running containers in clean-docker script

* chore: clean up docker state on mutable workers

* chore: use parallel branch name for filebeat logs

* fix: add forgotten double quotes

* chore: print Docker compose version On ARM

* chore: use aarch64 architecture for RPM packages

* chore: support ARM in tar installer

* fix: proper arch for TAR installer

* chore: log when the artifact is found

* chore: be defensive about no running containers

* chore: add tests for RPM artifact name

* chore: add tests for x86_64 TAR artifact

* chore: remove docker label from CI workers

* chore: separate ARM tests in a second parallel stage

This pretends to avoid using too many workers, adding a second layer for ARM.

* chore: update stage name

* fix: update arm target platform

* chore: use GOARCH in the junit file

* chore: use GOARCH in filebeat logs

* chore: run all stages in parallel

* chore: add platform to log when skipping tests

* fix: use a filebeat version that supports multiplatform

* chore: do not run integrations on ARM

See elastic/beats#25195

* chore: move test resources for ARM

* chore: use staging environment for package-registry

See https://github.com/elastic/package-storage

* fix: remove clean because arm workers are now immutable

* chore: use new multiplatform images

* chore: use new multiplatform base images for fleet-server

* chore: add platform to elastic-agent cloud flavour

* fix: use proper variable for platform

* chore: do not clean up installer's workspace

Because we are using ephemeral workers, we do not want to remove previous files

* Revert "chore: do not clean up installer's workspace"

This reverts commit 9104501.

* chore(ci): skip pull of images from code

* Revert "chore(ci): skip pull of images from code"

This reverts commit 01cc441.

* chore: set platform when pulling images

* chore: update packer cache with our systemd images

* fix: request proper platform image for the agent

* feat: exclude scenarios per platform

* chore: remove tags for endpoint, as they are not used at all

They are also blocking the addition of a parent tag at the feature level, as maximun number of tags is 3

* fix: skip endpoint integration on ARM64

* feat: provide supported platforms at the scenario level on CI

The scenario will be able to override its parent suite's platforms, making
possible to select different platforms at the scenario level.

* fix: use proper length method for groovy arrays

* fix: groovy arrays use size

* fix: override suite platforms per scenario

* chore: update Jenkins label for arm workers

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: simplify platforms, reusing stack

* chore: explicitly pass GOARCH to CI script

* Revert "chore: update Jenkins label for arm workers"

This reverts commit 9ef81ca.

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
(cherry picked from commit e1a97f3)

# Conflicts:
#	cli/config/compose/profiles/fleet/docker-compose.yml
#	cli/config/compose/profiles/metricbeat/docker-compose.yml
#	e2e/_suites/fleet/stand-alone.go
#	internal/installer/elasticagent_deb.go
#	internal/installer/elasticagent_docker.go
#	internal/installer/elasticagent_rpm.go
#	internal/installer/elasticagent_tar.go
#	internal/utils/utils_test.go
@mdelapenya mdelapenya added v7.14.0 and removed v7.14.0 labels Jun 1, 2021
@EricDavisX
Copy link
Contributor

Thx Manu - exciting to have it merged.

mdelapenya added a commit to mdelapenya/e2e-testing that referenced this pull request Jun 1, 2021
* feat: add support for running the tests in ARM

* fix: arm workers are not immutable yet

* chore: calculate artifact from the GOARCH variable

At this moment we support amd64 and arm64

* chore: run elasticsearch in the right platform

* chore: add log for architecture

* chore: print env variable for GOARCH

* chore: set GOARCH based in the worker

* fix: wrong copy&paste

* chore: install k8s dependencies on ARM

* chore: install godog using GOARCH

* fix: proper values for GO env in Makefile

* chore: add log when requesting a node

* fix: define Groovy variable

* chore: use proper platform for kibana

* chore: simplify platforms selection logic

* fix: use proper CI label for node selector

* chore: remove running containers in clean-docker script

* chore: clean up docker state on mutable workers

* chore: use parallel branch name for filebeat logs

* fix: add forgotten double quotes

* chore: print Docker compose version On ARM

* chore: use aarch64 architecture for RPM packages

* chore: support ARM in tar installer

* fix: proper arch for TAR installer

* chore: log when the artifact is found

* chore: be defensive about no running containers

* chore: add tests for RPM artifact name

* chore: add tests for x86_64 TAR artifact

* chore: remove docker label from CI workers

* chore: separate ARM tests in a second parallel stage

This pretends to avoid using too many workers, adding a second layer for ARM.

* chore: update stage name

* fix: update arm target platform

* chore: use GOARCH in the junit file

* chore: use GOARCH in filebeat logs

* chore: run all stages in parallel

* chore: add platform to log when skipping tests

* fix: use a filebeat version that supports multiplatform

* chore: do not run integrations on ARM

See elastic/beats#25195

* chore: move test resources for ARM

* chore: use staging environment for package-registry

See https://github.com/elastic/package-storage

* fix: remove clean because arm workers are now immutable

* chore: use new multiplatform images

* chore: use new multiplatform base images for fleet-server

* chore: add platform to elastic-agent cloud flavour

* fix: use proper variable for platform

* chore: do not clean up installer's workspace

Because we are using ephemeral workers, we do not want to remove previous files

* Revert "chore: do not clean up installer's workspace"

This reverts commit 9104501.

* chore(ci): skip pull of images from code

* Revert "chore(ci): skip pull of images from code"

This reverts commit 01cc441.

* chore: set platform when pulling images

* chore: update packer cache with our systemd images

* fix: request proper platform image for the agent

* feat: exclude scenarios per platform

* chore: remove tags for endpoint, as they are not used at all

They are also blocking the addition of a parent tag at the feature level, as maximun number of tags is 3

* fix: skip endpoint integration on ARM64

* feat: provide supported platforms at the scenario level on CI

The scenario will be able to override its parent suite's platforms, making
possible to select different platforms at the scenario level.

* fix: use proper length method for groovy arrays

* fix: groovy arrays use size

* fix: override suite platforms per scenario

* chore: update Jenkins label for arm workers

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: simplify platforms, reusing stack

* chore: explicitly pass GOARCH to CI script

* Revert "chore: update Jenkins label for arm workers"

This reverts commit 9ef81ca.

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
mdelapenya added a commit to mdelapenya/e2e-testing that referenced this pull request Jun 1, 2021
* feat: add support for running the tests in ARM

* fix: arm workers are not immutable yet

* chore: calculate artifact from the GOARCH variable

At this moment we support amd64 and arm64

* chore: run elasticsearch in the right platform

* chore: add log for architecture

* chore: print env variable for GOARCH

* chore: set GOARCH based in the worker

* fix: wrong copy&paste

* chore: install k8s dependencies on ARM

* chore: install godog using GOARCH

* fix: proper values for GO env in Makefile

* chore: add log when requesting a node

* fix: define Groovy variable

* chore: use proper platform for kibana

* chore: simplify platforms selection logic

* fix: use proper CI label for node selector

* chore: remove running containers in clean-docker script

* chore: clean up docker state on mutable workers

* chore: use parallel branch name for filebeat logs

* fix: add forgotten double quotes

* chore: print Docker compose version On ARM

* chore: use aarch64 architecture for RPM packages

* chore: support ARM in tar installer

* fix: proper arch for TAR installer

* chore: log when the artifact is found

* chore: be defensive about no running containers

* chore: add tests for RPM artifact name

* chore: add tests for x86_64 TAR artifact

* chore: remove docker label from CI workers

* chore: separate ARM tests in a second parallel stage

This pretends to avoid using too many workers, adding a second layer for ARM.

* chore: update stage name

* fix: update arm target platform

* chore: use GOARCH in the junit file

* chore: use GOARCH in filebeat logs

* chore: run all stages in parallel

* chore: add platform to log when skipping tests

* fix: use a filebeat version that supports multiplatform

* chore: do not run integrations on ARM

See elastic/beats#25195

* chore: move test resources for ARM

* chore: use staging environment for package-registry

See https://github.com/elastic/package-storage

* fix: remove clean because arm workers are now immutable

* chore: use new multiplatform images

* chore: use new multiplatform base images for fleet-server

* chore: add platform to elastic-agent cloud flavour

* fix: use proper variable for platform

* chore: do not clean up installer's workspace

Because we are using ephemeral workers, we do not want to remove previous files

* Revert "chore: do not clean up installer's workspace"

This reverts commit 9104501.

* chore(ci): skip pull of images from code

* Revert "chore(ci): skip pull of images from code"

This reverts commit 01cc441.

* chore: set platform when pulling images

* chore: update packer cache with our systemd images

* fix: request proper platform image for the agent

* feat: exclude scenarios per platform

* chore: remove tags for endpoint, as they are not used at all

They are also blocking the addition of a parent tag at the feature level, as maximun number of tags is 3

* fix: skip endpoint integration on ARM64

* feat: provide supported platforms at the scenario level on CI

The scenario will be able to override its parent suite's platforms, making
possible to select different platforms at the scenario level.

* fix: use proper length method for groovy arrays

* fix: groovy arrays use size

* fix: override suite platforms per scenario

* chore: update Jenkins label for arm workers

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: simplify platforms, reusing stack

* chore: explicitly pass GOARCH to CI script

* Revert "chore: update Jenkins label for arm workers"

This reverts commit 9ef81ca.

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
mdelapenya added a commit that referenced this pull request Jun 1, 2021
…1223)

* feat: add support for running the tests in ARM (#707)

* feat: add support for running the tests in ARM

* fix: arm workers are not immutable yet

* chore: calculate artifact from the GOARCH variable

At this moment we support amd64 and arm64

* chore: run elasticsearch in the right platform

* chore: add log for architecture

* chore: print env variable for GOARCH

* chore: set GOARCH based in the worker

* fix: wrong copy&paste

* chore: install k8s dependencies on ARM

* chore: install godog using GOARCH

* fix: proper values for GO env in Makefile

* chore: add log when requesting a node

* fix: define Groovy variable

* chore: use proper platform for kibana

* chore: simplify platforms selection logic

* fix: use proper CI label for node selector

* chore: remove running containers in clean-docker script

* chore: clean up docker state on mutable workers

* chore: use parallel branch name for filebeat logs

* fix: add forgotten double quotes

* chore: print Docker compose version On ARM

* chore: use aarch64 architecture for RPM packages

* chore: support ARM in tar installer

* fix: proper arch for TAR installer

* chore: log when the artifact is found

* chore: be defensive about no running containers

* chore: add tests for RPM artifact name

* chore: add tests for x86_64 TAR artifact

* chore: remove docker label from CI workers

* chore: separate ARM tests in a second parallel stage

This pretends to avoid using too many workers, adding a second layer for ARM.

* chore: update stage name

* fix: update arm target platform

* chore: use GOARCH in the junit file

* chore: use GOARCH in filebeat logs

* chore: run all stages in parallel

* chore: add platform to log when skipping tests

* fix: use a filebeat version that supports multiplatform

* chore: do not run integrations on ARM

See elastic/beats#25195

* chore: move test resources for ARM

* chore: use staging environment for package-registry

See https://github.com/elastic/package-storage

* fix: remove clean because arm workers are now immutable

* chore: use new multiplatform images

* chore: use new multiplatform base images for fleet-server

* chore: add platform to elastic-agent cloud flavour

* fix: use proper variable for platform

* chore: do not clean up installer's workspace

Because we are using ephemeral workers, we do not want to remove previous files

* Revert "chore: do not clean up installer's workspace"

This reverts commit 9104501.

* chore(ci): skip pull of images from code

* Revert "chore(ci): skip pull of images from code"

This reverts commit 01cc441.

* chore: set platform when pulling images

* chore: update packer cache with our systemd images

* fix: request proper platform image for the agent

* feat: exclude scenarios per platform

* chore: remove tags for endpoint, as they are not used at all

They are also blocking the addition of a parent tag at the feature level, as maximun number of tags is 3

* fix: skip endpoint integration on ARM64

* feat: provide supported platforms at the scenario level on CI

The scenario will be able to override its parent suite's platforms, making
possible to select different platforms at the scenario level.

* fix: use proper length method for groovy arrays

* fix: groovy arrays use size

* fix: override suite platforms per scenario

* chore: update Jenkins label for arm workers

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: simplify platforms, reusing stack

* chore: explicitly pass GOARCH to CI script

* Revert "chore: update Jenkins label for arm workers"

This reverts commit 9ef81ca.

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* fix: remove packer_cache from maintenance branches

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
mdelapenya added a commit that referenced this pull request Jun 1, 2021
#1224)

* feat: add support for running the tests in ARM (#707)

* feat: add support for running the tests in ARM

* fix: arm workers are not immutable yet

* chore: calculate artifact from the GOARCH variable

At this moment we support amd64 and arm64

* chore: run elasticsearch in the right platform

* chore: add log for architecture

* chore: print env variable for GOARCH

* chore: set GOARCH based in the worker

* fix: wrong copy&paste

* chore: install k8s dependencies on ARM

* chore: install godog using GOARCH

* fix: proper values for GO env in Makefile

* chore: add log when requesting a node

* fix: define Groovy variable

* chore: use proper platform for kibana

* chore: simplify platforms selection logic

* fix: use proper CI label for node selector

* chore: remove running containers in clean-docker script

* chore: clean up docker state on mutable workers

* chore: use parallel branch name for filebeat logs

* fix: add forgotten double quotes

* chore: print Docker compose version On ARM

* chore: use aarch64 architecture for RPM packages

* chore: support ARM in tar installer

* fix: proper arch for TAR installer

* chore: log when the artifact is found

* chore: be defensive about no running containers

* chore: add tests for RPM artifact name

* chore: add tests for x86_64 TAR artifact

* chore: remove docker label from CI workers

* chore: separate ARM tests in a second parallel stage

This pretends to avoid using too many workers, adding a second layer for ARM.

* chore: update stage name

* fix: update arm target platform

* chore: use GOARCH in the junit file

* chore: use GOARCH in filebeat logs

* chore: run all stages in parallel

* chore: add platform to log when skipping tests

* fix: use a filebeat version that supports multiplatform

* chore: do not run integrations on ARM

See elastic/beats#25195

* chore: move test resources for ARM

* chore: use staging environment for package-registry

See https://github.com/elastic/package-storage

* fix: remove clean because arm workers are now immutable

* chore: use new multiplatform images

* chore: use new multiplatform base images for fleet-server

* chore: add platform to elastic-agent cloud flavour

* fix: use proper variable for platform

* chore: do not clean up installer's workspace

Because we are using ephemeral workers, we do not want to remove previous files

* Revert "chore: do not clean up installer's workspace"

This reverts commit 9104501.

* chore(ci): skip pull of images from code

* Revert "chore(ci): skip pull of images from code"

This reverts commit 01cc441.

* chore: set platform when pulling images

* chore: update packer cache with our systemd images

* fix: request proper platform image for the agent

* feat: exclude scenarios per platform

* chore: remove tags for endpoint, as they are not used at all

They are also blocking the addition of a parent tag at the feature level, as maximun number of tags is 3

* fix: skip endpoint integration on ARM64

* feat: provide supported platforms at the scenario level on CI

The scenario will be able to override its parent suite's platforms, making
possible to select different platforms at the scenario level.

* fix: use proper length method for groovy arrays

* fix: groovy arrays use size

* fix: override suite platforms per scenario

* chore: update Jenkins label for arm workers

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: simplify platforms, reusing stack

* chore: explicitly pass GOARCH to CI script

* Revert "chore: update Jenkins label for arm workers"

This reverts commit 9ef81ca.

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* chore: remove packer_cache from maintenance branch

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area:test Anything related to the Test automation priority:high Product teams are waiting on features to make progress on their roadmap size:XL several weeks Team:Obs-DC Label for the Data Collection team triaged Triaged issues will end up in Backlog column in Robots GH Project v7.13.0 v7.14.0
Projects
None yet
7 participants