feat: support building centos/debian Docker images in multiplatform format #1091

mdelapenya · 2021-04-26T16:06:23Z

What does this PR do?

This PR adds a pipeline to build and push the Docker images for Centos and Debian (used by the elastic-agent) in multiplatform format: ARM64 and AMD64.

To achieve that we:

copied the Dockerfiles from the upstream project to this repository
created a script for building and pushing the images for each OS (centos and debian). Each image's Docker name will contain the platform as suffix: centos-systemd-arm64:latest
created a script for pushing the multiplatform manifest for each image (centos and debian), using Infra's manifest-tool. This tool is bundled into a Docker image.

This script leverages infra's tool to write the manifest, which needs to be ran right after the images have been built and pushed. Therefore, the tool will write the manifest for both platforms (AMD/ARM), inspecting the existing platform-specific repositories, combining them into the target.

create a regular pipeline to orchestrate the build&push. This pipeline has 4 different stages:
1. git checkout the e2e-testing repo
2. build & push the AMD images, grabbing an AMD worker (ubuntu-20)
3. build & push the ARM images, grabbing an ARM worker
4. push the multiplatform manifest

Why is it important?

In order to ran the Elastic Agent in ARM, we need to have Centos and Debian containers for ARM. This is not the case of integrations, which can be run elsewhere.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have run the Unit tests for the CLI, and they are passing locally
I have run the End-2-End tests for the suite I'm working on, and they are passing locally
I have noticed new Go dependencies (run make notice in the proper directory)

Author's Checklist

Pipeline scheduler
JJBB

How to test this PR locally

It's possible to run the scripts to build&push the images, but a docker login must be performed first. The multiplatform manifest needs the ARM images to be created.

Related issues

See https://github.com/CentOS/CentOS-Dockerfiles/tree/master/systemd/centos7

See https://github.com/alehaa/docker-debian-systemd

…ebian This script leverages infra's tool to write the manifest, which needs to be ran right after the images have been built and pushed. Therefore, the tool will write the manifest for both platforms (AMD/ARM), inspecting the existing platform-specific repositories, combining them into the target. FYI, the '-ARCH' placeholder will be replaced with the values in the '--platforms' argument

.ci/build-docker-images.groovy

mdelapenya · 2021-04-26T16:14:58Z

.ci/build-docker-images.groovy

+        stash allowEmpty: true, name: 'source', useDefaultExcludes: false
+      }
+    }
+    stage('Build AMD Docker images'){


The build stages could be run in parallel, maybe in a follow-up

mdelapenya · 2021-04-26T16:21:16Z

cli/config/compose/services/centos-systemd/docker-compose.yml

@@ -4,6 +4,7 @@ services:
    image: centos/systemd:${centos_systemdTag:-latest} 


I'm not changing the images yet. They will be eventually replaced with:

image: docker.elastic.co/observability-ci/centos-systemd:latest

elasticmachine · 2021-04-26T16:38:38Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #1091 opened
Start Time: 2021-04-26T16:06:36.529+0000
Duration: 32 min 56 sec
Commit: 9f3f41a

Test stats 🧪

Test	Results
Failed	0
Passed	153
Skipped	0
Total	153

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	153
Skipped	0
Total	153

* master: Support fleet-server-service-token (elastic#1096) fix: use wider selector for ARM workers (elastic#1103) feat: bootstrap fleet-server for the deployment of regular elastic-agents (elastic#1078) fix: use proper variable name (elastic#1102) fix: branch_specifier is needed (elastic#1097) Move kubernetes/kubectl/kind code to internal project layout (elastic#1092) fix: update JJBB with proper values (elastic#1093) feat: support building centos/debian Docker images in multiplatform format (elastic#1091) Kubernetes autodiscover suite (elastic#1064)

…ormat (elastic#1091) * chore: copy Centos+Systemd Dockerfile from origin See https://github.com/CentOS/CentOS-Dockerfiles/tree/master/systemd/centos7 * chore: copy debian+systemd Dockerfile from origin See https://github.com/alehaa/docker-debian-systemd * chore: add script to build&push ARCH-based images for centos and debian * chore: add script to push the multiplatform manifest for centos and debian This script leverages infra's tool to write the manifest, which needs to be ran right after the images have been built and pushed. Therefore, the tool will write the manifest for both platforms (AMD/ARM), inspecting the existing platform-specific repositories, combining them into the target. FYI, the '-ARCH' placeholder will be replaced with the values in the '--platforms' argument * chore: add regular pipeline to build the docker images * fix: default arch variable value * chore: abstract image name from file system * chore: couple agent's base box with stack platform

* feat: support building centos/debian Docker images in multiplatform format (#1091) * chore: copy Centos+Systemd Dockerfile from origin See https://github.com/CentOS/CentOS-Dockerfiles/tree/master/systemd/centos7 * chore: copy debian+systemd Dockerfile from origin See https://github.com/alehaa/docker-debian-systemd * chore: add script to build&push ARCH-based images for centos and debian * chore: add script to push the multiplatform manifest for centos and debian This script leverages infra's tool to write the manifest, which needs to be ran right after the images have been built and pushed. Therefore, the tool will write the manifest for both platforms (AMD/ARM), inspecting the existing platform-specific repositories, combining them into the target. FYI, the '-ARCH' placeholder will be replaced with the values in the '--platforms' argument * chore: add regular pipeline to build the docker images * fix: default arch variable value * chore: abstract image name from file system * chore: couple agent's base box with stack platform * Move kubernetes/kubectl/kind code to internal project layout (#1092) This is mainly a cleanup to keep all internal related code that could be reusable in our `internal` directory layout. Next steps would be to take what's in `internal/kubectl` and merge with this code. Signed-off-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com> * feat: bootstrap fleet-server for the deployment of regular elastic-agents (#1078) * chore: provide a fleet-server base image based on centos/debian with systemd * WIP * fix: remove duplicated fields after merge conflicts * fix: update method call after merge conflicts * chore: extract service name calculation to a method * chore: extract container name calculation to a method * chore: refactor get container name method * chore: refactor method even more * chore: use installer state to retrieve container name * chore: use installer when calculating service name * fix: adapt service names for fleet server * chore: enrich log when creating an installer * fix: use fleet server host when creating fleet config * fix: use https when connecting to fleet-server It's creating its own self-signed certs * feat: bootstrap a fleet server before a regular agent is deployed to fleet It will define the server host to be used when enrolling agents * fix: use fleet policy for agents, not the server one * fix: get different installers for fleet-server and agents * fix: use the old step for deploying regular agents * chore: rename variable with consistent name * chore: rename fleet-server scenario * fix: use proper container name for standalone mode * chore: save two variables * chore: rename standalone scenario for bootstrapping fleet-server * chore: rename bootstrap methods * chore: encapsulate bootstrap fleet-server logic * Update fleet.go * chore: remove Fleet Server CI parallel execution * chore: remove feature file for fleet-server * chore: boostrap fleet server only once We want to have it bootstrapped for the entire test suite, not for each scenario * fix: an agent was needed when adding integrations Co-authored-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com> * apm-server tests (#1083) * some tests for apm-server * clean op dir on init instead of after * fix agent uninstall (#1111) * Auto bootstrap fleet during initialize scenario (#1116) Signed-off-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com> Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com> * feat: support running k8s autodiscover suite for Beats PRs and local repositories (#1115) * chore: add license * chore: initialise configurations before test suite * chore: use timeout_factor from env * fix: tell kind to skip pulling beats images * chore: add a method to load images into kind * feat: support running k8s autodiscover for Beats PRs or local filesystem * chore: add license header * chore: expose logger and use it, simplifying initialisation * fix: only run APM services for local APM environment * Revert "chore: expose logger and use it, simplifying initialisation" This reverts commit a89325c. * chore: log scenario name * fix: always cache beat version for podName * chore: reduce log level Co-authored-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com> Co-authored-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com> Co-authored-by: Juan Álvarez <juan.alvarez@elastic.co>

mdelapenya added 8 commits April 26, 2021 17:40

chore: copy Centos+Systemd Dockerfile from origin

630e045

See https://github.com/CentOS/CentOS-Dockerfiles/tree/master/systemd/centos7

chore: copy debian+systemd Dockerfile from origin

54e9727

See https://github.com/alehaa/docker-debian-systemd

chore: add script to build&push ARCH-based images for centos and debian

e3db97e

chore: add regular pipeline to build the docker images

b5dae6f

fix: default arch variable value

26f712b

chore: abstract image name from file system

bcbc80c

chore: couple agent's base box with stack platform

9f3f41a

mdelapenya self-assigned this Apr 26, 2021

mdelapenya requested a review from a team April 26, 2021 16:06

kuisathaverat reviewed Apr 26, 2021

View reviewed changes

.ci/build-docker-images.groovy Show resolved Hide resolved

mdelapenya commented Apr 26, 2021

View reviewed changes

mdelapenya marked this pull request as ready for review April 26, 2021 16:20

mdelapenya commented Apr 26, 2021

View reviewed changes

kuisathaverat approved these changes Apr 26, 2021

View reviewed changes

adam-stokes merged commit b97edfd into elastic:master Apr 26, 2021

mdelapenya mentioned this pull request Apr 26, 2021

fix: update JJBB with proper values #1093

Merged

8 tasks

mdelapenya deleted the build-docker-images branch April 30, 2021 09:36

mdelapenya mentioned this pull request May 4, 2021

chore: backports for 7.x #1126

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support building centos/debian Docker images in multiplatform format #1091

feat: support building centos/debian Docker images in multiplatform format #1091

mdelapenya commented Apr 26, 2021 •

edited

Loading

mdelapenya Apr 26, 2021

mdelapenya Apr 26, 2021

elasticmachine commented Apr 26, 2021

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

		@@ -4,6 +4,7 @@ services:
		image: centos/systemd:${centos_systemdTag:-latest}

feat: support building centos/debian Docker images in multiplatform format #1091

feat: support building centos/debian Docker images in multiplatform format #1091

Conversation

mdelapenya commented Apr 26, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

mdelapenya Apr 26, 2021

Choose a reason for hiding this comment

mdelapenya Apr 26, 2021

Choose a reason for hiding this comment

elasticmachine commented Apr 26, 2021

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

mdelapenya commented Apr 26, 2021 •

edited

Loading