fix: update backports for 7.x (#1172)

* chore: initialise timeout factor next to the declaration (#1118)

* chore: initialise timeout factor on its own package

* chore: reuse timeout factor from common

* chore: remove unused code (#1119)

* chore: remove unused code

* chore: remove all references to fleet server hostname

Because we assume it's a runtime dependency, provided by the initial
compose file, we do not need to calculate service names, or URIs for the
fleet-service endpoint. Instead, we assume it's listening in the 8220 port
in the "fleet-server" hostname, which is accessible from the network
created by docker-compose.

* fix: use HTTP to connect to fleet-server

* chore: remove fleet server policy code

We do not need it anymore, as the fleet server is already bootstrapped

* chore: remove all policies but system and fleet_server

* Update policies.go

* Update fleet.go

* Update stand-alone.go

Co-authored-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>

* Support multiple deployment backends (#1130)

* Abstract out deployment

Provides ability to plugin different deployment backends for use in testing.
Current deployment backends supported are "docker" and "kubernetes"

* remove unused import
* remove unsetting of fleet server hostname as it's not needed
* add deployer support to stand-alone
* add elastic-agent to k8s deployment specs
* Update internal/docker/docker.go

Signed-off-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>
Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>

* chore: abstract process checks to the deployer (#1156)

* chore: abstract process checks to the deployer

* chore: rename variable in log entry

* docs: improve comment

* fix: go-fmt

* feat: simplify the initialisation of versions (#1159)

* chore: use fixed version in shell scripts

* chore: move retry to utils

We could move it to its own package, but at this moment it's very small

* chore: initialise stackVesion at one single place

* chore: initialise agent version base at one single place

* chore: initialise agent version at one single place

* chore: reduce the number of requests to Elastic's artifacts endpoint

* chore: rename AgentVersionBase variable to BeatVersionBase

* chore: rename AgentVersion variable to BeatVersion

* chore: use Beat version in metricbeat test suite

* chore: check if the version must use the fallback after coming from a Git SHA

* feat: support flavours in services, specially in the elastic-agent (#1162)

* chore: move compose to deploy package

* feat: use a ServiceRequest when adding services

* feat: add service flavour support

* chore: remove unused centos/debian services

* fixup: add service flavour

* chore: move docker client to the deploy package

We will need another abstraction to represent the Docker client operations,
as it's clear what is a deployment and what is an operation in the deployment.
Maybe a Client struct for each provider will help out in differenciate it

* chore: use ServiceRequest everywhere

* chore: run agent commands with a ServiceRequest

* chore: use ServiceRequest in metricbeat test suite

* chore: pass flavours to installers

* chore: add a step to install the agent for the underlying OS

* chore: always add flavour

* fix: use installer for fleet_mode when removing services at the end of the scenario

* fix: update broken references in metricbeat test suite

* fix: update broken references in helm test suite

* fix: standalone does not have an installer

* fix: use service instead of image to get a service request for the agent

* feat: support for scaling services in compose

* fix: run second agent using compose scale option

* fix: update kibana's default Docker namespace

* fix: there are 2 metricbeat instances

* fix: wait for 2 filebeat instances

* fix: fleet backend processes count

* chore: use 2 instances for background processes

Co-authored-by: Adam Stokes <51892+adam-stokes@users.noreply.github.com>

.ci/scripts/clean-docker.sh

@@ -9,7 +9,7 @@ set -euxo pipefail
 # Build and test the app using the install and test make goals.
 #
 
-readonly VERSION="7.x-SNAPSHOT"
+readonly VERSION="$(cat $(pwd)/.stack-version)"
 
 main() {
   # refresh docker images

.ci/scripts/fleet-test.sh

@@ -9,10 +9,10 @@ set -euxo pipefail
 # Run the functional tests for fleets using the functional-test wrapper
 #
 # Parameters:
-#   - STACK_VERSION - that's the version of the stack to be tested. Default '8.0.0-SNAPSHOT'.
+#   - STACK_VERSION - that's the version of the stack to be tested. Default is stored in '.stack-version'.
 #
 
-STACK_VERSION=${1:-'7.x-SNAPSHOT'}
+STACK_VERSION=${1:-"$(cat $(pwd)/.stack-version)"}
 SUITE='fleet'
 
 # Exclude the nightly tests in the CI.

.ci/scripts/functional-test.sh

@@ -12,14 +12,16 @@ set -euxo pipefail
 # Parameters:
 #   - SUITE - that's the suite to be tested. Default '' which means all of them.
 #   - TAGS - that's the tags to be tested. Default '' which means all of them.
-#   - STACK_VERSION - that's the version of the stack to be tested. Default '7.x-SNAPSHOT'.
-#   - BEAT_VERSION - that's the version of the metricbeat to be tested. Default '7.x-SNAPSHOT'.
+#   - STACK_VERSION - that's the version of the stack to be tested. Default is stored in '.stack-version'.
+#   - BEAT_VERSION - that's the version of the metricbeat to be tested. Default is stored in '.stack-version'.
 #
 
+BASE_VERSION="$(cat $(pwd)/.stack-version)"
+
 SUITE=${1:-''}
 TAGS=${2:-''}
-STACK_VERSION=${3:-'7.x-SNAPSHOT'}
-BEAT_VERSION=${4:-'7.x-SNAPSHOT'}
+STACK_VERSION=${3:-"${BASE_VERSION}"}
+BEAT_VERSION=${4:-"${BASE_VERSION}"}
 
 ## Install the required dependencies for the given SUITE
 .ci/scripts/install-test-dependencies.sh "${SUITE}"

.ci/scripts/metricbeat-test.sh

@@ -9,12 +9,14 @@ set -euxo pipefail
 # Run the functional tests for metricbeat using the functional-test wrapper
 #
 # Parameters:
-#   - STACK_VERSION - that's the version of the stack to be tested. Default '8.0.0-SNAPSHOT'.
-#   - BEAT_VERSION - that's the version of the metricbeat to be tested. Default '8.0.0-SNAPSHOT'.
+#   - STACK_VERSION - that's the version of the stack to be tested. Default is stored in '.stack-version'.
+#   - BEAT_VERSION - that's the version of the metricbeat to be tested. Default is stored in '.stack-version'.
 #
 
-STACK_VERSION=${1:-'7.x-SNAPSHOT'}
-BEAT_VERSION=${2:-'7.x-SNAPSHOT'}
+BASE_VERSION="$(cat $(pwd)/.stack-version)"
+
+STACK_VERSION=${1:-"${BASE_VERSION}"}
+BEAT_VERSION=${2:-"${BASE_VERSION}"}
 SUITE='metricbeat'
 
 .ci/scripts/functional-test.sh "${SUITE}" "" "${STACK_VERSION}" "${BEAT_VERSION}"

.stack-version

@@ -0,0 +1 @@
+8.0.0-SNAPSHOT

cli/cmd/deploy.go

@@ -8,7 +8,7 @@ import (
 	"context"
 
 	"github.com/elastic/e2e-testing/cli/config"
-	"github.com/elastic/e2e-testing/internal/compose"
+	"github.com/elastic/e2e-testing/internal/deploy"
 	log "github.com/sirupsen/logrus"
 
 	"github.com/spf13/cobra"
@@ -63,12 +63,16 @@ func buildDeployServiceCommand(srv string) *cobra.Command {
 		Short: `Deploys a ` + srv + ` service`,
 		Long:  `Deploys a ` + srv + ` service, adding it to a running profile, identified by its name`,
 		Run: func(cmd *cobra.Command, args []string) {
-			serviceManager := compose.NewServiceManager()
+			serviceManager := deploy.NewServiceManager()
 
 			env := map[string]string{}
 			env = config.PutServiceEnvironment(env, srv, versionToRun)
 
-			err := serviceManager.AddServicesToCompose(context.Background(), deployToProfile, []string{srv}, env)
+			err := serviceManager.AddServicesToCompose(
+				context.Background(),
+				deploy.NewServiceRequest(deployToProfile),
+				[]deploy.ServiceRequest{deploy.NewServiceRequest(srv)},
+				env)
 			if err != nil {
 				log.WithFields(log.Fields{
 					"profile":  deployToProfile,
@@ -85,12 +89,16 @@ func buildUndeployServiceCommand(srv string) *cobra.Command {
 		Short: `Undeploys a ` + srv + ` service`,
 		Long:  `Undeploys a ` + srv + ` service, removing it from a running profile, identified by its name`,
 		Run: func(cmd *cobra.Command, args []string) {
-			serviceManager := compose.NewServiceManager()
+			serviceManager := deploy.NewServiceManager()
 
 			env := map[string]string{}
 			env = config.PutServiceEnvironment(env, srv, versionToRun)
 
-			err := serviceManager.RemoveServicesFromCompose(context.Background(), deployToProfile, []string{srv}, env)
+			err := serviceManager.RemoveServicesFromCompose(
+				context.Background(),
+				deploy.NewServiceRequest(deployToProfile),
+				[]deploy.ServiceRequest{deploy.NewServiceRequest(srv)},
+				env)
 			if err != nil {
 				log.WithFields(log.Fields{
 					"profile":  deployToProfile,

cli/cmd/run.go

@@ -10,7 +10,7 @@ import (
 	"strings"
 
 	"github.com/elastic/e2e-testing/cli/config"
-	"github.com/elastic/e2e-testing/internal/compose"
+	"github.com/elastic/e2e-testing/internal/deploy"
 	log "github.com/sirupsen/logrus"
 
 	"github.com/spf13/cobra"
@@ -64,7 +64,7 @@ func buildRunServiceCommand(srv string) *cobra.Command {
 		Short: `Runs a ` + srv + ` service`,
 		Long:  `Runs a ` + srv + ` service, spinning up a Docker container for it and exposing its internal configuration so that you are able to connect to it in an easy manner`,
 		Run: func(cmd *cobra.Command, args []string) {
-			serviceManager := compose.NewServiceManager()
+			serviceManager := deploy.NewServiceManager()
 
 			env := config.PutServiceEnvironment(map[string]string{}, srv, versionToRun)
 
@@ -76,7 +76,8 @@ func buildRunServiceCommand(srv string) *cobra.Command {
 				env[k] = v
 			}
 
-			err := serviceManager.RunCompose(context.Background(), false, []string{srv}, env)
+			err := serviceManager.RunCompose(
+				context.Background(), false, []deploy.ServiceRequest{deploy.NewServiceRequest(srv)}, env)
 			if err != nil {
 				log.WithFields(log.Fields{
 					"service": srv,
@@ -96,7 +97,7 @@ Example:
   go run main.go run profile fleet -s elastic-agent:8.0.0-SNAPSHOT
 `,
 		Run: func(cmd *cobra.Command, args []string) {
-			serviceManager := compose.NewServiceManager()
+			serviceManager := deploy.NewServiceManager()
 
 			env := map[string]string{
 				"profileVersion": versionToRun,
@@ -110,14 +111,15 @@ Example:
 				env[k] = v
 			}
 
-			err := serviceManager.RunCompose(context.Background(), true, []string{key}, env)
+			err := serviceManager.RunCompose(
+				context.Background(), true, []deploy.ServiceRequest{deploy.NewServiceRequest(key)}, env)
 			if err != nil {
 				log.WithFields(log.Fields{
 					"profile": key,
 				}).Error("Could not run the profile.")
 			}
 
-			composeNames := []string{}
+			composeNames := []deploy.ServiceRequest{}
 			if len(servicesToRun) > 0 {
 				for _, srv := range servicesToRun {
 					arr := strings.Split(srv, ":")
@@ -137,10 +139,10 @@ Example:
 					}).Trace("Adding service")
 
 					env = config.PutServiceEnvironment(env, image, tag)
-					composeNames = append(composeNames, image)
+					composeNames = append(composeNames, deploy.NewServiceRequest(image))
 				}
 
-				err = serviceManager.AddServicesToCompose(context.Background(), key, composeNames, env)
+				err = serviceManager.AddServicesToCompose(context.Background(), deploy.NewServiceRequest(key), composeNames, env)
 				if err != nil {
 					log.WithFields(log.Fields{
 						"profile":  key,

cli/cmd/stop.go

@@ -8,7 +8,7 @@ import (
 	"context"
 
 	"github.com/elastic/e2e-testing/cli/config"
-	"github.com/elastic/e2e-testing/internal/compose"
+	"github.com/elastic/e2e-testing/internal/deploy"
 	log "github.com/sirupsen/logrus"
 
 	"github.com/spf13/cobra"
@@ -55,9 +55,10 @@ func buildStopServiceCommand(srv string) *cobra.Command {
 		Short: `Stops a ` + srv + ` service`,
 		Long:  `Stops a ` + srv + ` service, stoppping its Docker container`,
 		Run: func(cmd *cobra.Command, args []string) {
-			serviceManager := compose.NewServiceManager()
+			serviceManager := deploy.NewServiceManager()
 
-			err := serviceManager.StopCompose(context.Background(), false, []string{srv})
+			err := serviceManager.StopCompose(
+				context.Background(), false, []deploy.ServiceRequest{deploy.NewServiceRequest(srv)})
 			if err != nil {
 				log.WithFields(log.Fields{
 					"service": srv,
@@ -73,9 +74,10 @@ func buildStopProfileCommand(key string, profile config.Profile) *cobra.Command
 		Short: `Stops the ` + profile.Name + ` profile`,
 		Long:  `Stops the ` + profile.Name + ` profile, stopping the Services that compound it`,
 		Run: func(cmd *cobra.Command, args []string) {
-			serviceManager := compose.NewServiceManager()
+			serviceManager := deploy.NewServiceManager()
 
-			err := serviceManager.StopCompose(context.Background(), true, []string{key})
+			err := serviceManager.StopCompose(
+				context.Background(), true, []deploy.ServiceRequest{deploy.NewServiceRequest(key)})
 			if err != nil {
 				log.WithFields(log.Fields{
 					"profile": key,

cli/config/compose/profiles/fleet/docker-compose.yml

@@ -29,7 +29,7 @@ services:
       test: "curl -f http://localhost:5601/login | grep kbn-injected-metadata 2>&1 >/dev/null"
       retries: 600
       interval: 1s
-    image: "docker.elastic.co/${kibanaDockerNamespace:-beats}/kibana:${kibanaVersion:-7.x-SNAPSHOT}"
+    image: "docker.elastic.co/${kibanaDockerNamespace:-kibana}/kibana:${kibanaVersion:-7.x-SNAPSHOT}"
     ports:
       - "5601:5601"
     volumes:

cli/config/compose/services/centos-systemd/docker-compose.yml → cli/config/compose/services/elastic-agent/centos/docker-compose.yml

@@ -1,6 +1,6 @@
 version: '2.4'
 services:
-  centos-systemd:
+  elastic-agent:
     image: centos/systemd:${centos_systemdTag:-latest} 
     container_name: ${centos_systemdContainerName}
     entrypoint: "/usr/sbin/init"

cli/config/compose/services/debian-systemd/docker-compose.yml → cli/config/compose/services/elastic-agent/debian/docker-compose.yml

@@ -1,6 +1,6 @@
 version: '2.4'
 services:
-  debian-systemd:
+  elastic-agent:
     image: alehaa/debian-systemd:${debian_systemdTag:-stretch}
     container_name: ${debian_systemdContainerName} 
     entrypoint: "/sbin/init"

cli/config/config.go

@@ -73,16 +73,14 @@ func FileExists(configFile string) (bool, error) {
 
 // GetComposeFile returns the path of the compose file, looking up the
 // tool's workdir
-func GetComposeFile(isProfile bool, composeName string, composeFileName ...string) (string, error) {
-	if isProfile || composeFileName == nil || composeFileName[0] == "" {
-		composeFileName = []string{"docker-compose.yml"}
-	}
+func GetComposeFile(isProfile bool, composeName string) (string, error) {
+	composeFileName := "docker-compose.yml"
 	serviceType := "services"
 	if isProfile {
 		serviceType = "profiles"
 	}
 
-	composeFilePath := path.Join(Op.Workspace, "compose", serviceType, composeName, composeFileName[0])
+	composeFilePath := path.Join(Op.Workspace, "compose", serviceType, composeName, composeFileName)
 	found, err := io.Exists(composeFilePath)
 	if found && err == nil {
 		log.WithFields(log.Fields{

cli/config/kubernetes/overlays/deployment.yaml

@@ -0,0 +1,25 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: elastic-agent
+  labels:
+    app: elastic-agent
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: elastic-agent
+  template:
+    metadata:
+      labels:
+        app: elastic-agent
+    spec:
+      containers:
+      - name: elastic-agent
+        image: centos/systemd:latest
+        command: ["/usr/sbin/init"]
+        securityContext:
+          allowPrivilegeEscalation: true
+          runAsUser: 0
+          capabilities:
+            add: ["SYS_ADMIN"]

cli/config/kubernetes/overlays/kustomization.yaml

@@ -0,0 +1,5 @@
+bases:
+  - ../../base
+
+resources:
+- deployment.yaml

e2e/_suites/fleet/features/backend_processes.feature

@@ -6,8 +6,8 @@ Feature: Backend Processes
 Scenario Outline: Deploying the <os> agent
   Given a "<os>" agent is deployed to Fleet with "tar" installer
   When the "elastic-agent" process is in the "started" state on the host
-  Then there are "1" instances of the "filebeat" process in the "started" state
-    And there are "1" instances of the "metricbeat" process in the "started" state
+  Then there are "2" instances of the "filebeat" process in the "started" state
+    And there are "2" instances of the "metricbeat" process in the "started" state
 
 @centos
 Examples: Centos
@@ -21,10 +21,10 @@ Examples: Debian
 
 @enroll
 Scenario Outline: Deploying the <os> agent with enroll and then run on rpm and deb
-  Given a "<os>" agent is deployed to Fleet with "systemd" installer
+  Given a "<os>" agent is deployed to Fleet
   When the "elastic-agent" process is in the "started" state on the host
-  Then there are "1" instances of the "filebeat" process in the "started" state
-    And there are "1" instances of the "metricbeat" process in the "started" state
+  Then there are "2" instances of the "filebeat" process in the "started" state
+    And there are "2" instances of the "metricbeat" process in the "started" state
 
 @centos
 Examples: Centos
@@ -57,8 +57,8 @@ Examples: Debian
 Scenario Outline: Restarting the installed <os> agent
   Given a "<os>" agent is deployed to Fleet with "tar" installer
   When the "elastic-agent" process is "restarted" on the host
-  Then there are "1" instances of the "filebeat" process in the "started" state
-    And there are "1" instances of the "metricbeat" process in the "started" state
+  Then there are "2" instances of the "filebeat" process in the "started" state
+    And there are "2" instances of the "metricbeat" process in the "started" state
 
 @centos
 Examples: Centos
@@ -75,8 +75,8 @@ Scenario Outline: Restarting the <os> host with persistent agent restarts backen
   Given a "<os>" agent is deployed to Fleet with "tar" installer
   When the host is restarted
   Then the "elastic-agent" process is in the "started" state on the host
-    And there are "1" instances of the "filebeat" process in the "started" state
-    And there are "1" instances of the "metricbeat" process in the "started" state
+    And there are "2" instances of the "filebeat" process in the "started" state
+    And there are "2" instances of the "metricbeat" process in the "started" state
 
 @centos
 Examples: Centos

e2e/_suites/fleet/features/fleet_mode_agent.feature

@@ -21,7 +21,7 @@ Examples: Debian
 
 @enroll
 Scenario Outline: Deploying the <os> agent with enroll and then run on rpm and deb
-  Given a "<os>" agent is deployed to Fleet with "systemd" installer
+  Given a "<os>" agent is deployed to Fleet
   When the "elastic-agent" process is in the "started" state on the host
   Then the agent is listed in Fleet as "online"
     And system package dashboards are listed in Fleet