This doc gives short instructions how to run e2e tests. For the developing e2e tests, please refer to Developing E2E tests.
-
Make sure that
make
is availableTo install make:
- Ubuntu
sudo apt install build-essential
- Centos
sudo yum groupinstall "Development Tools"
-
Make sure that 'docker run' works without sudo, you can check it with
docker ps
Else add the user to
docker
group:sudo groupadd docker && sudo usermod -aG docker $USER && newgrp docker
Run e2e tests with
make test-e2e
After a finished test, cleanup is performed automatically. If the test setup failed or the test was interrupted, you can perform cleanup manually:
make clean -C /opt/metal3-dev-env/metal3-dev-env/
sudo rm -rf /opt/metal3-dev-env/
The e2e tests currently include three different sets:
- Pivoting based feature tests
- Remediation based feature tests
- clusterctl upgrade tests
- K8s upgrade tests
- Live ISO test
Because these tests run mainly in the target cluster, they are dependent on the pivoting test and need to run in the following order:
- Pivoting
- Certificate rotation
- Node reuse
- Re-pivoting
However, in case we need to run them in the ephemeral cluster pivoting and re-pivoting should be ignored.
Independent from the previous tests and can run independently includes:
- Remediation
- Inspection¹
- Metal3Remediation
¹ Inspection is actually run in the middle of the remediation test for practical reasons at the moment.
The ephemeral cluster is first launched using metal3-dev-env. The remediation, inspection and Metal3Remediation tests are then run with the controllers still in the ephemeral cluster either before pivoting or after re-pivoting.
- Upgrade BMO
- Upgrade Ironic
- Upgrade CAPI/CAPM3
tests | CAPM3 from | CAPM3 to | CAPI from | CAPI to |
---|---|---|---|---|
v1.7=>current | v1.7.0 | main | v1.7.3 | latest release |
v1.6=>current | v1.6.1 | main | v1.6.3 | latest release |
v1.5=>current | v1.5.3 | main | v1.5.7 | latest release |
Kubernetes version upgrade in target nodes. We run three latest version upgrades in k8s-upgrade tests for main branch and one kubernetes upgrade version for each release branch. When a new Kubernetes minor release is available, we will try to support it in an upcoming CAPM3 patch release (only in the latest supported CAPM3 minor release).
For example:
Main branch k8s-upgrade tests:
v1.27
=>v1.28
v1.28
=>v1.29
v1.29
=>v1.30
Release 1.7 branch k8s-upgrade test:
v1.29
=>v1.30
Release 1.6 branch k8s-upgrade test:
v1.28
=>v1.29
Release 1.5 branch k8s-upgrade test:
v1.26
=>v1.27
When Kubernetes 1.31 is released, k8s-upgrade v1.30
=> v1.31
will be
supported in v1.7.x (but not in v1.6.x)
Independent from the previous tests and can run independently. This is testing the booting of target cluster's nodes with the live ISO.
- Tests should be defined in a new file and separate test spec, unless the new test depends on existing tests.
- Tests are categorized with a custom label that can be used to filter a set of
E2E tests to be run. To run a subset of tests, a combination of either one or
both of the
GINKGO_FOCUS
andGINKGO_SKIP
env variables can be set. The labels are defined in the description of theDescribe
container between[]
, for example:[clusterctl-upgrade]
=> runs only existing upgrade tests including CAPI, CAPM3, Ironic and Baremetal Operator.[remediation]
=> runs only remediation and inspection tests.[live-iso]
=> runs only live ISO test.[k8s-upgrade]
=> runs only k8s upgrade tests.
For instance, to skip the upgrade E2E tests set GINKGO_SKIP="[upgrade]"
GINKGO_FOCUS
and GINKGO_SKIP
are defined in the Makefile, and should be
initialized in the CI JJBs to select the target e2e tests.
Both Ironic and BMO upgrade tests currently only check that a new version can be
rolled out (by going from latest
to main
). However, they do not actually
upgrade from some older release to a newer since they are not yet integrated
with the e2e upgrade test. The idea is to move them to the e2e upgrade test and
then upgrade Ironic and BMO together with CAPM3 from the previous minor release
to the latest.
Current e2e tests use the following Kubernetes versions for source and target clusters:
tests | bootstrap cluster | metal3 cluster init | metal3 cluster final |
---|---|---|---|
integration | v1.29.0 | v1.29.0 | x |
remediation | v1.29.0 | v1.29.0 | x |
pivot based feature | v1.29.0 | v1.28.1 | v1.29.0 |
upgrade | v1.29.0 | v1.28.1 | v1.29.0 |