This repository has been archived by the owner on Aug 19, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 14
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add E2E tests against our examples on real clusters
- Do not error out when deleting a non-existing namespace
- Stream command output to the GinkgoWriter in real-time as well
This allows following what happens when calling potentially long-running commands
- Implement airgap test mode
- Ignore error when creating a namespace that already exists
- Allow to use existing mirror registry in airgap scenario
- Extract constants for test modes
- Add documentation
- Find an easier way to determine the IMG variable, using the Makefile
- Add more examples to README.md
- Add note about clusters with hosted control planes
- Support k3d clusters
- Support Minikube clusters
- Load image into local clusters using an archive instead
This allows this logic to be agnostic to the container engine used to
build the image. We rely on the container image to export the image to
an archive ('{podman,docker} image save').
- Run E2E test nightly on main and release branch- Loading branch information
Showing
8 changed files
with
1,042 additions
and
2 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,82 @@ | ||
| name: Nightly checks | ||
|
|
||
| on: | ||
| # workflow_dispatch so that it can be triggered manually if needed | ||
| workflow_dispatch: | ||
| schedule: | ||
| - cron: "34 23 * * *" | ||
|
|
||
| jobs: | ||
| e2e-tests: | ||
| runs-on: ubuntu-latest | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| branch: [ main, 1.1.x ] | ||
| name: E2E Tests - ${{ matrix.branch }} | ||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ matrix.branch }} | ||
| cancel-in-progress: true | ||
| env: | ||
| CONTAINER_ENGINE: podman | ||
| steps: | ||
| - uses: actions/checkout@v4 # default branch will be checked out by default on scheduled workflows | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - if: ${{ matrix.branch != 'main' }} | ||
| name: Checkout ${{ matrix.branch }} branch | ||
| run: git switch ${{ matrix.branch }} | ||
|
|
||
| # check changes in this commit for regex include and exclude matches; pipe to an env var | ||
| - name: Check for changes to build | ||
| run: | | ||
| # don't fail if nothing returned by grep | ||
| set +e | ||
| CHANGES="$(git diff --name-only HEAD~1 | \ | ||
| grep -E "workflows/.+-container-build.yaml|Makefile|bundle/|config/|go.mod|go.sum|.+\.go|docker/|\.dockerignore" | \ | ||
| grep -v -E ".+_test.go|/.rhdh/")"; | ||
| echo "Changed files for this commit:" | ||
| echo "==============================" | ||
| echo "$CHANGES" | ||
| echo "==============================" | ||
| { | ||
| echo 'CHANGES<<EOF' | ||
| echo $CHANGES | ||
| echo EOF | ||
| } >> "$GITHUB_ENV" | ||
| - name: Determine built operator image | ||
| # run this stage only if there are changes that match the includes and not the excludes | ||
| if: ${{ env.CHANGES != '' }} | ||
| run: | | ||
| SHORT_SHA=$(git rev-parse --short HEAD) | ||
| BASE_VERSION=$(grep -E "^VERSION \?=" Makefile | sed -r -e "s/.+= //") # 0.1.0 | ||
| echo "OPERATOR_IMAGE=quay.io/janus-idp/operator:${BASE_VERSION}-${SHORT_SHA}" >> $GITHUB_ENV | ||
| - name: Wait until image exists in registry or timeout is reached | ||
| # run this stage only if there are changes that match the includes and not the excludes | ||
| if: ${{ env.CHANGES != '' }} | ||
| timeout-minutes: 10 | ||
| run: | | ||
| echo "Waiting until operator image is found or timeout expires: ${{ env.OPERATOR_IMAGE }}..." | ||
| until ${CONTAINER_ENGINE} image pull "${{ env.OPERATOR_IMAGE }}"; do | ||
| sleep 2 | ||
| echo ... | ||
| done | ||
| echo "... operator image found: ${{ env.OPERATOR_IMAGE }}." | ||
| - name: Start Minikube | ||
| # run this stage only if there are changes that match the includes and not the excludes | ||
| if: ${{ env.CHANGES != '' }} | ||
| uses: medyagh/setup-minikube@606b71970c783154fe49b711486c717f5780f485 # v0.0.15 | ||
| with: | ||
| addons: ingress | ||
|
|
||
| - name: Run E2E tests | ||
| # run this stage only if there are changes that match the includes and not the excludes | ||
| if: ${{ env.CHANGES != '' }} | ||
| env: | ||
| BACKSTAGE_OPERATOR_TESTS_PLATFORM: minikube | ||
| IMG: ${{ env.OPERATOR_IMAGE }} | ||
| run: make test-e2e |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| ## End-to-end tests | ||
|
|
||
| The end-to-end tests use the [Ginkgo framework](https://onsi.github.io/ginkgo/) and allow to test the operator against a real cluster in the following scenarios: | ||
| - building and deploying the operator image off of the current code | ||
| - using a specific image or a specific downstream build | ||
|
|
||
| Deployment of the operator itself can be done by: | ||
| - deploying with or without OLM, | ||
| - or deploying the downstream bundle in both online and air-gapped scenarios | ||
|
|
||
| To run the end-to-end tests, you can use: | ||
| ```shell | ||
| $ make test-e2e | ||
| ``` | ||
|
|
||
| ### Configuration | ||
|
|
||
| The behavior is configurable using the following environment variables: | ||
|
|
||
| | Name | Type | Description | Default value | Example | | ||
| |------------------------------------------------------------------------------------------------|--------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------|---------------------------------------------------------| | ||
| | `BACKSTAGE_OPERATOR_TEST_MODE` | string | The test mode:<br>- if not set, it will call `make deploy`<br>- `olm`: it will call `make deploy-olm`<br>- `rhdh-latest` or `rhdh-next`: it will install the operator using the [`install-rhdh-catalog-source.sh`](../../.rhdh/scripts/install-rhdh-catalog-source.sh) script<br>- `rhdh-airgap`: it will install the operator using the [`prepare-restricted-environment.sh`](../../.rhdh/scripts/prepare-restricted-environment.sh) script. | | `rhdh-latest` | | ||
| | `IMG` (or any variables from the Makefile that are used by `make deploy` or `make deploy-olm`) | string | The image to use. Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm` | `VERSION` defined in [`Makefile`](../../Makefile) | `quay.io/janus-idp/operator:0.0.1-latest` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES` | bool | If set to `true`, it will build the operator image with `make image-build`.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm`. | | `false` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_PUSH_IMAGES` | bool | If set to `true`, it will push the operator image with `make image-push`.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm`. | | `false` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_PLATFORM` | string | The platform type, to directly load the operator image if supported instead of pushing it.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm`.br>Supported values: [`kind`](#building-and-testing-local-changes-on-kind), [`k3d`](#building-and-testing-local-changes-on-k3d), [`minikube`](#building-and-testing-local-changes-on-minikube) | | `kind` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_KIND_CLUSTER` | string | Name of the local KinD cluster to use. Relevant only if `BACKSTAGE_OPERATOR_TESTS_PLATFORM` is `kind`. | `kind` | `kind-local-k8s-cluster` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_K3D_CLUSTER` | string | Name of the local k3d cluster to use. Relevant only if `BACKSTAGE_OPERATOR_TESTS_PLATFORM` is `k3d`. | `k3s-default` | `k3d-local-k8s-cluster` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_AIRGAP_INDEX_IMAGE` | string | Index image to use in the airgap scenario.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is `rhdh-airgap`. | `quay.io/rhdh/iib:latest-v4.14-x86_64` | `registry.redhat.io/redhat/redhat-operator-index:v4.14` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_AIRGAP_OPERATOR_VERSION` | string | Operator version to use in the airgap scenario.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is `rhdh-airgap`. | `v1.1.0` | `v1.1.0` | | ||
| | `BACKSTAGE_OPERATOR_TESTS_AIRGAP_MIRROR_REGISTRY` | string | Existing mirror registry to use in the airgap scenario.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is `rhdh-airgap`<br>. | | `my-registry.example.com` | | ||
|
|
||
| ### Examples | ||
|
|
||
| #### Testing a specific version | ||
|
|
||
| This should work on any Kubernetes cluster: | ||
|
|
||
| ```shell | ||
| $ make test-e2e VERSION=0.0.1-latest | ||
| ``` | ||
|
|
||
| #### Building and testing local changes on [kind](https://kind.sigs.k8s.io/) | ||
|
|
||
| ```shell | ||
| $ kind create cluster | ||
| $ make test-e2e BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES=true BACKSTAGE_OPERATOR_TESTS_PLATFORM=kind | ||
| ``` | ||
|
|
||
| #### Building and testing local changes on [k3d](https://k3d.io/) | ||
|
|
||
| ```shell | ||
| $ k3d cluster create | ||
| $ make test-e2e BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES=true BACKSTAGE_OPERATOR_TESTS_PLATFORM=k3d | ||
| ``` | ||
|
|
||
| #### Building and testing local changes on [minikube](https://minikube.sigs.k8s.io/docs/) | ||
|
|
||
| ```shell | ||
| $ minikube start | ||
| $ make test-e2e BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES=true BACKSTAGE_OPERATOR_TESTS_PLATFORM=minikube | ||
| ``` | ||
|
|
||
| #### Testing a specific image (e.g. PR image) | ||
|
|
||
| ```shell | ||
| $ make test-e2e IMG=quay.io/janus-idp/operator:0.0.1-pr-201-7d08c24 | ||
| ``` | ||
|
|
||
| #### Testing a specific version using OLM | ||
|
|
||
| This requires the [Operator Lifecycle Manager (OLM)](https://olm.operatorframework.io/) to be installed in the cluster: | ||
|
|
||
| ```shell | ||
| $ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=olm | ||
| ``` | ||
|
|
||
| #### Testing a downstream build of RHDH | ||
|
|
||
| This requires an OpenShift cluster. If testing a CI build, please follow the instructions in [Installing CI builds of Red Hat Developer Hub](../../.rhdh/docs/installing-ci-builds.adoc) to add your Quay token to the cluster. | ||
|
|
||
| ```shell | ||
| # latest | ||
| $ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=rhdh-latest | ||
|
|
||
| # or next | ||
| $ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=rhdh-next | ||
| ``` | ||
|
|
||
| #### Airgap testing of RHDH | ||
|
|
||
| This requires an OpenShift cluster. | ||
| Please also read the prerequisites in [Installing Red Hat Developer Hub (RHDH) in restricted environments](../../.rhdh/docs/airgap.adoc). | ||
|
|
||
| ```shell | ||
| $ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=rhdh-airgap | ||
| ``` |
Oops, something went wrong.