Skip to content
This repository has been archived by the owner on Aug 19, 2024. It is now read-only.

Commit

Permalink
Add E2E tests against our examples on real clusters
Browse files Browse the repository at this point in the history 
- Do not error out when deleting a non-existing namespace

- Stream command output to the GinkgoWriter in real-time as well

This allows following what happens when calling potentially long-running commands

- Implement airgap test mode

- Ignore error when creating a namespace that already exists

- Allow to use existing mirror registry in airgap scenario

- Extract constants for test modes

- Add documentation

- Find an easier way to determine the IMG variable, using the Makefile

- Add more examples to README.md

- Add note about clusters with hosted control planes
  • Loading branch information
@rm3l
rm3l committed Feb 17, 2024
1 parent 813c05c commit 0391172
Show file tree
Hide file tree
Showing 9 changed files with 899 additions and 2 deletions.
2 changes: 2 additions & 0 deletions .rhdh/docs/airgap.adoc
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
==== Installing Red Hat Developer Hub (RHDH) in restricted environments

WARNING: The procedure below will not work properly on hosted clusters like link:https://hypershift-docs.netlify.app/[HyperShift] or link:https://www.redhat.com/en/blog/red-hat-openshift-service-aws-hosted-control-planes-now-available[ROSA with hosted control planes] due to a limitation preventing link:https://docs.openshift.com/container-platform/4.14/rest_api/operator_apis/imagecontentsourcepolicy-operator-openshift-io-v1alpha1.html[`ImageContentSourcePolicy`] from being propagated to the cluster nodes. There is currently no workaround for these clusters.

On an OpenShift cluster operating in a restricted network, public resources are not available.
However, deploying the RHDH Operator and running RHDH requires the following public resources:

Expand Down
2 changes: 2 additions & 0 deletions .rhdh/docs/installing-ci-builds.adoc
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
== Installing CI builds of Red Hat Developer Hub

WARNING: The procedure below will not work properly on hosted clusters like link:https://hypershift-docs.netlify.app/[HyperShift] or link:https://www.redhat.com/en/blog/red-hat-openshift-service-aws-hosted-control-planes-now-available[ROSA with hosted control planes] due to a limitation preventing link:https://docs.openshift.com/container-platform/4.14/rest_api/operator_apis/imagecontentsourcepolicy-operator-openshift-io-v1alpha1.html[`ImageContentSourcePolicy`] from being propagated to the cluster nodes. There is currently no workaround for these clusters.

*Prerequisites*

* You are logged in as an administrator on the OpenShift web console.
Expand Down
37 changes: 36 additions & 1 deletion Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ VERSION ?= 0.1.0-dev
# Using docker or podman to build and push images
CONTAINER_ENGINE ?= docker

PKGS := $(shell go list ./... | grep -v /tests)

# CHANNELS define the bundle channels used in the bundle.
# Add a new line here if you would like to change its default config. (E.g CHANNELS = "candidate,fast,stable")
# To re-generate a bundle for other specific channels without changing the standard setup, you can:
Expand Down Expand Up @@ -127,7 +129,7 @@ vet: ## Run go vet against code.
.PHONY: test
test: manifests generate fmt vet envtest ## Run tests. We need LOCALBIN=$(LOCALBIN) to get correct default-config path
mkdir -p $(LOCALBIN)/default-config && cp config/manager/$(CONF_DIR)/* $(LOCALBIN)/default-config
LOCALBIN=$(LOCALBIN) KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" go test ./... -coverprofile cover.out
LOCALBIN=$(LOCALBIN) KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" go test $(PKGS) -coverprofile cover.out

##@ Build

Expand Down Expand Up @@ -227,6 +229,9 @@ GOSEC_VERSION ?= v2.18.2
GOSEC_FMT ?= sarif # for other options, see https://github.com/securego/gosec#output-formats
GOSEC_OUTPUT_FILE ?= gosec.sarif

GINKGO ?= $(LOCALBIN)/ginkgo
GINKGO_VERSION ?= v2.9.5

KUSTOMIZE_INSTALL_SCRIPT ?= "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"
.PHONY: kustomize
kustomize: $(KUSTOMIZE) ## Download kustomize locally if necessary.
Expand Down Expand Up @@ -365,3 +370,33 @@ catalog-update: ## Update catalog source in the default namespace for catalogsou
.PHONY: deploy-openshift
deploy-openshift: release-build release-push catalog-update ## Deploy the operator on openshift cluster

# After this time, Ginkgo will emit progress reports, so we can get visibility into long-running tests.
POLL_PROGRESS_INTERVAL := 120s
TIMEOUT ?= 14400s

GINKGO_FLAGS_ALL = $(GINKGO_TEST_ARGS) --randomize-all --poll-progress-after=$(POLL_PROGRESS_INTERVAL) --poll-progress-interval=$(POLL_PROGRESS_INTERVAL) -timeout $(TIMEOUT) --no-color

# Flags for tests that may be run in parallel
GINKGO_FLAGS=$(GINKGO_FLAGS_ALL) -nodes=$(TEST_EXEC_NODES)
# Flags to run one test per core.
GINKGO_FLAGS_AUTO = $(GINKGO_FLAGS_ALL) -p
ifdef TEST_EXEC_NODES
TEST_EXEC_NODES := $(TEST_EXEC_NODES)
else
TEST_EXEC_NODES := 1
endif

.PHONY: ginkgo
ginkgo: $(GINKGO) ## Download Ginkgo locally if necessary.
$(GINKGO): $(LOCALBIN)
test -s $(LOCALBIN)/ginkgo || GOBIN=$(LOCALBIN) go install github.com/onsi/ginkgo/v2/ginkgo@$(GINKGO_VERSION)

.PHONY: test-e2e
test-e2e: ginkgo ## Run end-to-end tests. See the 'tests/e2e/README.md' file for more details.
$(GINKGO) $(GINKGO_FLAGS) tests/e2e

show-img:
@echo -n $(IMG)

show-container-engine:
@echo -n $(CONTAINER_ENGINE)
2 changes: 1 addition & 1 deletion examples/rhdh-cr-with-app-configs.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
apiVersion: rhdh.redhat.com/v1alpha1
kind: Backstage
metadata:
name: my-backstage-app-with-app-config
name: bs-app-config
spec:
database:
enableLocalDb: true
Expand Down
97 changes: 97 additions & 0 deletions tests/e2e/README.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
## End-to-end tests

The end-to-end tests use the [Ginkgo framework](https://onsi.github.io/ginkgo/) and allow to test the operator against a real cluster in the following scenarios:
- building and deploying the operator image off of the current code
- using a specific image or a specific downstream build

Deployment of the operator itself can be done by:
- deploying with or without OLM,
- or deploying the downstream bundle in both online and air-gapped scenarios

To run the end-to-end tests, you can use:
```shell
$ make test-e2e
```

### Configuration

The behavior is configurable using the following environment variables:

| Name | Type | Description | Default value | Example |
|------------------------------------------------------------------------------------------------|--------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------|---------------------------------------------------------|
| `BACKSTAGE_OPERATOR_TEST_MODE` | string | The test mode:<br>- if not set, it will call `make deploy`<br>- `olm`: it will call `make deploy-olm`<br>- `rhdh-latest` or `rhdh-next`: it will install the operator using the [`install-rhdh-catalog-source.sh`](../../.rhdh/scripts/install-rhdh-catalog-source.sh) script<br>- `rhdh-airgap`: it will install the operator using the [`prepare-restricted-environment.sh`](../../.rhdh/scripts/prepare-restricted-environment.sh) script. | | `rhdh-latest` |
| `IMG` (or any variables from the Makefile that are used by `make deploy` or `make deploy-olm`) | string | The image to use. Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm` | `VERSION` defined in [`Makefile`](../../Makefile) | `quay.io/janus-idp/operator:0.0.1-latest` |
| `BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES` | bool | If set to `true`, it will build the operator image with `make image-build`.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm`. | | `false` |
| `BACKSTAGE_OPERATOR_TESTS_PUSH_IMAGES` | bool | If set to `true`, it will push the operator image with `make image-push`.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm`. | | `false` |
| `BACKSTAGE_OPERATOR_TESTS_PLATFORM` | string | The platform type, to directly load the operator image if supported instead of pushing it.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is not set or set to `olm`.br>Supported values: [`kind`](#building-and-testing-local-changes-on-kind), [`k3d`](#building-and-testing-local-changes-on-k3d), [`minikube`](#building-and-testing-local-changes-on-minikube) | | `kind` |
| `BACKSTAGE_OPERATOR_TESTS_KIND_CLUSTER` | string | Name of the local KinD cluster to use. Relevant only if `BACKSTAGE_OPERATOR_TESTS_PLATFORM` is `kind`. | `kind` | `kind-local-k8s-cluster` |
| `BACKSTAGE_OPERATOR_TESTS_K3D_CLUSTER` | string | Name of the local k3d cluster to use. Relevant only if `BACKSTAGE_OPERATOR_TESTS_PLATFORM` is `k3d`. | `k3s-default` | `k3d-local-k8s-cluster` |
| `BACKSTAGE_OPERATOR_TESTS_AIRGAP_INDEX_IMAGE` | string | Index image to use in the airgap scenario.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is `rhdh-airgap`. | `quay.io/rhdh/iib:latest-v4.14-x86_64` | `registry.redhat.io/redhat/redhat-operator-index:v4.14` |
| `BACKSTAGE_OPERATOR_TESTS_AIRGAP_OPERATOR_VERSION` | string | Operator version to use in the airgap scenario.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is `rhdh-airgap`. | `v1.1.0` | `v1.1.0` |
| `BACKSTAGE_OPERATOR_TESTS_AIRGAP_MIRROR_REGISTRY` | string | Existing mirror registry to use in the airgap scenario.<br>Relevant if `BACKSTAGE_OPERATOR_TEST_MODE` is `rhdh-airgap`<br>. | | `my-registry.example.com` |

### Examples

#### Testing a specific version

This should work on any Kubernetes cluster:

```shell
$ make test-e2e VERSION=0.0.1-latest
```

#### Building and testing local changes on [kind](https://kind.sigs.k8s.io/)

```shell
$ kind create cluster
$ make test-e2e BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES=true BACKSTAGE_OPERATOR_TESTS_PLATFORM=kind
```

#### Building and testing local changes on [k3d](https://k3d.io/)

```shell
$ k3d cluster create
$ make test-e2e BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES=true BACKSTAGE_OPERATOR_TESTS_PLATFORM=k3d
```

#### Building and testing local changes on [minikube](https://minikube.sigs.k8s.io/docs/)

```shell
$ minikube start
$ make test-e2e BACKSTAGE_OPERATOR_TESTS_BUILD_IMAGES=true BACKSTAGE_OPERATOR_TESTS_PLATFORM=minikube
```

#### Testing a specific image (e.g. PR image)

```shell
$ make test-e2e IMG=quay.io/janus-idp/operator:0.0.1-pr-201-7d08c24
```

#### Testing a specific version using OLM

This requires the [Operator Lifecycle Manager (OLM)](https://olm.operatorframework.io/) to be installed in the cluster:

```shell
$ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=olm
```

#### Testing a downstream build of RHDH

This requires an OpenShift cluster. If testing a CI build, please follow the instructions in [Installing CI builds of Red Hat Developer Hub](../../.rhdh/docs/installing-ci-builds.adoc) to add your Quay token to the cluster.

```shell
# latest
$ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=rhdh-latest

# or next
$ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=rhdh-next
```

#### Airgap testing of RHDH

This requires an OpenShift cluster.
Please also read the prerequisites in [Installing Red Hat Developer Hub (RHDH) in restricted environments](../../.rhdh/docs/airgap.adoc).

```shell
$ make test-e2e BACKSTAGE_OPERATOR_TEST_MODE=rhdh-airgap
```

0 comments on commit 0391172

Please sign in to comment.