✨ Add upgrade e2e tests #299
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #299 +/- ##
=======================================
Coverage 33.53% 33.53%
=======================================
Files 15 15
Lines 656 656
=======================================
Hits 220 220
Misses 413 413
Partials 23 23 ☔ View full report in Codecov by Sentry. |
test/tools/imageregistry/registry.sh
Outdated
| # Install cert-manager | ||
| kubectl apply -f "https://github.com/cert-manager/cert-manager/releases/download/v1.13.1/cert-manager.yaml" | ||
| kubectl apply -f "https://github.com/cert-manager/cert-manager/releases/download/${CERT_MGR_VERSION}/cert-manager.yaml" |
There was a problem hiding this comment.
I think we can get rid of this completely. Please take a look: #300
.github/workflows/upgrade-e2e.yaml
Outdated
| - main | ||
|
|
||
| jobs: | ||
| upgrade-e2e: |
There was a problem hiding this comment.
Can we have this job in .github/workflows/e2e.yaml? It will be easier to consistently manage on triggers if we decide to change something.
Makefile
Outdated
| @@ -106,6 +106,28 @@ verify: tidy fmt vet generate ## Verify the current code generation and lint | |||
| lint: $(GOLANGCI_LINT) ## Run golangci linter. | |||
| $(GOLANGCI_LINT) run $(GOLANGCI_LINT_ARGS) | |||
|
|
|||
| .PHONY: test-upgrade-e2e | |||
| test-upgrade-e2e: kind-cluster cert-manager build-container kind-load image-registry run-latest-release wait pre-upgrade-setup post-upgrade-checks deploy wait wait-cert ## Run upgrade e2e tests on a local kind cluster | |||
There was a problem hiding this comment.
Why do we have post-upgrade-checks here (after pre-upgrade-setup) and below?
Makefile
Outdated
| .PHONY: wait-cert | ||
| wait-cert: | ||
| kubectl wait --for=condition=Ready --namespace=olmv1-system certificate/catalogd-catalogserver-cert # Avoid upgrade test flakes when reissuing cert |
There was a problem hiding this comment.
Is wait-cert specific for upgrades? Or should it be included here?
Lines 186 to 187 in 7c0bcb4
| # delete any pods persisting from earlier tests | ||
| kubectl delete --ignore-not-found --wait -f test/tools/imageregistry/compare-catalog.pod.yaml | ||
| kubectl create -f test/tools/imageregistry/compare-catalog.pod.yaml | ||
| # wait till pod terminates | ||
| kubectl wait -f test/tools/imageregistry/compare-catalog.pod.yaml --for=jsonpath='{.status.containerStatuses[0].state.terminated}' | ||
| kubectl logs --all-containers=true -f -n catalogd-e2e catalogd-compare |
There was a problem hiding this comment.
Should we consider using kubectl port-forward for this? Sounds like it is going to be easier to request the catalog: no need for a temporary pod.
| containers: | ||
| - image: radial/busyboxplus:curl | ||
| imagePullPolicy: IfNotPresent | ||
| command: ["sh", "-c", "curl -k https://catalogd-catalogserver.olmv1-system.svc.cluster.local/catalogs/test-catalog/all.json | diff build-contents/expected_all.json -"] |
There was a problem hiding this comment.
Would be good to get rid of hardcoded test-catalog
Signed-off-by: Ankita Thomas <ankithom@redhat.com>
|
|
||
| .PHONY: compare-catalog | ||
| compare-catalog: | ||
| test/tools/imageregistry/compare-catalog.sh # compare test catalog with expected contents |
There was a problem hiding this comment.
Could the actual upgrade comparisons be written in a separate Go test suite instead of using a shell script?
I think this would make it easier to communicate with the catalogd HTTP server and be easier to extend the test suite in the future.
There was a problem hiding this comment.
@ankitathomas In operator-framework/operator-controller#1003 I started with simple bash, but we ended up extending checks there so I replacing post-upgrade-checks with go test.
You might want to take a similar approach.
I think this would make it easier to communicate with the catalogd HTTP server and be easier to extend the test suite in the future.
We run go test from outside of the cluster. Instead of kubectl port-forward & curl which I suggested earlier we will have to proxy somehow with client-go. We cam probably take some inspiration from kubectl port-forward
|
Closing in favour of #314 |
Description
Testing upgrade from the latest release to the current commit.
Fixes #271