Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding e2e make target, makefile adjustments, and e2e test framework #81

Merged
merged 6 commits into from
Jan 5, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .github/workflows/e2e.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: e2e

on:
workflow_dispatch:
pull_request:
push:
branches:
- main

jobs:
e2e-kind:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: actions/setup-go@v3
with:
go-version-file: "go.mod"
- name: Run e2e tests
run: |
make e2e
59 changes: 52 additions & 7 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,6 +1,15 @@

###########################
# Configuration Variables #
###########################
# Image URL to use all building/pushing image targets
IMG ?= controller:latest
export IMAGE_REPO ?= quay.io/operator-framework/operator-controller
export IMAGE_TAG ?= devel
export GO_BUILD_TAGS ?= upstream
IMG?=$(IMAGE_REPO):$(IMAGE_TAG)

OPERATOR_CONTROLLER_NAMESPACE ?= operator-controller-system
KIND_CLUSTER_NAME ?= operator-controller

# ENVTEST_K8S_VERSION refers to the version of kubebuilder assets to be downloaded by envtest binary.
ENVTEST_K8S_VERSION = 1.25.0
joelanford marked this conversation as resolved.
Show resolved Hide resolved

Expand All @@ -16,6 +25,9 @@ endif
SHELL = /usr/bin/env bash -o pipefail
.SHELLFLAGS = -ec

# Disable -j flag for make
.NOTPARALLEL:

.PHONY: all
all: build

Expand Down Expand Up @@ -54,25 +66,46 @@ fmt: ## Run go fmt against code.
vet: ## Run go vet against code.
go vet ./...

.PHONY: test
test: manifests generate fmt vet envtest ## Run tests.
.PHONY: test test-e2e e2e kind-load kind-cluster kind-cluster-cleanup
test: manifests generate fmt vet envtest test-e2e ## Run all tests.
KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" go test ./... -coverprofile cover.out

FOCUS := $(if $(TEST),-v -focus "$(TEST)")
E2E_FLAGS ?= ""
test-e2e: ginkgo ## Run the e2e tests
$(GINKGO) --tags $(GO_BUILD_TAGS) $(E2E_FLAGS) -trace -progress $(FOCUS) test/e2e

e2e: KIND_CLUSTER_NAME=operator-controller-e2e
e2e: run test-e2e kind-cluster-cleanup ## Run e2e test suite on local kind cluster
joelanford marked this conversation as resolved.
Show resolved Hide resolved

kind-load: kind ## Loads the currently constructed image onto the cluster
$(KIND) load docker-image $(IMG) --name $(KIND_CLUSTER_NAME)

kind-cluster: kind kind-cluster-cleanup ## Standup a kind cluster
$(KIND) create cluster --name ${KIND_CLUSTER_NAME}
$(KIND) export kubeconfig --name ${KIND_CLUSTER_NAME}

kind-cluster-cleanup: kind ## Delete the kind cluster
$(KIND) delete cluster --name ${KIND_CLUSTER_NAME}
Comment on lines +81 to +89
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Looking at consistency, we might want to rename these. If we wanted to go with k8s' V-O verbiage:
e.g.
kind-cluster -> kind-create-cluster
kind-cluster-cleanup -> kind-delete-cluster
kind-load -> kind-load-image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with your reasoning for these, but I also named them this way for the sake of consistency with rukpak, so we'd need to change them in both places if we also wanted to be internally consistent. If it's alright with you, I'd prefer to leave these at least for now.


##@ Build

.PHONY: build
build: manifests generate fmt vet ## Build manager binary.
go build -o bin/manager main.go

.PHONY: run
run: manifests generate fmt vet ## Run a controller from your host.
go run ./main.go
run: docker-build kind-cluster kind-load install deploy wait ## Build the operator-controller then deploy it into a new kind cluster.
dtfranz marked this conversation as resolved.
Show resolved Hide resolved

.PHONY: wait
wait:
kubectl wait --for=condition=Available --namespace=$(OPERATOR_CONTROLLER_NAMESPACE) deployment/operator-controller-controller-manager --timeout=60s

# If you wish built the manager image targeting other platforms you can use the --platform flag.
# (i.e. docker build --platform linux/arm64 ). However, you must enable docker buildKit for it.
# More info: https://docs.docker.com/develop/develop-images/build_enhancements/
.PHONY: docker-build
docker-build: test ## Build docker image with the manager.
docker-build: generate ## Build docker image with the operator-controller.
docker build -t ${IMG} .

.PHONY: docker-push
Expand Down Expand Up @@ -129,12 +162,24 @@ $(LOCALBIN):
## Tool Binaries
KUSTOMIZE ?= $(LOCALBIN)/kustomize
CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen
KIND ?= $(LOCALBIN)/kind
GINKGO ?= $(LOCALBIN)/ginkgo
ENVTEST ?= $(LOCALBIN)/setup-envtest

## Tool Versions
KUSTOMIZE_VERSION ?= v4.5.7
CONTROLLER_TOOLS_VERSION ?= v0.10.0

.PHONY: kind
kind: $(KIND) ## Download kind locally if necessary.
$(KIND): $(LOCALBIN)
test -s $(LOCALBIN)/kind || GOBIN=$(LOCALBIN) go install sigs.k8s.io/kind@v0.15.0

.PHONY: ginkgo
ginkgo: $(GINKGO) ## Download ginkgo locally if necessary.
$(GINKGO): $(LOCALBIN)
test -s $(LOCALBIN)/ginkgo || GOBIN=$(LOCALBIN) go install github.com/onsi/ginkgo/v2/ginkgo@v2.1.4
Comment on lines +176 to +181
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test -s $(LOCALBIN)/<binName> tests are sufficient for CI because they'll always fail and we'll always get the correct version.

But locally, these can be annoying when we bump the version of one of these and you still have an old version from an earlier commit. If there's some diff in behavior between your local version and the one specified here, you'll be scratching your head trying to figure out why.

There are two solutions I've seen for this:

  1. Rukpak has a separate tools go module with a separate go.mod that lists all the tool versions with make targets that always go build
  2. A somewhat curated script that knows how to check the presence and version of the existing binary to see if a new one is needed called like this. It's mostly set and forget.

Option 1 is good if all of our tools are Go-based
Option 2 is probably slightly faster when actually using the tools, but requires some tweaks every now and then when a tool changes its release artifacts.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be a follow-up for sure.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did notice number 1 there in rukpak's makefile. I initially was working with that but it ended up turning into a bit of a rabbit-hole with the amount of changes needed, since it appeared as though it would require re-working every built-in make target we got from kubebuilder, and then at that point it felt like I was going to end up with just a copy-paste of rukpak's makefile which would require re-testing all of the targets. We definitely need to take care of it at some point, if you think it's a good thing to take care of now I could give it another go. It just felt like it was getting a bit out-of-scope to me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've created issue #85 to track this.


KUSTOMIZE_INSTALL_SCRIPT ?= "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh"
.PHONY: kustomize
kustomize: $(KUSTOMIZE) ## Download kustomize locally if necessary. If wrong version is installed, it will be removed before downloading.
Expand Down
6 changes: 6 additions & 0 deletions config/manager/kustomization.yaml
Original file line number Diff line number Diff line change
@@ -1,2 +1,8 @@
resources:
- manager.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
images:
- name: controller
newName: quay.io/operator-framework/operator-controller
newTag: devel
1 change: 1 addition & 0 deletions config/manager/manager.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ spec:
args:
- --leader-elect
image: controller:latest
imagePullPolicy: IfNotPresent
name: manager
securityContext:
allowPrivilegeEscalation: false
Expand Down
38 changes: 38 additions & 0 deletions test/e2e/e2e_suite_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
package e2e

import (
"testing"
"time"

. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/client-go/rest"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"

operatorv1alpha1 "github.com/operator-framework/operator-controller/api/v1alpha1"
)

var (
cfg *rest.Config
c client.Client
)

func TestE2E(t *testing.T) {
RegisterFailHandler(Fail)
SetDefaultEventuallyTimeout(1 * time.Minute)
SetDefaultEventuallyPollingInterval(1 * time.Second)
RunSpecs(t, "E2E Suite")
}

var _ = BeforeSuite(func() {
cfg = ctrl.GetConfigOrDie()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could these tests go under suite_test.go? Just because we have the test framework already scaffolded by the tool?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just saw Joe's comment #81 (comment), please feel free to ignore my review :)

Only one nit (we could do in a follow up), we could use envtest even for the unit (case 1) tests (though it is mainly intended for integration) if we foresee the interaction of multiple individual components, and would have to test "Reconcile" anyway. This in the case (1) where we need deppy entity source, and essentially mock the functioning of controller. This is just because previously we had faced version incompatibility issues by using locally available binaries and had a hard time figuring the cause.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Varsha! I have to admit, my knowledge on the envtest stuff is lacking here, so if it's alright with you I'd like to reserve that for a follow up.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries, I have created an issue to follow up #86. We can start using envtest when we get to writing tests for reconciler depending on the complexity of those. Till then its good :)


scheme := runtime.NewScheme()
err := operatorv1alpha1.AddToScheme(scheme)
Expect(err).To(Not(HaveOccurred()))

c, err = client.New(cfg, client.Options{Scheme: scheme})
Expect(err).To(Not(HaveOccurred()))
})
46 changes: 46 additions & 0 deletions test/e2e/reconcile_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
package e2e

import (
"context"
"fmt"

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/rand"

. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
operatorv1alpha1 "github.com/operator-framework/operator-controller/api/v1alpha1"
)

var _ = Describe("Operator Install", func() {
When("a valid Operator CR specifying a package", func() {
var (
operator *operatorv1alpha1.Operator
ctx context.Context
err error
)
BeforeEach(func() {
ctx = context.Background()

operator = &operatorv1alpha1.Operator{
ObjectMeta: metav1.ObjectMeta{
Name: fmt.Sprintf("operator-%s", rand.String(8)),
},
Spec: operatorv1alpha1.OperatorSpec{
PackageName: "my-cool-package",
},
}
err = c.Create(ctx, operator)
Expect(err).To(Not(HaveOccurred()))
})
AfterEach(func() {
By("deleting the testing Operator resource")
err = c.Delete(ctx, operator)
Expect(err).To(Not(HaveOccurred()))
})
PIt("installs the specified package", func() {
// Pending until we actually have some code to test
// Expect that a CRD and Deployment were successfully installed by rukpak
})
})
})