Create E2E test suite #26

adamhosier · 2019-10-16T13:34:16Z

this pr:

Adds an external service definition to allow us to reach the client port from outside the cluster. required for the e2e tests to dial the cluster to verify its health.
An e2e test which will
1. Start a Kind cluster
2. Deploy CRDs to this cluster
3. Build the operator image & load it to the kind cluster
4. Deploy the operator
5. Deploy the custom resources in example/cluster.yaml
6. Wait for an etcd cluster to come up

this pr does not:

Add the step to CI (coming soon)
Add any in-depth testing, just a broad health check

run this yourself with

go test ./internal/test/e2e --kind --repo-root=$(pwd) -v

adamhosier · 2019-10-16T13:50:35Z

see #7

examples/cluster.yaml

examples/operator.yaml

adamhosier · 2019-10-17T15:28:13Z

@JamesLaverack @munnerz i've pushed a commit here which consistently uses config/ for all resources. there's a couple fixes bundled which popped up during tests. PTAL

wallrj

@adamhosier This looks really neat.

I ran the e2e tests locally and got a failure. See below.
I added one or two nits in the comments below.
It might be nice to add a new Make target, either here or in a follow up branch.
Finally, it'd be nice to add some brief documentation for this, perhaps in the CONTRIBUTING.md doc for now.

 richard   feature/e2e-test  ~  projects  improbable-eng  etcd-cluster-operator  go test ./internal/test/e2e --kind --repo-root=$(pwd) -v 
go: downloading sigs.k8s.io/kind v0.5.1
go: downloading go.etcd.io/etcd v3.3.17+incompatible
go: extracting sigs.k8s.io/kind v0.5.1
go: downloading github.com/sirupsen/logrus v1.4.1
go: downloading sigs.k8s.io/kustomize/v3 v3.1.1-0.20190821175718-4b67a6de1296
go: extracting github.com/sirupsen/logrus v1.4.1
go: extracting sigs.k8s.io/kustomize/v3 v3.1.1-0.20190821175718-4b67a6de1296
go: downloading github.com/go-openapi/spec v0.19.2
go: downloading github.com/emicklei/go-restful v2.9.6+incompatible
go: extracting github.com/emicklei/go-restful v2.9.6+incompatible
go: extracting go.etcd.io/etcd v3.3.17+incompatible
go: extracting github.com/go-openapi/spec v0.19.2
go: downloading github.com/go-openapi/jsonreference v0.19.2
go: downloading github.com/go-openapi/jsonpointer v0.19.2
go: downloading github.com/go-openapi/swag v0.19.2
go: extracting github.com/go-openapi/jsonreference v0.19.2
go: downloading github.com/PuerkitoBio/purell v1.1.1
go: extracting github.com/go-openapi/jsonpointer v0.19.2
go: extracting github.com/go-openapi/swag v0.19.2
go: downloading github.com/mailru/easyjson v0.0.0-20190620125010-da37f6c1e481
go: extracting github.com/PuerkitoBio/purell v1.1.1
go: downloading github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
go: extracting github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578
go: extracting github.com/mailru/easyjson v0.0.0-20190620125010-da37f6c1e481
go: downloading github.com/coreos/etcd v3.3.17+incompatible
go: extracting github.com/coreos/etcd v3.3.17+incompatible
=== RUN   TestE2E_Kind
 • Ensuring node image (kindest/node:v1.15.3) 🖼  ...
time="2019-10-18T11:16:08+01:00" level=info msg="Image: kindest/node:v1.15.3 present locally"
 ✓ Ensuring node image (kindest/node:v1.15.3) 🖼
 • Preparing nodes 📦  ...
 ✓ Preparing nodes 📦
 • Creating kubeadm config 📜  ...
 ✓ Creating kubeadm config 📜
 • Starting control-plane 🕹️  ...
 ✓ Starting control-plane 🕹️
 • Installing CNI 🔌  ...
 ✓ Installing CNI 🔌
 • Installing StorageClass 💾  ...
 ✓ Installing StorageClass 💾
Cluster creation complete. You can now use the cluster with:

export KUBECONFIG="$(kind get kubeconfig-path --name="etcd-e2e")"
kubectl cluster-info



--- FAIL: TestE2E_Kind (319.15s)
    e2e_test.go:159: Starting Kind cluster
    e2e_test.go:63: Applying CRDs
    kubectl.go:17: Running kubectl apply --kustomize /home/richard/projects/improbable-eng/etcd-cluster-operator/config/crd
    kubectl.go:26: customresourcedefinition.apiextensions.k8s.io/etcdclusters.etcd.improbable.io created
        customresourcedefinition.apiextensions.k8s.io/etcdpeers.etcd.improbable.io created
        
    e2e_test.go:68: Building operator image
    e2e_test.go:86: Loading image in to Kind cluster
    e2e_test.go:95: Applying operator
    kubectl.go:17: Running kubectl apply --kustomize /home/richard/projects/improbable-eng/etcd-cluster-operator/config/rbac
    kubectl.go:26: role.rbac.authorization.k8s.io/leader-election-role created
        clusterrole.rbac.authorization.k8s.io/manager-role created
        clusterrole.rbac.authorization.k8s.io/proxy-role created
        rolebinding.rbac.authorization.k8s.io/leader-election-rolebinding created
        clusterrolebinding.rbac.authorization.k8s.io/manager-rolebinding created
        clusterrolebinding.rbac.authorization.k8s.io/proxy-rolebinding created
        service/controller-manager-metrics-service created
        
    kubectl.go:17: Running kubectl apply --kustomize /home/richard/projects/improbable-eng/etcd-cluster-operator/config/manager
    kubectl.go:26: deployment.apps/etcd-cluster-operator created
        
    kubectl.go:17: Running kubectl patch --kustomize /home/richard/projects/improbable-eng/etcd-cluster-operator/config/manager --type json --patch [
        			{
        				"op": "replace", 
        				"path": "/spec/template/spec/containers/0/image", 
        				"value": "etcd-cluster-operator:test",
        			},
        			{
        				"op": "replace", 
        				"path": "/spec/template/spec/containers/0/imagePullPolicy", 
        				"value": "Never",
        			},
        		]
    kubectl.go:33: deployment.apps/etcd-cluster-operator patched
        
    kubectl.go:17: Running kubectl get deploy etcd-cluster-operator -o=jsonpath='{.status.readyReplicas}'
    e2e_test.go:133: Running tests
    kubectl.go:17: Running kubectl apply --filename /home/richard/projects/improbable-eng/etcd-cluster-operator/internal/test/e2e/fixtures/cluster-client-service.yaml
    kubectl.go:26: service/my-cluster-external created
        
    kubectl.go:17: Running kubectl apply --filename /home/richard/projects/improbable-eng/etcd-cluster-operator/config/samples/etcd_v1alpha1_etcdcluster.yaml
    kubectl.go:26: etcdcluster.etcd.improbable.io/my-cluster created
        
    e2e_test.go:213: Checking if ETCD is available
    e2e_test.go:213: Checking if ETCD is available
    e2e_test.go:213: Checking if ETCD is available
    e2e_test.go:213: Checking if ETCD is available
    e2e_test.go:213: Checking if ETCD is available
    e2e_test.go:213: Checking if ETCD is available
    e2e_test.go:213: Checking if ETCD is available
    e2e_test.go:213: Checking if ETCD is available
    e2e_test.go:213: Checking if ETCD is available
    e2e_test.go:213: Checking if ETCD is available
    e2e_test.go:213: Checking if ETCD is available
    require.go:794: 
        	Error Trace:	e2e_test.go:224
        	            				e2e_test.go:134
        	Error:      	Received unexpected error:
        	            	client: etcd cluster is unavailable or misconfigured; error #0: client: endpoint http://127.0.0.1:2379 exceeded header timeout
        	Test:       	TestE2E_Kind
    e2e_test.go:185: Stopping Kind cluster
=== RUN   TestE2E_CurrentContext
--- SKIP: TestE2E_CurrentContext (0.00s)
    e2e_test.go:139: 
FAIL
FAIL	github.com/improbable-eng/etcd-cluster-operator/internal/test/e2e	319.204s
FAIL

I'll spend a few minutes digging into that e2e test failure.

wallrj · 2019-10-18T09:42:41Z

internal/test/e2e/e2e_test.go

+	require.NoError(t, err)
+
+	// Ensure the operator starts.
+	err = try.Eventually(func() error {


Unrelated to this branch, but I noticed that testify has its own Eventually, so perhaps we can use that in future:

https://godoc.org/github.com/stretchr/testify/assert#Eventually

internal/test/e2e/e2e_test.go

wallrj · 2019-10-18T10:11:16Z

config/manager/manager.yaml

-  labels:
-    control-plane: controller-manager
-  name: system
---


I see why this is a problem, but I think the kubebuilder uses this namespace intentionally with the expectation that you will add a kustomization which adds a namespace prefix. e.g.

# Adds namespace to all resources. namespace: e2e # Value of this field is prepended to the # names of all resources, e.g. a deployment named # "wordpress" becomes "alices-wordpress". # Note that it should also match with the prefix (text before '-') of the namespace # field above. namePrefix: e2e- # Labels to add to all resources and selectors. #commonLabels:

See:

https://github.com/kubernetes-sigs/kustomize/blob/master/examples/wordpress/kustomization.yaml

So consider just updating the kustomization.yaml flie instead.

i found it puzzling why the default value was not default

config/manager/manager.yaml

internal/test/e2e/e2e_test.go

wallrj · 2019-10-18T11:00:41Z

It's quite frustrating that the logged progress messages (via t.Log) only show up at the end of the test run.
Others have asked for a -test.streamlogs flag but it hasn't been implemented yet: golang/go#24929
Consider using logging those messages with logr too.

internal/test/e2e/e2e_test.go

wallrj · 2019-10-18T11:32:55Z

Ignore my comment (above) about the tests failing. I'd forgotten that this branch hasn't got a EtcdCluster controller which creates peers, so it was never going to pass.
I merged in #27 branch and then encountered a bug in that branch.

It'd be nice if this E2E would grab the logs from the operator in case of test failures, but that can be a future improvement.

wallrj · 2019-10-18T13:38:20Z

I also notice that the test leaves behind internal/test/e2e/.kube/ containing various cache files.
It'd be nice if it didn't do this,
Perhaps you need to pass through the $HOME variable when you run kubectl so that it uses the standard ~/.kube directory.

adamhosier · 2019-10-21T09:38:26Z

@wallrj thanks for the review. i've set the homedir in the kubectl helpers & that seems to avoid building the cache inside the repo. on logs, my strategy has been to include high level checkpoints in the test's logs, which will allow people to dig in to the more specific logs of the failing component. agree that streaming logs would be a nice feature

…ler-gen-update Revert "update controller gen and kustomize versions"

e2e

bc051d6

adamhosier requested review from munnerz, wallrj and JamesLaverack October 16, 2019 13:34

import order

c185867

JamesLaverack reviewed Oct 16, 2019

View reviewed changes

examples/cluster.yaml Outdated Show resolved Hide resolved

examples/cluster.yaml Outdated Show resolved Hide resolved

examples/cluster.yaml Outdated Show resolved Hide resolved

examples/operator.yaml Outdated Show resolved Hide resolved

cr

d8d6039

JamesLaverack approved these changes Oct 16, 2019

View reviewed changes

munnerz reviewed Oct 16, 2019

View reviewed changes

examples/operator.yaml Outdated Show resolved Hide resolved

munnerz reviewed Oct 16, 2019

View reviewed changes

examples/operator.yaml Outdated Show resolved Hide resolved

JamesLaverack assigned adamhosier Oct 17, 2019

JamesLaverack added the enhancement New feature or request label Oct 17, 2019

use config/

8b07e32

wallrj approved these changes Oct 18, 2019

View reviewed changes

wallrj reviewed Oct 18, 2019

View reviewed changes

internal/test/e2e/e2e_test.go Show resolved Hide resolved

cr

febac8a

kubectl homedir

ee8f675

adamhosier merged commit e22d7a7 into improbable-eng:master Oct 21, 2019

adamhosier deleted the feature/e2e-test branch October 21, 2019 09:40

gottwald pushed a commit to digitalocean/etcd-cluster-operator that referenced this pull request Jun 9, 2022

Merge pull request improbable-eng#26 from digitalocean/revert-control…

0bc7b6e

…ler-gen-update Revert "update controller gen and kustomize versions"

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create E2E test suite #26

Create E2E test suite #26

adamhosier commented Oct 16, 2019 •

edited

Loading

adamhosier commented Oct 16, 2019

adamhosier commented Oct 17, 2019

wallrj left a comment

wallrj Oct 18, 2019

wallrj Oct 18, 2019

wallrj Oct 18, 2019

adamhosier Oct 18, 2019

wallrj commented Oct 18, 2019

wallrj commented Oct 18, 2019

wallrj commented Oct 18, 2019

adamhosier commented Oct 21, 2019

Create E2E test suite #26

Create E2E test suite #26

Conversation

adamhosier commented Oct 16, 2019 • edited Loading

adamhosier commented Oct 16, 2019

adamhosier commented Oct 17, 2019

wallrj left a comment

Choose a reason for hiding this comment

wallrj Oct 18, 2019

Choose a reason for hiding this comment

wallrj Oct 18, 2019

Choose a reason for hiding this comment

wallrj Oct 18, 2019

Choose a reason for hiding this comment

adamhosier Oct 18, 2019

Choose a reason for hiding this comment

wallrj commented Oct 18, 2019

wallrj commented Oct 18, 2019

wallrj commented Oct 18, 2019

adamhosier commented Oct 21, 2019

adamhosier commented Oct 16, 2019 •

edited

Loading