Various fixes for running e2e tests #67

wallrj · 2017-10-30T17:26:09Z

See #45 for discussion

NONE

…ve it from the travis script.

``` + kubectl logs -c apiserver -l app=navigator,component=apiserver error: a container cannot be specified when using a selector (-l) ```

jetstack-bot · 2017-10-30T17:26:11Z

@wallrj: Adding do-not-merge/release-note-label-needed because the release note process has not been followed.

One of the following labels is required "release-note", "release-note-action-required", or "release-note-none".
Please see: https://github.com/kubernetes/community/blob/master/contributors/devel/pull-requests.md#write-release-notes-if-needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wallrj · 2017-10-30T17:41:23Z

In #45 (comment) @munnerz wrote:

Can we switch this back? These test two different things - the api may be available whilst the controller is in a crash loop.

I don't mind putting it back but my thinking is that we shouldn't be poking around for the state of controller in e2e tests...unless the tests fail.
If the Navigator API is ready we should be able to create a Navigator database and then expect the database to start up in some time.

And in any case, the controller is running in the tests right now, but failing because RBAC policy prevents its leader election routine to run.

ERROR: logging before flag.Parse: I1030 17:40:31.035544       1 round_trippers.go:417] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: navigator-controller/v1.8.2 (linux/amd64) kubernetes/$Format/leader-election" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im5hdi1lMmUtbmF2aWdhdG9yLWNvbnRyb2xsZXItdG9rZW4tenRsNDUiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibmF2LWUyZS1uYXZpZ2F0b3ItY29udHJvbGxlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImI5MzM3ZmI3LWJkOTYtMTFlNy04ZWQ4LTUyNTQwMDg2NzBlYyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0Om5hdi1lMmUtbmF2aWdhdG9yLWNvbnRyb2xsZXIifQ.rq5OBsceqNzikBLbjIViy-yk1A22nn-dfRuRF_MWZiFjbRwBmMe4ZACk2O06mSPb-GaDvHS6ryeAaEwXNQZB_cyIKKgxPUabcRprgTh0-Ghl6K2w4d77s2gdERb-yBgRjffaa1QGAj_n8M0MbAGOVfHPvs4x8M83QnjIDfwmkIDw0u_-GboOWS1qKyb42sU3tFu7ByoMPqvlV7VX5gXmdJWcSKBoyv7GkoNFP_1bNp_2NieCC_XgmcmGAHOMxUgawFD4idtrMr3I3ReCIoC_p_mKJyqMaQKyNbuINRQ72lqgna3ZgCg7Nlo2h8eqoFvxOvKFWnsPA5HzAeRE7jxJRA" https://10.0.0.1:443/api/v1/namespaces/kube-system/endpoints/navigator-controller
ERROR: logging before flag.Parse: I1030 17:40:31.158865       1 round_trippers.go:436] GET https://10.0.0.1:443/api/v1/namespaces/kube-system/endpoints/navigator-controller 403 Forbidden in 123 milliseconds
ERROR: logging before flag.Parse: I1030 17:40:31.158889       1 round_trippers.go:442] Response Headers:
ERROR: logging before flag.Parse: I1030 17:40:31.158894       1 round_trippers.go:445]     Date: Mon, 30 Oct 2017 17:40:31 GMT
ERROR: logging before flag.Parse: I1030 17:40:31.158898       1 round_trippers.go:445]     Content-Type: text/plain
ERROR: logging before flag.Parse: I1030 17:40:31.158901       1 round_trippers.go:445]     X-Content-Type-Options: nosniff
ERROR: logging before flag.Parse: I1030 17:40:31.158905       1 round_trippers.go:445]     Content-Length: 118
ERROR: logging before flag.Parse: I1030 17:40:31.158928       1 request.go:836] Response Body: User "system:serviceaccount:default:nav-e2e-navigator-controller" cannot get endpoints in the namespace "kube-system".
ERROR: logging before flag.Parse: E1030 17:40:31.158961       1 leaderelection.go:224] error retrieving resource lock kube-system/navigator-controller: User "system:serviceaccount:default:nav-e2e-navigator-controller" cannot get endpoints in the namespace "kube-system". (get endpoints navigator-controller)
ERROR: logging before flag.Parse: I1030 17:40:31.158970       1 leaderelection.go:180] failed to acquire lease kube-system/navigator-controller

We should fix that.

munnerz · 2017-10-31T10:46:54Z

@wallrj agreed that it'd be nice to do a full test like that, but right now we don't have the testing infra to stand up full DB clusters due to resource limits on Travis. I'm working to fix this via our new test-infra, but for the time being, it's important we know whether the controller will work with the default configuration, so I'd like to keep it in.

Yep we should fix that (#68), and also should update the tests so they fail as a result of the leader election failing (through a health check perhaps). But this second part is a bit lower priority, as failing leader election is unlikely to be the cause of test failures (once we fix #68)

… my laptop

wallrj · 2017-10-31T16:20:54Z

.travis.yml

@@ -18,9 +18,6 @@ jobs:
      # Create a cluster. We do this as root as we are using the 'docker' driver.
      # We enable RBAC on the cluster too, to test the RBAC in Navigators chart
      - sudo -E CHANGE_MINIKUBE_NONE_USER=true minikube start --vm-driver=none --kubernetes-version="$KUBERNETES_VERSION" --extra-config=apiserver.Authorization.Mode=RBAC
-      - while true; do if kubectl get nodes; then break; fi; echo "Waiting 5s for kubernetes to be ready..."; sleep 5; done
-      # Fix problems with kube-dns + rbac
-      - kubectl create serviceaccount --namespace kube-system kube-dns


I removed these steps because they are already performed in the prepare-e2e.sh script.

wallrj · 2017-10-31T16:21:51Z

hack/e2e.sh

-retry TIMEOUT=600 kubectl get nodes
-
-echo "Waiting for tiller to be ready..."
-retry TIMEOUT=60 helm version


And these steps are also performed in prepare-e2e.sh, so I removed them.

wallrj · 2017-10-31T16:23:33Z

hack/e2e.sh

-    kubectl logs -c controller -l app=navigator,component=controller
-    exit 1
+    FAILURE_COUNT=$(($FAILURE_COUNT+1))
+    echo "TEST FAILURE: $1"
 }


This is so that all the tests are performed rather failing on the first test.

wallrj · 2017-10-31T16:24:19Z

hack/e2e.sh

+         --namespace "${USER_NAMESPACE}" \
+         service es-demo; then
+        fail_test "Navigator controller failed to create elasticsearchcluster service"
+    fi


Added a check to see that a service is created....a clear indication that the navigator controller is reacting to the new escluster resource.

wallrj · 2017-10-31T16:24:49Z

hack/prepare-e2e.sh

-CERT_DIR="$CONFIG_DIR/certs"
-mkdir -p $CERT_DIR
-TEST_DIR="$CONFIG_DIR/tmp"
-mkdir -p $TEST_DIR


None of this stuff is needed in this script.

wallrj · 2017-10-31T16:25:35Z

hack/prepare-e2e.sh

+  kind: ServiceAccount
+  metadata:
+    name: kube-dns
+    namespace: kube-system


Added this here rather than creating the service in the travis.yml file.

My reasoning behind the separation is so that we are not opinionated in our e2e test scripts about using minikube. It'd be ideal if the e2e tests could be run against any valid/functioning kubernetes cluster (this is more relevant when it comes to us running our tests on GCE/GKE).

Happy to accept this for now, but I think there'll be more refactoring of this stuff soon 😄

Ah, got it. Well thanks for merging this anyway.

wallrj · 2017-10-31T16:28:22Z

@munnerz Ok. I've added back the check for apiserver and controller replicas and added another check for kubectl get esc which was also failing when I run the tests locally.

I know you're planning to create e2e tests with Ginkgo / prow / gce, but I implore you to merge this change so that I can continue to perform some quick local e2e tests on my laptop.

wallrj · 2017-11-01T11:20:53Z

e2e test failed because of failed kubectl get esc which I suspect was caused by kube-dns failing:

https://travis-ci.org/jetstack/navigator/jobs/295687871#L853

+kubectl get pods --all-namespaces

NAMESPACE     NAME                                                                 READY     STATUS             RESTARTS   AGE

default       nav-e2e-navigator-apiserver-4058778657-jdmx9                         2/2       Running            0          1m

default       nav-e2e-navigator-controller-2991077948-7l4s5                        1/1       Running            0          1m

kube-system   kube-addon-manager-travis-job-345a1919-cc3d-44d6-8177-e50007723a0a   1/1       Running            0          8m

kube-system   kube-dns-1326421443-q1p8c                                            2/3       CrashLoopBackOff   7          8m

kube-system   kubernetes-dashboard-qb2lw                                           0/1       CrashLoopBackOff   6          8m

kube-system   tiller-deploy-3341511835-k8zxt                                       1/1       Running            0          1m

/retest

munnerz · 2017-11-01T11:22:16Z

@wallrj /retest does not work for travis jobs :(

I've hit restart. Are you able to restart jobs on travis?

wallrj · 2017-11-01T11:23:35Z

Ah yes, I see it here: https://travis-ci.org/jetstack/navigator/pull_requests

Thanks @munnerz

munnerz · 2017-11-01T15:04:22Z

/lgtm
/approve

jetstack-bot · 2017-11-01T15:04:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: munnerz

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

~~OWNERS~~ [munnerz]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

jetstack-bot · 2017-11-01T15:04:51Z

Automatic merge from submit-queue.

wallrj added 10 commits October 17, 2017 13:08

Allow 5 minutes for APIs to be registered

de95b72

A simpler check for navigator readiness

ec68368

Move helm health check into the e2e prepare script

0b07407

Remove unused variables and set up from e2e prepare script

3c5392f

put all the Kube-dns rbac fix ups in one place

9e24d78

The e2e prepare script already waits for kubernetes to be ready. Remo…

afe7a62

…ve it from the travis script.

Removing these lines because they fail with the following error message:

b188a4e

``` + kubectl logs -c apiserver -l app=navigator,component=apiserver error: a container cannot be specified when using a selector (-l) ```

Run all the tests instead of failing after the first

adad267

Create a debug log on failures

34d77e8

Merge branch 'master' into 27-e2e-fixes

e1752c8

jetstack-bot added do-not-merge/release-note-label-needed size/L labels Oct 30, 2017

jetstack-bot assigned simonswine and munnerz Oct 30, 2017

jetstack-bot added release-note-none and removed do-not-merge/release-note-label-needed labels Oct 30, 2017

wallrj mentioned this pull request Oct 30, 2017

Various fixes for running e2e tests locally #45

Closed

wallrj added 7 commits October 31, 2017 15:35

Merge branch 'master' into 27-e2e-fixes

c104775

Re-instate the check for apiserver and controller replica count

d5b2d09

Add a check for navigator api types to the readiness check

75aa826

Test that the controller adds a service for the elasticsearch cluster

9685a76

It can take up to 5 minutes for kubernetes to delete the namespace on…

9da37b9

… my laptop

Travis doesn't save files....so print debug information to stdout

22cf3af

Travis doesn't save files....so print debug information to stdout

4f3273c

wallrj commented Oct 31, 2017

View reviewed changes

Merge branch 'master' into 27-e2e-fixes

2496ce4

jetstack-bot added the lgtm label Nov 1, 2017

jetstack-bot added the approved label Nov 1, 2017

jetstack-bot merged commit 3c8a77e into jetstack:master Nov 1, 2017

wallrj deleted the 27-e2e-fixes branch November 1, 2017 15:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Various fixes for running e2e tests #67

Various fixes for running e2e tests #67

wallrj commented Oct 30, 2017 •

edited

Loading

jetstack-bot commented Oct 30, 2017

wallrj commented Oct 30, 2017

munnerz commented Oct 31, 2017

wallrj Oct 31, 2017 •

edited

Loading

wallrj Oct 31, 2017

wallrj Oct 31, 2017

wallrj Oct 31, 2017

wallrj Oct 31, 2017

wallrj Oct 31, 2017

munnerz Nov 1, 2017

wallrj Nov 1, 2017

wallrj commented Oct 31, 2017

wallrj commented Nov 1, 2017

munnerz commented Nov 1, 2017

wallrj commented Nov 1, 2017

munnerz commented Nov 1, 2017

jetstack-bot commented Nov 1, 2017

jetstack-bot commented Nov 1, 2017

Various fixes for running e2e tests #67

Various fixes for running e2e tests #67

Conversation

wallrj commented Oct 30, 2017 • edited Loading

jetstack-bot commented Oct 30, 2017

wallrj commented Oct 30, 2017

munnerz commented Oct 31, 2017

wallrj Oct 31, 2017 • edited Loading

Choose a reason for hiding this comment

wallrj Oct 31, 2017

Choose a reason for hiding this comment

wallrj Oct 31, 2017

Choose a reason for hiding this comment

wallrj Oct 31, 2017

Choose a reason for hiding this comment

wallrj Oct 31, 2017

Choose a reason for hiding this comment

wallrj Oct 31, 2017

Choose a reason for hiding this comment

munnerz Nov 1, 2017

Choose a reason for hiding this comment

wallrj Nov 1, 2017

Choose a reason for hiding this comment

wallrj commented Oct 31, 2017

wallrj commented Nov 1, 2017

munnerz commented Nov 1, 2017

wallrj commented Nov 1, 2017

munnerz commented Nov 1, 2017

jetstack-bot commented Nov 1, 2017

jetstack-bot commented Nov 1, 2017

wallrj commented Oct 30, 2017 •

edited

Loading

wallrj Oct 31, 2017 •

edited

Loading