Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🏃 🛠 fix prow presubmits #1201

Merged
merged 1 commit into from
Nov 25, 2019
Merged

🏃 🛠 fix prow presubmits #1201

merged 1 commit into from
Nov 25, 2019

Conversation

alexeldeib
Copy link
Contributor

Fixes prow e2e optional presubmit (maybe?)

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Nov 14, 2019
@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Nov 14, 2019
@alexeldeib
Copy link
Contributor Author

needs kubernetes/test-infra#15288

@alexeldeib
Copy link
Contributor Author

/hold

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 14, 2019
@alexeldeib alexeldeib changed the title 🛠 fix test_e2e GOBIN 🛠 fix prow e2e Nov 14, 2019
mengqiy
mengqiy approved these changes Nov 15, 2019
View reviewed changes
Copy link
Member

@mengqiy mengqiy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 15, 2019
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alexeldeib, mengqiy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 15, 2019
@mengqiy mengqiy changed the title 🛠 fix prow e2e 🏃 🛠 fix prow e2e Nov 15, 2019
@mengqiy
Copy link
Member

mengqiy commented Nov 15, 2019

/test pull-kubebuilder-e2e

@alexeldeib
Copy link
Contributor Author

Hmm, seems like they moved the runner in test-infra? Looking at Kind presumbits, they use runner.sh rather than /usr/local/bin/runner. Might need to switch this one more time :/

@alexeldeib
Copy link
Contributor Author

oh, the actual runner is /usr/local/bin/runner.sh -_-

@alexeldeib
Copy link
Contributor Author

fixing in kubernetes/test-infra#15291

@alexeldeib
Copy link
Contributor Author

cc @camilamacedo86

camilamacedo86
camilamacedo86 approved these changes Nov 17, 2019
View reviewed changes
Copy link
Member

@camilamacedo86 camilamacedo86 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@alexeldeib
Copy link
Contributor Author

/retest

1 similar comment
@mengqiy
Copy link
Member

mengqiy commented Nov 19, 2019

/retest

@mengqiy
Copy link
Member

mengqiy commented Nov 20, 2019

@alexeldeib Any update?
If you need more time to fix pull-kubebuilder-e2e, I'd suggest merging the fix for pull-kubebuilder-test first (maybe in a separate PR)

@alexeldeib alexeldeib changed the title 🏃 🛠 fix prow e2e 🏃 🛠 fix prow presubmit pt 1 Nov 20, 2019
@alexeldeib
Copy link
Contributor Author

@mengqiy sorry, I've been at Kubecon since the test-infra PR merged and it seems the test runner requires more adjustments on the invocation side. Happy to merge this as is to fix the vanilla test and finish e2e separately.

If that sounds ok to you, feel free to cancel the hold.

@alexeldeib
Copy link
Contributor Author

Oh, actually I think it's because the files changed to test_e2e_v1.sh and test_e2e_v2.sh. PR incoming...

@alexeldeib
Copy link
Contributor Author

kubernetes/test-infra#15342

If this PR still doesn't pass the e2e after that, I would go ahead and remove hold + merge anyway to get the other fix done.

@mengqiy mengqiy closed this Nov 20, 2019
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 21, 2019
@alexeldeib
Copy link
Contributor Author

@mengqiy tested pull-kubebuilder-e2e-1-16-2 prowjob locally with this PR + kubernetes/test-infra#15346 and things seem to be working now.

Apologies again for all the trial and error -- the new test-infra tooling should help avoid this kind of mess in the future.

@alexeldeib alexeldeib changed the title 🏃 🛠 fix prow presubmit pt 1 🏃 🛠 fix prow presubmits Nov 21, 2019
@alexeldeib
Copy link
Contributor Author

/retest

@mengqiy mengqiy closed this Nov 21, 2019
@mengqiy mengqiy reopened this Nov 21, 2019
@mengqiy
Copy link
Member

mengqiy commented Nov 21, 2019

@alexeldeib It seems we have some problem with kubeadm

@alexeldeib
Copy link
Contributor Author

alexeldeib commented Nov 21, 2019 •
edited
Loading

few additional references for kind using DinD:

kubernetes-sigs/kind#303
kubernetes-sigs/kind#759 --> think prow runner already handles this
kubernetes/test-infra#11510 --> preset for volumes

@alexeldeib
Copy link
Contributor Author

I can't repro this locally, but it seems even after fixing the immediate issue the clusters still fails to come up. Hoping kubernetes/test-infra#15348 will get the last bits fixed or else need to get some additional debug help from Ben w.r.t kind inside a pod using dind.

I don't want to make the CI more and more red with each attempt to fix it...I keep thinking I have it, but it's been a few rounds of the same. Open to direction on how to proceed without making a bigger mess 🙁

@alexeldeib
Copy link
Contributor Author

/retest

1 similar comment
@alexeldeib
Copy link
Contributor Author

/retest

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Nov 23, 2019 •
edited
Loading

@alexeldeib: The following tests failed, say /retest to rerun them all:

Test name Commit Details Rerun command
pull-kubebuilder-e2e-v1 a47bed1 link /test pull-kubebuilder-e2e-v1
pull-kubebuilder-e2e 0b747e2 link /test pull-kubebuilder-e2e

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@alexeldeib
Copy link
Contributor Author

/retest

@mengqiy
Copy link
Member

mengqiy commented Nov 23, 2019

It is great that this is finally passing!
Thanks for your hard work!
To merge it, can you please drop the change related to debugging and squashed the commits to a reasonable number?

@alexeldeib
Copy link
Contributor Author

Will do, I’m also relieved this finally works :)

I’m traveling for personal reasons over the weekend post-Kubecon, but will clean this up for merge Monday if that is okay?

@mengqiy
Copy link
Member

mengqiy commented Nov 23, 2019

Sure. Let's merge this on Monday.

@alexeldeib
Copy link
Contributor Author

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 25, 2019
@alexeldeib alexeldeib mentioned this pull request Nov 25, 2019
Merged
camilamacedo86
camilamacedo86 approved these changes Nov 25, 2019
View reviewed changes
Copy link
Member

@camilamacedo86 camilamacedo86 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It shows fine for me 👍
/lgtm /approved

@alexeldeib
Copy link
Contributor Author

/test pull-kubebuilder-e2e-k8s-1-15-3

@alexeldeib
Copy link
Contributor Author

@camilamacedo86 I think the bots didn't like the two commands in one line.

mengqiy
mengqiy reviewed Nov 25, 2019
View reviewed changes
scripts/setup.sh

# You can use --image flag to specify the cluster version you want, e.g --image=kindest/node:v1.13.6, the supported version are listed at https://hub.docker.com/r/kindest/node/tags
kind create cluster --config test/kind-config.yaml --image=kindest/node:$K8S_VERSION
kind create cluster -v 4 --retain --wait=1m --config test/kind-config.yaml --image=kindest/node:$K8S_VERSION
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-v 4 --retain --wait=1m

I wonder if these are related to debugging?
Re.--wait, do we have to wait for 1 minute? Is it causing any flakes? It seems the default wait time is 0s.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kind cluster is known to be fast to bring up.
Adding 1 minute wait time for bringing up the cluster slow it down ~10%. Look at https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kubebuilder/1201/pull-kubebuilder-e2e-k8s-1-16-2/1198845676235001858, the e2e test only takes <10 minutes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can remove this if desired, but I think it should stay. I matched it to what kind itself does in e2e: https://github.com/kubernetes-sigs/kind/blob/36eae75e067681658d59a31ed679e67e39d9cfc5/hack/ci/e2e-k8s.sh#L88-L93

Without this, kind won't wait at all for the nodes to come up: https://github.com/kubernetes-sigs/kind/blob/6a0de6124de068cd55b24025cf05e8e4ea172641/pkg/internal/cluster/create/actions/waitforready/waitforready.go#L45-L48

If anything, I can reduce -v from 4 to 3 I think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not adding a static 1 minute wait. It's only checking that the nodes actually come up.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note that it's also a fairly tight loop so there is not really additional wait time

@mengqiy
Copy link
Member

mengqiy commented Nov 25, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 25, 2019
@k8s-ci-robot k8s-ci-robot merged commit 74f69d6 into kubernetes-sigs:master Nov 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mengqiy mengqiy mengqiy approved these changes

@camilamacedo86 camilamacedo86 camilamacedo86 approved these changes

@pwittrock pwittrock Awaiting requested review from pwittrock

Assignees

@mengqiy mengqiy

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@alexeldeib @k8s-ci-robot @mengqiy @camilamacedo86