Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubeadm: fix broken e2e tests for 1.21 #2380

Closed
neolit123 opened this issue Jan 23, 2021 · 8 comments · Fixed by kubernetes/test-infra#20586
Closed

kubeadm: fix broken e2e tests for 1.21 #2380

neolit123 opened this issue Jan 23, 2021 · 8 comments · Fixed by kubernetes/test-infra#20586
Assignees
@neolit123
Labels
area/external kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. sig/release Categorizes an issue or PR as relevant to SIG Release. sig/testing Categorizes an issue or PR as relevant to SIG Testing.
Projects
CI Signal team (SIG Release)
Milestone
v1.21

Comments

@neolit123
Copy link
Member

neolit123 commented Jan 23, 2021 •
edited

we have a couple of failing jobs, which i think happened after the version bumps for 1.21:
kubernetes/kubernetes@7791bbe

  • skew latest on 1.20

https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-latest-on-1-20

this i failing in kubeadm init under kinder when the kubeadm config images list is called. i tried the isolated command and it works, so it seems to be failing only under kinder for some reason.

  • upgrade 1.20 -> latest

https://testgrid.k8s.io/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-upgrade-1-20-latest

upgrade is failing with:

this version of kubeadm only supports deploying clusters with the control plane version >= 1.20.0. Current version: v1.20.0-beta.2.96+98bc258bf5516b

then reset timeouts (runs more than 5 min)
@neolit123 neolit123 added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. labels Jan 23, 2021
@neolit123 neolit123 added this to the v1.21 milestone Jan 23, 2021
@neolit123 neolit123 self-assigned this Jan 23, 2021
@neolit123 neolit123 changed the title kubeadm: broken e2e tests for 1.21 kubeadm: fix broken e2e tests for 1.21 Jan 23, 2021
@neolit123
Copy link
Member Author

neolit123 commented Jan 24, 2021 •
edited

looks like the problem is the new bucket we switched to here.

https://console.cloud.google.com/storage/browser/_details/k8s-release-dev/ci/latest-1.20.txt resolves to v1.20.0-beta.2.96+98bc258bf5516b

the latest version reported by https://dl.k8s.io/ci/latest-1.20.txt is the valid one:
v1.20.3-rc.0.15+18194169ac684f
but this version and the artifacts are missing in the new bucket.

@neolit123 neolit123 mentioned this issue Jan 24, 2021
Closed
@neolit123 neolit123 added area/external sig/release Categorizes an issue or PR as relevant to SIG Release. sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. labels Jan 24, 2021
@neolit123 neolit123 mentioned this issue Jan 24, 2021
Closed
@SataQiu SataQiu mentioned this issue Jan 25, 2021
Merged
@thejoycekung thejoycekung added this to Under investigation (prioritized) in CI Signal team (SIG Release) Jan 26, 2021
CI Signal team (SIG Release) automation moved this from Under investigation (prioritized) to Observing (observe test failure/flake before marking as resolved) Jan 27, 2021
@neolit123
Copy link
Member Author

lets keep this open until the tests become green:
https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-upgrade-1-20-latest

checking https://console.cloud.google.com/storage/browser/_details/k8s-release-dev/ci/latest-1.20.txt still points to an old tag.

@neolit123 neolit123 reopened this Jan 27, 2021
CI Signal team (SIG Release) automation moved this from Observing (observe test failure/flake before marking as resolved) to Under investigation (prioritized) Jan 27, 2021
@SataQiu
Copy link
Member

SataQiu commented Jan 28, 2021

It seems that the job did not execute successfully.
https://testgrid.k8s.io/sig-release-1.20-blocking#build-1.20

@neolit123
Copy link
Member Author

looking at the logs i don't understand why it's failing. i see this:

W0128 15:18:14.861] subprocess.CalledProcessError: Command '('../release/push-build.sh', '--nomock', '--verbose', '--ci', '--bucket=k8s-release-dev', '--docker-registry=gcr.io/k8s-staging-ci-images', '--extra-version-markers=k8s-beta', '--allow-dup')' returned non-zero exit status 1

@neolit123 neolit123 mentioned this issue Jan 28, 2021
Merged
@spiffxp
Copy link
Member

spiffxp commented Jan 28, 2021

I opened a fix for this yesterday but test-infra is having merging issues (ref: kubernetes/test-infra#20586 (comment))

@neolit123
Copy link
Member Author

thanks @spiffxp

@neolit123
Copy link
Member Author

neolit123 commented Jan 28, 2021 •
edited

kubernetes/test-infra#20586 + kubernetes/test-infra#20624 indeed fixed the 2 failing tests 🎉
https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-kubeadm

/close

@k8s-ci-robot
Copy link
Contributor

@neolit123: Closing this issue.

In response to this:

kubernetes/test-infra#20624 indeed fixed the 2 failing tests 🎉
https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-kubeadm

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

CI Signal team (SIG Release) automation moved this from Under investigation (prioritized) to Observing (observe test failure/flake before marking as resolved) Jan 28, 2021
@thejoycekung thejoycekung moved this from Observing (observe test failure/flake before marking as resolved) to Resolved in CI Signal team (SIG Release) Feb 5, 2021
@thejoycekung thejoycekung moved this from Resolved to Resolved (2+ weeks) in CI Signal team (SIG Release) Apr 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@neolit123 neolit123

Labels
area/external kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. sig/release Categorizes an issue or PR as relevant to SIG Release. sig/testing Categorizes an issue or PR as relevant to SIG Testing.
Projects
CI Signal team (SIG Release)
  
Resolved (2+ weeks)
Milestone
v1.21
Development

Successfully merging a pull request may close this issue.

add ci-kubernetes-build-1-20-canary job SataQiu/test-infra
4 participants
@spiffxp @neolit123 @SataQiu @k8s-ci-robot