Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🐛 Fix flakiness in MD controller test #7668

Merged
merged 1 commit into from
Dec 12, 2022
Merged

🐛 Fix flakiness in MD controller test #7668

merged 1 commit into from
Dec 12, 2022

Conversation

killianmuldoon
Copy link
Contributor

Signed-off-by: killianmuldoon kmuldoon@vmware.com

Fix flakiness in MD controller test when verifying the deletion of old MachineSets.

@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 30, 2022
@k8s-ci-robot k8s-ci-robot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Nov 30, 2022
@killianmuldoon
Copy link
Contributor Author

/test ?

@k8s-ci-robot
Copy link
Contributor

@killianmuldoon: The following commands are available to trigger required jobs:

  • /test pull-cluster-api-build-main
  • /test pull-cluster-api-e2e-main
  • /test pull-cluster-api-test-main
  • /test pull-cluster-api-test-mink8s-main
  • /test pull-cluster-api-verify-main

The following commands are available to trigger optional jobs:

  • /test pull-cluster-api-apidiff-main
  • /test pull-cluster-api-e2e-full-main
  • /test pull-cluster-api-e2e-informing-ipv6-main
  • /test pull-cluster-api-e2e-informing-main
  • /test pull-cluster-api-e2e-workload-upgrade-1-25-latest-main

Use /test all to run the following jobs that were automatically triggered:

  • pull-cluster-api-apidiff-main
  • pull-cluster-api-build-main
  • pull-cluster-api-e2e-informing-ipv6-main
  • pull-cluster-api-e2e-informing-main
  • pull-cluster-api-e2e-main
  • pull-cluster-api-test-main
  • pull-cluster-api-test-mink8s-main
  • pull-cluster-api-verify-main

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@killianmuldoon
Copy link
Contributor Author

/test pull-cluster-api-test-main

4 similar comments
@killianmuldoon
Copy link
Contributor Author

/test pull-cluster-api-test-main

@killianmuldoon
Copy link
Contributor Author

/test pull-cluster-api-test-main

@killianmuldoon
Copy link
Contributor Author

/test pull-cluster-api-test-main

@killianmuldoon
Copy link
Contributor Author

/test pull-cluster-api-test-main

@killianmuldoon killianmuldoon changed the title [WIP] 🐛 Fix flakiness in MD controller test 🐛 Fix flakiness in MD controller test Dec 1, 2022
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 1, 2022
killianmuldoon
killianmuldoon commented Dec 1, 2022
View reviewed changes
internal/controllers/machinedeployment/machinedeployment_controller_test.go Outdated
@@ -394,7 +394,7 @@ func TestMachineDeploymentReconciler(t *testing.T) {
return false
}
return machineSets.Items[0].Status.Replicas == *deployment.Spec.Replicas
}, timeout*5).Should(BeTrue())
}, timeout*10).Should(BeTrue())
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't see an easy win here for this deletion - I think the best thing to do for now is to increase the timeout for the test from 150 seconds to 300 seconds and see if long tail reconciliation always happens within that window

@killianmuldoon killianmuldoon changed the title 🐛 Fix flakiness in MD controller test 🐛 [WIP] Fix flakiness in MD controller test Dec 1, 2022
@killianmuldoon
Fix flakiness in MD controller test
e9cce67 
Signed-off-by: killianmuldoon <kmuldoon@vmware.com>
@killianmuldoon killianmuldoon changed the title 🐛 [WIP] Fix flakiness in MD controller test 🐛 Fix flakiness in MD controller test Dec 7, 2022
@killianmuldoon
Copy link
Contributor Author

Taking another look at this I think we should merge as is and monitor to see if the test stays flaky. Seems to me there's a long tail on deletion in the test which is just hard to account for deterministically. If lengthening the max timeout doesn't work we can consider rewriting the test - possibly to make this check occur later in the order.

fabriziopandini
fabriziopandini reviewed Dec 12, 2022
View reviewed changes
Copy link
Member

@fabriziopandini fabriziopandini left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve
let's see if this improves the situation

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Dec 12, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fabriziopandini

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 12, 2022
@k8s-ci-robot k8s-ci-robot merged commit d294bfb into kubernetes-sigs:main Dec 12, 2022
@k8s-ci-robot k8s-ci-robot added this to the v1.4 milestone Dec 12, 2022
@killianmuldoon
Copy link
Contributor Author

/cherry-pick release-1.2

@killianmuldoon
Copy link
Contributor Author

/cherry-pick release-1.3

Cherry picking this fix as we've seen the flake a number of times in supported branches.

@k8s-infra-cherrypick-robot k8s-infra-cherrypick-robot mentioned this pull request Dec 13, 2022
Merged
@k8s-infra-cherrypick-robot
Copy link

@killianmuldoon: new pull request created: #7753

In response to this:

/cherry-pick release-1.2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-infra-cherrypick-robot
Copy link

@killianmuldoon: new pull request created: #7754

In response to this:

/cherry-pick release-1.3

Cherry picking this fix as we've seen the flake a number of times in supported branches.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-infra-cherrypick-robot k8s-infra-cherrypick-robot mentioned this pull request Dec 13, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vincepri vincepri vincepri left review comments

@fabriziopandini fabriziopandini fabriziopandini left review comments

@JoelSpeed JoelSpeed Awaiting requested review from JoelSpeed

Assignees

@fabriziopandini fabriziopandini

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
v1.4
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@killianmuldoon @k8s-ci-robot @k8s-infra-cherrypick-robot @vincepri @fabriziopandini