🐛 test/e2e check for machines being ready after provisioning on Runtime SDK test #8625

chrischdi · 2023-05-09T12:50:46Z

What this PR does / why we need it:

Sometimes when reaching the parts where we wait for the hooks, not all machines are ready. This results in the hook not even getting executed during the 30s period.

Failed test logs did show that it may only be a minute which is missing until everything would succeed.

This PR adds a check to ensure all machines are healthy before continuing.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

Workaround for #8486

We want to still try to track it down to see if we have an underlying issue.

cc @killianmuldoon

Note:

for control plane nodes we already check if this condition is true (edit: when upgrading a cluster):

cluster-api/test/framework/machine_helpers.go

Line 167 in e4144f8

if *m.Spec.Version == input.KubernetesUpgradeVersion && conditions.IsTrue(&m, clusterv1.MachineNodeHealthyCondition) {
but we don't do the same for e.g. machinedeployments (edit: when upgrading a cluster):

cluster-api/test/framework/machine_helpers.go

Line 207 in e4144f8

if *machine.Spec.Version == input.KubernetesUpgradeVersion {

elmiko

code logic looks ok to me, but the linter is unhappy ;)

chrischdi · 2023-05-09T13:49:30Z

code logic looks ok to me, but the linter is unhappy ;)

yep, fixed :-)

chrischdi · 2023-05-09T13:50:54Z

/cherry-pick release-1.4

k8s-infra-cherrypick-robot · 2023-05-09T13:50:56Z

@chrischdi: once the present PR merges, I will cherry-pick it on top of release-1.4 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

chrischdi · 2023-05-09T13:50:58Z

/cherry-pick release-1.3

k8s-infra-cherrypick-robot · 2023-05-09T13:50:59Z

@chrischdi: once the present PR merges, I will cherry-pick it on top of release-1.3 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

test/e2e/cluster_upgrade_runtimesdk.go

sbueringer · 2023-05-09T15:23:29Z

test/e2e/cluster_upgrade_runtimesdk.go

@@ -169,6 +169,23 @@ func clusterUpgradeWithRuntimeSDKSpec(ctx context.Context, inputGetter func() cl
 					clusterRef,
 					input.E2EConfig.GetIntervals(specName, "wait-cluster"))
 			},
+			PostMachinesProvisioned: func() {
+				Eventually(func() error {
+					machineList := &clusterv1.MachineList{}


Is the goal here to check that Nodes of all Machines are ready (CP + MD)?

If it's just about MDs wouldn't it be better to check the MD status replica fields?

~~It's about all nodes of the cluster.~~

Edit: more correct:

KCP should not be IsScaling

cluster-api/internal/controllers/topology/cluster/desired_state.go

Line 416 in 4adc1ec

cpScaling, err := contract.ControlPlane().IsScaling(s.Current.ControlPlane.Object)

No MD should be in RollingOut

cluster-api/internal/controllers/topology/cluster/desired_state.go

Line 430 in 4adc1ec

if s.Current.MachineDeployments.IsAnyRollingOut() {

So basically we use node readiness as a proxy for the things the topology controller is actually checking.

Let's please add a godoc comment that we are checking for node healthiness to ensure the cluster is ~ stable so that the upgrade we run below is not delayed by too much through the initial cluster creation.

I think the check is not really the same as MD IsAnyRollingOut as that one is also checking for available replicas which can have some delay depending on MinReadySeconds.

But overall I guess it's good enough, let's just make sure we document clearly why we do it.

Added a comment and squashed. PTAL

Sounds good, thank you!

ykakarap

Curious: The topology controller (the controllers that calls the hook) does not depend on the Machines being ready to call the hook. The hook response is also not dependent on the Machines being ready. I am curious how the machines not being ready are blocking the hook from getting called. 🤔

chrischdi · 2023-05-09T16:51:45Z

Curious: The topology controller (the controllers that calls the hook) does not depend on the Machines being ready to call the hook. The hook response is also not dependent on the Machines being ready. I am curious how the machines not being ready are blocking the hook from getting called. 🤔

We actually check the MD's and postpone calling the hook if a MD is rolling out

cluster-api/internal/controllers/topology/cluster/desired_state.go

Line 430 in 4adc1ec

if s.Current.MachineDeployments.IsAnyRollingOut() {

Also above that we check if the KCP IsScaling.

ykakarap · 2023-05-09T20:30:00Z

Ah! Good point.

… SDK test The topology controller checks if the ControlPlane IsScaling() or any MachineDeployment IsRollingOut() before running the BeforeClusterUpgrade hook. In some cases in e2e tests the nodes were not ready yet and the timeout for the BeforeClusterUpgrade hook was not sufficient for the nodes to get ready and running the hook.

killianmuldoon

/lgtm

Thanks again for taking this up!

k8s-ci-robot · 2023-05-11T18:39:38Z

LGTM label has been added.

Git tree hash: 82259d45f76afe4390de9bb90f4d2095f5a363e7

sbueringer · 2023-05-12T06:58:38Z

Thank you very much!

/approve

k8s-ci-robot · 2023-05-12T06:59:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sbueringer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-infra-cherrypick-robot · 2023-05-12T07:13:46Z

@chrischdi: new pull request created: #8646

In response to this:

/cherry-pick release-1.4

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-infra-cherrypick-robot · 2023-05-12T07:14:22Z

@chrischdi: new pull request created: #8647

In response to this:

/cherry-pick release-1.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

johannesfrey · 2023-06-05T05:38:08Z

/area testing

killianmuldoon · 2023-06-05T09:11:40Z

/area e2e-testing

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels May 9, 2023

k8s-ci-robot requested review from elmiko and ykakarap May 9, 2023 12:50

chrischdi changed the title ~~test/e2e check for machines being ready after provisioning on Runtime SDK test~~ 🐛 test/e2e check for machines being ready after provisioning on Runtime SDK test May 9, 2023

elmiko reviewed May 9, 2023

View reviewed changes

chrischdi force-pushed the pr/e2e-runtimesdk-increase-timeout branch from e4144f8 to 4adc1ec Compare May 9, 2023 13:49

chrischdi mentioned this pull request May 9, 2023

🐛 [WIP][DoNotReview] Reproduce issue #8486 #8492

Closed

sbueringer reviewed May 9, 2023

View reviewed changes

ykakarap reviewed May 9, 2023

View reviewed changes

chrischdi force-pushed the pr/e2e-runtimesdk-increase-timeout branch from c9998b8 to 642e16c Compare May 10, 2023 13:04

killianmuldoon reviewed May 11, 2023

View reviewed changes

k8s-ci-robot assigned killianmuldoon May 11, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 11, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 12, 2023

k8s-ci-robot merged commit 9278ddd into kubernetes-sigs:main May 12, 2023

k8s-ci-robot added this to the v1.5 milestone May 12, 2023

k8s-infra-cherrypick-robot mentioned this pull request May 12, 2023

[release-1.4] 🐛 test/e2e check for machines being ready after provisioning on Runtime SDK test #8646

Merged

k8s-infra-cherrypick-robot mentioned this pull request May 12, 2023

[release-1.3] 🐛 test/e2e check for machines being ready after provisioning on Runtime SDK test #8647

Merged

k8s-ci-robot added the area/testing Issues or PRs related to testing label Jun 5, 2023

k8s-ci-robot added the area/e2e-testing Issues or PRs related to e2e testing label Jun 5, 2023

killianmuldoon removed the area/testing Issues or PRs related to testing label Jun 5, 2023

chrischdi deleted the pr/e2e-runtimesdk-increase-timeout branch June 19, 2023 14:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 test/e2e check for machines being ready after provisioning on Runtime SDK test #8625

🐛 test/e2e check for machines being ready after provisioning on Runtime SDK test #8625

chrischdi commented May 9, 2023 •

edited

Loading

elmiko left a comment

chrischdi commented May 9, 2023

chrischdi commented May 9, 2023

k8s-infra-cherrypick-robot commented May 9, 2023

chrischdi commented May 9, 2023

k8s-infra-cherrypick-robot commented May 9, 2023

sbueringer May 9, 2023

chrischdi May 9, 2023 •

edited

Loading

sbueringer May 10, 2023

chrischdi May 10, 2023

sbueringer May 12, 2023

ykakarap left a comment

chrischdi commented May 9, 2023

ykakarap commented May 9, 2023

killianmuldoon left a comment

k8s-ci-robot commented May 11, 2023

sbueringer commented May 12, 2023

k8s-ci-robot commented May 12, 2023

k8s-infra-cherrypick-robot commented May 12, 2023

k8s-infra-cherrypick-robot commented May 12, 2023

johannesfrey commented Jun 5, 2023

killianmuldoon commented Jun 5, 2023

🐛 test/e2e check for machines being ready after provisioning on Runtime SDK test #8625

🐛 test/e2e check for machines being ready after provisioning on Runtime SDK test #8625

Conversation

chrischdi commented May 9, 2023 • edited Loading

elmiko left a comment

Choose a reason for hiding this comment

chrischdi commented May 9, 2023

chrischdi commented May 9, 2023

k8s-infra-cherrypick-robot commented May 9, 2023

chrischdi commented May 9, 2023

k8s-infra-cherrypick-robot commented May 9, 2023

sbueringer May 9, 2023

Choose a reason for hiding this comment

chrischdi May 9, 2023 • edited Loading

Choose a reason for hiding this comment

sbueringer May 10, 2023

Choose a reason for hiding this comment

chrischdi May 10, 2023

Choose a reason for hiding this comment

sbueringer May 12, 2023

Choose a reason for hiding this comment

ykakarap left a comment

Choose a reason for hiding this comment

chrischdi commented May 9, 2023

ykakarap commented May 9, 2023

killianmuldoon left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented May 11, 2023

sbueringer commented May 12, 2023

k8s-ci-robot commented May 12, 2023

k8s-infra-cherrypick-robot commented May 12, 2023

k8s-infra-cherrypick-robot commented May 12, 2023

johannesfrey commented Jun 5, 2023

killianmuldoon commented Jun 5, 2023

chrischdi commented May 9, 2023 •

edited

Loading

chrischdi May 9, 2023 •

edited

Loading