Consider dropping MachinePools from ClusterClass self-hosted tests

[killianmuldoon]

We should consider dropping MachinePools from the self-hosted ClusterClass tests in order to stabilize the signal ahead of the release of v1.6.0 on November 28th.

As described in #9522 the self-hosted ClusterClass tests are very flaky on the main branch of Cluster API. These tests are much more stable on release-1.4 and release-1.5.

At least some of the flakiness in the self-hosted tests is down to the addition of MachinePools to a number of test templates in #9393. There's a bug in the DockerMachinePool implementation - #9655 - that causes CAPI components in a self-hosted cluster using MachinePools to become unstable.

I don't think this new bug was introduced in the 1.6 release cycle. Before introducing MachinePools in ClusterClass there is only one e2e tests covering MachinePools - covering creation, scaling and deletion. There was no coverage for self-hosted Clusters using MachinePools.

In my view if these tests aren't stable on main by the publication of CAPI 1.6.0-rc.0, due to be published on 14th November, we should drop this test case from the release branch. We should maintain the tests on main with a view to fixing them, and if possible backporting the fixes.

[killianmuldoon]

/triage accepted

[CecileRobertMichon]

I don't think this new bug was introduced in the 1.6 release cycle. Before introducing MachinePools in ClusterClass there were no e2e tests covering MachinePools - e.g. the e2e tests running on release-1.5 have no MachinePools in the artifacts AFAICT.

I don't think this is true, there is a MachinePools test in the framework capi-e2e.[It] When testing MachinePools Should successfully create a cluster with machine pool machines https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-main

[killianmuldoon]

I don't think this is true, there is a MachinePools test in the framework capi-e2e.[It] When testing MachinePools Should successfully create a cluster with machine pool machines testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-main

Thanks for catching that - bad searching on my part - updated the issue to be more accurate.

[killianmuldoon]

/assign @nawazkh

/assign @adilGhaffarDev

[k8s-ci-robot]

@killianmuldoon: GitHub didn't allow me to assign the following users: adilGhaffarDev.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @nawazkh

/assign @adilGhaffarDev

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[CecileRobertMichon]

We shouldn't need to do this now that #8842 has merged. Let's keep monitoring the signal in the next few days and close this issue if we don't see the flake anymore.

[nawazkh]

We should be closing out this issue now since we have decided to not to drop the ClusterClass self-hosted tests and that #8842 has merged.
Moreover, the associated PR #9672 to this issue has been closed.

[nawazkh]

/close

[k8s-ci-robot]

@nawazkh: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.