TestReconcileMachinePoolScaleToFromZero is flaky

[killianmuldoon]

This unit test has been flaky for some time now and the flake needs investigation.

Link to the flakes:

• https://storage.googleapis.com/k8s-triage/index.html?job=.*-cluster-api-.*&xjob=.*-provider-.*#7cc32c63e433b1623bf8
• https://storage.googleapis.com/k8s-triage/index.html?job=.*-cluster-api-.*&xjob=.*-provider-.*#a4368a09373db948f1f9

/kind flake

[killianmuldoon]

@Jont828 given your recent work on MachinePools maybe you have some time to look into this one?

[a-hilaly]

I tried to run the test locally multiple times, not able to reproduce the errors seen in https://storage.googleapis.com/k8s-triage/index.html?job=.*-cluster-api-.*&xjob=.*-provider-.*
Is there a better way to chase this flaky test @killianmuldoon ?

[killianmuldoon]

@a-hilaly - you might have to run these tests thousands of times locally to catch the flake I'm afraid. The easiest way to do this is to add a for loop inside the test and run it locally as many times as you need to reliably catch the flake.

[Jont828]

@killianmuldoon I'm trying to wrap up some PRs in CAPI and CAPZ right now but I can take a look when I have some time. FYI I believe those links to the flakes aren't working as it's not in the last week of data now.

[killianmuldoon]

/triage accepted

[adilGhaffarDev]

@Jont828 Here are more persistent links:

• https://storage.googleapis.com/k8s-triage/index.html?text=TestReconcileMachinePoolScaleToFromZero&job=.*-cluster-api-.*&xjob=.*-provider-.*#a4368a09373db948f1f9
• https://storage.googleapis.com/k8s-triage/index.html?text=TestReconcileMachinePoolScaleToFromZero%2FShould_set_%60Running%60_when_scaled_from_zero_to_one&job=.*-cluster-api-.*&xjob=.*-provider-.*

[loktev-d]

@killianmuldoon hi! I'm new to the project and looking for ways to contribute, can I take a look at this issue?

[chrischdi]

@loktev-d , feel free to investigate. However finding the root cause and resolution seems not to be easy or straight forward as both are currently unknown and this is a very complex part of CAPI. This issue might not be a perfect fit for you to get started in the project.

[Jont828]

@loktev-d I was busy trying to push to get DockerMachinePools #8842 in and couldn't get to this. It looks like you opened a PR already, is it safe to say that you're picking this issue up?

[loktev-d]

@Jont828 Yes, forgot to assign myself

/assign