Dualstack tests failing

[killianmuldoon]

Which jobs are failing?

https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-dualstack-and-ipv6-main

Which tests are failing?

A number of tests are failing consistently. Logs etc. are available here: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-e2e-dualstack-and-ipv6-main/1692233880087564288

Since when has it been failing?

Since August 17th

Testgrid link

https://testgrid.k8s.io/sig-cluster-lifecycle-cluster-api#capi-e2e-dualstack-and-ipv6-main

Reason for failure (if possible)

No response

Anything else we need to know?

The dualstack tests for KIND were failing in the exact same way for a day or so - the solution in CAPI is likely the same as in KIND

Ref: https://testgrid.kubernetes.io/sig-network-kind#sig-network-kind,%20dual,%20master

Label(s) to be applied

/kind failing-test

@kubernetes-sigs/cluster-api-release-team
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[killianmuldoon]

/triage accepted

[sbueringer]

How is this related to #8816?

[killianmuldoon]

How is this related to #8816?

I don't think this is related at all. Different tests failing, and failing consistently.

[killianmuldoon]

This also impacts the 1.5 branch.

Marking this as release blocking - I think this is something that has to be investigated and remediated before releasing v1.5.1.

/kind release-blocking

@kubernetes-sigs/cluster-api-release-team

[sbueringer]

@chrischdi If you have some time, would be good if you can take a look. I assume it's a consequence of our 1.28 bump (?)

[chrischdi]

Jep, was the only one which was merged in that time window if I got it right. I'll take a look.

[killianmuldoon]

@chrischdi - just to underline this which is probably related:

The dualstack tests for KIND were failing in the exact same way for a day or so - the solution in CAPI is likely the same as in KIND
Ref: testgrid.kubernetes.io/sig-network-kind#sig-network-kind,%20dual,%20master

[chrischdi]

I think this is the related change:

Issue:

• Dual Stack tests are broken kubernetes/kubernetes#119927

PR:

• e2e WaitForServiceEndpointsNum take into account the address family kubernetes/kubernetes#119928

[killianmuldoon]

So I wonder are we just waiting for a kubekins auto-bump to fix this?

[chrischdi]

So I wonder are we just waiting for a kubekins auto-bump to fix this?

Nope, the conformance image is derived from the kubernetes version.

So we currently use registry.k8s.io/conformance:v1.28.0 (for ci versions: gcr.io/k8s-staging-ci-images/conformance), which does not have the above fix.

[chrischdi]

The fix currently get's cherry-picked into release-1.28:

• Automated cherry pick of #119928: e2e WaitForServiceEndpointsNum take into account the address kubernetes/kubernetes#119964

[sbueringer]

tl;dr if the cherry-pick gets merged, we're good after the 1.28.1 release? (and we only have to bump on our side, otherwise no changes needed)

[chrischdi]

tl;dr if the cherry-pick gets merged, we're good after the 1.28.1 release? (and we only have to bump on our side, otherwise no changes needed)

Yes, but with the caveat that: if there will not be a new kindest/node image for 1.28.1: we would always build it in our pipeline.

[sbueringer]

I think we could temporarily set the ConformanceImage here:

cluster-api/test/e2e/quick_start_test.go

Lines 124 to 128 in caaa744

	kubetest.RunInput{
	ClusterProxy: proxy.GetWorkloadCluster(ctx, namespace, clusterName),
	ArtifactsDirectory: artifactFolder,
	ConfigFilePath: "./data/kubetest/dualstack.yaml",
	},

(and then get rid of it either if we bump to 1.28.{>=1} or 1.29)

[furkatgofurov7]

The fix currently get's cherry-picked into release-1.28:

• Automated cherry pick of #119928: e2e WaitForServiceEndpointsNum take into account the address kubernetes/kubernetes#119964

I reached out to release managers over slack asking a help merging it

[furkatgofurov7]

The fix currently get's cherry-picked into release-1.28:

• Automated cherry pick of #119928: e2e WaitForServiceEndpointsNum take into account the address kubernetes/kubernetes#119964

I reached out to release managers over slack asking a help merging it

Looks like, it is a bit late for cherry-pick PR to make it to 1.28.1 due to cherry-picks being in a freeze right now, so it might need to wait for 1.28.2 scheduled in the September patch cycle.

[sbueringer]

Looks like, it is a bit late for cherry-pick PR to make it to 1.28.1 due to cherry-picks being in a freeze right now, so it might need to wait for 1.28.2 scheduled in the September patch cycle.

That's fine. More out of curiosity, are they already releasing 1.28.1 after the minor release last week? Or are they just releasing patch releases for the other minors?

[furkatgofurov7]

Looks like, it is a bit late for cherry-pick PR to make it to 1.28.1 due to cherry-picks being in a freeze right now, so it might need to wait for 1.28.2 scheduled in the September patch cycle.

That's fine. More out of curiosity, are they already releasing 1.28.1 after the minor release last week? Or are they just releasing patch releases for the other minors?

Yes and yes. They cut patch releases from all minor release branches at once / on the same day: https://kubernetes.io/releases/patch-releases/#upcoming-monthly-releases