🐛 KCP rollout maintains even distribution of failure domains

[benmoss]

What this PR does / why we need it:
Makes it so that KCP will not include outdated machines in its determination of which failure domain to place new machines into. This was causing us to get into situations where we'd have an uneven distribution of machines across failure domains.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #3396

[vincepri]

/assign
/milestone v0.3.8

[detiber]

/assign
reviewing this now

[detiber]

I'm wondering if it would make sense to push the logic for retrieving the KubeadmConfigs and related infrastructure resources into FilterableMachineCollection somehow, that might allow for a better way to join the resources in the various places we intend to use them without having to pass in full maps of KubeadmConfigs and infrastructure resources in all the places that are consuming it currently.

[benmoss]

/test pull-cluster-api-test
/test pull-cluster-api-e2e

[benmoss]

I'm wondering if it would make sense to push the logic for retrieving the KubeadmConfigs and related infrastructure resources into FilterableMachineCollection somehow, that might allow for a better way to join the resources in the various places we intend to use them without having to pass in full maps of KubeadmConfigs and infrastructure resources in all the places that are consuming it currently.

Yeah I thought about this too. We could change FilterableMachineCollection to be a collection of something like

type Machine struct {
    machine clusterv1.Machine
    infraMachine unstructured.Unstructured
    kubeadmConfig bootstrap.KubeadmConfig
}

to make it so we keep the connections between those objects in the object graph.

[vincepri]

@detiber wdyt of the above?

[detiber]

+1 from me to enriching the FilterableMachineCollection in the way suggested

[vincepri]

@benmoss Any updates here?

[benmoss]

+1 from me to enriching the FilterableMachineCollection in the way suggested

After looking into this a bit and spiking it a little here, I spoke with @detiber and we decided it was probably better to defer this refactoring since it's more involved than we imagined.

[detiber]

/test pull-cluster-api-e2e
/lgtm

[vincepri]

@benmoss Squash commits?

[vincepri]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[vincepri]

/assign @detiber @ncdc

[detiber]

/lgtm

[benmoss]

Looks like prow is just a lil busted right now

[vincepri]

/retest