✨ Add Metro Support

[cprivitere]

User Story

Support metro based provisioning. https://metal.equinix.com/developers/docs/locations/metros/

2023-05-01 Update (MJ): see also https://feedback.equinixmetal.com/changelog/bye-facilities-hello-again-metros

Detailed Description

As information becomes available this description will be updated and comments will be added to support the implementation changes.

Documentation should be broadly updated to demonstrate metros. A few smaller examples can demonstrate Facilities.

/kind feature

From a recent discussion on Slack/Zoom with Moath on the K8S Slack, these talking points came up:

• Metro and Facility should both be optional, there is exclusivity in the EM API and we could replicate that behavior in Cluster and Machine definitions
• Should control plane nodes be forced to reside in a single metro?
• These fields are not immutable today, and we should continue that practice. If a Facility is changed, the machine(s) would need to be recreated in the EM API. One exception to this would be that if a machine is converted from “facility: da11” to “metro: da”, the desired metro is already present. This would not necessitate a reprovision.
• The v1alpha3 types are being updated in this PR today. Do we want to introduce change to the old definitions?
• With respect to the ability to use private IPs (Create control plane without publicly addressable IPs #226):
  • Today, the control plane IP is public. With private networking, the control plane may not be accessible to all nodes.
  • Nodes in the same metro would share access to the same management network control plane (10/8)
  • Nodes in a project with project scoped backend-transfer could share the same management control plane (10/8) https://metal.equinix.com/developers/docs/networking/backend-transfer/
  • Nodes with hybrid-networking within the same metro would share access to the same VLAN network control plane (any IPs, the VLAN addresses and must configured via cloud-init)
• Projects may be hosting multiple clusters, so settings like “which metros are enrolled in backend transfer” may bleed over into other cluster settings.

Tasks

Beta Give feedback

1. Decide if metro and/or facility will be optional.
2. Create a plan for How to / should we upgrade users to metros if they currently use facilities?
3. Replicate metro exclusivity from the Equinix Metal API within Cluster and Machine definitions
4. Decide if control plane nodes should be forced to reside in a single metro.
5. Drop v1alpha3 types from the PR.

[davidspek]

I think it should be handle the same as the CPEM. It seems like the first choice is Facility, and if a Facility isn't defined it falls back to Metro, if also no Metro is defined it results in an error.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[cprivitere]

/remove-lifecycle stale

[cprivitere]

Sounds like we've decided facility and metro should be mutually exclusive arguments where one is required.