[RFE] Leverage resource-sync mechanism for resources that need created before cluster-agent comes up

[thatmidwesterncoder]

Is your feature request related to a problem? Please describe.

Currently there is a gap where certain resources that are required by system components (e.g. additionalManifests) are stored in plain text on the cluster object. After rancher/rancher#45718 is completed there will be a way to synchronize resources (at first just secrets) to the downstream clusters during the provisioning process by way of a specific annotation (docs soon).

The UI should use this feature where necessary to ensure that potentially sensitive manifests do not end up on disk.

Describe the solution you'd like

When creating a cluster through the wizard - the UI should create other resources (namely secrets) first with the appropriate annotation and then the cluster object, which would allow the provisioning process to bring up the cluster -> create those secrets -> then continue the provisioning process.

Describe alternatives you've considered

Mentioned in the r/r issue above, namely using additionalManifests or machineSelectorFiles. But those require the files to be placed on disk before being applied by RKE2.

Additional context

This will most likely complicate the UI in certain places (the additional config tab mostly) because creates will create the secret, then updates will update said secret. Hopefully it can be abstracted away to hook into it wherever there is something sensitive.

Related: Parent - SURE-5434, UI - SURE-9078

[richard-cox]

@Sahota1225 , @thatmidwesterncoder We need to discuss the implementation before starting this, was there an RFC?

[gaktive]

/backport v2.9.next2

[richard-cox]

See UI JIRA for implementation details SURE-9078

[richard-cox]

Blocked on SURE-5434

[richard-cox]

/backport v2.8.9

[richard-cox]

Blocked on #12066

[izaac]

Rancher version: v2.10-ae9edd7151ea0a2387b0dcf3d9387eff74d9d101-head
UI: master 4e82a4d

Tested on vSphere 8 in a vSphere local cluster using Downstream vSphere clusters.
The vSphere CPI and CSI apps create secrets on the local (Rancher) cluster. vsphere-config-secret and vsphere-cpi-creds

These are referenced during vSphere provisioning using the vSphere cloud provider. Deselecting auto generate secrets from the Add Ons, the secrets are automatically picked up and used for pre-bootstrap during provision.

Scope here was only vSphere. As per the Linked PRs.
Reference backport: #12065 (comment)