Velero CRD error in CI

[mjnagel]

Environment

Device and OS: github runner
App version: 0.17.0
Kubernetes distro being used: k3d
Other: this is in CI

Steps to reproduce

This is an intermittent issue in CI and hard to reproduce consistently, one example: https://github.com/defenseunicorns/uds-core/actions/runs/8422543929/job/23063288892?pr=190#step:8:1663

It appears that the velero CRDs are not getting installed in the proper order - maybe related to something weird with the hook job (does not appear to have run per the events?).

Expected result

UDS Core deploys successfully

Actual Result

      WARNING  Unable to complete helm chart install/upgrade, waiting 10 seconds and trying again: unable
 to build kubernetes objects from release manifest: [resource mapping not found for name:
 "default" namespace: "velero" from "": no matches for kind "BackupStorageLocation" in
 version "velero.io/v1" ensure CRDs are installed first, resource mapping not found for
 name: "velero-udsbackup" namespace: "velero" from "": no matches for kind "Schedule" in
 version "velero.io/v1" ensure CRDs are installed first]

Severity/Priority

Completely fails core installs when encountered - retries do not help.

Additional Context

It may be worth identifying if this is happening more frequently with upgrades? Could be some strange hook behavior on upgrade?

[mjnagel]

While this doesn't answer the why, separating out the CRDs into a separate chart/manifest references that happen before the main velero chart would help here. Reference https://www.github.com/vmware-tanzu/helm-charts/issues/421 - we could contribute to this issue.

[mjnagel]

More recent failure - https://github.com/defenseunicorns/uds-core/actions/runs/9212151193/job/25345235716?pr=407

[mjnagel]

Tentatively closing this one based on current CI results - have not seen this since the k3s update - #426.