Error on upgrade from 1.7.0 to 1.8.0 on OpenShift through OLM

[rockaut]

There is already an open Datadog case (1829977) for this. This is just a heads up for everybody else who runs into this ;-)

Describe what happened:
We upgraded the cluster through an installplan from 1.7.0 to 1.8.0. The upgrade got stuck in pending with an error on "possible data loss" because the removed v1alpha1 CRD of DatadogAgent was still in the CRDs .status.storedVersions . We then needed to tediously remove the installplans, crds and even the installed operator itself to get a clean state again.

The correct way to do this according to the k8s docs (https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definition-versioning/) is to remove the .status.storedVersion BEFORE you start the upgrade as with the upgrade the crd version will be removed.

We don't know if there even is a way to do this with OLM automatically but at least it should be noted in an upgrade notice.

Additional environment details (Operating System, Cloud provider, etc):
OpenShift 4.14
Operator 1.7.0 and 1.8.0 installed through OperatorHub

[celenechang]

Thank you for reporting @rockaut , and many apologies for the trouble. We will add documentation for this OLM upgrade issue to 1.8.

[tbavelier]

Thank you again for the report @rockaut and sorry for the inconvenience, we reproduced and added resolution steps that should be slightly less disruptive.