OpenShift 3.11 Projects with Grafana Operator and related CRs get stuck in "Terminating" state???

[jtudelag]

Hi,

First of all, thanks for this awesome operator, is very useful!!!

I'm seeing this behaviour consistently. I haven't gone into more details,
but when I have the Grafana Operator deployed on a project, along with Grafana, GrafanaDashboards and GrafanDataSources CRs, and I delete the Project, the project gets stuck in "Terminating" state.

I guess it has to be realated with the k8s finalizers and the order in which resources are deleted....

[jtudelag]

Contunuing with the troubleshooting, I see that Dashboards and DataSources objects both have finalizers, while Grafana does not...and I think the root cause...

kind: GrafanaDataSource
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"integreatly.org/v1alpha1","kind":"GrafanaDataSource","metadata":{"annotations":{},"name":"prometheus-ds","namespace":"jorge-prom"},"spec":{"datasources":[{"access":"proxy","editable":true,"isDefault":true,"jsonData":{"timeInterval":"5s","tlsSkipVerify":true},"name":"Prometheus","type":"prometheus","url":"http://prometheus-operated:9090","version":1}],"name":"prometheus-ds.yaml"}}
  creationTimestamp: 2019-11-22T18:47:41Z
  finalizers:
  - grafana.cleanup
  generation: 1
  name: prometheus-ds
.....

apiVersion: integreatly.org/v1alpha1
kind: GrafanaDashboard
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
  finalizers:
  - grafana.cleanup
  generation: 2
  labels:
    app: grafana
  name: kafka-dashboard
  namespace: puto-grafana
  resourceVersion: "20995558"
  selfLink: /apis/integreatly.org/v1alpha1/namespaces/puto-grafana/grafanadashboards/kafka-dashboard
  uid: 98f12603-10ee-11ea-9a3b-0654d1207a24
spec:
  json: |

[jtudelag]

Actually If I try to delete a dashboard of a data source, sth like this oc delete grafanadashboard kafka-dashboard , the object is not deleted, I have to delete the finalizer of the object, then the object dissapears... Once all the data sources and dashboards are deleted, the project is finally deleted by OCP ;).

[pb82]

@jtudelag Yes, that is a common problem with finalizers: the operator gets deleted before it has a chance to service the finalizers on the other resources. Kubernetes won't delete resources until all finalizers are removed and this propagates upwards so that the namespace can't be deleted because of the stuck resources.

One solution to this are ownerReferences that block the owner deletion. But we decided to get rid of finalizers alltogether. This work is currently ongoing in the development branch: https://github.com/integr8ly/grafana-operator/tree/development

It will be included in the next release (mid next week) along with other improvements.

[pb82]

@jtudelag Can you please give the new release (https://github.com/integr8ly/grafana-operator/releases/tag/v3.0.0) a try? It does no longer use finalizers and should solve your problem. It should soon also be available on Operatorhub.