ES secure settings finalizer hangs

[anyasabo]

Sometimes when I try to delete an ES instance, it can get hung with a malformed finalizer on it. Deleting the finalizer lets it delete, but I wonder why it happens in the first place. Here's the ES object once it gets into that state

apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
kind: Elasticsearch
metadata:
  annotations:
    common.k8s.elastic.co/controller-version: 0.9.0-SNAPSHOT
  creationTimestamp: "2019-07-17T21:34:16Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2019-07-18T00:08:32Z"
  finalizers:
  - secure-settings.finalizers..k8s.elastic.co
  generation: 2
  name: elasticsearch-sample
  namespace: e2e

and gist of the operator logs grepping for the ES name:
https://gist.github.com/anyasabo/11c93f3e016418efef9cea60bc66198a

You'll see there's a couple logs of it executing but not getting removed.

{"level":"info","ts":1563397407.7224836,"logger":"finalizer","msg":"Registering finalizer","finalizer_name":"secure-settings.finalizers..k8s.elastic.co","namespace":"e2e","name":"elasticsearch-sample"}
...
{"level":"info","ts":1563399256.390922,"logger":"finalizer","msg":"Registering finalizer","finalizer_name":"secure-settings.finalizers..k8s.elastic.co","namespace":"e2e","name":"elasticsearch-sample"}

I see we do some string manipulation here that looks like it might be responsible, since if getKind() returns the empty string then it would produce that finalizer. I don't know enough about how all this fits together to come up with a complete root cause, but wanted to open the issue to track it. I didn't see any existing ones similar to this at the moment.

[anyasabo]

Closing this for now because I can't reproduce it on master at the moment

[barkbay]

I think I had the same issue.
There's 2 related fixes in the controller-runtime: kubernetes-sigs/controller-runtime#212 and kubernetes-sigs/controller-runtime#389
We have the first one but not the second one.

I'm not sure we should consider getKind() as reliable for 0.9 and replace it with a constant.