[BUG] Pods started before InjectFinalizer is disabled are never deleted

[hamersaw]

Describe the bug

The k8s plugin configuration has an InjectFinalizer option. In FlytePropeller this is used to add finalizers to new pods and clear finalizers from completed pods. When a pod is started with the InjectFinalizer option enabled and then it is disabled before pod completion the flyte finalizer remains on the pod indefinitely. This means that pod is never deleted.

Expected behavior

Pods should be able to complete if the InjectFinalizer configuration option changes during evaluation.

Additional context to reproduce

1. Enable the InjectFinalizer configuration option

plugins:
  k8s:
    inject-finalizer: true

2. Start any FlyteWorkflow
3. Disable the InjectFinaizer configuration option

plugins:
  k8s:
    inject-finalizer: false

4. View pod to see it's stuck in Terminating state, has the Flyte finalizer, and has been deleted

hamersaw@ragnarok:~$ kubectl -n flytesnacks-development get pods
NAME              READY   STATUS        RESTARTS   AGE
f1m271n9tr-n0-0   0/1     Terminating   0          65s

hamersaw@ragnarok:~$ kubectl -n flytesnacks-development get pod f1m271n9tr-n0-0 -o yaml
apiVersion: v1
kind: Pod
metadata:
  ...
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2022-03-16T14:33:43Z"
  finalizers:
  - flyte/flytek8s
  ...

Screenshots

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes