Automatically deleted shovels reappear on topology operator restart

[Xploeng]

Describe the bug

Shovels that are declared using the topology operator reappear after they delete themselves when the topology operator restarts.

Shovels that have 'srcDeleteAfter' set to anything else than 'never' delete themselves at some point, but the Custom Resource Defintion in Kubernetes remains. That leads to the topology operator declaring the shovel again if it restarts.

To Reproduce

Steps to reproduce the behavior:

1. Declare an arbitrary shovel with srcDeleteAfter = queue-length or some int value
2. Wait until the shovel transfers the specified amount of messages and deletes itself
3. Restart the message topology operator
4. The shovel will be declared again

Expected behavior
I would expect the shovel to stay deleted even after topology operator restart. Not sure if the CRD should be deleted too.

Version and environment information

• Messaging Topology Operator: Bitnami chart 1.12.0
• RabbitMQ: RabbitMQ 3.11.23
• RabbitMQ Cluster Operator: Bitnami chart 2.5.0
• Kubernetes: v1.27.4

Additional context

I'm trying to use a shovel as a one time operation to transfer some messages from a stream to a work queue. The work queue then persists, and after the initial transfer, only gets messages via an exchange. This all works fine until the first restart of the topology operator deployment, when the shovel is declared again and moves the messages from the stream into the work queue again.

This is the topology operator log after restart. The declaration before the restart looks the same.

{"level":"info","ts":"2024-06-09T12:58:17Z","msg":"Start reconciling","controller":"shovel","controllerGroup":"rabbitmq.com","controllerKind":"Shovel","Shovel":{"name":"label-worker-sampled-frames-labeling-shovel","namespace":"oc-dev-jrau"},"namespace":"oc-dev-jrau","name":"label-worker-sampled-frames-labeling-shovel","reconcileID":"f90b30b0-fef1-4bac-9ece-344782ab1633","spec":"{\"name\":\"sampler_cvat_worker-shovel\",\"vhost\":\"/\",\"rabbitmqClusterReference\":{\"name\":\"oc\",\"namespace\":\"rabbitmq\"},\"uriSecret\":{\"name\":\"shovel-secret\"},\"ackMode\":\"on-confirm\",\"srcDeleteAfter\":\"1000\",\"destQueue\":\"sampler_cvat_worker\",\"srcQueue\":\"oc-dev-jrau_sampled_frames\",\"srcConsumerArgs\":{\"x-stream-offset\":\"first\"}}"}

{"level":"info","ts":"2024-06-09T12:58:18Z","msg":"Successfully declare shovel","controller":"shovel","controllerGroup":"rabbitmq.com","controllerKind":"Shovel","Shovel":{"name":"label-worker-sampled-frames-labeling-shovel","namespace":"oc-dev-jrau"},"namespace":"oc-dev-jrau","name":"label-worker-sampled-frames-labeling-shovel","reconcileID":"f90b30b0-fef1-4bac-9ece-344782ab1633"}

{"level":"info","ts":"2024-06-09T12:58:18Z","msg":"Finished reconciling","controller":"shovel","controllerGroup":"rabbitmq.com","controllerKind":"Shovel","Shovel":{"name":"label-worker-sampled-frames-labeling-shovel","namespace":"oc-dev-jrau"},"namespace":"oc-dev-jrau","name":"label-worker-sampled-frames-labeling-shovel","reconcileID":"f90b30b0-fef1-4bac-9ece-344782ab1633"}

[github-actions]

This issue has been marked as stale due to 60 days of inactivity. Stale issues will be closed after a further 30 days of inactivity; please remove the stale label in order to prevent this occurring.