zookeeper lock znode not getting cleared after workflows are finalized

[mithlesh135]

Describe the bug
As zookeeper takes lock for any operation, it creates znodes in below format
/conductor/conductor-lock/workflow-uuid/sequential-epephermal-uuid

Now these epephermal nodes get deleted but the the znode at workflow uuid is persistence once and is not getting deleted, and and we have more and more workflows getting processed, number of these persistence znodes increase inside the parent conductor-lock znode. Zookeeper suggests max size of znode to be 1 MB, this ever growing znode is a problem.

To Reproduce
Steps to reproduce the behavior:

1. Go to conductor
2. connect to zookeeper server
3. run a workflow and check znodes getting created for workflow
4. even after worklow execution has completed, workflow znode stays

Expected behavior
workflow znode should be deleted after workflow is finalized

[v1r3n]

Hi @mithlesh135 I would recommend using Redis for locks. Zookeeper support exists for the legacy reasons and is quite a heavy solution to maintain. Redis based locks are going to be much more lightweight.