[RFE] Clean up stale IP

[lubronzhan]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Found this doc https://gist.github.com/dougbtv/076d63f5204a35d65ab4952e77199d56
But it would be best if we could have a process that monitors whether there are stale IPs need to be cleaned up

Describe the solution you'd like
Have a separate process that cleans up stale IP. Might need to lock the allocation process when cleaning up in case of race condition

Describe alternatives you've considered

Additional context
Add any other context or screenshots about the feature request here.