[Bug] failed node is not removed

[shawncao]

We need have to have a regular routine to ensure all the nodes are up running, which means stale nodes and their data blocks need to be removed. Keeping data blocks up to date and fresh is critical.

Currently, there is a race condition that "dead nodes" could stay in meta data tracking list, we need mechanism to clean it up to ensure all query won't hit and timeout on those dead nodes.