Fix/lifecycler/stuck in leaving

[thorfour]

What this PR does: I've observed ingesters that were evicted and then rescheduled. They were unable to gracefully shutdown and left their state in the ring as LEAVING. On reschedule the ingester found itself in the LEAVING state. Set itself to the LEAVING state and was then unable to join the cluster.

This allows an ingester that finds itself in the LEAVING state with tokens to mark itself as ACTIVE.

Which issue(s) this PR fixes:
N/A

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[codesome]

Were you able to find out why the ingester was stuck in LEAVING state? I found myself in the same situation when working with the WAL and it turned out that I had termination grace period of Kubernetes set very low (30sec). Hence before the ingester could exit gracefully, it was killed, leaving the ring in LEAVING state.

Also, this takes care of this issue #1904 (created when I faced the exact same issue)

[pracucci]

I see the problem you're trying to fix. The change sounds reasonable to me, but I'm not familiar enough with all the ring's edge cases. @pstibrany what's your take on this?

[pstibrany]

Overall, this fixes the problem when ingesters reuse their names (eg. statefulsets).

[khaines]

This will certainly help the statefulset ingester setup when recovering from an unclean shutdown. Thank you!

[canghai118]

If ingester in deploy use k8s deployment, every time ingester recreate by k8s, it will register itself with different hostname and address. the zombie ingester will stay in ring for ever, and finally the clients try to read/write will report too many failed ingesters.

If the ingester is kill by oom-killer, it won't start the quit process, and will leave state in ACTIVE, I just encounter this problems on ruler #1956

By the way , after the success transfer or flush, ingester will sleep for -ingester.final-sleep, after that it unregister it self from ring. Default value for -ingester.final-sleep is 30seconds, the default grace termination period for k8s pod is 30s, it will probably cause trouble if both parameter use default value.

the same problem happens on ruler too.

May be we need a away to cleanup dead ingester from ring.

@thorfour @khaines @pstibrany

[thorfour]

@canghai118 I'm not sure there's a safe way to determine if an ingester is a "zombie" or if it's just crash looping and will come back eventually. I'd worry that trying to perform cleanups on what we believe to be a zombie ingester might cause more problems than it solves.