Memberlist: fix "forget"

[pstibrany]

What this PR does: This PR fixes bug in "forget" feature when using memberlist.

When using memberlist (gossip), "forgetting" an entry doesn't delete it from the KV store, but sets entry to the "LEFT" state instead. Reason is that memberlist needs something to gossip about, and it can gossip about "LEFT" entry but not deleted entry. Unfortunately during this step, we didn't set Timestamp to current time, but reused last Timestamp of the entry. If there was already another message in the gossip network with higher timestamp, such message would eventually propagate everywhere, "resurrecting" the forgotten entry (because of higher timestamp, even though it's still in the past).

Solution is to set entry timestamp to "now" when setting entry to LEFT state.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[ranton256]

LGTM except the tenant deletion proposal is included, and assume that was meant to be separate.

[pracucci]

Good finding, fix LGTM!

[nicolai86]

this change improves the issue of "ghost" entries in the ring after scaling down but doesn't seem to fix the issue entirely.

for verification of this patch we quickly scaled up store-gateways from 12 to 80 instances and back down to 12, simulating a burst in auto scaling. About 3 1/2 hours after downscaling without any further scaling activity the store gateway ring reports unhealthy members in ring every 20 minutes apart.

instance ids are random members from the previous scale up. Any ideas how to triage this issue and resolve it for good?

[pracucci]

this change improves the issue of "ghost" entries in the ring after scaling down but doesn't seem to fix the issue entirely.

@nicolai86 I would suggest you to open an issue instead of commenting on a merged PR (then the issue can reference this PR). Moving forward the discussion in an issue looks the more appropriate place.