Implementing Spread Minimizing Token Generator

[alanprot]

What this PR does:
This PR implements the Spread Minimize Token Generator algorithm based on grafana/dskit#321 by GL (thank you), but the implementation is slightly different.

The algorithm implemented in the DSKit package creates tokens in "order", and so, all the tokens need to be created using this algorithm and require a reshard for existing clusters.

The proposal in this PR is slightly different. Here, we create tokens based on the state of the ring right at a given moment, and so, we can achieve similar benefits as soon as new ingesters join the ring (no need to reshard all existing ingesters).

The algorithm is basically as follows:
When registering a new ingester, we first:

• Determine the expected ownership of the new ingester based on the existing ingesters.
• Determine the expected ownership of each token of this new ingester:
  • Look at all ingesters present in the same AZ and sort them (heap) based on their ownership.
  • For each new token that needs to be generated, we:
    • Pick the existing ingester with the higher ownership in the same AZ.
    • Pick a token of this ingester that has a distance greater than the expected distance (ED) for each new token.
    • Split that token, creating a new token with the expected distance ED.
    • Decrement the distance of the existing ingester by ED.

One edge case to be aware of is if we are scaling multiple ingesters at the same time, we could end up generating multiple tokens close together. To avoid this, if multiple ingesters are being scaled up simultaneously, we only use the new strategy on one of them. This seems to be working well so far, but if we encounter any problems with this approach, we can force the ingesters to join the ring one by one.

If we decide to merge this PR, I recommend leaving this feature as experimental for now and, therefore, hidden from the documentation.

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• [NA] Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[yeya24]

Great work. Any reason it is still a draft?