Consistent hashing

[fpetkovski]

This commit adds support for distributing series in Receivers using
consistent hashing based on the libketama algorithm.

The following article provides a great summary of all available options: https://dgryski.medium.com/consistent-hashing-algorithmic-tradeoffs-ef6b8e2fcae8.

We can also use Multi-Probe Consistent Hashing which might have a better performance.

Fixes #4972

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Add support for consistent hashrings in Thanos Receiver.

Verification

Unit tests.

[fpetkovski]

For the purpose of testing, I disabled flushing the TSDB when the hashring changes. Here's how head series look like now when a new receiver is added.

The receiver-3 instance came up around 11:16, but receivers 1 and 2 had no bump in head series.
One thing I noticed is that the sharding algorithm is not as balanced as hashmod, but this might not be a big problem in practice.

[GiedriusS]

LGTM, let's just use testutil.* helpers instead of t.Fatalf and so on

[bwplotka]

Amazing work, will check after my PTO (tomorrow+)! Kind Regards, Bartek Płotka ***@***.***)

…

On Tue, Jun 14, 2022 at 1:44 PM Giedrius Statkevičius < ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In pkg/receive/hashring_test.go <#5408 (comment)>: > + { + Name: "pod", + Value: "thanos", + }, + }, + }, + expectedNode: "node-3", + }, + } + + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + hashRing := newConsistentHashring(test.nodes, 10) + result, err := hashRing.GetN("tenant", test.ts, test.n) + if err != nil { + t.Error(err) Still not using any helpers here ;P — Reply to this email directly, view it on GitHub <#5408 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVA3O324ZSL5ZUQ2GNYHLTVPBWAFANCNFSM5YFXUAOQ> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[douglascamata]

Should this be called "ketama" instead of "consistent"? So that other consistent hashing algorithms can be added in the future without breaking backwards compatibility.

[fpetkovski]

I agree, we should give it a more specific name. We can change it to either ring-hash, ketama, dynamo or something along those lines. I'm open to suggestions :)

[douglascamata]

ketama sounds like a good one to me. It's one algorithm that uses a ring-hash. There could be other algorithms using ring-hashes and then we would have a name conflict.

[fpetkovski]

Sounds good 👍 Renamed to ketama

[bill3tt]

I'm wondering whether we want to provide users with the option to specify a hashing algorithm at all? Is consistent hashing the first best solution here?

I know I'm not the only one that thought Receive was already using consistent hashing, I'm wondering what use-cases folks would people have for using hashmod based algorithm 🤔

The downside to just changing the default algorithm would be in the upgrade path. Someone that was using hashmod, that then upgraded to consistent hashing would then see data being reallocated across Receivers.

[GiedriusS]

Good point! Maybe we could remove this option in 1.0? 🤔

[bwplotka]

Looks amazing! Just readability comment, otherwise let's merge, and check the prod!

[bwplotka]

Kind of sad our tooling does not put those values in help. Do you mean explicitly adding those options for people to know how to use it?

[bwplotka]

I need to agree with @douglascamata I think it would quite good to specify exactly what implementation we use. If it's based on libketama from https://dgryski.medium.com/consistent-hashing-algorithmic-tradeoffs-ef6b8e2fcae8 - let's mentioned that in comment, name of this ketama, and rename structs.

[fpetkovski]

👍

[fpetkovski]

Renamed consistent to ketama.

I also wonder if what @bill3tt mentioned makes sense, and whether we want to even expose different options to users. One advantage of hashmod is that series distribution is still a bit more even, so it might take a bit more time for the consistent hashing algorithm to get there.

[douglascamata]

@fpetkovski if I understood correctly, the ketama algorithm gets more even as the number of sections per node increases. So making this configurable by users helps solve the "is it even?" question: it'll be as even as you want it to be (and you pay some memory cost for it).

I don't think we will have a good answer yet to "why do people need hashmod instead of ketama hashring?" until people actually upgrade their setups to ketama and test it out.

My guess is that it will depend on how it performs at scale:

• How much resources ketama's hashring is going to use under heavy load when compared to hashmod?
• How much latency does it add?
• How much memory does it need to achieve a similarly even distribution as hashmod?

[bwplotka]

I think that's a good point to think about new default in 1.0 or before. For now let's treat this option as "feature flag" we can experiment with (:

[bwplotka]

Amazing, let's remove hack dir though (mistake?)

[bwplotka]

Should we remove hack dir? you can put .gitignore to not accidently commit those (:

[fpetkovski]

Ah yes, removed :)

[bwplotka]

👍🏽 Thanks!

[squat]

Maybe be clear about what "better" means. More "uniform"? more "localized"? More "random"? etc