Use deterministic method to distribute across fuzziness window

[replay]

We're currently relying on randomness, which makes this function hard to test. It would be better if we could reach distribution across the fuzziness window without involving randomness because then we could unit test if everything works as expected:
https://github.com/raintank/metrictank/blob/7cffa2d3d8eaaa2d59b07c149f5bae6deb674645/idx/cassandra/cassandra.go#L230

[woodsaj]

The reason behind the randomness was to ensure we didnt try and update all metricDefs at the same time.
So we could just use a hash of the metricId as an offset value.

eg

updateInterval := 21600
if time.Unix() > (def.LastUpdate + ( updateInterval - (def.LastUpdate % updateInterval)) + (hash(def.Id) % updateInterval) {
 // updated needed
}

assuming updateInterval = 21600 and the hash of our id is 1234

the metric is created at 1488825675, lastUpdate is set.
next update will be at 1488825675 + (21600 - (1488825675 mod 21600 )) + (1234 mod 21600)
which is 5.65hours later at 1488846034
but every update after that will be 6hours after the previous update.
1488846034 + (21600 - (1488846034 mod 21600)) + (1234 mod 21600)

for generating the hash, i recommend https://github.com/huichen/murmur as it is fast and simple to use.

[Dieterbe]

can't we simplify this formula a lot?
maybe something like:

if time.Now().Unix() - def.LastUpdate > hash(def.Id) % updateInterval {
    // update

[woodsaj]

if time.Now().Unix() - def.LastUpdate > hash(def.Id) % updateInterval

that wont work. That just updates every hash(def.id) % update interval, which could be a really small number.

[Dieterbe]

[replay]

@awoods in your formula i don't understand why this:

(def.LastUpdate + ( updateInterval - (def.LastUpdate % updateInterval)) + (hash(def.Id) % updateInterval))

is better than:

(def.LastUpdate + (def.LastUpdate % updateInterval) + (hash(def.Id) % updateInterval))

Wouldn't both result in an equal distribution (just upside down), but the second is more simple?

[woodsaj]

closed by #574