Consider observed latencies in weighing

[aarshkshah1992]

• Give nodes a weight boost if they're in the 80th <-> 99th percentile of the fastest(download speed) nodes.
• However, only do so once we have enough speed observations across many different nodes and a minimum threshold on the number of speed observations.
• Also introduces a cool off period on weight bump ups for speed to reward only those nodes who show consistency in their speed.

[willscott]

What i'd like at some point is a higher level sense that over a lot of requests we see reasonable steady-state behavior.

so, e.g.

• set up backends that work 90% of the time.
• send a bunch of requests while pushing forward time (or having a small de-bounce)
• make sure there's a healthy pool at the end

[aarshkshah1992]

@willscott

We already have the Bifrost staging environment we deploy to and collect pool heatlh metrics on and now we have L1 load distribution metrics on the Saturn side too. I am going to ask Lidel to deploy this PR and see how those metrics shape up over a day.

Or do you imagine a more automated setup here where we setup our own actual L1 backends for testing Caboose ?

[willscott]

Or do you imagine a more automated setup here where we setup our own actual L1 backends for testing Caboose ?

Happy to have synthetic L1s.
Something to have a sense of what the stable dynamics of changes will be that's lighter weight than deploying to real traffic. we've had to role back a couple times because that's the only way we have to test right now.

[aarshkshah1992]

@willscott Saturn does have a testnet with a few L1s. I'll sync up with the Saturn team and write a load testing tool or a script or something to test the Caboose <-> Saturn flow without having to deploy to prod. This makes sense to me.

[willscott]

if it's something we can run ourselves or have run against PRs in CI that would let us experiment much more than needing to wait on an external team's schedule for deployment

[lidel]

Deployed to bifrost-stage1-ny

2023/03/08 17:50:16 Starting bifrost-gateway 2023-03-08-7d6ef21

Looks good:

[lidel]

Deployed cd9c1d8 in ipfs-inactive/bifrost-gateway@9eac2e7 to staging:

root@bifrost-stage1-ny:~# docker logs -f bifrost-gw
2023/03/09 18:58:03 Starting bifrost-gateway 2023-03-09-9eac2e7

[lidel]

(I only reviewed metrics code, and deployed to staging, seems to work ok)

[willscott]

@aarshkshah1992 is this still relevant or has it been superseded by subsequent changes?

[aarshkshah1992]

@willscott I think we can close this as we have the shiny new L1 server timings now and can use that for weighing.