Make sure synapse_rate_limit_reject_affected_hosts does what it says it does

[MadLittleMods]

Spawning from #13541 (comment)

---

The synapse_rate_limit_reject_affected_hosts gauge is always evaluating to 0. The raw data in Prometheus also shows 0 for reference.

https://grafana.matrix.org/d/dYoRgTgVz/messages-timing?orgId=1&from=1661855196368&to=1661876796368&viewPanel=220

Even though we see individual requests being rejected (synapse_rate_limit_reject_total) which should mean at least 1 host,

But this could be a mismatch in how the guages were being reported because we were accidentally registering them twice, #13641

---

Now that we fixed the duplicate metric registering issue in #13649 and the fix was put on matrix.org this morning, we're seeing both at 0 now. This could mean that the previous rejections we were seeing were all from the UsernameAvailabilityRestServlet which we are no longer tracking. And we're not rejecting any requests in the federation servlets.

It is a bit suspicious though.

How can we know if it's right?

In order to confirm that synapse_rate_limit_reject_affected_hosts is working, it would be nice to see a non-zero value.

The reject_limit is 50 which I think means there has to be more than 50 requests within the 1 second federation_rc_window_size to start rejecting.

We do see the rate of slept requests go above 70 sometimes which I would expect to trigger this 🤔

https://grafana.matrix.org/d/dYoRgTgVz/messages-timing?orgId=1&from=1661873701156&to=1661877301156&viewPanel=223

Dev notes

The synapse_rate_limit_reject_affected_hosts metric was originally added in #13541 and updated in #13649

[DMRobertson]

Erm, isn't that sum is a sum across all workers?

I'm anxious that two different workers might separately decide to rate-limit example.com. If so, that'd mean that example.com is double-counted as two hosts in these graphs (I think?)

[MadLittleMods]

@DMRobertson Yes but it's the best we can do to count hosts AFAIK

[MadLittleMods]

Since the deploy, we have at least seen a blip of rejected requests on matrix.org but it looks like it wasn't sustained enough to be caught be the 15 second /collect interval that the rejected host gauge runs on.

Same blip (https://grafana.matrix.org/d/dYoRgTgVz/messages-timing?orgId=1&from=1662486044397&to=1662486894994&viewPanel=206):

[MadLittleMods]

We finally saw something to show up on the "Number of hosts being rejected by the rate limiter" graph (the second graph the image). But it doesn't correspond to any spikes in the number of rate limit rejected requests 🤔

https://grafana.matrix.org/d/000000012/synapse?orgId=1&from=1664997566912&to=1666293566912&viewPanel=237

[MadLittleMods]

Seeing this more active on other hosts (libera.chat):

Going to close as it seems to count something.