[WIP] External sharded cache

[Fizzadar]

WIP experimental external sharded Redis caching. Our synapse instance experiences extremely high calls to the _get_joined_profile_from_event_id cache, which also happens to never be invalidated. During normal running the misses (~5k/s min) cause a reasonable amount of DB load and during restarts this becomes a real problem.

Thus, because the cache is never invalidated, I've been looking into the possibility of another caching layer shared between workers and Redis seems like the best fit as it's already used. I chose to make this a new config section and recommend separate Redis instances which handle eviction of items from the cache (specifically the MSET Redis command does not support expiry - although it can be applied key-by-key afterwards).

The sharding is client side based using the jump hashing algorithm for speed and ease of deployment, plus txredis doesn't support Redis cluster (and all the complexities it brings).

Feel free to close/cancel this if not desired as well, this work is primarily to be rolled in our own synapse fork but I thought it may be of interest, especially as an optional configurable. I'm going to leave a few comments on the PR itself as well where I'm unsure about things.

Related: #2123

Signed off by Nick @ Beeper

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file.
• Pull request includes a sign off
• Code style is correct
  (run the linters)

[Fizzadar]

This all feels ripe for some kind of wrapper - I had a brief look at cachedList but I don't think it should live there because it's already pretty complex! Maybe some kind of util wrapper? Or out of scope for this PR?

[Fizzadar]

Not currently supported but could be fairly easily using a Redis transaction block and setting each key one by one.

[erikjohnston]

This looks super interesting! We're actually currently doing a bunch of work to reduce the amount of state we pull out of the DB, which may (or may not) help a lot. My inclination would be then to revisit this after that work has landed to see whether this is still useful.

On which point: I'd love to know what is triggering calls to _get_joined_profiles_from_event_ids for you, e.g. I think I've mostly seen it happen when calculation push actions for new events?

[Fizzadar]

@erikjohnston the _get_joined_profiles_from_event_ids calls are all from our worker and federation sender instances, which seems like this is unexpected! I'm currently trying to track down exactly where they're being called from and will get back to you here, it seems the function is really only triggered in a handful of locations...

[erikjohnston]

@erikjohnston the _get_joined_profiles_from_event_ids calls are all from our worker and federation sender instances, which seems like this is unexpected! I'm currently trying to track down exactly where they're being called from and will get back to you here, it seems the function is really only triggered in a handful of locations...

I think the federation sender instance one will be fixed by #12964, at the very least.

[Fizzadar]

@erikjohnston very happy to report 1.61 has removed near all reqs from the federation sender on that cache!

[erikjohnston]

Hi @Fizzadar, sorry I was out last week.

Great news that the cache request rate has dropped! I think this means that this PR is no longer needed for this particular case, but we'd certainly be open to using this approach for other cases if needed. So I'm going to close this, but feel free to reopen if/when there are other caches that would benefit from this.