Allow sharding of media repository

[erikjohnston]

Looks like the media repository on matrix.org has high CPU usage, so we should investigate how we can scale it out a bit.

The main sources of CPU usage seem to be:

• S3 storage provider
• URL previews (I think due to SSL overhead maybe?)

[ara4n]

URL previews (I think due to SSL overhead maybe?)

impossible - URL preview code is perfect in all ways!

[clokep]

URL previews (I think due to SSL overhead maybe?)

I think these already share SSL contexts due to #7094?

[erikjohnston]

URL previews (I think due to SSL overhead maybe?)

I think these already share SSL contexts due to #7094?

Probably, but I think that just reduces overhead from INSANE to normal levels, maybe

[clokep]

@erikjohnston So what's the thought here? How would one get started with this? Was there particular hot spots noticed while profiling or is profiling the first step?

[erikjohnston]

I think next steps are to look at the media repository code to see what looks easy to split out. Things to look out for:

1. Any background looping calls that need to only happen on one worker
2. DB functions that assume they're the only writer
3. Any locks or ratelimiting
4. Any cached database functions are correctly invalidated.

I don't think there is much of any of those going on in the media repository, so depending on what you find it might be quite easy to make it possible to e.g. run multiple media repositories for downloads or URL previews or what not. (It may not even requires any code changes, depending)