Add a primitive helper script for listing worker endpoints.

[reivilibre]

This is an extension of Patrick's branch.

Currently this only lists endpoints handleable by synapse.app.generic_worker and doesn't include any of the information needed for routing decisions,
but I think that can be added in time.

Part of: #12139
Base: develop

I recommend reviewing this as a whole, because the commit history is likely too messy to be useful.

---

Example output

# Client API requests
^/_matrix/client/(api/v1|r0|v3|unstable)/createRoom$
^/_matrix/client/(api/v1|r0|v3|unstable)/createRoom/.*$
^/_matrix/client/(api/v1|r0|v3|unstable)/events/.*$
^/_matrix/client/(api/v1|r0|v3|unstable)/joined_rooms$
^/_matrix/client/(api/v1|r0|v3|unstable)/publicRooms$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/context/.*$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/event/.*$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/joined_members$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/members$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state$
^/_matrix/client/(api/v1|r0|v3|unstable)/search$
^/_matrix/client/(api/v1|r0|v3|unstable)/voip/turnServer$
^/_matrix/client/(r0|v3)/rooms/.*/aliases$
^/_matrix/client/(r0|v3|unstable)/account/3pid$
^/_matrix/client/(r0|v3|unstable)/account/whoami$
^/_matrix/client/(r0|v3|unstable)/devices$
^/_matrix/client/(v1|unstable)/rooms/.*/relations/.*(/.*(/.*)?)?$
^/_matrix/client/unstable/org\.matrix\.msc2432/rooms/.*/aliases
^/_matrix/client/v1/rooms/.*/hierarchy$
^/_matrix/client/v1/rooms/.*/threads
^/_matrix/client/v1/rooms/.*/timestamp_to_event$
^/_matrix/client/versions$


# Sync requests
^/_matrix/client/(api/v1|r0|v3|unstable)/events$
^/_matrix/client/(api/v1|r0|v3|unstable)/initialSync$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/initialSync$
^/_matrix/client/(r0|v3|unstable)/sync$


# Event sending requests
^/_matrix/client/(api/v1|r0|v3|unstable)/join/.*$
^/_matrix/client/(api/v1|r0|v3|unstable)/join/.*/.*$
^/_matrix/client/(api/v1|r0|v3|unstable)/profile/.*
^/_matrix/client/(api/v1|r0|v3|unstable)/profile/.*/avatar_url
^/_matrix/client/(api/v1|r0|v3|unstable)/profile/.*/displayname
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/.*$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/.*/.*$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/redact/.*$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/redact/.*/.*$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/send/.*$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/send/.*/.*$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state/.*$
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state/.*/.*$
^/_matrix/client/(r0|v3|unstable)/knock/.*


# Client API requests (ALL FOR SAME ROOM MUST GO TO SAME WORKER)
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/messages$


# The typing stream
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/typing/.*$


# Registration/login requests
^/_matrix/client/(api/v1|r0|v3|unstable)/login$
^/_matrix/client/(r0|v3|unstable)/refresh$
^/_matrix/client/(r0|v3|unstable)/register$
^/_matrix/client/(v1|unstable)/register/m.login.registration_token/validity


# SSO requests needed for all SSO providers
^/_matrix/client/(api/v1|r0|v3|unstable)/login/(cas|sso)/redirect$
^/_matrix/client/(r0|v3)/login/sso/redirect/.*$


# Presence requests
^/_matrix/client/(api/v1|r0|v3|unstable)/presence/.*/status


# Push rule requests

GET    ^/_matrix/client/(api/v1|r0|v3|unstable)/.*$

# Encryption requests
^/_matrix/client/(r0|v3|unstable)/keys/changes$
^/_matrix/client/(r0|v3|unstable)/keys/claim$
^/_matrix/client/(r0|v3|unstable)/keys/query$
^/_matrix/client/(r0|v3|unstable)/keys/upload(/.*)?$
^/_matrix/client/(r0|v3|unstable)/room_keys/keys(/.*)?(/.*)?$
^/_matrix/client/(r0|v3|unstable)/room_keys/version$
^/_matrix/client/(r0|v3|unstable)/room_keys/version/.*$
^/_matrix/client/(r0|v3|unstable)/user/.*/filter
^/_matrix/client/(r0|v3|unstable)/user/.*/filter/.*


# Receipts requests
^/_matrix/client/(r0|v3|unstable)/rooms/.*/read_markers$
^/_matrix/client/(r0|v3|unstable)/rooms/.*/receipt/.*/.*$


# Account data requests
^/_matrix/client/(r0|v3|unstable)/user/.*/account_data/.*
^/_matrix/client/(r0|v3|unstable)/user/.*/rooms/.*/account_data/.*
^/_matrix/client/(r0|v3|unstable)/user/.*/rooms/.*/tags$
^/_matrix/client/(r0|v3|unstable)/user/.*/rooms/.*/tags/.*


# The to_device stream
^/_matrix/client/(r0|v3|unstable)/sendToDevice/.*/.*$


# User directory search requests
^/_matrix/client/(r0|v3|unstable)/user_directory/search$


# Inbound federation transaction request
^/_matrix/federation/v1/send/.*/?$


# Federation requests
^/_matrix/federation/v1/3pid/onbind$
^/_matrix/federation/v1/backfill/.*/?$
^/_matrix/federation/v1/event/.*/?$
^/_matrix/federation/v1/event_auth/.*/.*$
^/_matrix/federation/v1/exchange_third_party_invite/.*$
^/_matrix/federation/v1/get_missing_events/.*$
^/_matrix/federation/v1/hierarchy/.*$
^/_matrix/federation/v1/invite/.*/.*$
^/_matrix/federation/v1/make_join/.*/.*$
^/_matrix/federation/v1/make_knock/.*/.*$
^/_matrix/federation/v1/make_leave/.*/.*$
^/_matrix/federation/v1/openid/userinfo$
^/_matrix/federation/v1/publicRooms$
^/_matrix/federation/v1/query/.*$
^/_matrix/federation/v1/send_join/.*/.*$
^/_matrix/federation/v1/send_knock/.*/.*$
^/_matrix/federation/v1/send_leave/.*/.*$
^/_matrix/federation/v1/state/.*/?$
^/_matrix/federation/v1/state_ids/.*/?$
^/_matrix/federation/v1/timestamp_to_event/.*/?$
^/_matrix/federation/v1/user/devices/.*$
^/_matrix/federation/v1/user/keys/claim$
^/_matrix/federation/v1/user/keys/query$
^/_matrix/federation/v1/version$
^/_matrix/federation/v2/invite/.*/.*$
^/_matrix/federation/v2/send_join/.*/.*$
^/_matrix/federation/v2/send_leave/.*/.*$
^/_matrix/key/v2/query$
^/_matrix/key/v2/query/.*(/.*)?$


# Federation requests (unstable)
^/_matrix/federation/unstable/rooms/.*/complexity$

(I'm hoping it will eventually auto-generate the docs page — hence the categorisation — but it's not quite there yet.)

---

Original commit schedule, with full messages:

1. POC

2. Reduce patterns per servlet.

3. Move unstable to last entry.

4. Avoid duplicate entries.

5. More explicit paths for workers.

6. Handle capturing groups.

7. Sort results.

8. Add other client endpoints.

9. Add federation endpoints.

10. Add key resource.

11. Let the script be an HttpServer and just have paths registered onto it

12. Simplify path regexes, as before

13. Read categories from servlets and group the endpoints together by those

14. Category: Sync requests

15. Category: Federation requests

16. Categories: client API requests &c

17. Less obvious stragglers...

18. Add more stragglers

[clokep]

I think this is true now after #14605?

[clokep]

Not sure if this is useful or a comment to yourself?

[reivilibre]

hmm, open to feedback on it really. .* makes the regex simpler but broader than needed. I didn't want to leave this with .* without at least documenting that I'm on the fence about it, but perhaps we can agree on something here.

[MatMaul]

I think the simplification is fine after reviewing the use of the pattern.

[clokep]

I think i saw you add these? Or maybe not the OIDC/SAML/CAS specific ones.

[reivilibre]

Afraid that the /pick_idp is still not in the list (and some of the other OIDC/SAML/CAS ones). Possibly the homeserver has to be configured for them or they are not registered in the same way..

[clokep]

I think the homeserver doesn't register them unless there are configured providers.

[MatMaul]

We may want to just hardcode a servlets list for those specific cases. LGTM for a first version however.

[clokep]

A future improvement could be to only accept a single pattern per servlet, although I'm not 100% sure that's possible.

(Note that I think the changes in this file are valuable outside of this PR as it is fewer regexes to check / register.)

[reivilibre]

Multiple patterns might have its place, sometimes it might be clearer than amalgamating them into a single one?

[clokep]

I was surprised to learn that RestServlet doesn't define an abstract PATTERN property!

Not related to this PR, but I was going to suggest we document CATEGORY next to it?

[clokep]

I wrote a good chunk of this code so I don't think my review is sufficient, would be good to get other people's thoughts!

[MatMaul]

LGTM.

It could be interesting to add a GHA hook to generate the map file, and if different to the one currently in git, send a PR with the new updated one.

[reivilibre]

It could be interesting to add a GHA hook to generate the map file, and if different to the one currently in git, send a PR with the new updated one.

Yeah, there are plans for this script:

• generating the docs
• generating our path map for matrix.org
• generating the path map for Complement
• (I guess: generating the path map for EMS)
• generating the path map for SyTest

But had to leave this off and this felt like a sensible unit of work to put up as a PR anyway