Test Synapse against Complement with workers. #12810

reivilibre · 2022-05-20T11:16:07Z

Closes #12638 (once the tests pass, that is...).

For the time being (until #12848 is fixed), I've split them into 4 alphabetical groups.
I first went by dividing the alphabet up immediately, but then knocked together a quick script to give the best points to split based on runtime

SPLIT AT TestFilter
SPLIT AT TestMediaFilenames
SPLIT AT TestRoomCreate

Funnily enough, those are the quarter-points of the alphabet anyway (or 1 letter off).

It's not nice to need to do this, but it means we can enable Complement in CI right away until we can properly shave some time off. I have some ideas to do that, but they would definitely not be in scope for the same PR.

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

reivilibre · 2022-05-23T11:23:34Z

52 minutes! 😨

.github/workflows/tests.yml

richvdh · 2022-05-30T16:43:31Z

.github/workflows/tests.yml

+          # Test with workers
+          - workers: workers
+            regex: 'A-Ea-e'
+
+          - workers: workers
+            regex: 'F-Lf-l'
+
+          - workers: workers
+            regex: 'M-Qm-q'
+
+          - workers: workers
+            regex: 'R-Zr-z'


I have reservations over this is a useful thing to do.

We are constrained as an organisation as to the number of GHA jobs we can run concurrently (to 60, according to https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits). Splitting the tests up this way does nothing to reduce the total amount of GHA worker time used by our tests, but does increase the chances of us monopolising the workers.

An alternative might be to exclude worker-mode-complement from the tests-done job, so we don't need to wait for it. But that's fiddly too.

le sigh.

Right, fair enough, I didn't realise we had this limit (well, I assumed there must be a limit, but I didn't know how much it was or how concerned we were about it).
You're right that this only reduces the wall time, at the cost of increasing the total consumed time (since each image has the overheads of building the Complement images, etc.).

Perhaps it may make sense, for now, to:

restrict these worker tests to running on develop only(?) — gives us an indication if nothing else. Also prevents us having to 'wait' for it.

try and get some speed improvements landed (e.g. the forking launcher, which cuts it down to ~26 minutes which is only a few minutes longer than the next slowest thing.)

Whilst testing, I've been cheating a bit and making Complement start before the lints are done; this gives it a few minutes headstart and then it's not the slowest thing on the list. If we're interested that much in shaving some time off, it could be worth considering, though I'd rather get some real time shavings if possible.

Perhaps it may make sense, for now, to:

yeah, let's try what you suggest for now

Turns out you can't use a matrix in the if block of a job (YAML anchors are no good either), so I've duplicated it for now :/

Turns out you can't use a matrix in the if block of a job (YAML anchors are no good either), so I've duplicated it for now :/

Are you sure about this? We have examples of doing so, e.g.

synapse/.github/workflows/latest_deps.yml

Line 58 in 57fac2a

if: ${{ matrix.postgres-version }}

Oh, perhaps you mean the job-level if rather than the step-level if

Turns out you can't use a matrix in the if block of a job (YAML anchors are no good either), so I've duplicated it for now :/

Are you sure about this? We have examples of doing so, e.g.

synapse/.github/workflows/latest_deps.yml

Line 58 in 57fac2a

if: ${{ matrix.postgres-version }}

Yeah, see https://github.com/matrix-org/synapse/actions/runs/2414412187/workflow#L310

I don't think that example is using it at the same level with the same meaning (we want to skip a job, not a step)

…the ugly way

.github/workflows/tests.yml

richvdh

lgtm elsewise

.ci/scripts/checkout_complement.sh

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

@olmari

Synapse 1.61.0 (2022-06-14) =========================== This release removes support for the non-standard feature known both as 'groups' and as 'communities', which have been superseded by *Spaces*. See [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1610) for more details. Improved Documentation ---------------------- - Mention removed community/group worker endpoints in [upgrade.md](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1610s). Contributed by @olmari. ([\matrix-org#13023](matrix-org#13023)) Synapse 1.61.0rc1 (2022-06-07) ============================== Features -------- - Add new `media_retention` options to the homeserver config for routinely cleaning up non-recently accessed media. ([\matrix-org#12732](matrix-org#12732), [\matrix-org#12972](matrix-org#12972), [\matrix-org#12977](matrix-org#12977)) - Experimental support for [MSC3772](matrix-org/matrix-spec-proposals#3772): Push rule for mutually related events. ([\matrix-org#12740](matrix-org#12740), [\matrix-org#12859](matrix-org#12859)) - Update to the `check_event_for_spam` module callback: Deprecate the current callback signature, replace it with a new signature that is both less ambiguous (replacing booleans with explicit allow/block) and more powerful (ability to return explicit error codes). ([\matrix-org#12808](matrix-org#12808)) - Add storage and module API methods to get monthly active users (and their corresponding appservices) within an optionally specified time range. ([\matrix-org#12838](matrix-org#12838), [\matrix-org#12917](matrix-org#12917)) - Support the new error code `ORG.MATRIX.MSC3823.USER_ACCOUNT_SUSPENDED` from [MSC3823](matrix-org/matrix-spec-proposals#3823). ([\matrix-org#12845](matrix-org#12845), [\matrix-org#12923](matrix-org#12923)) - Add a configurable background job to delete stale devices. ([\matrix-org#12855](matrix-org#12855)) - Improve URL previews for pages with empty elements. ([\matrix-org#12951](matrix-org#12951)) - Allow updating a user's password using the admin API without logging out their devices. Contributed by @jcgruenhage. ([\matrix-org#12952](matrix-org#12952)) Bugfixes -------- - Always send an `access_token` in `/thirdparty/` requests to appservices, as required by the [Application Service API specification](https://spec.matrix.org/v1.1/application-service-api/#third-party-networks). ([\matrix-org#12746](matrix-org#12746)) - Implement [MSC3816](matrix-org/matrix-spec-proposals#3816): sending the root event in a thread should count as having 'participated' in it. ([\matrix-org#12766](matrix-org#12766)) - Delete events from the `federation_inbound_events_staging` table when a room is purged through the admin API. ([\matrix-org#12784](matrix-org#12784)) - Fix a bug where we did not correctly handle invalid device list updates over federation. Contributed by Carl Bordum Hansen. ([\matrix-org#12829](matrix-org#12829)) - Fix a bug which allowed multiple async operations to access database locks concurrently. Contributed by @sumnerevans @ Beeper. ([\matrix-org#12832](matrix-org#12832)) - Fix an issue introduced in Synapse 0.34 where the `/notifications` endpoint would only return notifications if a user registered at least one pusher. Contributed by Famedly. ([\matrix-org#12840](matrix-org#12840)) - Fix a bug where servers using a Postgres database would fail to backfill from an insertion event when MSC2716 is enabled (`experimental_features.msc2716_enabled`). ([\matrix-org#12843](matrix-org#12843)) - Fix [MSC3787](matrix-org/matrix-spec-proposals#3787) rooms being omitted from room directory, room summary and space hierarchy responses. ([\matrix-org#12858](matrix-org#12858)) - Fix a bug introduced in Synapse 1.54.0 which could sometimes cause exceptions when handling federated traffic. ([\matrix-org#12877](matrix-org#12877)) - Fix a bug introduced in Synapse 1.59.0 which caused room deletion to fail with a foreign key violation error. ([\matrix-org#12889](matrix-org#12889)) - Fix a long-standing bug which caused the `/messages` endpoint to return an incorrect `end` attribute when there were no more events. Contributed by @Vetchu. ([\matrix-org#12903](matrix-org#12903)) - Fix a bug introduced in Synapse 1.58.0 where `/sync` would fail if the most recent event in a room was a redaction of an event that has since been purged. ([\matrix-org#12905](matrix-org#12905)) - Fix a potential memory leak when generating thumbnails. ([\matrix-org#12932](matrix-org#12932)) - Fix a long-standing bug where a URL preview would break if the image failed to download. ([\matrix-org#12950](matrix-org#12950)) Improved Documentation ---------------------- - Fix typographical errors in documentation. ([\matrix-org#12863](matrix-org#12863)) - Fix documentation incorrectly stating the `sendToDevice` endpoint can be directed at generic workers. Contributed by Nick @ Beeper. ([\matrix-org#12867](matrix-org#12867)) Deprecations and Removals ------------------------- - Remove support for the non-standard groups/communities feature from Synapse. ([\matrix-org#12553](matrix-org#12553), [\matrix-org#12558](matrix-org#12558), [\matrix-org#12563](matrix-org#12563), [\matrix-org#12895](matrix-org#12895), [\matrix-org#12897](matrix-org#12897), [\matrix-org#12899](matrix-org#12899), [\matrix-org#12900](matrix-org#12900), [\matrix-org#12936](matrix-org#12936), [\matrix-org#12966](matrix-org#12966)) - Remove contributed `kick_users.py` script. This is broken under Python 3, and is not added to the environment when `pip install`ing Synapse. ([\matrix-org#12908](matrix-org#12908)) - Remove `contrib/jitsimeetbridge`. This was an unused experiment that hasn't been meaningfully changed since 2014. ([\matrix-org#12909](matrix-org#12909)) - Remove unused `contrib/experiements/cursesio.py` script, which fails to run under Python 3. ([\matrix-org#12910](matrix-org#12910)) - Remove unused `contrib/experiements/test_messaging.py` script. This fails to run on Python 3. ([\matrix-org#12911](matrix-org#12911)) Internal Changes ---------------- - Test Synapse against Complement with workers. ([\matrix-org#12810](matrix-org#12810), [\matrix-org#12933](matrix-org#12933)) - Reduce the amount of state we pull from the DB. ([\matrix-org#12811](matrix-org#12811), [\matrix-org#12964](matrix-org#12964)) - Try other homeservers when re-syncing state for rooms with partial state. ([\matrix-org#12812](matrix-org#12812)) - Resume state re-syncing for rooms with partial state after a Synapse restart. ([\matrix-org#12813](matrix-org#12813)) - Remove Mutual Rooms' ([MSC2666](matrix-org/matrix-spec-proposals#2666)) endpoint dependency on the User Directory. ([\matrix-org#12836](matrix-org#12836)) - Experimental: expand `check_event_for_spam` with ability to return additional fields. This enables spam-checker implementations to experiment with mechanisms to give users more information about why they are blocked and whether any action is needed from them to be unblocked. ([\matrix-org#12846](matrix-org#12846)) - Remove `dont_notify` from the `.m.rule.room.server_acl` rule. ([\matrix-org#12849](matrix-org#12849)) - Remove the unstable `/hierarchy` endpoint from [MSC2946](matrix-org/matrix-spec-proposals#2946). ([\matrix-org#12851](matrix-org#12851)) - Pull out less state when handling gaps in room DAG. ([\matrix-org#12852](matrix-org#12852), [\matrix-org#12904](matrix-org#12904)) - Clean-up the push rules datastore. ([\matrix-org#12856](matrix-org#12856)) - Correct a type annotation in the URL preview source code. ([\matrix-org#12860](matrix-org#12860)) - Update `pyjwt` dependency to [2.4.0](https://github.com/jpadilla/pyjwt/releases/tag/2.4.0). ([\matrix-org#12865](matrix-org#12865)) - Enable the `/account/whoami` endpoint on synapse worker processes. Contributed by Nick @ Beeper. ([\matrix-org#12866](matrix-org#12866)) - Enable the `batch_send` endpoint on synapse worker processes. Contributed by Nick @ Beeper. ([\matrix-org#12868](matrix-org#12868)) - Don't generate empty AS transactions when the AS is flagged as down. Contributed by Nick @ Beeper. ([\matrix-org#12869](matrix-org#12869)) - Fix up the variable `state_store` naming. ([\matrix-org#12871](matrix-org#12871)) - Faster room joins: when querying the current state of the room, wait for state to be populated. ([\matrix-org#12872](matrix-org#12872)) - Avoid running queries which will never result in deletions. ([\matrix-org#12879](matrix-org#12879)) - Use constants for EDU types. ([\matrix-org#12884](matrix-org#12884)) - Reduce database load of `/sync` when presence is enabled. ([\matrix-org#12885](matrix-org#12885)) - Refactor `have_seen_events` to reduce memory consumed when processing federation traffic. ([\matrix-org#12886](matrix-org#12886)) - Refactor receipt linearization code. ([\matrix-org#12888](matrix-org#12888)) - Add type annotations to `synapse.logging.opentracing`. ([\matrix-org#12894](matrix-org#12894)) - Remove PyNaCl occurrences directly used in Synapse code. ([\matrix-org#12902](matrix-org#12902)) - Bump types-jsonschema from 4.4.1 to 4.4.6. ([\matrix-org#12912](matrix-org#12912)) - Rename storage classes. ([\matrix-org#12913](matrix-org#12913)) - Preparation for database schema simplifications: stop reading from `event_edges.room_id`. ([\matrix-org#12914](matrix-org#12914)) - Check if we are in a virtual environment before overriding the `PYTHONPATH` environment variable in the demo script. ([\matrix-org#12916](matrix-org#12916)) - Improve the logging when signature checks on events fail. ([\matrix-org#12925](matrix-org#12925)) # -----BEGIN PGP SIGNATURE----- # # iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmKoaa0QHGVyaWtAbWF0 # cml4Lm9yZwAKCRClQuTtGw+sCVn3B/sF8hdhrZ7hWW40ST3eG9cEKNFrj/xZXiaI # zho3ryrxaQF68BSKot15AvZSprEdwBXWrb8WeTjyw+QH7vTKrCQDZ0p7qubn10Z7 # BuKq9hyYjyCLjBZrgy8d4U3Y8gsSByuO59YKHNLn+UTJLOs5GTH8Wprwh4mpU3Jl # +o+cC+lMSVcyZij2hihFymcSxWq/I9WL0dsjRif8x0BUQwRXwmXc6+mhlgLBe2Zs # 2dUouzJ8NVZcjfWvsg4noXPrNQ/IiyCVZlSIgaDftDIxVSPk5/rXiUUNex8Tn1+I # TnOgnXhpOQD1vwVGYS8LrcfA0ubSili7xUJ8k2e5TkCjpkaVnYNu # =q+7N # -----END PGP SIGNATURE----- # gpg: Signature made Tue Jun 14 11:57:49 2022 BST # gpg: using RSA key 053191DFF4670330465227F7A542E4ED1B0FAC09 # gpg: issuer "erik@matrix.org" # gpg: WARNING: server 'dirmngr' is older than us (2.2.34 < 2.3.6) # gpg: Note: Outdated servers may lack important security fixes. # gpg: Note: Use the command "gpgconf --kill all" to restart them. # gpg: Can't check signature: No public key # Conflicts: # docs/workers.md # synapse/handlers/message.py # synapse/handlers/pagination.py # synapse/push/bulk_push_rule_evaluator.py # synapse/push/push_rule_evaluator.py # synapse/rest/client/receipts.py # synapse/storage/databases/main/appservice.py # tests/push/test_push_rule_evaluator.py

reivilibre requested a review from a team as a code owner May 20, 2022 11:16

reivilibre marked this pull request as draft May 20, 2022 11:16

reivilibre removed the request for review from a team May 20, 2022 11:16

reivilibre mentioned this pull request May 20, 2022

Test Synapse in worker mode under Complement #12638

Closed

reivilibre added 2 commits May 20, 2022 17:39

Run Complement with workers in CI

be69bd2

Newsfile

322d22f

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>

reivilibre force-pushed the rei/complement_workers_in_ci branch from b4e3c2d to 322d22f Compare May 20, 2022 16:39

richvdh reviewed May 23, 2022

View reviewed changes

.github/workflows/tests.yml Outdated Show resolved Hide resolved

reivilibre force-pushed the rei/complement_workers_in_ci branch 2 times, most recently from 3ca3176 to c8c7f5a Compare May 23, 2022 16:24

Split workers' tests by regex into 4 different jobs for now

a974ccb

reivilibre force-pushed the rei/complement_workers_in_ci branch from 9888b74 to a974ccb Compare May 26, 2022 14:08

Merge branch 'develop' into rei/complement_workers_in_ci

a2ae63e

reivilibre force-pushed the rei/complement_workers_in_ci branch from 6883b5f to a2ae63e Compare May 26, 2022 14:50

reivilibre marked this pull request as ready for review May 27, 2022 12:21

reivilibre requested a review from a team May 30, 2022 13:41

richvdh reviewed May 30, 2022

View reviewed changes

reivilibre added 3 commits May 31, 2022 11:43

Restrict Complement workers tests to develop branch

9b3cdeb

Cut out the regex rubbish

7f9870e

Work around the lack of YAML anchors or other easy reuse primitives, …

d539339

…the ugly way

reivilibre requested a review from richvdh May 31, 2022 10:59

richvdh reviewed May 31, 2022

View reviewed changes

reivilibre added 4 commits May 31, 2022 12:33

Move checkout to a script

50a9264

Flatten develop-only

06b1221

Flatten normal (reversible later)

5f1efa4

Fix path to script

b998416

reivilibre requested a review from richvdh May 31, 2022 11:51

richvdh approved these changes May 31, 2022

View reviewed changes

.ci/scripts/checkout_complement.sh Outdated Show resolved Hide resolved

Update .ci/scripts/checkout_complement.sh

58819e1

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

reivilibre enabled auto-merge (squash) May 31, 2022 12:25

reivilibre merged commit bf01e51 into develop May 31, 2022

reivilibre deleted the rei/complement_workers_in_ci branch May 31, 2022 13:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test Synapse against Complement with workers. #12810

Test Synapse against Complement with workers. #12810

reivilibre commented May 20, 2022 •

edited

Loading

reivilibre commented May 23, 2022

richvdh May 30, 2022

reivilibre May 31, 2022

richvdh May 31, 2022

reivilibre May 31, 2022

DMRobertson May 31, 2022

DMRobertson May 31, 2022

reivilibre May 31, 2022

richvdh left a comment

Test Synapse against Complement with workers. #12810

Test Synapse against Complement with workers. #12810

Conversation

reivilibre commented May 20, 2022 • edited Loading

reivilibre commented May 23, 2022

richvdh May 30, 2022

Choose a reason for hiding this comment

reivilibre May 31, 2022

Choose a reason for hiding this comment

richvdh May 31, 2022

Choose a reason for hiding this comment

reivilibre May 31, 2022

Choose a reason for hiding this comment

DMRobertson May 31, 2022

Choose a reason for hiding this comment

DMRobertson May 31, 2022

Choose a reason for hiding this comment

reivilibre May 31, 2022

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

reivilibre commented May 20, 2022 •

edited

Loading