-
-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add end-to-end tests for End-to-end encryption #7312
Comments
the basic case is tested already at https://github.com/matrix-org/matrix-react-end-to-end-tests/blob/master/src/scenario.js#L60 but adding federation, and extended case would indeed be great. |
* Include Vietnamese language ([\element-hq#20029](element-hq#20029)). * Simple static location sharing ([\element-hq#19754](element-hq#19754)). * Add support for the Indonesian language ([\element-hq#20032](element-hq#20032)). Fixes element-hq#20030. Contributed by @Linerly. * Always unhide widgets on layout change (pinning a widget) ([\element-hq#7299](matrix-org/matrix-react-sdk#7299)). * Update status message in the member list and user info panel when it is changed ([\element-hq#7338](matrix-org/matrix-react-sdk#7338)). Fixes element-hq#20127. Contributed by @SimonBrandner. * Iterate space panel toggle collapse interaction ([\element-hq#7335](matrix-org/matrix-react-sdk#7335)). Fixes element-hq#20079. * Spotlight search labs ([\element-hq#7116](matrix-org/matrix-react-sdk#7116)). Fixes element-hq#19530. * Put room settings form elements in fieldsets ([\element-hq#7311](matrix-org/matrix-react-sdk#7311)). * Add descriptions to ambiguous links for screen readers ([\element-hq#7310](matrix-org/matrix-react-sdk#7310)). * Make tooltips keyboard accessible ([\#7281](matrix-org/matrix-react-sdk#7281)). * Iterate room context menus for DMs ([\element-hq#7308](matrix-org/matrix-react-sdk#7308)). Fixes element-hq#19527. * Update space panel expand mechanism ([\element-hq#7230](matrix-org/matrix-react-sdk#7230)). Fixes element-hq#17993. * Add CSS variable to make the UI gaps consistent and fix the resize handle position ([\element-hq#7234](matrix-org/matrix-react-sdk#7234)). Fixes element-hq#19904 and element-hq#19938. * Custom location sharing. ([\element-hq#7185](matrix-org/matrix-react-sdk#7185)). * Simple static location sharing ([\element-hq#7135](matrix-org/matrix-react-sdk#7135)). * Finish sending pending messages before leaving room ([\element-hq#7276](matrix-org/matrix-react-sdk#7276)). Fixes element-hq#4702. * Dropdown follow wai-aria practices for expanding on arrow keys ([\element-hq#7277](matrix-org/matrix-react-sdk#7277)). Fixes element-hq#3687. * Expose PL control for pinned events when lab enabled ([\element-hq#7278](matrix-org/matrix-react-sdk#7278)). Fixes element-hq#5396. * In People & Favourites metaspaces always show all rooms ([\element-hq#7288](matrix-org/matrix-react-sdk#7288)). Fixes element-hq#20048. * Don't allow calls when the connection the server has been lost ([\element-hq#7287](matrix-org/matrix-react-sdk#7287)). Fixes element-hq#2096. Contributed by @SimonBrandner. * Analytics opt in for posthog ([\element-hq#6936](matrix-org/matrix-react-sdk#6936)). * Don't inhibit current room notifications if user has Modal open ([\element-hq#7274](matrix-org/matrix-react-sdk#7274)). Fixes element-hq#1118. * Remove the `Screen sharing is here!` dialog ([\#7266](matrix-org/matrix-react-sdk#7266)). Fixes element-hq#18824. Contributed by @SimonBrandner. * Make composer buttons react to settings without having to change room ([\element-hq#7264](matrix-org/matrix-react-sdk#7264)). Fixes element-hq#20011. * Decorate view keyboard shortcuts link as a link ([\element-hq#7260](matrix-org/matrix-react-sdk#7260)). Fixes element-hq#20007. * Improve ease of focusing on Room list Search ([\element-hq#7255](matrix-org/matrix-react-sdk#7255)). Fixes matrix-org/element-web-rageshakes#7017. * Autofocus device panel entry when renaming device ([\element-hq#7249](matrix-org/matrix-react-sdk#7249)). Fixes element-hq#19984. * Update Space Panel scrollable region ([\element-hq#7245](matrix-org/matrix-react-sdk#7245)). Fixes element-hq#19978. * Replace breadcrumbs with recently viewed menu ([\element-hq#7073](matrix-org/matrix-react-sdk#7073)). Fixes element-hq#19528. * Tweaks to informational architecture 1.1 ([\element-hq#7052](matrix-org/matrix-react-sdk#7052)). Fixes element-hq#19526, element-hq#19379, element-hq#17792, element-hq#16450, element-hq#19881, element-hq#19892, element-hq#19300, element-hq#19324, element-hq#17307, element-hq#17468 element-hq#19932 and element-hq#19956. * [Release] Fix inline code block nowrap issue ([\element-hq#7407](matrix-org/matrix-react-sdk#7407)). * don't collapse spaces in inline code blocks (https ([\element-hq#7328](matrix-org/matrix-react-sdk#7328)). Fixes element-hq#6051. Contributed by @HarHarLinks. * Fix accessibility regressions ([\element-hq#7336](matrix-org/matrix-react-sdk#7336)). * Debounce User Info start dm "Message" button ([\element-hq#7357](matrix-org/matrix-react-sdk#7357)). Fixes element-hq#7763. * Fix thread filter being cut-off on narrow screens ([\element-hq#7354](matrix-org/matrix-react-sdk#7354)). Fixes element-hq#20146. * Fix upgraded rooms wrongly showing up in spotlight ([\element-hq#7341](matrix-org/matrix-react-sdk#7341)). Fixes element-hq#20141. * Show votes in replied-to polls (pass in getRelationsForEvent) ([\element-hq#7345](matrix-org/matrix-react-sdk#7345)). Fixes element-hq#20153. * Keep all previously approved widget capabilities when requesting new capabilities ([\element-hq#7340](matrix-org/matrix-react-sdk#7340)). Contributed by @dhenneke. * Only show poll previews when the polls feature is enabled ([\element-hq#7331](matrix-org/matrix-react-sdk#7331)). * No-op action:join if the user is already invited for scalar ([\element-hq#7334](matrix-org/matrix-react-sdk#7334)). Fixes element-hq#20134. * Don't show polls in timeline if polls are disabled ([\element-hq#7332](matrix-org/matrix-react-sdk#7332)). Fixes element-hq#20130. * Don't send a poll response event if you are voting for your current c… ([\element-hq#7326](matrix-org/matrix-react-sdk#7326)). Fixes element-hq#20129. * Don't show options button when the user can't modify widgets ([\element-hq#7324](matrix-org/matrix-react-sdk#7324)). Fixes element-hq#20114. Contributed by @SimonBrandner. * Add vertical spacing between buttons when they go over multiple lines ([\element-hq#7314](matrix-org/matrix-react-sdk#7314)). Contributed by @twigleingrid. * Improve accessibility of opening space create menu ([\element-hq#7316](matrix-org/matrix-react-sdk#7316)). * Correct tab order in room preview dialog ([\element-hq#7302](matrix-org/matrix-react-sdk#7302)). * Fix favourites and people metaspaces not rendering their content ([\#7315](matrix-org/matrix-react-sdk#7315)). Fixes element-hq#20070. * Make clear button images visible in high contrast theme ([\element-hq#7306](matrix-org/matrix-react-sdk#7306)). Fixes element-hq#19931. * Fix html exporting and improve output size ([\element-hq#7312](matrix-org/matrix-react-sdk#7312)). Fixes element-hq#19436 element-hq#20107 and element-hq#19441. * Fix textual message stripping new line ([\element-hq#7239](matrix-org/matrix-react-sdk#7239)). Fixes element-hq#15320. Contributed by @renancleyson-dev. * Fix issue with room list resizer getting clipped in firefox ([\element-hq#7303](matrix-org/matrix-react-sdk#7303)). Fixes element-hq#20076. * Fix wrong indentation with nested ordered list unnesting list on edit ([\element-hq#7300](matrix-org/matrix-react-sdk#7300)). Contributed by @renancleyson-dev. * Fix input field behaviour inside context menus ([\element-hq#7293](matrix-org/matrix-react-sdk#7293)). Fixes element-hq#19881. * Corrected the alignment of the Edit button on LoginPage. ([\element-hq#7292](matrix-org/matrix-react-sdk#7292)). Contributed by @ankur12-1610. * Allow sharing manual location without giving location permission ([\element-hq#7295](matrix-org/matrix-react-sdk#7295)). Fixes element-hq#20065. Contributed by @tulir. * Make emoji picker search placeholder localizable ([\element-hq#7294](matrix-org/matrix-react-sdk#7294)). * Fix jump to bottom on message send ([\element-hq#7280](matrix-org/matrix-react-sdk#7280)). Fixes element-hq#19859. Contributed by @SimonBrandner. * Fix: Warning: Unsupported style property pointer-events. Did you mean pointerEvents? ([\element-hq#7291](matrix-org/matrix-react-sdk#7291)). * Add edits and replies to the right panel timeline & prepare the timelineCard to share code with threads ([\element-hq#7262](matrix-org/matrix-react-sdk#7262)). Fixes element-hq#20012 and element-hq#19928. * Fix labs exploding when lab group is empty ([\element-hq#7290](matrix-org/matrix-react-sdk#7290)). Fixes element-hq#20051. * Update URL when room aliases are modified ([\element-hq#7289](matrix-org/matrix-react-sdk#7289)). Fixes element-hq#1616 and element-hq#1925. * Render mini user menu for when space panel is disabled ([\element-hq#7258](matrix-org/matrix-react-sdk#7258)). Fixes element-hq#19998. * When accepting DM from People metaspace don't switch to Home ([\element-hq#7272](matrix-org/matrix-react-sdk#7272)). Fixes element-hq#19995. * Fix CallPreview `room is null` ([\element-hq#7265](matrix-org/matrix-react-sdk#7265)). Fixes element-hq#19990, element-hq#19972, matrix-org/element-web-rageshakes#7004 matrix-org/element-web-rageshakes#6991 and matrix-org/element-web-rageshakes#6964. * Fixes more instances of double-translation ([\element-hq#7259](matrix-org/matrix-react-sdk#7259)). Fixes element-hq#20010. * Fix video calls ([\element-hq#7256](matrix-org/matrix-react-sdk#7256)). Fixes element-hq#20008. Contributed by @SimonBrandner. * Fix broken i18n in Forgot & Change password ([\element-hq#7252](matrix-org/matrix-react-sdk#7252)). Fixes element-hq#19989. * Fix setBotPower to not use `.content` ([\element-hq#7179](matrix-org/matrix-react-sdk#7179)). Fixes element-hq#19845. * Break long words in pinned messages to prevent overflow ([\element-hq#7251](matrix-org/matrix-react-sdk#7251)). Fixes element-hq#19985. * Disallow sending empty feedbacks ([\element-hq#7240](matrix-org/matrix-react-sdk#7240)). * Fix wrongly sized default sub-space icons in space panel ([\element-hq#7243](matrix-org/matrix-react-sdk#7243)). Fixes element-hq#19973. * Hide clear cache and reload button if crash is before client init ([\element-hq#7242](matrix-org/matrix-react-sdk#7242)). Fixes matrix-org/element-web-rageshakes#6996. * Fix automatic space switching wrongly going via Home for room aliases ([\element-hq#7247](matrix-org/matrix-react-sdk#7247)). Fixes element-hq#19974. * Fix links being parsed as markdown links improperly ([\element-hq#7200](matrix-org/matrix-react-sdk#7200)). Contributed by @Palid.
* Include Vietnamese language ([\element-hq#20029](element-hq#20029)). * Simple static location sharing ([\element-hq#19754](element-hq#19754)). * Add support for the Indonesian language ([\element-hq#20032](element-hq#20032)). Fixes element-hq#20030. Contributed by @Linerly. * Always unhide widgets on layout change (pinning a widget) ([\element-hq#7299](matrix-org/matrix-react-sdk#7299)). * Update status message in the member list and user info panel when it is changed ([\element-hq#7338](matrix-org/matrix-react-sdk#7338)). Fixes element-hq#20127. Contributed by @SimonBrandner. * Iterate space panel toggle collapse interaction ([\element-hq#7335](matrix-org/matrix-react-sdk#7335)). Fixes element-hq#20079. * Spotlight search labs ([\element-hq#7116](matrix-org/matrix-react-sdk#7116)). Fixes element-hq#19530. * Put room settings form elements in fieldsets ([\element-hq#7311](matrix-org/matrix-react-sdk#7311)). * Add descriptions to ambiguous links for screen readers ([\element-hq#7310](matrix-org/matrix-react-sdk#7310)). * Make tooltips keyboard accessible ([\#7281](matrix-org/matrix-react-sdk#7281)). * Iterate room context menus for DMs ([\element-hq#7308](matrix-org/matrix-react-sdk#7308)). Fixes element-hq#19527. * Update space panel expand mechanism ([\element-hq#7230](matrix-org/matrix-react-sdk#7230)). Fixes element-hq#17993. * Add CSS variable to make the UI gaps consistent and fix the resize handle position ([\element-hq#7234](matrix-org/matrix-react-sdk#7234)). Fixes element-hq#19904 and element-hq#19938. * Custom location sharing. ([\element-hq#7185](matrix-org/matrix-react-sdk#7185)). * Simple static location sharing ([\element-hq#7135](matrix-org/matrix-react-sdk#7135)). * Finish sending pending messages before leaving room ([\element-hq#7276](matrix-org/matrix-react-sdk#7276)). Fixes element-hq#4702. * Dropdown follow wai-aria practices for expanding on arrow keys ([\element-hq#7277](matrix-org/matrix-react-sdk#7277)). Fixes element-hq#3687. * Expose PL control for pinned events when lab enabled ([\element-hq#7278](matrix-org/matrix-react-sdk#7278)). Fixes element-hq#5396. * In People & Favourites metaspaces always show all rooms ([\element-hq#7288](matrix-org/matrix-react-sdk#7288)). Fixes element-hq#20048. * Don't allow calls when the connection the server has been lost ([\element-hq#7287](matrix-org/matrix-react-sdk#7287)). Fixes element-hq#2096. Contributed by @SimonBrandner. * Analytics opt in for posthog ([\element-hq#6936](matrix-org/matrix-react-sdk#6936)). * Don't inhibit current room notifications if user has Modal open ([\element-hq#7274](matrix-org/matrix-react-sdk#7274)). Fixes element-hq#1118. * Remove the `Screen sharing is here!` dialog ([\#7266](matrix-org/matrix-react-sdk#7266)). Fixes element-hq#18824. Contributed by @SimonBrandner. * Make composer buttons react to settings without having to change room ([\element-hq#7264](matrix-org/matrix-react-sdk#7264)). Fixes element-hq#20011. * Decorate view keyboard shortcuts link as a link ([\element-hq#7260](matrix-org/matrix-react-sdk#7260)). Fixes element-hq#20007. * Improve ease of focusing on Room list Search ([\element-hq#7255](matrix-org/matrix-react-sdk#7255)). Fixes matrix-org/element-web-rageshakes#7017. * Autofocus device panel entry when renaming device ([\element-hq#7249](matrix-org/matrix-react-sdk#7249)). Fixes element-hq#19984. * Update Space Panel scrollable region ([\element-hq#7245](matrix-org/matrix-react-sdk#7245)). Fixes element-hq#19978. * Replace breadcrumbs with recently viewed menu ([\element-hq#7073](matrix-org/matrix-react-sdk#7073)). Fixes element-hq#19528. * Tweaks to informational architecture 1.1 ([\element-hq#7052](matrix-org/matrix-react-sdk#7052)). Fixes element-hq#19526, element-hq#19379, element-hq#17792, element-hq#16450, element-hq#19881, element-hq#19892, element-hq#19300, element-hq#19324, element-hq#17307, element-hq#17468 element-hq#19932 and element-hq#19956. * [Release] Fix inline code block nowrap issue ([\element-hq#7407](matrix-org/matrix-react-sdk#7407)). * don't collapse spaces in inline code blocks (https ([\element-hq#7328](matrix-org/matrix-react-sdk#7328)). Fixes element-hq#6051. Contributed by @HarHarLinks. * Fix accessibility regressions ([\element-hq#7336](matrix-org/matrix-react-sdk#7336)). * Debounce User Info start dm "Message" button ([\element-hq#7357](matrix-org/matrix-react-sdk#7357)). Fixes element-hq#7763. * Fix thread filter being cut-off on narrow screens ([\element-hq#7354](matrix-org/matrix-react-sdk#7354)). Fixes element-hq#20146. * Fix upgraded rooms wrongly showing up in spotlight ([\element-hq#7341](matrix-org/matrix-react-sdk#7341)). Fixes element-hq#20141. * Show votes in replied-to polls (pass in getRelationsForEvent) ([\element-hq#7345](matrix-org/matrix-react-sdk#7345)). Fixes element-hq#20153. * Keep all previously approved widget capabilities when requesting new capabilities ([\element-hq#7340](matrix-org/matrix-react-sdk#7340)). Contributed by @dhenneke. * Only show poll previews when the polls feature is enabled ([\element-hq#7331](matrix-org/matrix-react-sdk#7331)). * No-op action:join if the user is already invited for scalar ([\element-hq#7334](matrix-org/matrix-react-sdk#7334)). Fixes element-hq#20134. * Don't show polls in timeline if polls are disabled ([\element-hq#7332](matrix-org/matrix-react-sdk#7332)). Fixes element-hq#20130. * Don't send a poll response event if you are voting for your current c… ([\element-hq#7326](matrix-org/matrix-react-sdk#7326)). Fixes element-hq#20129. * Don't show options button when the user can't modify widgets ([\element-hq#7324](matrix-org/matrix-react-sdk#7324)). Fixes element-hq#20114. Contributed by @SimonBrandner. * Add vertical spacing between buttons when they go over multiple lines ([\element-hq#7314](matrix-org/matrix-react-sdk#7314)). Contributed by @twigleingrid. * Improve accessibility of opening space create menu ([\element-hq#7316](matrix-org/matrix-react-sdk#7316)). * Correct tab order in room preview dialog ([\element-hq#7302](matrix-org/matrix-react-sdk#7302)). * Fix favourites and people metaspaces not rendering their content ([\#7315](matrix-org/matrix-react-sdk#7315)). Fixes element-hq#20070. * Make clear button images visible in high contrast theme ([\element-hq#7306](matrix-org/matrix-react-sdk#7306)). Fixes element-hq#19931. * Fix html exporting and improve output size ([\element-hq#7312](matrix-org/matrix-react-sdk#7312)). Fixes element-hq#19436 element-hq#20107 and element-hq#19441. * Fix textual message stripping new line ([\element-hq#7239](matrix-org/matrix-react-sdk#7239)). Fixes element-hq#15320. Contributed by @renancleyson-dev. * Fix issue with room list resizer getting clipped in firefox ([\element-hq#7303](matrix-org/matrix-react-sdk#7303)). Fixes element-hq#20076. * Fix wrong indentation with nested ordered list unnesting list on edit ([\element-hq#7300](matrix-org/matrix-react-sdk#7300)). Contributed by @renancleyson-dev. * Fix input field behaviour inside context menus ([\element-hq#7293](matrix-org/matrix-react-sdk#7293)). Fixes element-hq#19881. * Corrected the alignment of the Edit button on LoginPage. ([\element-hq#7292](matrix-org/matrix-react-sdk#7292)). Contributed by @ankur12-1610. * Allow sharing manual location without giving location permission ([\element-hq#7295](matrix-org/matrix-react-sdk#7295)). Fixes element-hq#20065. Contributed by @tulir. * Make emoji picker search placeholder localizable ([\element-hq#7294](matrix-org/matrix-react-sdk#7294)). * Fix jump to bottom on message send ([\element-hq#7280](matrix-org/matrix-react-sdk#7280)). Fixes element-hq#19859. Contributed by @SimonBrandner. * Fix: Warning: Unsupported style property pointer-events. Did you mean pointerEvents? ([\element-hq#7291](matrix-org/matrix-react-sdk#7291)). * Add edits and replies to the right panel timeline & prepare the timelineCard to share code with threads ([\element-hq#7262](matrix-org/matrix-react-sdk#7262)). Fixes element-hq#20012 and element-hq#19928. * Fix labs exploding when lab group is empty ([\element-hq#7290](matrix-org/matrix-react-sdk#7290)). Fixes element-hq#20051. * Update URL when room aliases are modified ([\element-hq#7289](matrix-org/matrix-react-sdk#7289)). Fixes element-hq#1616 and element-hq#1925. * Render mini user menu for when space panel is disabled ([\element-hq#7258](matrix-org/matrix-react-sdk#7258)). Fixes element-hq#19998. * When accepting DM from People metaspace don't switch to Home ([\element-hq#7272](matrix-org/matrix-react-sdk#7272)). Fixes element-hq#19995. * Fix CallPreview `room is null` ([\element-hq#7265](matrix-org/matrix-react-sdk#7265)). Fixes element-hq#19990, element-hq#19972, matrix-org/element-web-rageshakes#7004 matrix-org/element-web-rageshakes#6991 and matrix-org/element-web-rageshakes#6964. * Fixes more instances of double-translation ([\element-hq#7259](matrix-org/matrix-react-sdk#7259)). Fixes element-hq#20010. * Fix video calls ([\element-hq#7256](matrix-org/matrix-react-sdk#7256)). Fixes element-hq#20008. Contributed by @SimonBrandner. * Fix broken i18n in Forgot & Change password ([\element-hq#7252](matrix-org/matrix-react-sdk#7252)). Fixes element-hq#19989. * Fix setBotPower to not use `.content` ([\element-hq#7179](matrix-org/matrix-react-sdk#7179)). Fixes element-hq#19845. * Break long words in pinned messages to prevent overflow ([\element-hq#7251](matrix-org/matrix-react-sdk#7251)). Fixes element-hq#19985. * Disallow sending empty feedbacks ([\element-hq#7240](matrix-org/matrix-react-sdk#7240)). * Fix wrongly sized default sub-space icons in space panel ([\element-hq#7243](matrix-org/matrix-react-sdk#7243)). Fixes element-hq#19973. * Hide clear cache and reload button if crash is before client init ([\element-hq#7242](matrix-org/matrix-react-sdk#7242)). Fixes matrix-org/element-web-rageshakes#6996. * Fix automatic space switching wrongly going via Home for room aliases ([\element-hq#7247](matrix-org/matrix-react-sdk#7247)). Fixes element-hq#19974. * Fix links being parsed as markdown links improperly ([\element-hq#7200](matrix-org/matrix-react-sdk#7200)). Contributed by @Palid.
Rewriting this as a to-do list for Cypress crypto testing :) |
@richvdh do any of the test cases listed happen to have been added during the ER project? |
Hrm. https://github.com/matrix-org/matrix-react-end-to-end-tests is dead (and superceded by Cypress). I'm don't think any of the unchecked tests have been added explicitly. Most of the tests we've added have centered on things like verification, cross-signing, and event shields. Some cases would be easy enough to add; others are virtually impossible to test with an e2e test (such as racing to claim OTKs). And our cypress test infrastructure doesn't easily cater for federated homeservers, so those are hard to cover. Not entirely sure what to do with this issue. It feels a bit outdated. |
How about we reduce it to the ones you'd deem reasonable to add into our current E2E testing infrastructure? |
This is easy, but of questionable value. We cover it in other tests anyway, and if E2E breaks this badly we'll notice pretty quickly.
Don't know what this means. How can you send a message before you join?
We could probably usefully do something like this. It's slightly tricky because (obviously) you only actually run a single instance of EW and the other "devices" are effectively "bots" controlled directly by javascript. And you can only really have a max of 2 (3 at a stretch) active devices in cypress otherwise you hit Chrome's concurrent request limit.
I think this would work, though there are various subtests: with and without working key backup; with and without other active devices.
I think this is maybe about historical room key sharing. Likely we'll add tests for this when we come to that piece of work anyway.
We could test this, but I don't think it works currently... The other stuff I think would be hard to test in Cypress. |
I think this might refer to joining a room and sending a message before another user joins.
Do you think test cases like the ones above would be better handled via https://github.com/matrix-org/trafficlight than Cypress / Playwright? |
Ok, so: A sends a message, then B joins without an invite, and we see if B can see the message (which he can't, that being the point of E2E?)
Not that familiar with trafficlight to be honest. Possibly some of the "multiple client" tests could usefully be done that way. I don't think it makes it easy to do things like running multiple servers though. |
Err, yeah something doesn't add up there. 🤔 |
I think we probably need to discuss the best approach for E2E tests of E2EE as a wider team. I know @kegsay is thinking about this sort of thing, and no doubt folks like @michaelkaye would have useful input too. |
So trafficlight's usefulness depends a bit on where you want to cut the tests - if you want UI to UI tests then it's probably better than everything else we have; but if you don't want to pay the overhead of managing the UI interface (like kegan is thinking about) then it's not worth the overhead. In terms of running multiple servers; it did use to have a complement homerunner to start up servers and sets of servers alongside trafficlight, but practically that was a lot of work to make available for eg emulators in the cloud type services so has slipped by the wayside. At the moment we start with new users on a long-running server. |
My thoughts are now written up at element-hq/element-meta#2165 which feels like a better home than element-web. |
For the record, this ended up resulting in https://github.com/matrix-org/complement-crypto/ which as of this writing supports: User A on the test synapse makes an E2E room and invites user B on the same test synapse
|
As part of our epic quest against UISIs/UTDs (https://github.com/vector-im/riot-web/issues/2996\), we should use our fancy new E2E testing framework over at https://github.com/matrix-org/matrix-react-end-to-end-tests to exercise E2EE end-to-end and try to locate some failing tests.
Tests to write:
* [x] Can A decrypt B's messages? Start adding Cypress tests for crypto matrix-org/matrix-react-sdk#8577
* [ ] Can B decrypt A's messages?
* [ ] sending messages before the users join (currently shouldn't work at all if A and B both join a room)
* [ ] or adding devices on both a & b's side and see if they can then participate correctly
* [ ] having b leave the room, change device, rejoin the room, and see if they can participate correctly
* [ ] Having A invite B, and seeing if B can see A's messages
* [ ] Having A invite B, having B then change device, then B join, see if B can see A's message.
* [ ] Seeing what happens and if we can recover if a device runs out of published OTKs
* [ ] Seeing what happens if devices race on claiming OTKs and end up claiming the same OTK (https://github.com/vector-im/riot-web/issues/3868\)
All of these with A and B on different servers (if Cypress can spin up multiple federating synapse instances within our CI), as the federated & non-federated code paths have entirely different flows.
another interesting one is where you deliberately break federation between a & b and see how rapidly it recovers
(in terms of missing e2e keys or unsynced dev lists)
fwiw sytest tries to test all of ^ already, but when you add the whole js-sdk stack into the mix too i'm not so confident there aren't still bugs.
(plus protecting from regressions in js-sdk ftw)
The text was updated successfully, but these errors were encountered: