Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add end-to-end tests for End-to-end encryption #7312

Closed
ara4n opened this issue Sep 11, 2018 · 13 comments
Closed

Add end-to-end tests for End-to-end encryption #7312

ara4n opened this issue Sep 11, 2018 · 13 comments
Labels
A-E2EE A-Testing Testing, code coverage, etc. O-Frequent Affects or can be seen by most users regularly or impacts most users' first experience T-Task Tasks for the team like planning Team: Crypto

Comments

@ara4n
Copy link
Member

ara4n commented Sep 11, 2018

As part of our epic quest against UISIs/UTDs (https://github.com/vector-im/riot-web/issues/2996\), we should use our fancy new E2E testing framework over at https://github.com/matrix-org/matrix-react-end-to-end-tests to exercise E2EE end-to-end and try to locate some failing tests.

Tests to write:

  • User A on the test synapse makes an E2E room and invites user B on the same test synapse
    * [x] Can A decrypt B's messages? Start adding Cypress tests for crypto matrix-org/matrix-react-sdk#8577
    * [ ] Can B decrypt A's messages?
    * [ ] sending messages before the users join (currently shouldn't work at all if A and B both join a room)
    * [ ] or adding devices on both a & b's side and see if they can then participate correctly
    * [ ] having b leave the room, change device, rejoin the room, and see if they can participate correctly
    * [ ] Having A invite B, and seeing if B can see A's messages
    * [ ] Having A invite B, having B then change device, then B join, see if B can see A's message.
    * [ ] Seeing what happens and if we can recover if a device runs out of published OTKs
    * [ ] Seeing what happens if devices race on claiming OTKs and end up claiming the same OTK (https://github.com/vector-im/riot-web/issues/3868\)
  • All of these with A and B on different servers (if Cypress can spin up multiple federating synapse instances within our CI), as the federated & non-federated code paths have entirely different flows.

  • another interesting one is where you deliberately break federation between a & b and see how rapidly it recovers

(in terms of missing e2e keys or unsynced dev lists)

fwiw sytest tries to test all of ^ already, but when you add the whole js-sdk stack into the mix too i'm not so confident there aren't still bugs.
(plus protecting from regressions in js-sdk ftw)

@bwindels
Copy link
Contributor

the basic case is tested already at https://github.com/matrix-org/matrix-react-end-to-end-tests/blob/master/src/scenario.js#L60 but adding federation, and extended case would indeed be great.

@lampholder lampholder added T-Task Tasks for the team like planning P1 labels Sep 21, 2018
@uhoreg uhoreg added the A-E2EE label Jan 15, 2020
BBaoVanC added a commit to boba-best/element.boba.best that referenced this issue Dec 20, 2021
* Include Vietnamese language ([\element-hq#20029](element-hq#20029)).
* Simple static location sharing ([\element-hq#19754](element-hq#19754)).
* Add support for the Indonesian language ([\element-hq#20032](element-hq#20032)). Fixes element-hq#20030. Contributed by @Linerly.
* Always unhide widgets on layout change (pinning a widget) ([\element-hq#7299](matrix-org/matrix-react-sdk#7299)).
* Update status message in the member list and user info panel when it is changed ([\element-hq#7338](matrix-org/matrix-react-sdk#7338)). Fixes element-hq#20127. Contributed by @SimonBrandner.
* Iterate space panel toggle collapse interaction ([\element-hq#7335](matrix-org/matrix-react-sdk#7335)). Fixes element-hq#20079.
* Spotlight search labs ([\element-hq#7116](matrix-org/matrix-react-sdk#7116)). Fixes element-hq#19530.
* Put room settings form elements in fieldsets ([\element-hq#7311](matrix-org/matrix-react-sdk#7311)).
* Add descriptions to ambiguous links for screen readers ([\element-hq#7310](matrix-org/matrix-react-sdk#7310)).
* Make tooltips keyboard accessible ([\#7281](matrix-org/matrix-react-sdk#7281)).
* Iterate room context menus for DMs ([\element-hq#7308](matrix-org/matrix-react-sdk#7308)). Fixes element-hq#19527.
* Update space panel expand mechanism ([\element-hq#7230](matrix-org/matrix-react-sdk#7230)). Fixes element-hq#17993.
* Add CSS variable to make the UI gaps consistent and fix the resize handle position ([\element-hq#7234](matrix-org/matrix-react-sdk#7234)). Fixes element-hq#19904 and element-hq#19938.
* Custom location sharing. ([\element-hq#7185](matrix-org/matrix-react-sdk#7185)).
* Simple static location sharing ([\element-hq#7135](matrix-org/matrix-react-sdk#7135)).
* Finish sending pending messages before leaving room ([\element-hq#7276](matrix-org/matrix-react-sdk#7276)). Fixes element-hq#4702.
* Dropdown follow wai-aria practices for expanding on arrow keys ([\element-hq#7277](matrix-org/matrix-react-sdk#7277)). Fixes element-hq#3687.
* Expose PL control for pinned events when lab enabled ([\element-hq#7278](matrix-org/matrix-react-sdk#7278)). Fixes element-hq#5396.
* In People & Favourites metaspaces always show all rooms ([\element-hq#7288](matrix-org/matrix-react-sdk#7288)). Fixes element-hq#20048.
* Don't allow calls when the connection the server has been lost ([\element-hq#7287](matrix-org/matrix-react-sdk#7287)). Fixes element-hq#2096. Contributed by @SimonBrandner.
* Analytics opt in for posthog ([\element-hq#6936](matrix-org/matrix-react-sdk#6936)).
* Don't inhibit current room notifications if user has Modal open ([\element-hq#7274](matrix-org/matrix-react-sdk#7274)). Fixes element-hq#1118.
* Remove the `Screen sharing is here!` dialog ([\#7266](matrix-org/matrix-react-sdk#7266)). Fixes element-hq#18824. Contributed by @SimonBrandner.
* Make composer buttons react to settings without having to change room ([\element-hq#7264](matrix-org/matrix-react-sdk#7264)). Fixes element-hq#20011.
* Decorate view keyboard shortcuts link as a link ([\element-hq#7260](matrix-org/matrix-react-sdk#7260)). Fixes element-hq#20007.
* Improve ease of focusing on Room list Search ([\element-hq#7255](matrix-org/matrix-react-sdk#7255)). Fixes matrix-org/element-web-rageshakes#7017.
* Autofocus device panel entry when renaming device ([\element-hq#7249](matrix-org/matrix-react-sdk#7249)). Fixes element-hq#19984.
* Update Space Panel scrollable region ([\element-hq#7245](matrix-org/matrix-react-sdk#7245)). Fixes element-hq#19978.
* Replace breadcrumbs with recently viewed menu ([\element-hq#7073](matrix-org/matrix-react-sdk#7073)). Fixes element-hq#19528.
* Tweaks to informational architecture 1.1 ([\element-hq#7052](matrix-org/matrix-react-sdk#7052)). Fixes element-hq#19526, element-hq#19379, element-hq#17792, element-hq#16450, element-hq#19881, element-hq#19892, element-hq#19300, element-hq#19324, element-hq#17307, element-hq#17468 element-hq#19932 and element-hq#19956.
* [Release] Fix inline code block nowrap issue ([\element-hq#7407](matrix-org/matrix-react-sdk#7407)).
* don't collapse spaces in inline code blocks (https ([\element-hq#7328](matrix-org/matrix-react-sdk#7328)). Fixes element-hq#6051. Contributed by @HarHarLinks.
* Fix accessibility regressions ([\element-hq#7336](matrix-org/matrix-react-sdk#7336)).
* Debounce User Info start dm "Message" button ([\element-hq#7357](matrix-org/matrix-react-sdk#7357)). Fixes element-hq#7763.
* Fix thread filter being cut-off on narrow screens ([\element-hq#7354](matrix-org/matrix-react-sdk#7354)). Fixes element-hq#20146.
* Fix upgraded rooms wrongly showing up in spotlight ([\element-hq#7341](matrix-org/matrix-react-sdk#7341)). Fixes element-hq#20141.
* Show votes in replied-to polls (pass in getRelationsForEvent) ([\element-hq#7345](matrix-org/matrix-react-sdk#7345)). Fixes element-hq#20153.
* Keep all previously approved widget capabilities when requesting new capabilities ([\element-hq#7340](matrix-org/matrix-react-sdk#7340)). Contributed by @dhenneke.
* Only show poll previews when the polls feature is enabled ([\element-hq#7331](matrix-org/matrix-react-sdk#7331)).
* No-op action:join if the user is already invited for scalar ([\element-hq#7334](matrix-org/matrix-react-sdk#7334)). Fixes element-hq#20134.
* Don't show polls in timeline if polls are disabled ([\element-hq#7332](matrix-org/matrix-react-sdk#7332)). Fixes element-hq#20130.
* Don't send a poll response event if you are voting for your current c… ([\element-hq#7326](matrix-org/matrix-react-sdk#7326)). Fixes element-hq#20129.
* Don't show options button when the user can't modify widgets ([\element-hq#7324](matrix-org/matrix-react-sdk#7324)). Fixes element-hq#20114. Contributed by @SimonBrandner.
* Add vertical spacing between buttons when they go over multiple lines ([\element-hq#7314](matrix-org/matrix-react-sdk#7314)). Contributed by @twigleingrid.
* Improve accessibility of opening space create menu ([\element-hq#7316](matrix-org/matrix-react-sdk#7316)).
* Correct tab order in room preview dialog ([\element-hq#7302](matrix-org/matrix-react-sdk#7302)).
* Fix favourites and people metaspaces not rendering their content ([\#7315](matrix-org/matrix-react-sdk#7315)). Fixes element-hq#20070.
* Make clear button images visible in high contrast theme ([\element-hq#7306](matrix-org/matrix-react-sdk#7306)). Fixes element-hq#19931.
* Fix html exporting and improve output size ([\element-hq#7312](matrix-org/matrix-react-sdk#7312)). Fixes element-hq#19436 element-hq#20107 and element-hq#19441.
* Fix textual message stripping new line ([\element-hq#7239](matrix-org/matrix-react-sdk#7239)). Fixes element-hq#15320. Contributed by @renancleyson-dev.
* Fix issue with room list resizer getting clipped in firefox ([\element-hq#7303](matrix-org/matrix-react-sdk#7303)). Fixes element-hq#20076.
* Fix wrong indentation with nested ordered list unnesting list on edit ([\element-hq#7300](matrix-org/matrix-react-sdk#7300)). Contributed by @renancleyson-dev.
* Fix input field behaviour inside context menus ([\element-hq#7293](matrix-org/matrix-react-sdk#7293)). Fixes element-hq#19881.
* Corrected the alignment of the Edit button on LoginPage. ([\element-hq#7292](matrix-org/matrix-react-sdk#7292)). Contributed by @ankur12-1610.
* Allow sharing manual location without giving location permission ([\element-hq#7295](matrix-org/matrix-react-sdk#7295)). Fixes element-hq#20065. Contributed by @tulir.
* Make emoji picker search placeholder localizable ([\element-hq#7294](matrix-org/matrix-react-sdk#7294)).
* Fix jump to bottom on message send ([\element-hq#7280](matrix-org/matrix-react-sdk#7280)). Fixes element-hq#19859. Contributed by @SimonBrandner.
* Fix: Warning: Unsupported style property pointer-events. Did you mean pointerEvents? ([\element-hq#7291](matrix-org/matrix-react-sdk#7291)).
* Add edits and replies to the right panel timeline & prepare the timelineCard to share code with threads ([\element-hq#7262](matrix-org/matrix-react-sdk#7262)). Fixes element-hq#20012 and element-hq#19928.
* Fix labs exploding when lab group is empty ([\element-hq#7290](matrix-org/matrix-react-sdk#7290)). Fixes element-hq#20051.
* Update URL when room aliases are modified ([\element-hq#7289](matrix-org/matrix-react-sdk#7289)). Fixes element-hq#1616 and element-hq#1925.
* Render mini user menu for when space panel is disabled ([\element-hq#7258](matrix-org/matrix-react-sdk#7258)). Fixes element-hq#19998.
* When accepting DM from People metaspace don't switch to Home ([\element-hq#7272](matrix-org/matrix-react-sdk#7272)). Fixes element-hq#19995.
* Fix CallPreview `room is null` ([\element-hq#7265](matrix-org/matrix-react-sdk#7265)). Fixes element-hq#19990, element-hq#19972, matrix-org/element-web-rageshakes#7004 matrix-org/element-web-rageshakes#6991 and matrix-org/element-web-rageshakes#6964.
* Fixes more instances of double-translation ([\element-hq#7259](matrix-org/matrix-react-sdk#7259)). Fixes element-hq#20010.
* Fix video calls ([\element-hq#7256](matrix-org/matrix-react-sdk#7256)). Fixes element-hq#20008. Contributed by @SimonBrandner.
* Fix broken i18n in Forgot & Change password ([\element-hq#7252](matrix-org/matrix-react-sdk#7252)). Fixes element-hq#19989.
* Fix setBotPower to not use `.content` ([\element-hq#7179](matrix-org/matrix-react-sdk#7179)). Fixes element-hq#19845.
* Break long words in pinned messages to prevent overflow ([\element-hq#7251](matrix-org/matrix-react-sdk#7251)). Fixes element-hq#19985.
* Disallow sending empty feedbacks ([\element-hq#7240](matrix-org/matrix-react-sdk#7240)).
* Fix wrongly sized default sub-space icons in space panel ([\element-hq#7243](matrix-org/matrix-react-sdk#7243)). Fixes element-hq#19973.
* Hide clear cache and reload button if crash is before client init ([\element-hq#7242](matrix-org/matrix-react-sdk#7242)). Fixes matrix-org/element-web-rageshakes#6996.
* Fix automatic space switching wrongly going via Home for room aliases ([\element-hq#7247](matrix-org/matrix-react-sdk#7247)). Fixes element-hq#19974.
* Fix links being parsed as markdown links improperly ([\element-hq#7200](matrix-org/matrix-react-sdk#7200)). Contributed by @Palid.
@aaronraimist aaronraimist added the A-Testing Testing, code coverage, etc. label Dec 21, 2021
su-ex added a commit to SchildiChat/element-web that referenced this issue Dec 21, 2021
* Include Vietnamese language ([\element-hq#20029](element-hq#20029)).
* Simple static location sharing ([\element-hq#19754](element-hq#19754)).
* Add support for the Indonesian language ([\element-hq#20032](element-hq#20032)). Fixes element-hq#20030. Contributed by @Linerly.
* Always unhide widgets on layout change (pinning a widget) ([\element-hq#7299](matrix-org/matrix-react-sdk#7299)).
* Update status message in the member list and user info panel when it is changed ([\element-hq#7338](matrix-org/matrix-react-sdk#7338)). Fixes element-hq#20127. Contributed by @SimonBrandner.
* Iterate space panel toggle collapse interaction ([\element-hq#7335](matrix-org/matrix-react-sdk#7335)). Fixes element-hq#20079.
* Spotlight search labs ([\element-hq#7116](matrix-org/matrix-react-sdk#7116)). Fixes element-hq#19530.
* Put room settings form elements in fieldsets ([\element-hq#7311](matrix-org/matrix-react-sdk#7311)).
* Add descriptions to ambiguous links for screen readers ([\element-hq#7310](matrix-org/matrix-react-sdk#7310)).
* Make tooltips keyboard accessible ([\#7281](matrix-org/matrix-react-sdk#7281)).
* Iterate room context menus for DMs ([\element-hq#7308](matrix-org/matrix-react-sdk#7308)). Fixes element-hq#19527.
* Update space panel expand mechanism ([\element-hq#7230](matrix-org/matrix-react-sdk#7230)). Fixes element-hq#17993.
* Add CSS variable to make the UI gaps consistent and fix the resize handle position ([\element-hq#7234](matrix-org/matrix-react-sdk#7234)). Fixes element-hq#19904 and element-hq#19938.
* Custom location sharing. ([\element-hq#7185](matrix-org/matrix-react-sdk#7185)).
* Simple static location sharing ([\element-hq#7135](matrix-org/matrix-react-sdk#7135)).
* Finish sending pending messages before leaving room ([\element-hq#7276](matrix-org/matrix-react-sdk#7276)). Fixes element-hq#4702.
* Dropdown follow wai-aria practices for expanding on arrow keys ([\element-hq#7277](matrix-org/matrix-react-sdk#7277)). Fixes element-hq#3687.
* Expose PL control for pinned events when lab enabled ([\element-hq#7278](matrix-org/matrix-react-sdk#7278)). Fixes element-hq#5396.
* In People & Favourites metaspaces always show all rooms ([\element-hq#7288](matrix-org/matrix-react-sdk#7288)). Fixes element-hq#20048.
* Don't allow calls when the connection the server has been lost ([\element-hq#7287](matrix-org/matrix-react-sdk#7287)). Fixes element-hq#2096. Contributed by @SimonBrandner.
* Analytics opt in for posthog ([\element-hq#6936](matrix-org/matrix-react-sdk#6936)).
* Don't inhibit current room notifications if user has Modal open ([\element-hq#7274](matrix-org/matrix-react-sdk#7274)). Fixes element-hq#1118.
* Remove the `Screen sharing is here!` dialog ([\#7266](matrix-org/matrix-react-sdk#7266)). Fixes element-hq#18824. Contributed by @SimonBrandner.
* Make composer buttons react to settings without having to change room ([\element-hq#7264](matrix-org/matrix-react-sdk#7264)). Fixes element-hq#20011.
* Decorate view keyboard shortcuts link as a link ([\element-hq#7260](matrix-org/matrix-react-sdk#7260)). Fixes element-hq#20007.
* Improve ease of focusing on Room list Search ([\element-hq#7255](matrix-org/matrix-react-sdk#7255)). Fixes matrix-org/element-web-rageshakes#7017.
* Autofocus device panel entry when renaming device ([\element-hq#7249](matrix-org/matrix-react-sdk#7249)). Fixes element-hq#19984.
* Update Space Panel scrollable region ([\element-hq#7245](matrix-org/matrix-react-sdk#7245)). Fixes element-hq#19978.
* Replace breadcrumbs with recently viewed menu ([\element-hq#7073](matrix-org/matrix-react-sdk#7073)). Fixes element-hq#19528.
* Tweaks to informational architecture 1.1 ([\element-hq#7052](matrix-org/matrix-react-sdk#7052)). Fixes element-hq#19526, element-hq#19379, element-hq#17792, element-hq#16450, element-hq#19881, element-hq#19892, element-hq#19300, element-hq#19324, element-hq#17307, element-hq#17468 element-hq#19932 and element-hq#19956.
* [Release] Fix inline code block nowrap issue ([\element-hq#7407](matrix-org/matrix-react-sdk#7407)).
* don't collapse spaces in inline code blocks (https ([\element-hq#7328](matrix-org/matrix-react-sdk#7328)). Fixes element-hq#6051. Contributed by @HarHarLinks.
* Fix accessibility regressions ([\element-hq#7336](matrix-org/matrix-react-sdk#7336)).
* Debounce User Info start dm "Message" button ([\element-hq#7357](matrix-org/matrix-react-sdk#7357)). Fixes element-hq#7763.
* Fix thread filter being cut-off on narrow screens ([\element-hq#7354](matrix-org/matrix-react-sdk#7354)). Fixes element-hq#20146.
* Fix upgraded rooms wrongly showing up in spotlight ([\element-hq#7341](matrix-org/matrix-react-sdk#7341)). Fixes element-hq#20141.
* Show votes in replied-to polls (pass in getRelationsForEvent) ([\element-hq#7345](matrix-org/matrix-react-sdk#7345)). Fixes element-hq#20153.
* Keep all previously approved widget capabilities when requesting new capabilities ([\element-hq#7340](matrix-org/matrix-react-sdk#7340)). Contributed by @dhenneke.
* Only show poll previews when the polls feature is enabled ([\element-hq#7331](matrix-org/matrix-react-sdk#7331)).
* No-op action:join if the user is already invited for scalar ([\element-hq#7334](matrix-org/matrix-react-sdk#7334)). Fixes element-hq#20134.
* Don't show polls in timeline if polls are disabled ([\element-hq#7332](matrix-org/matrix-react-sdk#7332)). Fixes element-hq#20130.
* Don't send a poll response event if you are voting for your current c… ([\element-hq#7326](matrix-org/matrix-react-sdk#7326)). Fixes element-hq#20129.
* Don't show options button when the user can't modify widgets ([\element-hq#7324](matrix-org/matrix-react-sdk#7324)). Fixes element-hq#20114. Contributed by @SimonBrandner.
* Add vertical spacing between buttons when they go over multiple lines ([\element-hq#7314](matrix-org/matrix-react-sdk#7314)). Contributed by @twigleingrid.
* Improve accessibility of opening space create menu ([\element-hq#7316](matrix-org/matrix-react-sdk#7316)).
* Correct tab order in room preview dialog ([\element-hq#7302](matrix-org/matrix-react-sdk#7302)).
* Fix favourites and people metaspaces not rendering their content ([\#7315](matrix-org/matrix-react-sdk#7315)). Fixes element-hq#20070.
* Make clear button images visible in high contrast theme ([\element-hq#7306](matrix-org/matrix-react-sdk#7306)). Fixes element-hq#19931.
* Fix html exporting and improve output size ([\element-hq#7312](matrix-org/matrix-react-sdk#7312)). Fixes element-hq#19436 element-hq#20107 and element-hq#19441.
* Fix textual message stripping new line ([\element-hq#7239](matrix-org/matrix-react-sdk#7239)). Fixes element-hq#15320. Contributed by @renancleyson-dev.
* Fix issue with room list resizer getting clipped in firefox ([\element-hq#7303](matrix-org/matrix-react-sdk#7303)). Fixes element-hq#20076.
* Fix wrong indentation with nested ordered list unnesting list on edit ([\element-hq#7300](matrix-org/matrix-react-sdk#7300)). Contributed by @renancleyson-dev.
* Fix input field behaviour inside context menus ([\element-hq#7293](matrix-org/matrix-react-sdk#7293)). Fixes element-hq#19881.
* Corrected the alignment of the Edit button on LoginPage. ([\element-hq#7292](matrix-org/matrix-react-sdk#7292)). Contributed by @ankur12-1610.
* Allow sharing manual location without giving location permission ([\element-hq#7295](matrix-org/matrix-react-sdk#7295)). Fixes element-hq#20065. Contributed by @tulir.
* Make emoji picker search placeholder localizable ([\element-hq#7294](matrix-org/matrix-react-sdk#7294)).
* Fix jump to bottom on message send ([\element-hq#7280](matrix-org/matrix-react-sdk#7280)). Fixes element-hq#19859. Contributed by @SimonBrandner.
* Fix: Warning: Unsupported style property pointer-events. Did you mean pointerEvents? ([\element-hq#7291](matrix-org/matrix-react-sdk#7291)).
* Add edits and replies to the right panel timeline & prepare the timelineCard to share code with threads ([\element-hq#7262](matrix-org/matrix-react-sdk#7262)). Fixes element-hq#20012 and element-hq#19928.
* Fix labs exploding when lab group is empty ([\element-hq#7290](matrix-org/matrix-react-sdk#7290)). Fixes element-hq#20051.
* Update URL when room aliases are modified ([\element-hq#7289](matrix-org/matrix-react-sdk#7289)). Fixes element-hq#1616 and element-hq#1925.
* Render mini user menu for when space panel is disabled ([\element-hq#7258](matrix-org/matrix-react-sdk#7258)). Fixes element-hq#19998.
* When accepting DM from People metaspace don't switch to Home ([\element-hq#7272](matrix-org/matrix-react-sdk#7272)). Fixes element-hq#19995.
* Fix CallPreview `room is null` ([\element-hq#7265](matrix-org/matrix-react-sdk#7265)). Fixes element-hq#19990, element-hq#19972, matrix-org/element-web-rageshakes#7004 matrix-org/element-web-rageshakes#6991 and matrix-org/element-web-rageshakes#6964.
* Fixes more instances of double-translation ([\element-hq#7259](matrix-org/matrix-react-sdk#7259)). Fixes element-hq#20010.
* Fix video calls ([\element-hq#7256](matrix-org/matrix-react-sdk#7256)). Fixes element-hq#20008. Contributed by @SimonBrandner.
* Fix broken i18n in Forgot & Change password ([\element-hq#7252](matrix-org/matrix-react-sdk#7252)). Fixes element-hq#19989.
* Fix setBotPower to not use `.content` ([\element-hq#7179](matrix-org/matrix-react-sdk#7179)). Fixes element-hq#19845.
* Break long words in pinned messages to prevent overflow ([\element-hq#7251](matrix-org/matrix-react-sdk#7251)). Fixes element-hq#19985.
* Disallow sending empty feedbacks ([\element-hq#7240](matrix-org/matrix-react-sdk#7240)).
* Fix wrongly sized default sub-space icons in space panel ([\element-hq#7243](matrix-org/matrix-react-sdk#7243)). Fixes element-hq#19973.
* Hide clear cache and reload button if crash is before client init ([\element-hq#7242](matrix-org/matrix-react-sdk#7242)). Fixes matrix-org/element-web-rageshakes#6996.
* Fix automatic space switching wrongly going via Home for room aliases ([\element-hq#7247](matrix-org/matrix-react-sdk#7247)). Fixes element-hq#19974.
* Fix links being parsed as markdown links improperly ([\element-hq#7200](matrix-org/matrix-react-sdk#7200)). Contributed by @Palid.
@duxovni
Copy link
Contributor

duxovni commented May 27, 2022

Rewriting this as a to-do list for Cypress crypto testing :)

@SimonBrandner SimonBrandner added O-Frequent Affects or can be seen by most users regularly or impacts most users' first experience and removed P1 labels May 27, 2022
@Johennes
Copy link
Contributor

@richvdh do any of the test cases listed happen to have been added during the ER project?

@richvdh
Copy link
Member

richvdh commented Oct 24, 2023

Hrm. https://github.com/matrix-org/matrix-react-end-to-end-tests is dead (and superceded by Cypress).

I'm don't think any of the unchecked tests have been added explicitly. Most of the tests we've added have centered on things like verification, cross-signing, and event shields.

Some cases would be easy enough to add; others are virtually impossible to test with an e2e test (such as racing to claim OTKs). And our cypress test infrastructure doesn't easily cater for federated homeservers, so those are hard to cover.

Not entirely sure what to do with this issue. It feels a bit outdated.

@Johennes
Copy link
Contributor

How about we reduce it to the ones you'd deem reasonable to add into our current E2E testing infrastructure?

@richvdh
Copy link
Member

richvdh commented Oct 24, 2023

  • Can B decrypt A's messages?

This is easy, but of questionable value. We cover it in other tests anyway, and if E2E breaks this badly we'll notice pretty quickly.

  • sending messages before the users join (currently shouldn't work at all if A and B both join a room)

Don't know what this means. How can you send a message before you join?

  • or adding devices on both a & b's side and see if they can then participate correctly

We could probably usefully do something like this. It's slightly tricky because (obviously) you only actually run a single instance of EW and the other "devices" are effectively "bots" controlled directly by javascript. And you can only really have a max of 2 (3 at a stretch) active devices in cypress otherwise you hit Chrome's concurrent request limit.

  • having b leave the room, change device, rejoin the room, and see if they can participate correctly

I think this would work, though there are various subtests: with and without working key backup; with and without other active devices.

  • Having A invite B, and seeing if B can see A's messages

I think this is maybe about historical room key sharing. Likely we'll add tests for this when we come to that piece of work anyway.

  • Having A invite B, having B then change device, then B join, see if B can see A's message.

We could test this, but I don't think it works currently...

The other stuff I think would be hard to test in Cypress.

@Johennes
Copy link
Contributor

-[ ] sending messages before the users join (currently shouldn't work at all if A and B both join a room)

Don't know what this means. How can you send a message before you join?

I think this might refer to joining a room and sending a message before another user joins.

[...] you only actually run a single instance of EW and the other "devices" are effectively "bots" controlled directly by javascript

Do you think test cases like the ones above would be better handled via https://github.com/matrix-org/trafficlight than Cypress / Playwright?

@Johennes Johennes changed the title End-to-end tests for End-to-end encryption Add End-to-end tests for End-to-end encryption Oct 25, 2023
@Johennes Johennes changed the title Add End-to-end tests for End-to-end encryption Add end-to-end tests for End-to-end encryption Oct 25, 2023
@richvdh
Copy link
Member

richvdh commented Oct 25, 2023

-[ ] sending messages before the users join (currently shouldn't work at all if A and B both join a room)
Don't know what this means. How can you send a message before you join?

I think this might refer to joining a room and sending a message before another user joins.

Ok, so: A sends a message, then B joins without an invite, and we see if B can see the message (which he can't, that being the point of E2E?)

[...] you only actually run a single instance of EW and the other "devices" are effectively "bots" controlled directly by javascript

Do you think test cases like the ones above would be better handled via https://github.com/matrix-org/trafficlight than Cypress / Playwright?

Not that familiar with trafficlight to be honest. Possibly some of the "multiple client" tests could usefully be done that way. I don't think it makes it easy to do things like running multiple servers though.

@Johennes
Copy link
Contributor

Ok, so: A sends a message, then B joins without an invite, and we see if B can see the message (which he can't, that being the point of E2E?)

Err, yeah something doesn't add up there. 🤔

@richvdh
Copy link
Member

richvdh commented Oct 25, 2023

I think we probably need to discuss the best approach for E2E tests of E2EE as a wider team. I know @kegsay is thinking about this sort of thing, and no doubt folks like @michaelkaye would have useful input too.

@michaelkaye
Copy link
Contributor

So trafficlight's usefulness depends a bit on where you want to cut the tests - if you want UI to UI tests then it's probably better than everything else we have; but if you don't want to pay the overhead of managing the UI interface (like kegan is thinking about) then it's not worth the overhead.

In terms of running multiple servers; it did use to have a complement homerunner to start up servers and sets of servers alongside trafficlight, but practically that was a lot of work to make available for eg emulators in the cloud type services so has slipped by the wayside. At the moment we start with new users on a long-running server.

@kegsay
Copy link
Contributor

kegsay commented Oct 27, 2023

My thoughts are now written up at element-hq/element-meta#2165 which feels like a better home than element-web.

@Johennes Johennes closed this as not planned Won't fix, can't repro, duplicate, stale Nov 6, 2023
@kegsay
Copy link
Contributor

kegsay commented Nov 21, 2023

For the record, this ended up resulting in https://github.com/matrix-org/complement-crypto/ which as of this writing supports:

User A on the test synapse makes an E2E room and invites user B on the same test synapse

  • Can A decrypt B's messages? Start adding Cypress tests for crypto matrix-org/matrix-react-sdk#8577
  • Can B decrypt A's messages?
  • sending messages before the users join (currently shouldn't work at all if A and B both join a room)
  • or adding devices on both a & b's side and see if they can then participate correctly
  • having b leave the room, change device, rejoin the room, and see if they can participate correctly
  • Having A invite B, and seeing if B can see A's messages
  • FAILS: Having A invite B, having B then change device, then B join, see if B can see A's message.
  • Seeing what happens and if we can recover if a device runs out of published OTKs
  • Seeing what happens if devices race on claiming OTKs and end up claiming the same OTK

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-E2EE A-Testing Testing, code coverage, etc. O-Frequent Affects or can be seen by most users regularly or impacts most users' first experience T-Task Tasks for the team like planning Team: Crypto
Projects
None yet
Development

No branches or pull requests