Adopt the E2EE cryptography module from matrix-rust-sdk: "Element R"

[novocaine]

Context

Element Web currently uses a transpiled version of https://gitlab.matrix.org/matrix-org/olm originally written in C++

Using the rust-sdk's WASM bindings, we replace our entire crypto implementation for performance, stability, security, and maintenance reasons. This project is already approved for work to begin.

The intent is to support both old & new crypto implementations side-by-side through a labs flag up until we're ready to ship.

Update March 2024

Element Web and Element Desktop have been using the Rust crypto stack for new logins for several weeks now. All users on the develop.element.io instance of Element Web have also been migrated to the Rust stack.

Next step, hopefully starting this week (w/c 25 March 2024) is to roll out that migration to app.element.io users.

Update January 2024

We've been continuing to work on defects and missing features in Element Web R.

We've also developed a "migration" process, which will take a session using legacy crypto, and convert it to rust crypto. Once #26772 lands, you will be able to test it out by enabling the "Rust Cryptography" switch in the "labs" settings.

The next stage is to enable it for new logins on "stable" deployments (such as app.element.io, and Element Desktop). The things remaining to be sorted before that can happen are tracked at https://github.com/element-hq/element-web/issues?q=is%3Aissue+is%3Aopen+label%3AZ-Element-R-Blocker.

Update October 2023

https://develop.element.io has been updated so that new logins will use the rust cryptography implementation.

Update June 2023

We are now using a project board to track this work: https://github.com/orgs/vector-im/projects/76/views/15. "Prioritized Backlog" represents the work we consider to be required for the "MVP" of enabling on https://develop.element.io.

To try out the work-in-progress, see matrix-org/matrix-react-sdk#10080.

Plan from December 2022

High level overview of approach

(as of December 2022)

• 1. Expose a config.json setting which will make the js-sdk use matrix-sdk-crypto-js instead of libolm. At this stage there will be no support for migrating existing sessions between libolm and rust.
• 2. Build up functionality based on the rust sdk:
  • basic encryption and decryption support
  • attachment/media support
  • Key backup
  • Cross signing
  • Key sharing
    ... plus other things TBD.
• 3. Implement a one-way migration from libolm to rust, and allow people to opt into it via the labs flag.
• 4. Further work to achieve parity with the libolm implementation. (Matrix Content Scanner integration? Extensions etc for Element Call and widgets?)
• 5. Finish migration:
  • Enable the labs flag by default for new sessions
  • Forcibly migrate existing sessions to rust sdk
  • Get rid of the config flag and legacy code

After step 2, Element R web should be good enough to replace libolm-based Element Web as a daily driver. We'll begin to encourage users to try out a hosted instance of it and give feedback, while continuing with step 3 and beyond.

Detailed plans from April 2022

Phases

Phasing is approximate and used as an estimation tool. This list will be updated often - check back frequently.

Time estimates are not guarantees or even accurate - they are relative. They will also be updated as needed.

Phase 1: Prove it works (1-2 months)

• [--] Bindings exist on NPM (in progress)
• [ 5] Introduce labs flag for js-sdk
• [13] Hook up rust stores to storage. Rudimentary one-way migration.
• [13] Hook up to sync loop, basic encryption and decryption support.
  • Including attachment/media support
• [ 8] Key backup (protocol level, not UI)
• [ 8] Key sharing (protocol level, not UI)
• [13] Cross-signing (protocol level, not UI)
• [21] Two-way migration for labs flag, allowing users to "switch" between implementations
  • Potential to implement this as an implementation which writes to both stores saving us from migration

Phase 2: Establish trust in the plan (1-2 months)

• [ 8] Benchmark old crypto & new crypto to establish performance targets
• [??] Tests which prove old crypto was working
  • "Working" is defined as behavioural traits, not necessarily bug-free.
• [??] Tests which prove the new crypto isn't any more broken than old

Phase 3: Build a plausible client (2-4 months with questionable accuracy)

• [??] Self verification
• [??] Other user verification
• [??] Visual indicators for user trust (cross-signing, 4S)
• [??] UI bits for key backup, key sharing, and cross-signing
• [??] Manual key export
• [??] Historical key sharing (for room history)
• [??] Device management

Phase 4: Polish and remaining bits (1-2 months)

• [13] Support for customisation endpoints/modules as needed
• [13] Matrix Content Scanner integration (if needed)
• [??] Device dehydration
• [??] Functions required by Element Call and widgets (custom to-device messages?)
• [??] Posthog metrics
• [??] TBD stuff from https://github.com/matrix-org/matrix-rust-sdk/milestone/1
• [??] Get design involved for migration experience

Phase 5: Stability (1-2 months)

• [??] Enable by default on Nightly and Develop (not production, EMS, app, or staging)
• [??] Fix bugs & build comfort
• [??] Enable by default in production (EMS, app, and staging alongside existing develop channels)
• [??] Fix bugs & build comfort

Phase 6: Release (1 month)

• [??] Convert migration to a one-way migration
• [??] Remove old crypto code (keep migrator)
• [??] Remove labs flag
• [??] Eternal maintenance

Internal references

myhours: https://app.myhours.com/#/projects/1943744/overview

[turt2live]

waiting on poc to prove this is feasible, then will formulate and enact a plan

[novocaine]

@Hywan are there some sub issues or PRs that we can link to this epic?

[turt2live]

matrix-org/matrix-rust-sdk#675 covers the WASM-specific bindings. Crypto team is currently working on a simple encrypt/decrypt demo that will need adapting into production-safe code (which is my problem).

[Hywan]

@novocaine Nothing I'm aware of except the one mentionned by @turt2live hereinabove.

[turt2live]

issue description updated with scope

[richvdh]

Update on the current state of play:

We are working towards enabling Rust Crypto on https://develop.element.io. (Once we do this, existing logins will be unaffected, but new logins will use Rust Crypto).

The "Prioritised Backlog" on the project board is an up-to-date view of the remaining work we see as important to complete before that happens (the "MVP") but in summary, the big ticket items are:

• Element-R: Cross-user verification #25752: Cross-user verification. This is so close to working we can almost smell it.
• Element R: support for Key backup #24828: Key backup. Also nearly done.
• Element-R: Fix up message shields #25321: I have this largely working locally but need to put up some PRs and improve test coverage.
• Element-R: Use the "pickleKey" to encrypt the indexeddb crypto store #24967

[richvdh]

#26291 proposes to enable rust crypto on https://develop.element.io

[richvdh]

We are tracking remaining blockers for deployment to develop.element.io at https://github.com/vector-im/element-web/issues?q=is%3Aissue+is%3Aopen+label%3AZ-Element-R-Blocker

[richvdh]

Just to keep this issue up to date: #26291 has now been merged, meaning that new logins on https://develop.element.io will use rust crypto.

[richvdh]

Update:

We've been continuing to work on defects and missing features in Element Web R.

We've also developed a "migration" process, which will take a session using legacy crypto, and convert it to rust crypto. Once #26772 lands, you will be able to test it out by enabling the "Rust Cryptography" switch in the "labs" settings.

The next stage is to enable it for new logins on "stable" deployments (such as app.element.io, and Element Desktop). The things remaining to be sorted before that can happen are tracked at https://github.com/element-hq/element-web/issues?q=is%3Aissue+is%3Aopen+label%3AZ-Element-R-Blocker.

[Johennes]

The https://github.com/matrix-org/matrix-js-sdk?tab=readme-ov-file#end-to-end-encryption-support section is marked as outdated, pointing at this issue but neither gives any advice on how to set up E2EE using Rust crypto. Is there any recommendation for client developers other than reverse-engineering it from Element Web?

[richvdh]

Unfortunately improving the documentation is still on the todo list (matrix-org/matrix-js-sdk#4132).