Add ephemeral messages support (MSC2228) #6409

babolivier · 2019-11-25T14:25:29Z

Implement part MSC2228. The parts that differ are:

the feature is hidden behind a configuration flag (enable_ephemeral_messages)
self-destruction doesn't happen for state events
only implement support for the m.self_destruct_after field (not the m.self_destruct one)
doesn't send synthetic redactions to clients because for this specific case we consider the clients to be able to destroy an event themselves, instead we just censor it (by pruning its JSON) in the database

The actual redaction isn't there yet though

This is a slightly modified copy of the event_store_labels background update, adapted to extract and store expiry timestamps for events.

richvdh

good start, but needs a bit more work I think.

I don't think you've thought about how it is going to work with workers: which workers does this stuff run on, and how will you arrange for that to be so?

synapse/api/constants.py

synapse/handlers/message.py

synapse/handlers/federation.py

synapse/storage/data_stores/main/events.py

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

stale

richvdh

lgtm!

richvdh · 2019-12-03T15:20:19Z

synapse/handlers/message.py

+        now_ms = self.clock.time_msec()
+        delay = (expiry_ts - now_ms) / 1000
+
+        # callLater doesn't support negative delays, so trim the delay to 0 if we're
+        # in that case.
+        if delay < 0:
+            delay = 0


fwiw I'd be inclined to write all of this as:

delay = max(0, expiry_ts - self.clock.time_msec()) / 1000

but no harm in making it all explicit as you have.

richvdh · 2019-12-03T15:22:28Z

synapse/storage/data_stores/main/events.py

+    def _delete_event_expiry_txn(self, txn, event_id):
+        """Delete the expiry timestamp associated with an event ID without deleting the
+        actual event.
+
+        Args:
+            txn (LoggingTransaction): The transaction to use to perform the deletion.
+            event_id (str): The event ID to delete the associated expiry timestamp of.
+        """
+        return self._simple_delete_txn(
+            txn=txn, table="event_expiry", keyvalues={"event_id": event_id}
+        )


i'd inline this, but ymmv

This reverts commit c1102d6.

This reverts commit 8d2ee04.

Synapse 1.7.0rc1 (2019-12-09) ============================= Features -------- - Implement per-room message retention policies. ([\#5815](#5815), [\#6436](#6436)) - Add etag and count fields to key backup endpoints to help clients guess if there are new keys. ([\#5858](#5858)) - Add `/admin/v2/users` endpoint with pagination. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](#5925)) - Require User-Interactive Authentication for `/account/3pid/add`, meaning the user's password will be required to add a third-party ID to their account. ([\#6119](#6119)) - Implement the `/_matrix/federation/unstable/net.atleastfornow/state/<context>` API as drafted in MSC2314. ([\#6176](#6176)) - Configure privacy-preserving settings by default for the room directory. ([\#6354](#6354)) - Add ephemeral messages support by partially implementing [MSC2228](matrix-org/matrix-spec-proposals#2228). ([\#6409](#6409)) - Add support for [MSC 2367](matrix-org/matrix-spec-proposals#2367), which allows specifying a reason on all membership events. ([\#6434](#6434)) Bugfixes -------- - Transfer non-standard power levels on room upgrade. ([\#6237](#6237)) - Fix error from the Pillow library when uploading RGBA images. ([\#6241](#6241)) - Correctly apply the event filter to the `state`, `events_before` and `events_after` fields in the response to `/context` requests. ([\#6329](#6329)) - Fix caching devices for remote users when using workers, so that we don't attempt to refetch (and potentially fail) each time a user requests devices. ([\#6332](#6332)) - Prevent account data syncs getting lost across TCP replication. ([\#6333](#6333)) - Fix bug: TypeError in `register_user()` while using LDAP auth module. ([\#6406](#6406)) - Fix an intermittent exception when handling read-receipts. ([\#6408](#6408)) - Fix broken guest registration when there are existing blocks of numeric user IDs. ([\#6420](#6420)) - Fix startup error when http proxy is defined. ([\#6421](#6421)) - Fix error when using synapse_port_db on a vanilla synapse db. ([\#6449](#6449)) - Fix uploading multiple cross signing signatures for the same user. ([\#6451](#6451)) - Fix bug which lead to exceptions being thrown in a loop when a cross-signed device is deleted. ([\#6462](#6462)) - Fix `synapse_port_db` not exiting with a 0 code if something went wrong during the port process. ([\#6470](#6470)) - Improve sanity-checking when receiving events over federation. ([\#6472](#6472)) - Fix inaccurate per-block Prometheus metrics. ([\#6491](#6491)) - Fix small performance regression for sending invites. ([\#6493](#6493)) - Back out cross-signing code added in Synapse 1.5.0, which caused a performance regression. ([\#6494](#6494)) Improved Documentation ---------------------- - Update documentation and variables in user contributed systemd reference file. ([\#6369](#6369), [\#6490](#6490)) - Fix link in the user directory documentation. ([\#6388](#6388)) - Add build instructions to the docker readme. ([\#6390](#6390)) - Switch Ubuntu package install recommendation to use python3 packages in INSTALL.md. ([\#6443](#6443)) - Write some docs for the quarantine_media api. ([\#6458](#6458)) - Convert CONTRIBUTING.rst to markdown (among other small fixes). ([\#6461](#6461)) Deprecations and Removals ------------------------- - Remove admin/v1/users_paginate endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](#5925)) - Remove fallback for federation with old servers which lack the /federation/v1/state_ids API. ([\#6488](#6488)) Internal Changes ---------------- - Add benchmarks for structured logging and improve output performance. ([\#6266](#6266)) - Improve the performance of outputting structured logging. ([\#6322](#6322)) - Refactor some code in the event authentication path for clarity. ([\#6343](#6343), [\#6468](#6468), [\#6480](#6480)) - Clean up some unnecessary quotation marks around the codebase. ([\#6362](#6362)) - Complain on startup instead of 500'ing during runtime when `public_baseurl` isn't set when necessary. ([\#6379](#6379)) - Add a test scenario to make sure room history purges don't break `/messages` in the future. ([\#6392](#6392)) - Clarifications for the email configuration settings. ([\#6423](#6423)) - Add more tests to the blacklist when running in worker mode. ([\#6429](#6429)) - Refactor data store layer to support multiple databases in the future. ([\#6454](#6454), [\#6464](#6464), [\#6469](#6469), [\#6487](#6487)) - Port synapse.rest.client.v1 to async/await. ([\#6482](#6482)) - Port synapse.rest.client.v2_alpha to async/await. ([\#6483](#6483)) - Port SyncHandler to async/await. ([\#6484](#6484))

Synapse 1.7.0 (2019-12-13) ========================== This release changes the default settings so that only local authenticated users can query the server's room directory. See the [upgrade notes](UPGRADE.rst#upgrading-to-v170) for details. Support for SQLite versions before 3.11 is now deprecated. A future release will refuse to start if used with an SQLite version before 3.11. Administrators are reminded that SQLite should not be used for production instances. Instructions for migrating to Postgres are available [here](docs/postgres.md). A future release of synapse will, by default, disable federation for servers using SQLite. No significant changes since 1.7.0rc2. Synapse 1.7.0rc2 (2019-12-11) ============================= Bugfixes -------- - Fix incorrect error message for invalid requests when setting user's avatar URL. ([\#6497](#6497)) - Fix support for SQLite 3.7. ([\#6499](#6499)) - Fix regression where sending email push would not work when using a pusher worker. ([\#6507](#6507), [\#6509](#6509)) Synapse 1.7.0rc1 (2019-12-09) ============================= Features -------- - Implement per-room message retention policies. ([\#5815](#5815), [\#6436](#6436)) - Add etag and count fields to key backup endpoints to help clients guess if there are new keys. ([\#5858](#5858)) - Add `/admin/v2/users` endpoint with pagination. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](#5925)) - Require User-Interactive Authentication for `/account/3pid/add`, meaning the user's password will be required to add a third-party ID to their account. ([\#6119](#6119)) - Implement the `/_matrix/federation/unstable/net.atleastfornow/state/<context>` API as drafted in MSC2314. ([\#6176](#6176)) - Configure privacy-preserving settings by default for the room directory. ([\#6355](#6355)) - Add ephemeral messages support by partially implementing [MSC2228](matrix-org/matrix-spec-proposals#2228). ([\#6409](#6409)) - Add support for [MSC 2367](matrix-org/matrix-spec-proposals#2367), which allows specifying a reason on all membership events. ([\#6434](#6434)) Bugfixes -------- - Transfer non-standard power levels on room upgrade. ([\#6237](#6237)) - Fix error from the Pillow library when uploading RGBA images. ([\#6241](#6241)) - Correctly apply the event filter to the `state`, `events_before` and `events_after` fields in the response to `/context` requests. ([\#6329](#6329)) - Fix caching devices for remote users when using workers, so that we don't attempt to refetch (and potentially fail) each time a user requests devices. ([\#6332](#6332)) - Prevent account data syncs getting lost across TCP replication. ([\#6333](#6333)) - Fix bug: TypeError in `register_user()` while using LDAP auth module. ([\#6406](#6406)) - Fix an intermittent exception when handling read-receipts. ([\#6408](#6408)) - Fix broken guest registration when there are existing blocks of numeric user IDs. ([\#6420](#6420)) - Fix startup error when http proxy is defined. ([\#6421](#6421)) - Fix error when using synapse_port_db on a vanilla synapse db. ([\#6449](#6449)) - Fix uploading multiple cross signing signatures for the same user. ([\#6451](#6451)) - Fix bug which lead to exceptions being thrown in a loop when a cross-signed device is deleted. ([\#6462](#6462)) - Fix `synapse_port_db` not exiting with a 0 code if something went wrong during the port process. ([\#6470](#6470)) - Improve sanity-checking when receiving events over federation. ([\#6472](#6472)) - Fix inaccurate per-block Prometheus metrics. ([\#6491](#6491)) - Fix small performance regression for sending invites. ([\#6493](#6493)) - Back out cross-signing code added in Synapse 1.5.0, which caused a performance regression. ([\#6494](#6494)) Improved Documentation ---------------------- - Update documentation and variables in user contributed systemd reference file. ([\#6369](#6369), [\#6490](#6490)) - Fix link in the user directory documentation. ([\#6388](#6388)) - Add build instructions to the docker readme. ([\#6390](#6390)) - Switch Ubuntu package install recommendation to use python3 packages in INSTALL.md. ([\#6443](#6443)) - Write some docs for the quarantine_media api. ([\#6458](#6458)) - Convert CONTRIBUTING.rst to markdown (among other small fixes). ([\#6461](#6461)) Deprecations and Removals ------------------------- - Remove admin/v1/users_paginate endpoint. Contributed by Awesome Technologies Innovationslabor GmbH. ([\#5925](#5925)) - Remove fallback for federation with old servers which lack the /federation/v1/state_ids API. ([\#6488](#6488)) Internal Changes ---------------- - Add benchmarks for structured logging and improve output performance. ([\#6266](#6266)) - Improve the performance of outputting structured logging. ([\#6322](#6322)) - Refactor some code in the event authentication path for clarity. ([\#6343](#6343), [\#6468](#6468), [\#6480](#6480)) - Clean up some unnecessary quotation marks around the codebase. ([\#6362](#6362)) - Complain on startup instead of 500'ing during runtime when `public_baseurl` isn't set when necessary. ([\#6379](#6379)) - Add a test scenario to make sure room history purges don't break `/messages` in the future. ([\#6392](#6392)) - Clarifications for the email configuration settings. ([\#6423](#6423)) - Add more tests to the blacklist when running in worker mode. ([\#6429](#6429)) - Refactor data store layer to support multiple databases in the future. ([\#6454](#6454), [\#6464](#6464), [\#6469](#6469), [\#6487](#6487)) - Port synapse.rest.client.v1 to async/await. ([\#6482](#6482)) - Port synapse.rest.client.v2_alpha to async/await. ([\#6483](#6483)) - Port SyncHandler to async/await. ([\#6484](#6484))

SoftwareSchlosser · 2020-02-13T11:54:34Z

Hi,

I already posted this in the matrix-doc issues (sorry guys!). Maybe you can help me. Here's a copy of my post:

I'm currently trying to use this nice new feature because I have a python script that sends messages automatically which should be deleted after a few days.

The script uses matrix-nio and here's what my message sending looks like:
_content={
	"msgtype": "m.text",
	"body": "Test",
	"m.self_destruct_after": 30000
}

client.room_send(room_id=_dstRoom, message_type="m.room.message", content=_content)
Unfortunately it doesn't work or I didn't do it right.

Server version is 1.9.1

Also tried to add "m.self_destruct": "True" and putting quotation marks around the time value.

What am I doing wrong?

Oh, and what about media? I'm actually sending jpeg images and it would be great if those would also be deleted upon the message self destruction.

babolivier · 2020-02-13T12:01:10Z

Hey @SoftwareSchlosser, please note that the Synapse implementation of this feature is very experimental.

To turn it on, you need to add enable_ephemeral_messages: true to your server's configuration file.
On top of that, since the MSC hasn't been merged into the spec yet, you need to use org.matrix.self_destruct_after instead of m.self_destruct_after.

Feel free to join #synapse:matrix.org if you want to discuss it further, it'll better conditions than on the comments of a merged PR :)

SoftwareSchlosser · 2020-02-13T16:30:15Z

Oh yes, I forgot to mention that I already enabled enable_ephemeral_messages in my config but anyway, using org.matrix.self_destruct_after now works as expected so thank you!
Now I have to get rid of the corresponding media files on my home server but that's another story.

babolivier · 2020-02-13T16:50:19Z

Support for media with this feature isn't implemented currently, as it's quite a tricky thing to do (especially in e2e encrypted rooms). Also that's not part of MSC2228.
Anyway, glad it works :)

SoftwareSchlosser · 2020-02-13T17:27:42Z

Well, as the messages I want to be deleted are sent by a bot-user I could write a script that deletes all media sent only by that user. I wonder if it's enough to delete the corresponding entries in local_media_repository, local_media_repository_thumbnails and local_media_repository_url_cache or if there are any more tables to consider.

Hexalyse · 2020-04-23T14:16:37Z

Quick question : is there any Matrix client supporting this feature ? I've tried to search if Riot included it, but nope... there are even Issues on the Riot github repo discussing such a feature, and people don't seem aware that it has now been implemented in Synapse. (example : https://github.com/vector-im/riot-web/issues/12165 )

babolivier · 2020-04-27T10:44:18Z

As far as I know, no, no client are currently supporting this feature. Note that this is only a partial implementation of the MSC, and that this implementation is very experimental and not recommended for production, so further work on this is needed before we can safely tell people to use it.

Hexalyse · 2020-04-28T05:25:40Z

Ok, thanks for the clarification. Any estimated ETA for the complete implementation?

babolivier · 2020-04-28T08:46:30Z

Not yet I'm afraid.

Apex69 · 2020-06-20T00:25:52Z

Not yet I'm afraid.

Any news?

babolivier added 5 commits November 25, 2019 11:43

Add database support for ephemeral messages

c982380

Add helper functions to redact messages on expiry

6ad5e92

The actual redaction isn't there yet though

Plug redaction scheduling in the right places

2628e19

Changelog

8255b9e

Lint

8563e2b

babolivier changed the title ~~Add ephemeral messages support (MSC2228)~~ [WIP] Add ephemeral messages support (MSC2228) Nov 25, 2019

babolivier added 10 commits November 26, 2019 15:21

Actually delete events on expiry

f2c6018

Rename functions + doc

01d1876

Lint

274612a

Move the field name to a constant

d424549

Hide the feature behind a configuration flag

052294e

Don't expire state events

dee234f

Merge branch 'develop' into babolivier/ephemeral-messages

344d287

Test case and various fixes

36230a5

Lint

89765f4

Add background update

ceab0a8

This is a slightly modified copy of the event_store_labels background update, adapted to extract and store expiry timestamps for events.

babolivier changed the title ~~[WIP] Add ephemeral messages support (MSC2228)~~ Add ephemeral messages support (MSC2228) Nov 27, 2019

babolivier requested a review from a team November 27, 2019 12:29

Lint

26dec61

richvdh previously requested changes Nov 27, 2019

View reviewed changes

babolivier and others added 3 commits November 27, 2019 16:43

Update synapse/api/constants.py

2ac7869

Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Incorporate part of the review

a4307c6

Change the scheduling flow

a6461c0

babolivier self-assigned this Nov 28, 2019

babolivier added 4 commits November 28, 2019 11:23

Use the same code path for censoring redactions and expired events

c33d2b4

Merge branch 'develop' into babolivier/ephemeral-messages

b766861

Lint and docstring

9a531be

Lint

964c6da

babolivier requested a review from a team November 28, 2019 11:31

babolivier added 2 commits December 3, 2019 14:14

Typos

d5640c3

Incorporate review

1573a5e

richvdh approved these changes Dec 3, 2019

View reviewed changes

babolivier added 5 commits December 3, 2019 16:27

Merge branch 'develop' into babolivier/ephemeral-messages

44741a0

Add flakey tests to the black list

8d2ee04

Moar flakey tests

c1102d6

Revert "Moar flakey tests"

3e007c0

This reverts commit c1102d6.

Revert "Add flakey tests to the black list"

7648299

This reverts commit 8d2ee04.

babolivier merged commit 54dd5dc into develop Dec 3, 2019

babolivier deleted the babolivier/ephemeral-messages branch January 9, 2020 15:46

richvdh mentioned this pull request Feb 13, 2020

Implement MSC2228 Self destructing messages #6287

Open

lrq3000 mentioned this pull request Feb 15, 2020

RTC/Riot: warn about media and centralization on matrix.org? privacytools/privacytools.io#1395

Closed

Hexalyse mentioned this pull request Apr 22, 2020

Message lifetimes element-hq/element-meta#364

Closed

matrix-org locked as resolved and limited conversation to collaborators Jun 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ephemeral messages support (MSC2228) #6409

Add ephemeral messages support (MSC2228) #6409

babolivier commented Nov 25, 2019 •

edited

Loading

richvdh left a comment

richvdh left a comment

richvdh Dec 3, 2019

richvdh Dec 3, 2019

SoftwareSchlosser commented Feb 13, 2020

babolivier commented Feb 13, 2020

SoftwareSchlosser commented Feb 13, 2020 •

edited

Loading

babolivier commented Feb 13, 2020 •

edited

Loading

SoftwareSchlosser commented Feb 13, 2020

Hexalyse commented Apr 23, 2020

babolivier commented Apr 27, 2020

Hexalyse commented Apr 28, 2020 via email •

edited

Loading

babolivier commented Apr 28, 2020

Apex69 commented Jun 20, 2020

Add ephemeral messages support (MSC2228) #6409

Add ephemeral messages support (MSC2228) #6409

Conversation

babolivier commented Nov 25, 2019 • edited Loading

richvdh left a comment

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

richvdh Dec 3, 2019

Choose a reason for hiding this comment

richvdh Dec 3, 2019

Choose a reason for hiding this comment

SoftwareSchlosser commented Feb 13, 2020

babolivier commented Feb 13, 2020

SoftwareSchlosser commented Feb 13, 2020 • edited Loading

babolivier commented Feb 13, 2020 • edited Loading

SoftwareSchlosser commented Feb 13, 2020

Hexalyse commented Apr 23, 2020

babolivier commented Apr 27, 2020

Hexalyse commented Apr 28, 2020 via email • edited Loading

babolivier commented Apr 28, 2020

Apex69 commented Jun 20, 2020

babolivier commented Nov 25, 2019 •

edited

Loading

SoftwareSchlosser commented Feb 13, 2020 •

edited

Loading

babolivier commented Feb 13, 2020 •

edited

Loading

Hexalyse commented Apr 28, 2020 via email •

edited

Loading