Stuck notifications

[Johennes]

We are experiencing multiple issues in the area of "stuck" notifications and unread markers. Many are related to the way receipts interact with threaded messages.

Symptoms

We are seeing different symptoms of the problem:

• "Stuck" unread dots and notification counters: Even when I have read all messages in a room, the room is still marked as unread or having notifications.
• Unread dots or notification counters returning on app startup: When I restart the app (or refresh the page on Web) rooms which I have already read re-appear as unread.
• Notification counters growing and shrinking spontaneously after entering or scrolling in a room.

Spec-level causes

Message ordering

Fundamentally, in order to interpret the meaning of a receipt that says "I have read everything up to here", we need to know what order messages are in. This is not clear in the spec, and we propose to make it clear and explicit in MSC4033.

In the meantime, Element Web uses a combination of "sync" order (the implicit order of events arriving via a /sync request) and "timestamp" order (using the ts property within events).

Some of the existing bugs are probably caused by this inconsistency, but it is not clear yet how many: we believe there are also bugs in the implementation that cause additional problems, and this theoretical inconsistency is only the cause of a few problems.

Which thread the root belongs to

The spec has what we consider a bug when it talks about which thread the root message belongs to, which has been reflected in client code, making it inconsistent with the server implementation (at least on the Synapse server). We have a proposal to fix this bug in MSC4037.

Identifying which thread any message is in

It is sometimes difficult for clients to identify which thread an event belongs to, meaning that a receipt pointing to it is sometimes ignored. We have begun drafting MSC4023 to address this.

Other

Previously, we believed that MSC3981 (recursive relations) would solve some of the problems, but since that MSC does not solve the event-ordering problem (because the events from the /relations API are returned in "topological" order) we no longer believe it is important, except as a performance optimisation.
Code-level causes

Code-level causes

We have found and fixed several bugs in the Element Web code that were caused by an incomplete understanding of the meaning of threaded and unthreaded read receipts. We anticipate that some more exist.

(We believe that the primary reason why we're not seeing the same problems on mobile is that the apps persist events they've received whereas Element Web has to re-fetch from scratch after every launch. As a result, any issue in the unread state logic, strikes again and again. The apps also use a single timeline whereas Element Web maintains one timeline per thread in addition to the main timeline in every room.)

High-level plan of actions

Tasks

Preview Give feedback

1. Set up POC branch / deployment with threads disabled – Experiment with putting threads back behind a beta toggle #25676
2. Fix read-receipt behavior around thread roots – Fix read receipt sending behaviour around thread roots matrix-org/matrix-js-sdk#3600
3. Fix read-receipt behavior around non-thread relations to thread roots – Fix edge cases around non-thread relations to thread roots and read receipts matrix-org/matrix-js-sdk#3607
4. Fix missing message issues due to replies to unknown events – Fix how Room::eventShouldLiveIn handles replies to unknown parents matrix-org/matrix-js-sdk#3615
5. Fix unread count returning from zero after reload – Message in a thread is read and becomes unread when I refresh #25806
6. Fix notification counters sometimes being doubled – Notification count is doubled in e2ee rooms on startup #25803
7. Set up unread & notification test suite – Test cases for stuck unreads and notifs #25449
8. Unthreaded receipt is not returned in the next syncs, confirm bug and get fixed - Unthreaded receipt is not returned in the next syncs synapse#17247
9. Pre-investigate issues listed after the threads model refactoring to see if any of them are fixable before
10. Refactor threads model code and remove serverless / legacy code path to simplify further changes
11. Fix missing notifications – Thread replies sent from Element Android cause neither unread count nor unread dot on Element Desktop #25621
12. Fix incorrect thread reply count – Reply count in thread is wrong #24636
13. Fix notification counters not adding up between spaces and rooms – Notification badges in left panel contradict each other #20372
14. Fix unread counter explosion – Unread count goes from zero through the roof after restarting the app #25479
15. Fix Zombie notifications from old threads – ? issue ?
16. Fix read receipts / fully read marker jumping backwards – ? issue ?
17. Triage existing issues to increase confidence that the root causes will actually be fixed through the MSCs mentioned above.
18. Fix remaining bugs found in no-threads POC as they likely impact the threaded experience as well – Experiment with putting threads back behind a beta toggle #25676

We believe a lot of progress can still be made without spec changes. So we're slightly deprioritising work on the MSCs.

• MSC4037: Thread root is not in the thread matrix-org/matrix-spec-proposals#4037
  We claim it is implemented in EW
  Synapse will need small fixes when this is merged
• MSC4033: Explicit ordering of events for receipts matrix-org/matrix-spec-proposals#4033
  Needs implementation on server and client but we believe it should mostly only cover edge cases
• MSC4023: Thread ID for 2nd order-relation matrix-org/matrix-spec-proposals#4023
  There is disagreement about the way to specify this and possible interference with MSC3051: A scalable relation format matrix-org/matrix-spec-proposals#3051
• MSC3389: Relation redactions matrix-org/matrix-spec-proposals#3389
  There is likely no good workaround other than implementing this MSC

New issue inbox

The following is a holding area for newly reported issues that require review. Once reviewed, issues should either be moved to one of the other task lists below or, if not applicable, removed from this epic.

Tasks

Preview Give feedback

1. Missing notifications badges vector-im/element-web#25331
   A-Notif-Panel A-Notifications O-Occasional S-Major T-Defect X-Regression +6
2. Stuck notifications due to unread threads not updating rapidly. vector-im/element-web#25420
   A-Notifications A-Threads O-Frequent S-Major T-Defect +5
3. Room stuck as unread despite all messages/threads marked as read (to my knowledge) vector-im/element-web#25481
   A-Notifications T-Defect X-Needs-Investigation +3
4. Stuck notification after redacting thread root vector-im/element-web#25480
   A-Notifications A-Threads O-Occasional S-Major T-Defect +5
5. Unread count goes from zero through the roof after restarting the app vector-im/element-web#25479
   A-Notifications O-Occasional S-Major T-Defect +4
6. https://github.com/matrix-org/element-web-rageshakes/issues/21717
7. https://github.com/matrix-org/element-web-rageshakes/issues/21724
8. Stuck notification case vector-im/element-web#25513
   A-Notifications O-Occasional S-Critical T-Defect +4
9. https://github.com/matrix-org/element-web-rageshakes/issues/21737
10. Stuck notifications due to edits in threads vector-im/element-web#25482
    A-Notifications A-Threads O-Occasional S-Major T-Defect +5
11. Constant unread highlight notifications in one particular room vector-im/element-web#25528
    A-Notifications O-Occasional S-Critical T-Defect +4
12. Starting nightly for the first time seems to trigger notification counter mess in normal Desktop vector-im/element-web#25541
    A-Notifications O-Occasional S-Major T-Defect +4
13. Thread replies sent from Element Android cause neither unread count nor unread dot on Element Desktop vector-im/element-web#25621
    A-Notifications A-Threads O-Occasional S-Major T-Defect +5
14. Room with thread message not shown at top of list when „Show rooms with unread messages first“ is checked vector-im/element-web#24547
    A-Threads O-Frequent S-Minor T-Defect +4
15. Stuck unread when latest message is in a thread vector-im/element-web#25623
    A-Notifications A-Threads O-Occasional S-Minor T-Defect +5
    
16. Errors in console likely linked to stuck unreads vector-im/element-web#25642
    A-Notifications O-Occasional S-Minor T-Defect +4
    
17. Persistent Threads dot when last message in a thread is redacted vector-im/element-web#25670
    A-Notifications A-Redaction A-Threads O-Occasional S-Minor T-Defect +6
18. Experiment with putting threads back behind a beta toggle vector-im/element-web#25676
    1 of 2
    A-Notifications A-Threads T-Task +3
19. Permanent stuck unread-notification when starting a thread with a reply message vector-im/element-web#23976
    A-Notifications A-Threads O-Occasional S-Major T-Defect X-Cannot-Reproduce X-Needs-Info +7
20. Read message is marked as unread vector-im/element-web#25804
    A-Read-Marker O-Uncommon S-Major T-Defect X-Regression +5
21. Message in a thread is read and becomes unread when I refresh vector-im/element-web#25806
    A-Notifications O-Occasional S-Minor T-Defect +4
    
22. Read status of room resetting itself and not shared to other devices vector-im/element-web#24111
    A-Room-List O-Frequent S-Minor T-Defect +4
23. impossible to dismiss notification vector-im/element-web#25408
    A-Notifications O-Frequent S-Minor T-Defect +4
24. New threads/new thread messages always show "new messages" at already 20x times read other threads too vector-im/element-web#25907
    A-Notifications A-Threads O-Occasional S-Major T-Defect +5
25. Audible ping but no badge on notifications for bot messages vector-im/element-web#25904
    A-Notifications T-Enhancement Z-GetYourUpdates Z-Labs +4
26. Stuck unread dot on v1.11.38 vector-im/element-web#25929
    A-Notifications O-Occasional S-Major T-Defect +4
27. https://github.com/matrix-org/element-web-rageshakes/issues/22226
28. https://github.com/matrix-org/element-web-rageshakes/issues/22363
29. https://github.com/matrix-org/element-web-rageshakes/issues/22414
30. Stuck notifications in a room with no posting permissions vector-im/element-web#25975
    O-Frequent S-Minor T-Defect +3
31. Persistent stuck unread in encrypted room #25984
    A-Notifications O-Frequent S-Major T-Defect +4
32. New message count not clearing in DM vector-im/element-web#25950
    A-Notifications O-Occasional S-Minor T-Defect +4
33. Stuck notification in DM #26063
    A-Notifications O-Occasional S-Major T-Defect +4
34. https://github.com/matrix-org/element-web-rageshakes/issues/22526
35. Threads might skip processing some replies from the sync response matrix-org/matrix-js-sdk#3665
    +0
36. Notification for unread redacted message in thread gets stuck #26933
    A-Notifications A-Threads O-Uncommon S-Major T-Defect +5

Tasks not blocked by spec work

Tasks

Preview Give feedback

1. Stuck notifications: Create devtools room stuck notifications debug tool vector-im/element-web#24388
   A-DevTools T-Task +2
   
2. Stuck notifications: Reacting to a not-latest message in a thread causes a stuck notification vector-im/element-web#24000
   A-Threads O-Uncommon S-Minor T-Defect +4
   
3. Stuck notifications: DM stuck as unread vector-im/element-web#23991
   A-DMs A-Notifications O-Frequent S-Major T-Defect Z-MadLittleMods +6
   
4. Stuck notifications: New DMs have stuck notifications #23685
   A-Notifications O-Frequent S-Major T-Defect Z-MadLittleMods Z-Synapse +6
5. https://github.com/matrix-org/element-web-rageshakes/issues/21667
6. Read messages reappear as "new" after restarting the app. vector-im/element-web#24629
   A-Notifications O-Frequent S-Major T-Defect +4
   
7. „Mark as read“ button does not send the correct receipt for the last event in a thread vector-im/element-web#25207
   A-Read-Receipts A-Threads O-Frequent S-Minor T-Defect +5
   
8. Unthreaded read receipts are sent after reading the main timeline vector-im/element-web#25196
   A-Notifications A-Read-Receipts A-Threads O-Uncommon S-Minor T-Defect +6
   
9. Test how we handle receiving an unthreaded read receipt vector-im/element-web#25212
   A-Read-Receipts +1
   
10. Rooms incorrectly marked as unread vector-im/element-web#10954
    A-Read-Marker A-Room-List T-Defect +3
    
11. https://github.com/matrix-org/element-web-rageshakes/issues/21641
12. Cannot mark room as read. vector-im/element-web#25411
    T-Defect X-Release-Blocker +2
    
13. New reaction to old event ends up on the main timeline instead of the thread vector-im/element-web#25450
    O-Uncommon S-Major T-Defect X-Release-Blocker +4
    
14. Test cases for stuck unreads and notifs #25449
    A-Notifications T-Enhancement +2
    
15. Experiment with putting threads back behind a beta toggle vector-im/element-web#25676
    1 of 2
    A-Notifications A-Threads T-Task +3
16. Missing reactions for some messages vector-im/element-web#25596
    A-Reactions O-Uncommon S-Major T-Defect +4
    

Tasks that are related to or dependent on spec work

We've written the following MSCs to try and address the root causes in a reliable and performant way:

• MSC4033: Explicit ordering of events for receipts matrix-org/matrix-spec-proposals#4033
  A client-side workaround for this based on timestamp ordering has been implemented. This is imperfect and easily abused though.
• MSC4023: Thread ID for 2nd order-relation matrix-org/matrix-spec-proposals#4023
  A client-side workaround for this based on calling /event to fetch the parent has been implemented. This is functionally correct but has a noticeable performance impact.
• MSC4037: Thread root is not in the thread matrix-org/matrix-spec-proposals#4037
  The client-side code has already been modeled to reflect the behavior proposed in the MSC (which is also what Synapse already does today).
• MSC3981: /relations recursion matrix-org/matrix-spec-proposals#3981
  We expect this to not help with the ordering problems but it will be a performance improvement.

Tasks

Preview Give feedback

1. Write / shepherd MSC3981 vector-im/element-meta#1350
   A-Notifications T-Task +2
   
2. Implement MSC3981 in Synapse matrix-org/synapse#15377
   A-Threads O-Occasional S-Minor T-Enhancement +4
   
3. Implement MSC3981 in matrix-react-sdk vector-im/element-web#25021
   A-Notifications O-Frequent S-Major T-Task Team: App +5
   
4. Room stuck as unread after mention in a thread #24312
   A-Threads O-Occasional S-Minor T-Defect +4
5. Stuck unread with reaction + start thread vector-im/element-web#24394
   A-Notifications A-Threads O-Frequent S-Major T-Defect +5
6. Can't get rid of unread status of a room vector-im/element-web#24595
   A-Notifications O-Frequent S-Major T-Defect +4
7. Thread not being marked as read vector-im/element-web#24442
   A-Notifications A-Threads O-Occasional S-Major T-Defect +5
8. impossible to dismiss notification vector-im/element-web#25408
   A-Notifications O-Frequent S-Minor T-Defect +4
9. https://github.com/matrix-org/element-web-rageshakes/issues/21679
10. Stuck notifications due to edits in threads vector-im/element-web#25482
    A-Notifications A-Threads O-Occasional S-Major T-Defect +5
11. Write / shepherd MSC4023 element-meta#1714
    App: web +1
12. Implement MSC4023: Thread ID for 2nd order-relation in Synapse matrix-org/synapse#15701
    A-Relations-Endpoint A-Threads O-Occasional S-Minor T-Task roadmap +6
13. Excessive calls to GET /relations and GET /event, with MSC3981 enabled and working vector-im/element-web#25395
    O-Frequent S-Major T-Defect +3
    
14. Stuck notifications when there's no new message after a thread vector-im/element-web#25893
    A-Notifications O-Uncommon S-Major T-Defect +4

Issues that are related but out of scope

• Several places in the code use timestamp to determine event order matrix-org/matrix-js-sdk#3325

Time sheeting

WEB: Stuck notifications

[Johennes]

Descoping #24373 as this doesn't have the same level of severity as the other issues.

[Johennes]

As mentioned in the issue description, the remaining bugs will require a spec change. We're currently in the process of drafting the MSC but it'll, unfortunately, be more involved than just a simple client-side bug fix.

[jplatte]

I don't think that is true. If you look at the comments in #24595, you will find some from rooms where threads were never used and IME clearing cache often fixes these issues.

[Johennes]

We're focusing on the most prominent problems first which are in the interplay of threads and other relations. There may be further issues beyond that.

[AlBundy33]

I had the same issue in some rooms.
Because I don't participate that much in this rooms I tried to write a test-message and it seems that the stuck notifications are gone.

[Johennes]

@8227846265 the threads-related stuck notification issues are blocking on matrix-org/matrix-spec-proposals#3981.

[Johennes]

We've completed initial implementations for it last week and are now starting to prepare for testing it. Unfortunately, it'll still take some more time to get this landed due to the nature of the spec process.

[leonardehrenfried]

@Johennes Thanks for this explanation. If there is an environment where a regular user could help test it, I'd love to hear about it.

[Johennes]

Thanks @leonardehrenfried. We will most likely be enabling it on beta.matrix.org as to not impact regular users but you can sign-up for a separate account there if you want to help testing.

[alexander-potemkin]

@Johennes , thanks for the explanation indeed! How soon it could be expected to land to the self hosted solutions?

[Johennes]

I'd expect the initial PRs (matrix-org/synapse#15315 & matrix-org/matrix-js-sdk#3248) to land this week.

[leonardehrenfried]

For everybody subscribed to this ticket: the backend and frontend PRs have been merged now so I expect a rollout quite soon.

(I'm not an Element/Matrix employee, just an interested user.)

[Johennes]

@8227846265 we have one more fix in the making that doesn't depend on the MSC (#25196) and should land soon. If that one doesn't help your case, we'd be interested to hear more details about your particular error scenario.

[mutantcornholio]

Also, opening chats for the first time after restarting element, when the messages are loaded, marks some of the loaded ancient messages from threads as unread.

One more thing that I've noticed, is a lot of times the ancient messages are stuff like Alice invited Bob, or Alice changed the power level of Bob from Default to Admin, or Alice made the room invite only. Happens far more often in comparison to active chats, where people actually send messages.

[ajkessel]

Has progress stalled? I've had the same stuck unread notifications for several months now, both on every production release of Element as well as running bleeding edge from develop.element.io. It does seem to be the same rooms/old messages that consistently show as unread. Is there anything I can do as an end-user to troubleshoot the circumstances, particularly where the issue is reproducible at least for some fraction of the time?

[mutantcornholio]

Any update?

[JMoVS]

Is there still work on threads becoming unread again and again all the time? Happens quite frequently to me nowadays

[MTRNord]

Is there still work on threads becoming unread again and again all the time? Happens quite frequently to me nowadays

I also have this. In foundation room I get a loud ping for a mention in a thread each time a new message is added to the room. Its annoying me.

[dreirund]

Is there still work on threads becoming unread again and again all the time? Happens quite frequently to me nowadays

I have this, too.

[HelderFSFerreira]

I have more than 30 rooms stuck notifications.

Mark everything a read does not work. Let me know if I should provide any logs.

[JMoVS]

are there more logs required or anything? This issue has been around for more than a full year now and recently progress seems to have stalled

[langleyd]

Thanks all for the continued patience and feedback on this issue and apologies for it becoming stale.

A lot of progress was made in 2023 and Q12024 on spec, bug fixes, tests and also improvements to the UX on notifications including:

• Removing threads activity(white dots caused by threaded messages) from the main room list
• Adding the Thread Activity Center(thread button bottom left) specifically for thread notifications.

You can check out this blog post for some more details on the improvements.

We hope these efforts have brought a somewhat calmer and improved experience for notifications, especially on the main room list.

We are aware there are still problems, particularly in the threads experience. We suspect some of the bugs in threads related to activity indicators(white dots) are caused by a bug in synapse and are pursuing this as a next port of call. (@JMoVS We suspect this is the source of the bug you and others have highlighted above).

Aside from ongoing efforts to fix particular known bugs, we know where we are currently with notifications is not the destination. For example, while we believe the new threads activity centre creates a calmer experience there is a risk of missing out on threaded messages. One thing we would like to do in future is introduce Thread Participation to opt in to notifications for specific threads. This would allow us to improve the UX and value delivered by the Threads Activity Center. However before we tackle another major iteration on Notifications we need to first make some foundational improvements to how both room list/spaces and the timeline work. We’ll update again when we have more to share.

[mutantcornholio]

Love the progress, but this still doesn't say how are we doing on actually fixing bugs related to notifications. Why are there still notifications reappearing after restart, and what's the timeline on fixing those?

[Valodim]

Concrete question on this: Will there be a change in behavior of notifications with sliding sync? Given that clients see much less data overall, I suspect they should be less prone to false notifications from local misinterpretations of room state and loaded backlog. Is that the case, or will this be largely the same?

Overall, the current state of affairs is not ideal, but bearable. If the sliding sync overhaul affects this in a fundamental way, it might be wiser to focus efforts on that.

[wielinde]

Another source of frustration is UX related: I have about 100 rooms in various spaces. The list does not fit on a single screen. The room list is not sorted by last activity which makes me miss notifications below the first fold. If I miss notifications the tool loses the trust. If Element had a room list ordered by activity then this would constitute a real inbox and resolve a lot of anxiety of missing messages.

BTW I also believe that Threads should ideally have titles so that it becomes easier to jump back. Also threads could then be treated similar to rooms within such inbox. However, one should be able to mark a thread as closed/resolved. Zulip does that part very well.

[vranki]

Another source of frustration is UX related: I have about 100 rooms in various spaces. The list does not fit on a single screen. The room list is not sorted by last activity which makes me miss notifications below the first fold. If I miss notifications the tool loses the trust. If Element had a room list ordered by activity then this would constitute a real inbox and resolve a lot of anxiety of missing messages.

This is not related to "Stuck notifications" issue so better report and discuss it elsewhere. But anyway right-click on "Rooms" header and you can choose to show rooms with activity first.

[mehlkopf]

That's my Element on iOS. I do not know what is wrong there, as the situation is much better on my Mac client, but neither mark all as read nor deleting the cache helps clearing these unread notifications. So for me personally, this is still a dumpster fire.

[nebocamin]

i dont know where it fits but want to share an observation.
i have a lot stuck unreads, but i dont see them with desktop element under macos, but on linux and it doesnt matter if i use stable or nightly element (2nd load after deleting cache).

[leonardehrenfried]

An observation: For me clearing all cookies and local storage has managed to get rid of a very persistent notification. Not that it should be necessary though...

[nebocamin]

An observation: For me clearing all cookies and local storage has managed to get rid of a very persistent notification. Not that it should be necessary though...

i just started a completely new session on a new device using linux and after the second start, all my 8 friends (well known stuck notifications) are here again. in the meantime on macos using element-desktop still clean. i have no clue.

[andybalaam]

I just cleaned my cache from the Settings menu, and a new stuck unread appeared!

Note that "user read up to" matches "Last event", which is surprising.

Rageshake is here: https://github.com/element-hq/element-web-rageshakes/issues/28074