Timeline needs to refresh when we see a MSC2716 marker event

[MadLittleMods]

Inform the client that historical messages were imported in the timeline and they should refresh the timeline in order to see the new events.

Companion matrix-react-sdk PR: matrix-org/matrix-react-sdk#8354

The marker events are being used as state now because this way they can't be lost in a timeline gap. Regardless of when they were sent, we will still have the latest version of the state to compare against. Any time we see our latest state value change for marker events, prompt the user that the timeline needs to refresh.

In a sync meeting with @ara4n, we came up with the idea to make the marker events as state events. When the client sees that the m.room.marker state changed to a different event ID, it can throw away all of the timeline and re-fetch as needed.

For homeservers where the same problem can happen, we probably don't want to throw away the whole timeline but it can go up the unsigned.replaces_state chain of the m.room.marker state events to get them all.

In terms of state performance, there could be thousands of marker events in a room but it's no different than room members joining and leaving over and over like an IRC room.

-- matrix-org/matrix-spec-proposals#2716 (comment)

Why are we just setting timlineNeedsRefresh (and prompting the user) instead of automatically refreshing the timeline for the user?

If we refreshed the timeline automatically, someone could cause your Element client to constantly refresh the timeline by just sending marker events over and over. Granted, you probably want to leave a room like this 🤷. Perhaps also some sort of DOS vector since everyone will be refreshing and hitting the server at the exact same time.

In order to avoid the timeline maybe going blank during the refresh, we could re-fetch the new events first, then replace the timeline. But the points above still stand on why we shouldn't.

Todo

• Add tests
• Add tests for refreshLiveTimeline
  • Added as e2e tests in Ask to refresh timeline when historical messages are imported (MSC2716) matrix-react-sdk#8354
• Add method to refresh the timeline to use in Ask to refresh timeline when historical messages are imported (MSC2716) matrix-react-sdk#8303 (comment)

Dev notes

yarn jest spec/integ/matrix-client-syncing.spec.js --silent=false --verbose=false

---

        Error.stackTraceLimit = Infinity;
        console.log(`setStateEvents timelineWasEmpty=${markerFoundOptions && markerFoundOptions.timelineWasEmpty} stateEvents:\n`, stateEvents.map((ev) => {
            return `\t${ev.getType()} ${ev.getId()}`;
        }).join('\n'), new Error().stack);

---

Functions with overloads:

eventTimelineset.addLiveEvent
eventTimelineSet.addEventToTimeline
eventTimeline.addEvent
room.addLiveEvent
room.addLiveEvents

---

room.resetLiveTimeline(null, null);

addEventsToTimeline
addEventToTimeline
addLiveEvent

resetLiveTimeline

resetAllTimelines

removeFilteredTimelineSet


syncFromCache
processSyncResponse
processRoomEvents
room.addLiveEvents


sessionStore


processSyncResponse
processRoomEvents
initialiseState
setStateEvents

load
getEventTimeline
initialiseState
setStateEvents

hasSyncedBefore
getSyncState()

function serializeDebugInfoFromTimelineSets(timelineSets): { [key: string]: string[] }[] {
    const serializedEventIdsInTimelineSet = timelineSets.map((timelineSet) => {
        const timelineMap = {};

        const timelines = timelineSet.getTimelines();
        const liveTimeline = timelineSet.getLiveTimeline();

        timelines.forEach((timeline, index) => {
            const forwardPaginationToken = timeline.getPaginationToken(EventTimeline.FORWARDS);
            const backwardPaginationToken = timeline.getPaginationToken(EventTimeline.BACKWARDS);

            // Add a special label when it is the live timeline so we can tell
            // it apart from the others
            const isLiveTimeline = timeline === liveTimeline;
            const paginationString = `--f=${forwardPaginationToken}--b=${backwardPaginationToken}`;
            timelineMap[isLiveTimeline ? `${index}-${timeline.toString()}--liveTimeline${paginationString}` : `${index}-${timeline.toString()}${paginationString}`] = timeline.getEvents().map(ev => ev.getId());
        });

        return timelineMap;
    });

    return serializedEventIdsInTimelineSet;
}

console.log('refresh timeline serializeDebugInfoFromTimelineSets', serializeDebugInfoFromTimelineSets(this.timelineSets));

---

Here's what your changelog entry will look like:

✨ Features

• Timeline needs to refresh when we see a MSC2716 marker event (#2299). Contributed by @MadLittleMods.

[MadLittleMods]

Ideas welcome ^

In my initial thoughts, I wasn't too keen on adding an extra field to the marker event to say where the history was inserted next to.

I'm not too keen to add an extra field to the marker or insertion event indicating where the history is imported next to as there isn't a way to enforce that it's true (a homeserver could lie and make it up). I'd rather keep the source of truth in the prev_events.

-- #2282 (comment)

But I don't think I'm leaning as far away now since a homeserver could also make up the m.insertion_id field that we have now. We could add a m.insertion_prev_events field.

{
    "type": "m.room.marker",
    "sender": "@appservice:example.org",
    "content": {
        "m.insertion_id": insertion_event.event_id,
        "m.insertion_prev_events": insertion_event.prev_events
    },
    "room_id": "!jEsUZKDJdhlrceRyVU:example.org",
    "origin_server_ts": 1626914158639,
}

We can also iterate this in another PR.

[MadLittleMods]

We can also iterate this in another PR.

Opting for this option ⏩. This is a nice to have.

[t3chguy]

This is getting very close now

[t3chguy]

thanks for perservering with this!

[MadLittleMods]

For reference, I updated this because room can be null which this didn't catch before (only caught undefined)

[MadLittleMods]

Thanks for the all of the review @t3chguy! 🐝 Lots of strain to get it up to snuff but came out with a lot more polish and more confidence in this new timeline concept working!