Sometimes we have entries in event_to_state_groups for outliers, and I don't know why

[richvdh]

Generally we don't expect to have the state at an outlier event, so having entries in event_to_state_groups is surprising to me. I haven't investigated why this happens, or if it causes any real problems.

[MadLittleMods]

This seems very plausible (by the name of the function): We have various functions which run into _auth_and_persist_outliers and makes it way down to _store_event_state_mappings_txn which upserts into event_to_state_groups.

Call stack:

_auth_and_persist_outliers
_auth_and_persist_outliers_inner
persist_events_and_notify
persist_events
enqueue
_event_persist_queue -> _persist_event_batch
_persist_events_and_state_updates
_persist_events_txn
_store_event_state_mappings_txn (upserts `event_to_state_groups`)

[richvdh]

but _store_event_state_mappings_txn shouldn't do anything for outliers. It contains the following:

            if event.internal_metadata.is_outlier():
                # ...
                continue

[MadLittleMods]

@richvdh Good call 🤦‍♂️

What is the date of the most recent entry in event_to_state_groups that is an outlier? (share your database queries if you look it up) I'm wondering if it's still an ongoing problem of accidentally adding outliers or possibly a previous bug that is now fixed.

My other guess is that we're updating an event to outlier: True again after it was already persisted as a normal event.

[richvdh]

I've not been able to find any examples on matrix.org, but I don't know if that means they don't exist, or just that a couple of hours of table-scanning failed find them.

I did find a number of examples on sw1v.org, with this query:

select to_timestamp(received_ts/1000), * from events e join event_to_state_groups esg using (event_id) where e.outlier order by stream_ordering desc limit 10;

The most recent examples there are from 2021-12-02, so it may have been fixed since.

Some other things I noticed about these entries:

• they all have positive stream_orderings
• they also have entries in ex_outlier_stream, with negative stream orderings.

That means they were originally persisted with positive stream-orderings, and we later attempted to de-outlier them with a negative stream-ordering. There's nothing particularly wrong with that; what is odd is that they are still marked as outliers after going through the de-outliering process, given that we always set outlier=False whenever we add an entry to ex_outlier_stream: https://github.com/matrix-org/synapse/blob/release-v1.53/synapse/storage/databases/main/events.py#L1320-L1332.

It feels like #11632 (which caused backfilled events to be marked as outliers incorrectly) may be involved here but I don't really know how that would cause an event to be in ex_outlier_stream and yet still be marked as an outlier.

[richvdh]

Maybe this was caused by #12584 (before the assertion was added)? I still can't really see how.