Messages from certain remote users missing / being rejected #13716
Comments
|
The key log line: Can you see any logs that mention Additionally, what do the following database queries return? SELECT COUNT(*) FROM events WHERE event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU';
SELECT COUNT(*) FROM event_json WHERE event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU'; |
DMRobertson
added
T-Defect
No, only that one log line above. I believe my log history doesn't go that far back since the containers have been recreated to update synapse.
|
|
It's possible that the user's membership, Can you run the following query to confirm? SELECT event_id, type, outlier, rejection_reason FROM events WHERE event_id IN (SELECT '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU' UNION SELECT auth_id FROM event_auth WHERE event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU'); |
There seems to be no rejection reason |
|
Odd. I wonder if you've been hit by a state reset (#8629) due to the weird corner (#12943). I'd like to do a little more investigation---let me concoct some queries for you to run. In the worst case scenario, I'd suggest purging the room with the admin API and rejoining it. |
|
@tonkku107, could you run this query and paste its output please? SELECT current_state_delta_stream.* FROM events JOIN current_state_delta_stream ON events.stream_ordering = current_state_delta_stream.stream_id WHERE events.event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU';
SELECT * FROM current_state_events WHERE event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU'; |
I'm guessing that last one isn't supposed to return nothing |
Yeah. To help understand why, can you run this query? WITH groups AS (
SELECT state_group_edges.*
FROM state_group_edges
JOIN event_to_state_groups USING(state_group)
JOIN events USING(event_id)
WHERE events.event_id = '$jWCsQ39xw6rhTIPuRjHqh6KspCEmWzbycTKjqBfcEtU'
)
SELECT state_groups_state.*
FROM state_groups_state
JOIN groups ON state_groups_state.state_group IN (groups.state_group, groups.prev_state_group)
; |
|
|
Okay, looks like the event was accepted when it was received. Can you run this query to see if there any changes since then? (May take a while) SELECT * FROM current_state_delta_stream WHERE room_id='!GtIfdsfQtQIgbQSxwJ:archlinux.org' AND type=' m.room.member' AND '@rajabinkz:matrix.org' ORDER BY stream_id ASC; |
I had to do some edits to the query, I believe this is what you wanted |
|
Ahh yes, sorry. I think I screwed up my copying and pasting. (Thanks, the query is just as I should have typed it!) What happened at stream_id WITH groups AS (
SELECT state_group_edges.*
FROM state_group_edges
WHERE state_group = 653360
)
SELECT state_groups_state.*
FROM state_groups_state
JOIN groups ON state_groups_state.state_group IN (groups.state_group, groups.prev_state_group)
; |
nothing 🤔 |
|
Sorry, I think I missed a JOIN and got the first WHERE clause wrong. What about: WITH groups AS (
SELECT state_group_edges.*
FROM events
JOIN event_to_state_groups USING(event_id)
JOIN state_group_edges USING (state_group)
WHERE stream_ordering=653360
)
SELECT state_groups_state.*
FROM state_groups_state
JOIN groups ON state_groups_state.state_group IN (groups.state_group, groups.prev_state_group)
; |
|
Still nothing... |
|
One last try then. WITH RECURSIVE state(state_group) AS (
(SELECT state_group FROM events NATURAL JOIN event_to_state_groups WHERE stream_ordering = 653360)
UNION ALL
SELECT prev_state_group FROM state_group_edges e, state s
WHERE s.state_group = e.state_group
)
SELECT state_group, type, state_key, event_id
FROM state_groups_state
WHERE state_group IN (
SELECT state_group FROM state
)
ORDER BY state_group DESC, type, state_key
; |
|
This one returned 16536 rows. Gonna upload the results as a file |
|
Thanks for bearing with us @tonkku107 Hmm, that Like @DMRobertson says, I think this will be due to either the "weird corner" and/or a state reset. The former has been fixed, and we're working on the latter. Having all users on your server leave the room and rejoin should "fix" this for you. I'm going to close this issue for now in favour of the issues @DMRobertson linked. |
|
(Obviously shout if leaving/rejoining doesn't work!) |
|
I am getting an error |
|
@erikjohnston things just got worse. I can't even leave or delete the room anymore... Leaving errors with |
When you say "it seems to be having trouble getting missing prev_events", what is "it"? Can you paste the curl command line you used to invoke the admin API? Make sure to omit or redact access tokens. |
"it" is referring to synapse
I used the synapse admin ui to delete the room Edit: This was due to not restarting synapse before rejoining. |
|
This has just happened again and in the same room as well. |
|
If you have logs illustrating the problem (e.g. describing an event id from matrix.org that's referred to, but isn't visible to you) we can take a quick look. |
|
I should've saved a copy of the logs for this issue... Went and lost them this morning with a restart. I also tried checking those SQL queries again yesterday and this time the response to the last query was empty (using the |
Description
I've noticed people replying to events that don't exist for me and looking at the event source they are replying to specific users from the matrix.org homeserver. I am not receiving messages from those members but I do see them typing and their read receipts.
In the case of the log output below, they have been in the room for close to a month and the referenced
m.room.memberevent can be found via element's dev tools, yet at the time of receiving the event synapse fails to calculate that as an auth_event.Steps to reproduce
Homeserver
tonkku.me
Synapse Version
1.66.0
Installation Method
Docker (matrixdotorg/synapse)
Platform
Running in docker containers on a server with Ubuntu 20.04.3, AMD Ryzen 5 3600, 64GB RAM.
Relevant log output
Anything else that would be useful to know?
No response
The text was updated successfully, but these errors were encountered: