Prevent locking up while processing batched_auth_events

[ggogel]

This PR aims to fix #16895, caused by a regression in #7 and not fixed by #16903. The PR #16903 only fixes a starvation issue, where the CPU isn't released. There is a second issue, where the execution is blocked. This theory is supported by the flame graphs provided in #16895 and the fact that I see the CPU usage reducing and far below the limit.

Since the changes in #7, the method check_state_independent_auth_rules is called with the additional parameter batched_auth_events:

synapse/synapse/handlers/federation_event.py

Lines 1741 to 1743 in 6fa13b4

	await check_state_independent_auth_rules(
	self._store, event, batched_auth_events=auth_map
	)

It makes the execution enter this if clause, introduced with #15195

synapse/synapse/event_auth.py

Lines 178 to 189 in 6fa13b4

	if batched_auth_events:
	# Copy the batched auth events to avoid mutating them.
	auth_events = dict(batched_auth_events)
	needed_auth_event_ids = set(event.auth_event_ids()) - batched_auth_events.keys()
	if needed_auth_event_ids:
	auth_events.update(
	await store.get_events(
	needed_auth_event_ids,
	redact_behaviour=EventRedactBehaviour.as_is,
	allow_rejected=True,
	)
	)

There are two issues in the above code snippet.

First, there is the blocking issue. I'm not entirely sure if this is a deadlock, starvation, or something different. In the beginning, I thought the copy operation was responsible. It wasn't. Then I investigated the nested store.get_events inside the function update. This was also not causing the blocking issue. Only when I replaced the set difference operation (- ) with a list comprehension, the blocking was resolved. Creating and comparing sets with a very large amount of events seems to be problematic.

This is how the flamegraph looks now while persisting outliers. As you can see, the execution no longer locks up in the above function.

Second, the copying here doesn't serve any purpose, because only a shallow copy is created. This means the same objects from the original dict are referenced. This fails the intention of protecting these objects from mutation. The review of the original PR matrix-org/synapse#15195 had an extensive discussion about this matter.

Various approaches to copying the auth_events were attempted:

1. Implementing a deepcopy caused issues due to builtins.EventInternalMetadata not being pickleable.
2. Creating a dict with new objects akin to a deepcopy.
3. Creating a dict with new objects containing only necessary attributes.

Concluding, there is no easy way to create an actual copy of the objects. Opting for a deepcopy can significantly strain memory and CPU resources, making it an inefficient choice. I don't see why the copy is necessary in the first place. Therefore I'm proposing to remove it altogether.

After these changes, I was able to successfully join these rooms, without the main worker locking up:

• #synapse:matrix.org
• #element-android:matrix.org
• #element-web:matrix.org
• #ecips:matrix.org
• #ipfs-chatter:ipfs.io
• #python:matrix.org
• #matrix:matrix.org

[CLAassistant]

All committers have signed the CLA.

[ggogel]

@anoadragon453 @jaywink @erikjohnston

Would you mind taking a look at this PR when you get a chance? Any feedback would be appreciated!

I've been running the build with the latest changes for three days now without issues.

[anoadragon453]

Hi @ggogel. Hugely appreciate you debugging this issue, pushing up a fix and confirming that it works on your own deployment.

I apologise that we've been so slow to respond over the past couple weeks. It's a direct symptom of a recently reduced team size and a separate (internal) project that's taking up most of the team's time. We'll be freed up again in a couple weeks, but until then this is where we're at.

Regardless, this is a small change and we'll find time to review it before the next release cycle (v1.103.0rc1). Thanks again for your patience.

I've approved CI to run on this PR. Could you fix up the lint/typechecking in the meantime?

[ggogel]

Hi @anoadragon453, no need to apologize at all. I completely understand that team priorities can shift and that everyone's plate can get pretty full at times. I appreciate your acknowledgment of the fix.

I got the linting and type checking sorted out. Looks like the CI workflow needs approval again.

Looking forward to collaborating further on this and getting it ready for the upcoming release cycle.

[ggogel]

Linting and type checking are good now but it timed out during sytest.

2024-03-07 10:28:23.781 UTC [2642] FATAL:  database "sytest_template" does not exist
    Test 152 Remote users can join room by alias... WARN: DBI connect('dbname=sytest_template;host=localhost','postgres',...) failed: FATAL:  database "sytest_template" does not exist at lib/SyTest/Homeserver.pm line 456.
2024-03-07 10:28:23.790 UTC [2644] FATAL:  database "sytest_template" does not exist

[ggogel]

The typing change in dfac2ca caused the issue to reappear.

- auth_events = batched_auth_events
+ auth_events = dict(batched_auth_events)

If someone can shed light on why employing dict() in this context causes a deadlock again, I would appreciate the insight.

[ggogel]

I've found a fix that passes mypy and prevents the locking issue by iterating over the mapping and adding each item individually.

My best guess so far is that dict() causes the whole mapping to get locked and I believe the same happens when using the set difference operation.

[ggogel]

I'm testing the latest build 3e642ed by repeatedly joining and leaving rooms.

Successful:

• #ipfs-chatter:ipfs.ioorg
• #archlinux:archlinux.org
• #ecips:matrix.org
• #fdroid:f-droid.org

Keeps failing:

• #matrix:matrix.org

I'm unsure if it's failing where the changes are or somewhere else.

I'm working on yet another change at the moment...

[ggogel]

I've finally found a solution that has good performance and makes mypy happy.

My initial working solution was the fastest because no copying was done at all. I simply assigned the dictionary to another variable. However, due to batched_auth_events being defined as Mapping and not as MutableMapping, it doesn't have an update function.

auth_events = batched_auth_events

Then I changed it to use dict() because of a type error in mypy. This is not simply a type cast, it creates a new dictionary by copying the old one.

auth_events = dict(batched_auth_events)

Since batched_auth_events can get extremely large, this can be problematic and cause the process to lock up for a while. The largest I've seen was 165k items. Depending on the room complexity this function can be called several thousand times. If it only yields every 1000th loop as implemented in #16903 this accumulates to 165 million individual item copy operations without yielding the processor once. Some debug logs:

2024-03-08 23:05:14,189 - synapse.event_auth - 192 - WARNING - _process_incoming_pdus_in_room_inner-0-$eZHQIOFFcm5JjBrgYZBNW_-5j5Q6KMohRc6DycXw5gs-$Hli_I4rl2cSIn4GUe_gLnwXsOy4LIHbldqBFPKqGMKc-$igefq-4nOjAhDi2Kc4ZSJHfXzXqP53EAn2dboERIsnQ-$p4HEbuvaaz4ssS06nUht6gjTeBJAaNWJdspPZBcP5vo- batched_auth_events length: 165451
2024-03-08 23:05:14,189 - synapse.event_auth - 211 - WARNING - _process_incoming_pdus_in_room_inner-0-$eZHQIOFFcm5JjBrgYZBNW_-5j5Q6KMohRc6DycXw5gs-$Hli_I4rl2cSIn4GUe_gLnwXsOy4LIHbldqBFPKqGMKc-$igefq-4nOjAhDi2Kc4ZSJHfXzXqP53EAn2dboERIsnQ-$p4HEbuvaaz4ssS06nUht6gjTeBJAaNWJdspPZBcP5vo- needed_auth_events length: 0

After that, I tried the following approach of copying each item one by one into a new dictionary. This was much faster, but still caused issues with the humongous room #matrix:matrix.org. It was locked for 2 minutes and got killed by the liveness probe. Joining this room seems to be the most difficult benchmark for synapse. All other rooms I tried, worked fine with this solution.

for key, value in batched_auth_events.items():
            auth_events[key] = value

I explored other options such as using the dictionary unpacking operator ** and the operator | to merge batched_auth_events and needed_auth_events into a new dict. This approach is also slow because again it copies every item. It's also slower than using .update().

As a side note: When using .update() it is important to update on the larger of both dictionaries. Since the two dictionaries in this function can't have any equal keys, the order doesn't matter here. In my tests, I always found that needed_auth_events was empty, while batched_auth_events can be very large. If we would execute update on needed_auth_events it would essentially again result in a full copy of batched_auth_events.

I finally ended up with this solution, which matches my initial solution but also satisfies mypy by using cast.

auth_events = cast(MutableMapping[str, "EventBase"], batched_auth_events)

[ggogel]

Following a discussion with @erikjohnston, I've implemented a change to utilize ChainMap.

I've successfully tested the change on:

• #ipfs-chatter:ipfs.io
• #archlinux:matrix.org
• #matrix:matrix.org

[erikjohnston]

Thanks @ggogel for digging into this and trying out various of the fixes!