Faster room joins: avoid blocking when pulling events with missing prevs

[squahtx]

Avoid blocking on full state in _resolve_state_at_missing_prevs and
return a new flag indicating whether the resolved state is partial.
Thread that flag around so that it makes it into the event context.

Resolves #13002.

---

Corresponding Complement tests: matrix-org/complement#419
Best reviewed commit by commit.
The first commit is from #13354.

[richvdh]

generally looks great!

[richvdh]

should we check that partial_state is false?

[squahtx]

sure, keeping in mind that it won't catch the "calculate-the-state-yourself" case. I can't think of how to handle a second partial state apart from raising an exception.

[richvdh]

sorry yes, I'd envisaged raising an exception, on the grounds we could give a clearer idea of what's going wrong.

But if you'd expect it to behave correctly (ish) in the case where we've rejoined, then I guess that's not appropriate and we should just leave this.

[squahtx]

I've yet to figure out what happens when we rejoin (not that we can even leave right now!) so am happy to raise an exception.

[richvdh]

I'd find it helpful to enumerate the possibilities here. AIUI, they are:

• state_ids and partial_state are both None, if we had all the prev_events (some of which may have partial state).
• state_ids is non-empty, and partial_state is False, meaning we were missing some or all prev_events, and have calculated the state after the prev_events
• state_ids is non-empty, and partial_state is True, meaning we were missing some prev_events, but others had partial-state. We have calculated the state after all the prev_events, but it is still partial. There is no point building an event context in this case.

[richvdh]

... I can't help feeling that the interface for _resolve_state_at_missing_prevs is excessively baroque and coupled to that of compute_event_context. (Should we replace _resolve_state_at_missing_prevs with something that returns an EventContext, and we pass an EventContext into _process_received_pdu instead of state_ids and partial_state?)

But that probably requires a bunch of refactoring that isn't justified right now.

[squahtx]

I've added a big comment listing the cases, borrowing some of your wording here.

I agree that _resolve_state_at_missing_prevs is really unwieldy after this change. Is the proposal to have it *always* return an EventContext? I'll try that refactor in a follow on PR.

[richvdh]

as a random point for future reference: there is a convention that the subject line of a git commit message should be limited to 50 characters. That helps avoid this sort of thing:

There's an art to writing such a pithy summary, of course. In this case I would probably just say:

`get_state_groups_ids`: add `await_full_state` parameter

... and put "so that it can return partial state" in the body of the commit message.

[squahtx]

interesting, I wasn't aware that the subject line had a shorter recommended length.

Up until now I've been using a limit of 72 characters in merge commits (and ignoring it in commits that get squashed, like the one above). 50 characters certainly makes things tricky!

[richvdh]

ignoring it in commits that get squashed, like the one above

Yeah, the problem comes when we're doing a commit-by-commit review: it's much nicer if there's a pithy summary for each commit. It's certainly not worth stressing overmuch over though.

50 characters certainly makes things tricky!

Indeed. github actually seems to truncate at 66 chars, so that seems a more reasonable compromise.

[richvdh]

lgtm