Catch-up after Federation Outage (bonus): Catch-up on Synapse Startup

[reivilibre]

This PR makes Synapse attempt a catch-up on start-up for destinations which need it and are not being backed off from.

The need for this can come about in at least two situations that I can think of:

• s1 is sending to s2 which is offline or unreachable. s1 gets restarted before we retry long enough for s2 to get backed off from.
• s1 has queued events or outstanding catch-up for s2 and gets restarted.

Thoughts:

• when a server's backoff expires, no mechanism exists to go and poke it before we hear from it or otherwise have a PDU or EDU to send. Presence EDUs can cause enough chattiness.
• when a failing server is finally brought online, it doesn't go and intentionally poke other servers unless it already has an excuse to go talking (EDUs/PDUs). I think this one may be worth solving (in another PR) for small homeservers' sakes, but how? Again, presence EDUs can already cause enough chattiness.

But if you don't have presence, then a small homeserver that has wound up on harsh enough backoffs would need to go and send messages to all other servers before anything further will be received. (This is pre-existing behaviour but since I'm in the area, feels like a good time to think about this and at least put something on the issue list about it?)

Part of #2528

[MayeulC]

when a failing server is finally brought online, it doesn't go and intentionally poke other servers unless it already has an excuse to go talking (EDUs/PDUs). I think this one may be worth solving (in another PR) for small homeservers' sakes, but how? Again, presence EDUs can already cause enough chattiness.

An idea could be to send a dummy event somewhere (where?).

Alternatively, presence could be introduced for homeservers, maybe with a much lower resolution?

[clokep]

I think I generally understand what's happening, I left a few questions as well as a couple of changes. I think that it makes sense overall.

[clokep]

Where is this last_successful_stream_ordering coming from?

[reivilibre]

Is it clearer now?:

WHERE destination_rooms.stream_ordering >
  destinations.last_successful_stream_ordering

[clokep]

This attempts to find destinations which have never been backed off from or which are beyond their retry interval?

Does this mean it will just poke all servers when it wakes up?

[reivilibre]

Yes, this part means have never been backed off from or have an expired backoff.

                WHERE destination IN (
                    SELECT destination FROM destination_rooms
                        WHERE destination_rooms.stream_ordering >
                            destinations.last_successful_stream_ordering
                )

restricts this to only destinations with catch-up needed.

[clokep]

Why is this necessary? Is it because _wake_destinations_needing_catchup doesn't protect itself from being called multiple times?

[reivilibre]

Yes, since the reactor gets pumped then calling it manually ourselves doesn't change the fact that it is called automatically after the timer.

[clokep]

If you're just trying to compare the values are the same (in any order), use assertCountEqual:

Suggested change

	self.assertEqual(set(woken.keys()), set(server_names[:-1]))
	self.assertCountEqual(woken.keys(), server_names[:-1])

[reivilibre]

It's good to see there is a built-in function for this, but I'm not sure I like it — the name is completely misleading?

(sounds like it is doing len(woken.keys()) == len(server_names[:-1])…)

[clokep]

I agree that the name isn't great, but using the standardized form is easier instead of parsing the logic IMO.