Deflake leave-via-kick/ban faster joins tests

[squahtx]

We need to allow the homeserver under test to finish making requests to
the Complement homeserver before tearing it down, otherwise the
homeserver under test may mark the hostname:port combination as down and
refuse to contact it in subsequent tests.

Signed-off-by: Sean Quah seanq@matrix.org

---

NB: you can tease out the flakes by forcing Complement to reuse ports whenever possible:

diff --git a/internal/federation/server.go b/internal/federation/server.go
index ed7974d..4c4ccbe 100644
--- a/internal/federation/server.go
+++ b/internal/federation/server.go
@@ -455,7 +455,10 @@ func (s *Server) Listen() (cancel func()) {
        var wg sync.WaitGroup
        wg.Add(1)

-       ln, err := net.Listen("tcp", ":0") //nolint
+       ln, err := net.Listen("tcp", ":63333") //nolint
+       if err != nil {
+               ln, err = net.Listen("tcp", ":63334") //nolint
+       }
        if err != nil {
                s.t.Fatalf("ListenFederationServer: net.Listen failed: %s", err)
        }

[squahtx]

This is incredibly hacky. Alternative suggestions welcome.

[DMRobertson]

Top of my head: we could make the homeserver under test take an action in the room that requires knowledge of the full state, and wait for complement's HS to see it. (Is there an example of a federation API that blocks on full state?)

(Personally I can live with the hack though)

[squahtx]

I think the difficulty is that the homeserver under test leaves the room independently of, and before the sync process finishes. Any request we might try would be met with a not-in-the-room response even though the sync process may still be running.

[DMRobertson]

otherwise the homeserver under test may mark the hostname:port combination as down and refuse to contact it in subsequent tests.

Should we add some kind of escape hatch for this? E.g. an admin API which could be used to mark a destination as up, that complement could call whenever a test begins? (Not ideal, but we seem to hit this class of flake a lot...)

[DMRobertson]

Arg, this is one of my tests. I think we did talk about it at the time but clearly I missed this class of flakes/interaction with other tests---sorry.

[DMRobertson]

Top of my head: we could make the homeserver under test take an action in the room that requires knowledge of the full state, and wait for complement's HS to see it. (Is there an example of a federation API that blocks on full state?)

(Personally I can live with the hack though)

[squahtx]

otherwise the homeserver under test may mark the hostname:port combination as down and refuse to contact it in subsequent tests.

Should we add some kind of escape hatch for this? E.g. an admin API which could be used to mark a destination as up, that complement could call whenever a test begins? (Not ideal, but we seem to hit this class of flake a lot...)

We could do. It's very Synapse and faster joins test-specific though. We only hit this class of flake because we reuse the same Synapse with lots of different Complement instances in the faster joins test suite.