Help E2E Model Based Testing

[davidjohnbell]

I want to implement fast-check into a software system that consists of a web frontend and statefull backend. The frontend and the backend communicate over websocket. For simplicity sake the software behaves like a game (i.e., join, produce events, respond to events, leave). The system would be deterministic if it weren't for IO related issues (e.g., slow connections, failed requests). I am hoping to use fast-check to tease out how the system responds to these issues, if possible can you provide some advice? Thank you.

[gruhn]

In general that sounds like a job for Model-based Testing. Presumably, the frontend is completely state-less or has no state that is relevant here? Then you probably would create a model for the backend state and create a Command for each state modifying interaction that you want to test (JoinCommand, LeaveCommand, ...). It's absolutely crucial that you can reset the backend state completely. Otherwise you might get bogus counterexamples. For example,

• fast-check tries random command sequence: [join, sendEvent3] (no problem detected)
• (state is not reset)
• fast-check tries random command sequence: [sendEvent4] (problem detected!)

Now fast-check reports that the command sequence [sendEvent4] is problematic, although the actual problem is [join, sendEvent3, sendEvent4]. So individual test runs must be independent to a make them reproducible.

To test things like slow connections and failed requests, my first instinct is to create dedicated commands for that. Something like:

class FailedJoinCommand implements fc.Command<Model, System> {
  run(m: Model, r: System): void {
    // trigger join message but maybe intercept the request or something so it doesn't reach the backend.
  }
}

So fast-check can consciously simulate these IO issues.

[davidjohnbell]

Thankyou for your speedy reply. The application is a peer-to-peer video player (HTTP Live Stream). When someone requests the video they send a join event and receive incoming-connection events and multiple outgoing-connection events. Of particular importance is potentially emitting stream-quality events which boils down to poor download/IO conditions for you or somebody upstream from you. This would cause the server to tell you to connect to someone else. Given perfect network conditions both the frontend and backend are state-reducers (state, event) -> state. The tricky part is testing e2e how the system responds to poor IO.
The direction you gave was helpful. I was able to implement commands for join and leave but am a little stuck with the async nature and how to test IO. The assert case for my model is probing the HTML video element and checking it has advanced.
Thanks for your help, it's greatly appreciated :)

[gruhn]

Ok so is there a central server and a bunch of web clients that are streaming peer-to-peer? Damn, I didn't even know that's possible. I think I vaguely understand what's going on.

It sounds like you can't predict the messages coming from the backend. The Model-Based Testing approach kind of assumes that every state update is controlled via a Command. In a classic client/server setup that's ok. In the command you trigger a request and the await the response + state update. But in your case the backend can just send messages at any time, right? I think you would need control over the sending of both client and server messages to apply Model-Based Testing here. Your state model would be the gods eye view of the entire distributed system. Something like:

type StateModel = {
   client1: { sendMessages: [...], receivedMessages: [...] },
   client2: { sendMessages: [...], receivedMessages: [...] },
   server: { sendMessages: [...], receivedMessages: [...] },
}

But that's not feasible, right?

[davidjohnbell]

Yeah I think you have a picture of whats going on. It is more deterministic than it may seem at first - the backend is just responding to other viewers (e.g., someone isn't receiving video from their parent so they get places under you, you receive a seemingly random connection event but it's not actually random insofar as you can trace it back to your new peer complaining they weren't getting enough data). Such is the case with all the events.

I think we're on the right track here. There would be events that process the incoming and outgoing messages, add and remove messages, etc. I like this, but do you suppose there are other methods that would allow me to treat the whole thing more like a black box?

Thanks for your help, I appreciate it. I'll mark it as an answer - don't feel like you have to keep helping 😄