Investigate simulating GraphQL responses for testing

[kinow]

We started with the Suite Graph View issue (#74) but then we had a meeting, and after that others joined the discussion and now we have another broader issue for all Suite Views (#75), and it was also mentioned that it would be important to work on the testing part.

One part of the testing required is some simulated data, to be used by many tests to simulate possible states of a workflow, and pass through our code to verify its behaviour.

In #74, I was going to exactly work on the looking if a VCR-like tool could be used to record-persist-replay the requests of real workflows. And then use this persisted data for tests.

Now with the #75, I think it might be easier to work on this new issue, instead of documenting this part of testing in #74.

[kinow]

There is a complication, as most of the existing tools were used by traditional web applications, where a user would send a request to Service#1, then get the response, call Service#2, and so on.

In our case, we have one endpoint only (/graphql), where we are sending the same request (?query=QUERY), but the response is varying with time, as the workflow state changes.

Which means we would have to find a tool that records multiple interactions. Most of the tools I've seen so far overwrite or raise an error when the same requests appears more than once.

[kinow]

The best JS options were:

• https://github.com/linkedin/sepia
  • Looks promising, from Linkedin, quite a few stars on GitHub. But last commit from 2015.
• https://github.com/flickr/yakbak
  • From Flickr, but with last commit from Jan 2018 too.

So will probably check the Python alternatives instead. We could use it to record the requests, and import into this cylc-ui project as test resources.

[hjoliver]

It might be a lot simpler (if less elegant) to hack the workflow service and do a JSON dump (to file) of the suite data structure, then manually massage that into the new GraphQL schema? That would be easy for a small test suite at least. It might be painful for the 1000+ task complex demo suite. But we can start small!

[kinow]

For Python the most famous is vcrpy. Even the deprecated uber/cassette now recommends users to use it too.

[kinow]

It might be a lot simpler (if less elegant) to hack cylc-7 and do a JSON dump (to file) of the suite data structure, then manually massage that into the new GraphQL schema? That would be easy for a small test suite at least. It might be painful for the 1000+ task complex demo suite. But we can start small!

That's an alternative too. We have already some methods in Cylc 8 branch of @dwsutherland that convert data from the pools and other objects into Protobuf messages, that are used to produce the GraphQL output... so that could be simpler indeed. It won't take long to verify if vcrpy works. If not, then will start evaluating writing something from scratch.

[hjoliver]

(sorry used cylc-7 above by mistake, already edited that comment)

[kinow]

Tried the following tools:

• https://docs.mitmproxy.org/stable/concepts-modes/
  • complicated, probably better for pentesting
• https://github.com/jjyr/zerotest
  • bit buggy, got a few tracebacks, one looked like an issue with something from py37
• https://netflix.github.io/pollyjs/
  • not really able to easily record and replay messages, unless you write some js code. The cli is only for listening.
• https://github.com/buger/goreplay
  • DEB-Package contains Mach-O binary buger/goreplay#661 first, and then couldn't get it to replay any GraphQL
• Postman
  • asks you to create log in to create mock servers now

[kinow]

In the end vcrpy was the best solution. But we have to use a custom url matcher with some interval, otherwise it matches the URL /users/kinow/graphql/graphiql and refuses to record it.

Quick code at https://github.com/kinow/cylc-cassettes, and the example GraphQL response from the workflow five, recorded every 5 seconds after it started: https://github.com/kinow/cylc-cassettes/tree/master/cassettes/five

Now the idea is to write a quick JS code within cylc-ui to a) read the VCR tapes, b) return suite.yaml for the SuiteService#getSuites() when in development, and return the current tasks-?.yaml for the SuiteService#getSuiteTasks() when in development.

The current task must be simply the index in a circular buffer... starting at 0, and going until the last tape/cassette found for tasks-?.yaml, then start over again. The index must change in intervals, something like setTimeout(function() { ... change index ... }, 3000);.

This way, we should be able to quickly simulate workflows states without the need to run cylc-flow and some command like cylc run, or have to worry about the suite state. Plus, we can later customize the service to return a fixed state, which may be useful for developing or testing certain workflow or task states.