Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test (Cypress GitHub Backend): Introduce an option to record cypress tests data and run tests using recorded data by default #2764

Closed
wants to merge 1 commit into from

Conversation

erezrokah
Copy link
Contributor

GitHub actions version of this PR

Follow up on the GraphQL PR, specifically this comment.

The main purpose of this PR it to allow running Cypress GitHub backend tests using mock/recorded data.
I've looked into several solutions:

  1. http://www.mock-server.com/ - A proxy server to record traffic and match requests.
  2. https://github.com/Netflix/pollyjs - A framework that uses stubbing to record and playback requests (supports browsers and node) - enabling it broke Cypress completely for some reason.
  3. https://github.com/nock/nock - Similar to Polly.JS, but only works for node (overrides http.request and http.ClientRequest).
  4. http://www.wheresrhys.co.uk/fetch-mock/ Similar to Nock, but works with fetch.
  5. Cypress built in mocking solution - Only works with XMLHttpRequests. There are some workarounds for supporting fetch, but another main issue is that you can only match by URL and method (no post data) making it unusable for most use cases. Again there are some workarounds here and here but they seem overly complex.

The method I'm using is described in the Readme:
For recording - set up Cypress with a proxy (http://www.mock-server.com/), sanitize and transform the recorded data to an easier to process format.

For playback - Cypress exposes a sinon instance so I'm using that to stub window.fetch and match requests.

Several problems I experienced in matching requests:

  • Requests are sent with a ts parameter to bust browser caching. This is solved by using pattern matching on the URL.
  • Creating posts uses the current date by default, which changes the content of some post requests.
    I initially solved it by entering a fixed date into the date field, but this wasn't good enough as slugs are formatted with the current date regardless of the post date.
    Cypress has a clock function that mocks the Date object, but it created another issue as some of the UI fields are de-bounced, thus ignoring value changes unless the the clock is advanced.
    The solution was to advanced the clock after updating posts just enough to trigger the de-bounced input field, while still keeping the slug generation consistent. We could also consider generating the slug based on the post date (if it has such a field).
  • A specific open authoring test can return entry to draft and delete it was passing on Chrome but failing on Electron consistently (same behavior for REST and GraphQL). This revealed another race condition: When moving an entry from In Review to Draft while in open authoring mode, the PR that was created on the origin repo is closed (the forked repo branch is not deleted). One can click "Delete new entry" before the change status operation completes (moving from In Review to Draft), thus introducing the race. This doesn't have a negative effect on human users, but since the test validate toasts messages in order, it fails.
  • Cypress doesn't clear indexedDB between tests. This has to be done manually.

Refactoring:

  • Moved the tests utility methods into relevant files to keep plugin/index.js clean.

Bug fixes:

  • Remove redundant call to new Date().
  • Add null check before calling this.api.reset.
  • Fix the race condition described above.

CI improvements:

  • Run specs in parallel. This is done by splitting the CI process into 2 parts (build and e2e tests). Very similar to the azure pipeline original change described here, only without yarn caching as it is not supported yet.

@erezrokah erezrokah closed this Oct 22, 2019
@erezrokah erezrokah deleted the test/github_backend/enable_e2e_tests branch October 22, 2019 15:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant