Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Improve testing mode and support in sdk-utils based tests #1031

Merged
merged 6 commits into from
Aug 12, 2022

Conversation

wwilsman
Copy link
Contributor

What is this?

Currently, all SDKs that use @percy/sdk-utils utilize its testing utils, including a small testing server that emulates some CLI API features to make the SDKs think they are interacting with the real CLI. When the .NET SDK was implemented, it used a new CLI feature made specifically to help test SDKs: testing mode (#984).

This PR adds a few more features to testing mode:

  • Support for sabotaging remote logging connections
  • Tracking API requests and allowing access to tracked requests
  • Ability to mimic a failed build for specific error handling

These additional features then allow @percy/sdk-utils test helpers to work with testing mode so we can remove the CLI API emulation script from our SDK tests and test against the CLI directly. The rest of this PR replaces all @percy/sdk-utils test helpers with new helpers that allow working with testing mode from consuming SDKs:

  • await helpers.setupTest() - Replaces helpers.setup(), and now automatically resets testing mode.
  • await helpers.test(cmd[, arg]) - Will make a POST request to the /test/api/:cmd endpoint with any provided second argument as the request body.
  • await helpers.get(what[, map]) - Will make a GET request from the /test/{logs|requests} endpoint and optionally map the resulting response data. By default, logs are mapped to their message property.
  • helpers.testSnapshotURL - A getter that returns the URL of a small HTML page at /test/snapshot.

Changes for SDK tests

It's important to note that none of these changes have an effect on SDKs' functionality, but will require changes to be made to SDK tests to use the new @percy/sdk-utils/test/helpers. This list serves as a reference while we're making appropriate changes to those SDK test suites.

helpers.setup() & helpers.teardown()

The helpers.setup() function is now async and named helpers.setupTest(). The helpers.teardown() function is no longer required and was removed completely.


helpers.mockSite() & helpers.closeSite()

These helpers were used to stand up and start a mock site for taking snapshots of in respective SDKs. The new CLI testing mode has a /test/snapshot endpoint that serves a basic HTML page for this purpose. If an SDK needs a more advance snapshot to capture for a specific reason we can circle back to this new endpoint and allow it to be customized using another /test/api/:cmd command.


helpers.logger.*

All logging helpers were removed. Local tracked logs and the loglevel are reset during helpers.setupTest(). While in testing mode, all logs are sent remotely to the CLI API via a WebSocket connection. These logs may be accessed with the new helpers.get() function.

Before:

helpers.logger.mock();
helpers.logger.reset();

expect(helpers.logger.stdout).toEqual([
  '[percy] Some log made by the SDK'
]);
expect(helpers.logger.stderr).toEqual([
  '[percy] Some error made by the SDK'
]);

After:

The remote logs are, by default, mapped to messages only. However, a second argument to helpers.get() can map any other value, or provide an identity function to assert on raw logs.

expect(await helper.get('logs')).toEqual([
  '[percy] Some log made by the SDK',
  '[percy] Some error made by the SDK'
]);

// or

expect(await helper.get('logs', i => i)).toEqual([
  { debug: 'sdk-name', level: 'info', message: 'Some log made by the SDK', ... },
  { debug: 'sdk-name', level: 'error', message: 'Some error made by the SDK', ... }
]);

helpers.testFailure(path[, error][, responsebody])

Before:

Contrary to the name of the helper, this helper sets an endpoint to respond with an error message. It's the most common test helper for altering CLI API behavior.

await helpers.testFailure('/percy/healthcheck');
await helpers.testFailure('/percy/healthcheck', 'Error message');
await helpers.testFailure('/percy/healthcheck', 'Build failed', { build: { failed: true } });

After:

Appropriately named error, this command uses an internal error message. Testing failed builds, while not done by SDKs, can be enabled with another command, build-failure.

await helpers.test('error', '/percy/healthcheck');
await helpers.test('build-failure');

helpers.testError(path)

Before:

Less common than helpers.testFailure(path), but also conversely named, this helper sets an endpoint to fail to respond to a request at all by disconnecting as soon as possible.

await helpers.testError('/percy/healthcheck');

After:

More aptly named, the new test command is disconnect, and performs the same functionality as before.

await helpers.test('disconnect', '/percy/healthcheck');

helpers.getRequests()

Before:

Used to get requests made to the emulated CLI API, the returned payload was an array of tuples representing the URL and any associated payload.

expect(await helpers.getRequests()).toEqual([
  ['/percy/healthcheck'],
  ['/percy/dom.js'],
  ['/percy/snapshot', { /*...*/ }],
]);

After:

In testing mode, the real CLI API will now track requests which can be similarly fetched with the using helpers.get(). However, most SDKs will want to test against logs instead of requests, since we tend to test user-facing things.

expect(await helpers.get('requests')).toEqual([
  { url: '/percy/healthcheck', method: 'GET' },
  { url: '/percy/dom.js', method: 'GET' },
  { url: '/percy/snapshot', method: 'POST', body: { /*...*/ } },
]);

// or 

expect(await helpers.get('requests', r => r.url))
  .toEqual(['/percy/healthcheck', '/percy/dom.js', '/percy/snapshot']);

helpers.testReply(path, reply)

This was used to create additional endpoints in the emulated CLI API. If used, it is likely to create fake pages to snapshot, which should no longer be necessary.


helpers.testSerialize([fn])

This was used to replace the emulated API CLI's dummy DOM serialization script. It is no longer an available feature since the real @percy/dom serialization script will be available to SDKs when using testing mode.


helpers.call(...)

Any test that uses this directly will need to be adjusted to use testing mode features. No SDK should be doing this, but if they are, they likely can be migrated to one of the other new testing mode commands.

- Support sabotaging remote logging connections
- Track API requests and allow access to tracked requests
- Allow setting a build error to propigate for failed requests
- Store the full error stack trace when logging an error message
- Include boolean error value in entry like core's logger
This fixes an issue, especially in tests, where setting a loglevel may race to be overridden when
immediately following a fresh remote connection
This util is designed to only make requests to the CLI API only. Since the CLI API is always
accessible for sdk-utils tests, there is never a missing response error message to allow the
fallback message to be constructed from generic response properties.
@wwilsman wwilsman added the ✨ enhancement New feature or request label Aug 12, 2022
@wwilsman wwilsman requested a review from Robdel12 August 12, 2022 01:21
Copy link
Contributor

@Robdel12 Robdel12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🏁 VERY nice PR description and excellent work 🔥

@wwilsman wwilsman merged commit 1a86546 into master Aug 12, 2022
@wwilsman wwilsman deleted the ww/sdk-utils-testing-mode branch August 12, 2022 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
✨ enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants