✨ Improve testing mode and support in sdk-utils based tests #1031
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is this?
Currently, all SDKs that use
@percy/sdk-utils
utilize its testing utils, including a small testing server that emulates some CLI API features to make the SDKs think they are interacting with the real CLI. When the .NET SDK was implemented, it used a new CLI feature made specifically to help test SDKs: testing mode (#984).This PR adds a few more features to testing mode:
These additional features then allow
@percy/sdk-utils
test helpers to work with testing mode so we can remove the CLI API emulation script from our SDK tests and test against the CLI directly. The rest of this PR replaces all@percy/sdk-utils
test helpers with new helpers that allow working with testing mode from consuming SDKs:await helpers.setupTest()
- Replaceshelpers.setup()
, and now automatically resets testing mode.await helpers.test(cmd[, arg])
- Will make a POST request to the/test/api/:cmd
endpoint with any provided second argument as the request body.await helpers.get(what[, map])
- Will make a GET request from the/test/{logs|requests}
endpoint and optionally map the resulting response data. By default,logs
are mapped to theirmessage
property.helpers.testSnapshotURL
- A getter that returns the URL of a small HTML page at/test/snapshot
.Changes for SDK tests
It's important to note that none of these changes have an effect on SDKs' functionality, but will require changes to be made to SDK tests to use the new
@percy/sdk-utils/test/helpers
. This list serves as a reference while we're making appropriate changes to those SDK test suites.helpers.setup()
&helpers.teardown()
The
helpers.setup()
function is now async and namedhelpers.setupTest()
. Thehelpers.teardown()
function is no longer required and was removed completely.helpers.mockSite()
&helpers.closeSite()
These helpers were used to stand up and start a mock site for taking snapshots of in respective SDKs. The new CLI testing mode has a
/test/snapshot
endpoint that serves a basic HTML page for this purpose. If an SDK needs a more advance snapshot to capture for a specific reason we can circle back to this new endpoint and allow it to be customized using another/test/api/:cmd
command.helpers.logger.*
All logging helpers were removed. Local tracked logs and the loglevel are reset during
helpers.setupTest()
. While in testing mode, all logs are sent remotely to the CLI API via a WebSocket connection. These logs may be accessed with the newhelpers.get()
function.Before:
After:
The remote
logs
are, by default, mapped to messages only. However, a second argument tohelpers.get()
can map any other value, or provide an identity function to assert on raw logs.helpers.testFailure(path[, error][, responsebody])
Before:
Contrary to the name of the helper, this helper sets an endpoint to respond with an error message. It's the most common test helper for altering CLI API behavior.
After:
Appropriately named
error
, this command uses an internal error message. Testing failed builds, while not done by SDKs, can be enabled with another command,build-failure
.helpers.testError(path)
Before:
Less common than
helpers.testFailure(path)
, but also conversely named, this helper sets an endpoint to fail to respond to a request at all by disconnecting as soon as possible.After:
More aptly named, the new test command is
disconnect
, and performs the same functionality as before.helpers.getRequests()
Before:
Used to get requests made to the emulated CLI API, the returned payload was an array of tuples representing the URL and any associated payload.
After:
In testing mode, the real CLI API will now track requests which can be similarly fetched with the using
helpers.get()
. However, most SDKs will want to test againstlogs
instead ofrequests
, since we tend to test user-facing things.helpers.testReply(path, reply)
This was used to create additional endpoints in the emulated CLI API. If used, it is likely to create fake pages to snapshot, which should no longer be necessary.
helpers.testSerialize([fn])
This was used to replace the emulated API CLI's dummy DOM serialization script. It is no longer an available feature since the real
@percy/dom
serialization script will be available to SDKs when using testing mode.helpers.call(...)
Any test that uses this directly will need to be adjusted to use testing mode features. No SDK should be doing this, but if they are, they likely can be migrated to one of the other new testing mode commands.