Master ticket: Make all tests runnable in the browser

[ryanhamley]

Motivation

In an effort to reduce environment-specific bugs, we want to make all tests runnable in browsers, rather than just in Node. Node is actually a somewhat unrepresentative test environment and we frequently see bugs related specifically to environment variables such as browser and GPU. While we do manual browser testing during each release process, this is not as efficient or comprehensive as our full test suite. By ensuring that our tests can be run directly in these environments, we will be able to catch more environment-specific bugs before they make it to a release and will have an easier time debugging these issues when they arise in the future.

Design Alternatives

The main alternative is what we have now. Traditional Node-based test suites. This works fine for the most part, but as described above, it limits our ability to find environment-specific bugs so doing nothing isn't an option.

Another more traditional approach is end-to-end testing with libraries such as Selenium. We've already begun adding Selenium tests in #9245 but these have limited functionality for us because they are designed for use with DOM elements, whereas most of GL JS's functionality is within a canvas element and there are typically only a small number of DOM elements (usually just the canvas, some control buttons and maybe HTML markers).

Design

We already made query tests runnable in the browser with #8584 That PR uses a very simple layout which is all we really need.

Mock-Up

We could replace the Node tests entirely so that existing commands such as yarn test-query and yarn test-render work, just running the tests in the browser instead of Node. #8584 adds a new yarn watch-query command instead of replacing the older Node test command. If we no longer need the ability to run the tests in Node though, we could keep the older testing interface to simplify adoption.

Concepts

There shouldn't really be any new concepts in the public facade of this. Writing query, render and unit tests likely won't change. It's just how the tests are run that changes. #8584 has documentation on the new process, including clear documentation on how to run a single test which should be somewhat simpler now than it was with the Node tests.

Implementation

#8584 provides an extensible framework for running render tests in the browser. Early experimentations with running render tests in the browser have shown that this approach is viable.

One notable challenge of the implementation in #8584 is that the library which runs the test suite testem co-opts the console output, making it somewhat difficult to have logs show up in the browser, a common approach to debugging.

However, experiments with running unit tests have shown that a somewhat different approach is necessary. We currently rely on the unit test runner node-tap which is incompatible with a browser environment (The Node version of Javascript is slightly different from what runs in browser engines and tap relies too heavily on Node-specific APIs). @kkaefer has spent some time experimenting with a different test runner called node-tape which is compatible with browser environments. According to Konstantin, we need to use the node-tape prerelease version since it exposes a few assert functions that we currently use. Most issues with node-tape are around it not supporting beforeEach/afterEach methods which allow for setting up and tearing down functions and objects common to a test suite (e.g. creating a map). This means we have to wrap a few tests in functions that setup + teardown a sandbox for the Sinon library which allows for stubs, mocks and spies in tests.

Konstantin shared this wrapper function for test/util/test.js which creates a Sinon sandbox.

export const withSandbox = fn => t => {
    let sandbox;
    t.test(`cleanup: ${t.name}`, t => {
        sandbox.restore();
        t.end();
    });
    sandbox = sinon.createSandbox({
        properties: ['spy', 'stub', 'mock']
    });
    return fn(t, sandbox);
};

Eventually, we will want to automate these tests against a representative set of devices and browsers. We've done some research into using platforms such as AWS Device Farm, but implementation of this is currently blocked for various technical reasons. For the time being, we will be able to manually run our test suite against a handful of the most common devices and browsers.

Types of Tests

• Query - already runnable in browsers (Running query tests in browser #8584)
• Render
• Unit
• Browser - Selenium tests run in browser (Add browser testing with Selenium/WebDriver #9245)

[ryanhamley]

I've been experimenting with render tests in https://github.com/mapbox/mapbox-gl-js/tree/render-tests-in-browser There's not a lot there right now as I've mostly been able to work within the existing framework. Once I get the tests passing, I'll need to add a separate command.

[arindam1993]

Super on point summary @ryanhamley !

re: tape.

The query test runner actually uses tape. It wraps each test as a tape test. This is what makes generating test reporting data easier, since tape will output junit test reports which circleCI can generate nicely formatted reports of.