Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scripts: Add test-e2e script to wp-scripts #12437

Merged
merged 7 commits into from
Dec 19, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
563 changes: 295 additions & 268 deletions package-lock.json

Large diffs are not rendered by default.

5 changes: 2 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -90,13 +90,12 @@
"jest-puppeteer": "3.2.1",
"jsdom": "11.12.0",
"lerna": "3.4.3",
"lint-staged": "7.2.0",
"lint-staged": "7.3.0",
Copy link
Member Author

@gziolo gziolo Nov 29, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to bump it to ensure it uses the latest version of jest-validate. Otherwise, it shows some warnings for the configuration which is invalid.

"lodash": "4.17.10",
"mkdirp": "0.5.1",
"node-sass": "4.11.0",
"pegjs": "0.10.0",
"phpegjs": "1.0.0-beta7",
"puppeteer": "1.6.1",
"react-dom": "16.6.3",
"react-test-renderer": "16.6.3",
"redux": "4.0.0",
Expand Down Expand Up @@ -179,7 +178,7 @@
"publish:prod": "npm run build:packages && lerna publish",
"test": "npm run lint && npm run test-unit",
"pretest-e2e": "concurrently \"./bin/reset-e2e-tests.sh\" \"npm run build\"",
"test-e2e": "cross-env JEST_PUPPETEER_CONFIG=test/e2e/puppeteer.config.js wp-scripts test-unit-js --config test/e2e/jest.config.json --runInBand",
"test-e2e": "wp-scripts test-e2e --config test/e2e/jest.config.json",
gziolo marked this conversation as resolved.
Show resolved Hide resolved
"test-e2e:watch": "npm run test-e2e -- --watch",
"test-php": "npm run lint-php && npm run test-unit-php",
"test-unit": "wp-scripts test-unit-js --config test/unit/jest.config.json",
Expand Down
1 change: 1 addition & 0 deletions packages/babel-preset-default/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
},
"main": "index.js",
"dependencies": {
"@babel/core": "^7.0.0",
"@babel/plugin-proposal-async-generator-functions": "^7.0.0",
"@babel/plugin-proposal-object-rest-spread": "^7.0.0",
"@babel/plugin-transform-react-jsx": "^7.0.0",
Expand Down
1 change: 1 addition & 0 deletions packages/scripts/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
### New Features

- Added support for `check-engines` script ([#12721](https://github.com/WordPress/gutenberg/pull/12721))
- Added support for `test-e2e` script ([#12437](https://github.com/WordPress/gutenberg/pull/12437)).
- Update default config provided for `lint-js` script ([#12845](https://github.com/WordPress/gutenberg/pull/12845)).

## 2.4.4 (2018-11-20)
Expand Down
74 changes: 52 additions & 22 deletions packages/scripts/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,11 @@ _Example:_
{
"scripts": {
"check-engines": "wp-scripts check-engines",
"check-licenses": "wp-scripts check-licenses --production",
"lint:js": "wp-scripts lint-js .",
"lint:pkg-json": "wp-scripts lint-pkg-json .",
"test": "wp-scripts test-unit-js"
"test:e2e": "wp-scripts test-e2e",
"test:unit": "wp-scripts test-unit-js"
}
}
```
Expand All @@ -45,8 +48,28 @@ _Example:_
This is how you execute the script with presented setup:
* `npm run check-engines` - checks installed version of `node` and `npm`.

### `check-licenses`

### `wp-scripts lint-js`
Validates that all dependencies of a project are compatible with the project's own license.

_Example:_

```json
{
"scripts": {
"check-licenses": "wp-scripts check-licenses --prod --gpl2 --ignore=abab"
}
}
```

_Flags_:

- `--prod` (or `--production`): When present, validates only `dependencies` and not `devDependencies`
- `--dev` (or `--development`): When present, validates both `dependencies` and `devDependencies`
- `--gpl2`: Validates against [GPLv2 license compatibility](https://www.gnu.org/licenses/license-list.en.html)
- `--ignore=a,b,c`: A comma-separated set of package names to ignore for validation. This is intended to be used primarily in cases where a dependency's `license` field is malformed. It's assumed that any `ignored` package argument would be manually vetted for compatibility by the project owner.

### `lint-js`

Helps enforce coding style guidelines for your JavaScript files. It uses [eslint](https://eslint.org/) with the set of recommended rules defined in [@wordpress/eslint-plugin](https://www.npmjs.com/package/@wordpress/eslint-plugin) npm package. You can override default rules with your own as described in [eslint docs](https://eslint.org/docs/rules/).

Expand All @@ -63,7 +86,7 @@ _Example:_
This is how you execute the script with presented setup:
* `npm run lint:js` - lints JavaScripts files in the whole project's.

### `wp-scripts lint-pkg-json`
### `lint-pkg-json`

Helps enforce standards for your package.json files. It uses [npm-package-json-lint](https://www.npmjs.com/package/npm-package-json-lint) with the set of recommended rules defined in [@wordpress/npm-package-json-lint-config](https://www.npmjs.com/package/@wordpress/npm-package-json-lint-config) npm package. You can override default rules with your own as described in [npm-package-json-lint wiki](https://github.com/tclindner/npm-package-json-lint/wiki).

Expand All @@ -80,52 +103,59 @@ _Example:_
This is how you execute those scripts using the presented setup:
* `npm run lint:pkg-json` - lints `package.json` file in the project's root folder.

### `wp-scripts test-unit-js`
### `test-e2e`

_Alias_: `wp-scripts test-unit-jest`
Launches the End-To-End (E2E) test runner. It uses [Jest](https://facebook.github.io/jest/) behind the scenes and you are able to utilize all of its [CLI options](https://facebook.github.io/jest/docs/en/cli.html). You can also run `./node_modules/.bin/wp-scripts test:e2e --help` or `npm run test:e2e:help` (as presented below) to view all of the available options.

Launches the test runner. It uses [Jest](https://facebook.github.io/jest/) behind the scenes and you are able to utilize all of its [CLI options](https://facebook.github.io/jest/docs/en/cli.html). You can also run `./node_modules/.bin/wp-scripts test-unit-js --help` or `npm run test:help` (as presented below) to view all of the available options. By default, it uses the set of recommended options defined in [@wordpress/jest-preset-default](https://www.npmjs.com/package/@wordpress/jest-preset-default) npm package. You can override them with your own options as described in [Jest documentation](https://jestjs.io/docs/en/configuration).
Writing tests can be done using Puppeteer API:

> [Puppeteer](https://pptr.dev/) is a Node library which provides a high-level API to control Chrome or Chromium over the [DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/). Puppeteer runs [headless](https://developers.google.com/web/updates/2017/04/headless-chrome) by default, but can be configured to run full (non-headless) Chrome or Chromium.

_Example:_

```json
{
"scripts": {
"test": "wp-scripts test-unit-js",
"test:help": "wp-scripts test-unit-js --help",
"test:watch": "wp-scripts test-unit-js --watch"
"test:e2e": "wp-scripts test-e2e",
"test:e2e:help": "wp-scripts test-e2e --help"
}
}
```

This is how you execute those scripts using the presented setup:
* `npm run test` or `npm test` - runs all unit tests.
* `npm run test:help` - prints all available options to configure unit tests runner.
* `npm run test:watch` - runs all unit tests in the watch mode.
* `npm run test:e2e` - runs all unit tests.
* `npm run test:e2e:help` - prints all available options to configure unit tests runner.

### `wp-scripts check-licenses`
This script automatically detects the best config to start Puppeteer but sometimes you may need to specify custom options:
- You can add a `jest-puppeteer.config.js` at the root of the project or define a custom path using `JEST_PUPPETEER_CONFIG` environment variable. Check [jest-puppeteer](https://github.com/smooth-code/jest-puppeteer#jest-puppeteerconfigjs) for more details.

Validates that all dependencies of a project are compatible with the project's own license.
We enforce that all tests run serially in the current process using [--runInBand](https://jestjs.io/docs/en/cli#runinband) Jest CLI option to avoid conflicts between tests caused by the fact that they share the same WordPress instance.

### `test-unit-js`

_Alias_: `test-unit-jest`

Launches the unit test runner. It uses [Jest](https://facebook.github.io/jest/) behind the scenes and you are able to utilize all of its [CLI options](https://facebook.github.io/jest/docs/en/cli.html). You can also run `./node_modules/.bin/wp-scripts test-unit-js --help` or `npm run test:unit:help` (as presented below) to view all of the available options. By default, it uses the set of recommended options defined in [@wordpress/jest-preset-default](https://www.npmjs.com/package/@wordpress/jest-preset-default) npm package. You can override them with your own options as described in [Jest documentation](https://jestjs.io/docs/en/configuration).

_Example:_

```json
{
"scripts": {
"check-licenses": "wp-scripts check-licenses --prod --gpl2 --ignore=abab",
"test:unit": "wp-scripts test-unit-js",
"test:unit:help": "wp-scripts test-unit-js --help",
"test:unit:watch": "wp-scripts test-unit-js --watch"
}
}
```

_Flags_:

- `--prod` (or `--production`): When present, validates only `dependencies` and not `devDependencies`
- `--dev` (or `--development`): When present, validates both `dependencies` and `devDependencies`
- `--gpl2`: Validates against [GPLv2 license compatibility](https://www.gnu.org/licenses/license-list.en.html)
- `--ignore=a,b,c`: A comma-separated set of package names to ignore for validation. This is intended to be used primarily in cases where a dependency's `license` field is malformed. It's assumed that any `ignored` package argument would be manually vetted for compatibility by the project owner.
This is how you execute those scripts using the presented setup:
* `npm run test:unit` - runs all unit tests.
* `npm run test:unit:help` - prints all available options to configure unit tests runner.
* `npm run test:unit:watch` - runs all unit tests in the watch mode.

## Inspiration

This is inspired by [react-scripts](https://www.npmjs.com/package/react-scripts) and [kcd-scripts](https://www.npmjs.com/package/kcd-scripts).
This package is inspired by [react-scripts](https://www.npmjs.com/package/react-scripts) and [kcd-scripts](https://www.npmjs.com/package/kcd-scripts).

<br/><br/><p align="center"><img src="https://s.w.org/style/images/codeispoetry.png?1" alt="Code is Poetry." /></p>
26 changes: 26 additions & 0 deletions packages/scripts/config/jest-e2e.config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
/**
gziolo marked this conversation as resolved.
Show resolved Hide resolved
* External dependencies
*/
const path = require( 'path' );

/**
* Internal dependencies
*/
const { hasBabelConfig } = require( '../utils' );

const jestE2EConfig = {
preset: 'jest-puppeteer',
testMatch: [
'**/__tests__/**/*.js',
'**/?(*.)(spec|test).js',
'**/test/*.js',
],
};

if ( ! hasBabelConfig() ) {
jestE2EConfig.transform = {
'^.+\\.jsx?$': path.join( __dirname, 'babel-transform' ),
};
}

module.exports = jestE2EConfig;
21 changes: 21 additions & 0 deletions packages/scripts/config/jest-unit.config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
/**
* External dependencies
*/
const path = require( 'path' );

/**
* Internal dependencies
*/
const { hasBabelConfig } = require( '../utils' );

const jestUnitConfig = {
preset: '@wordpress/jest-preset-default',
};

if ( ! hasBabelConfig() ) {
jestUnitConfig.transform = {
'^.+\\.jsx?$': path.join( __dirname, 'babel-transform' ),
};
}

module.exports = jestUnitConfig;
28 changes: 0 additions & 28 deletions packages/scripts/config/jest.config.js

This file was deleted.

2 changes: 2 additions & 0 deletions packages/scripts/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,9 @@
"cross-spawn": "^5.1.0",
"eslint": "^4.19.1",
"jest": "^23.6.0",
"jest-puppeteer": "3.2.1",
"npm-package-json-lint": "^3.3.1",
"puppeteer": "1.6.1",
"read-pkg-up": "^1.0.1",
"resolve-bin": "^0.4.0"
},
Expand Down
45 changes: 45 additions & 0 deletions packages/scripts/scripts/test-e2e.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
// Do this as the first thing so that any code reading it knows the right env.
process.env.BABEL_ENV = 'test';
process.env.NODE_ENV = 'test';

// Makes the script crash on unhandled rejections instead of silently
// ignoring them. In the future, promise rejections that are not handled will
// terminate the Node.js process with a non-zero exit code.
process.on( 'unhandledRejection', ( err ) => {
throw err;
} );

/**
* External dependencies
*/
const jest = require( 'jest' );

/**
* Internal dependencies
*/
const {
fromConfigRoot,
getCliArgs,
hasCliArg,
hasProjectFile,
hasJestConfig,
} = require( '../utils' );

// Provides a default config path for Puppeteer when jest-puppeteer.config.js
// wasn't found at the root of the project or a custom path wasn't defined
// using JEST_PUPPETEER_CONFIG environment variable.
if ( ! hasProjectFile( 'jest-puppeteer.config.js' ) && ! process.env.JEST_PUPPETEER_CONFIG ) {
process.env.JEST_PUPPETEER_CONFIG = fromConfigRoot( 'puppeteer.config.js' );
}

const config = ! hasJestConfig() ?
Copy link
Member

@aduth aduth May 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For end-to-end tests, should this be checking for jest.config.js (or package.json jest)? Or only jest-puppeteer.config.js? How should we expect to give people control over whether to override configuration for either unit tests or end-to-end tests?

I'm running into this in my Dones project, where I do choose to override the default configuration for unit tests, but unfortunately it seems to bleed over into end-to-end test scripts as well, where I would prefer to simply use the default. Or at least, it's not clear how I can configure for both unit tests and end-to-end tests.

It kinda feels like this should be limited to checking for the presence of jest-e2e.config.js perhaps?

Copy link
Member Author

@gziolo gziolo May 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It kinda feels like this should be limited to checking for the presence of jest-e2e.config.js perhaps?

An interesting challenge and I'm surprised it didn't come up earlier :)

How about we add higher priority for non-standard:

  • jest-e2e.config.js
  • jest-unit.config.js

To give folks a way to target only one or both of those tools without the need for using --config flag.

As a temporary workaround, you can use the non-standard name of the file and pass it as a config option to the test script. In fact, this is how it works in Gutenberg:

"test-e2e": "wp-scripts test-e2e --config packages/e2e-tests/jest.config.js",

"test-unit": "wp-scripts test-unit-js --config test/unit/jest.config.js",

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In retrospect, it seems a bit more obvious in retrospect in in observing that the default configurations folder includes variations of these two files (jest-e2e.config.js, jest-unit.config.js), yet only one is supported for end-users. It's an interesting exercise to consider for other potential issues. Is there a similar problem with .eslintrc-md.js and .eslintrc.js ?

How about we add higher priority for non-standard:

I think this could be a good solution. I was worried if it might have to be a "Breaking Change" if we only start considering one of these two, but (a) IIRC jest.config.js is a "standard" file supported by Jest and (b) it could be viewed as a rare, advanced use-case where one opts to configure both.

As a temporary workaround, you can use the non-standard name of the file and pass it as a config option to the test script. In fact, this is how it works in Gutenberg:

Yeah, I ended up using something like this, or at least observing that higher priority is given to --config, also observing that the E2E configuration is quite small and I wasn't entirely on board with the specs forced folder convention of the default configuration 😅

wp-scripts test-e2e --config {\"preset\":\"jest-puppeteer\"} e2e

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request: #22477

[ '--config', JSON.stringify( require( fromConfigRoot( 'jest-e2e.config.js' ) ) ) ] :
[];

const hasRunInBand = hasCliArg( '--runInBand' ) ||
hasCliArg( '-i' );
const runInBand = ! hasRunInBand ?
[ '--runInBand' ] :
[];

jest.run( [ ...config, ...runInBand, ...getCliArgs() ] );
19 changes: 5 additions & 14 deletions packages/scripts/scripts/test-unit-jest.js
Original file line number Diff line number Diff line change
Expand Up @@ -18,22 +18,13 @@ const jest = require( 'jest' );
* Internal dependencies
gziolo marked this conversation as resolved.
Show resolved Hide resolved
*/
const {
fromConfigRoot,
getCliArgs,
hasCliArg,
hasProjectFile,
hasPackageProp,
hasJestConfig,
} = require( '../utils' );

const args = getCliArgs();

const hasJestConfig = hasCliArg( '-c' ) ||
hasCliArg( '--config' ) ||
hasProjectFile( 'jest.config.js' ) ||
hasProjectFile( 'jest.config.json' ) ||
hasPackageProp( 'jest' );

const config = ! hasJestConfig ?
[ '--config', JSON.stringify( require( '../config/jest.config' ) ) ] :
const config = ! hasJestConfig() ?
[ '--config', JSON.stringify( require( fromConfigRoot( 'jest-unit.config.js' ) ) ) ] :
[];

jest.run( [ ...config, ...args ] );
jest.run( [ ...config, ...getCliArgs() ] );
Loading