Test: Collapse passed tests in Travis. #16755

epiqueras · 2019-07-25T16:58:11Z

Description

This PR replaces the default jest reporter with a subclass that folds passed tests when running in Travis CI.

How has this been tested?

Tests were run locally and the default reporter was used. Now we need to verify the new reporter works as expected in Travis.

Types of Changes

Build Tooling: Collapse passed tests in Travis.

Checklist:

My code is tested.
My code follows the WordPress code style.
My code follows the accessibility standards.
My code has proper inline documentation.
I've included developer documentation if appropriate.

epiqueras · 2019-07-25T18:50:21Z

It works! 🎉

epiqueras · 2019-07-25T20:10:25Z

Now the fold has an actual description/header 🚀

nerrad

Nice! The only question I have with this is whether it's possible to (or should) be included in the @wordpress/jest-preset-defaults package instead? That way the alternative reporter is available in a wider scope?

epiqueras · 2019-07-26T11:38:33Z

Nice! The only question I have with this is whether it's possible to (or should) be included in the @wordpress/jest-preset-defaults package instead? That way the alternative reporter is available in a wider scope?

I added it to all the test suites in this repo. I don't know if we want it to be part of the package that other repos might use or if it's something that should be specific to this one. Personally, I do think that having it in the package is a good idea.

We can also always easily move it later.

cc @youknowriad

youknowriad · 2019-07-26T11:39:24Z

Having it in the preset works for me 👍

epiqueras · 2019-07-26T14:31:06Z

Done 🎉

nerrad · 2019-07-26T14:44:48Z

packages/jest-preset-default/jest-preset.json

@@ -23,5 +23,6 @@
 	"transform": {
 		"^.+\\.[jt]sx?$": "<rootDir>/node_modules/babel-jest"
 	},
-	"verbose": true
+	"verbose": true,
+	"reporters": [ "../../../@wordpress/jest-preset-default/scripts/travis-fold-passes-reporter.js" ]


hmm... is this path right?

Yes, <rootDir> does not work there in .json files for some reason.

Did you test it outside of Gutenberg repository? In other places we use <rootDir>, e.g.:
<rootDir>/node_modules/@wordpress/jest-preset-default/scripts/setup-globals.js

As far as I remember this is the only way to make it work for Jest presets. At least this was the case in the past.

huh, I would have thought the path would be relative to the jest-preset.json file. I'm just wondering how well/if this will work things consuming this package (other than gutenberg). So with this being a relative path, would ./scripts/travis-fold-passes-reporter.js not work?

It's relative to the core jest package.

https://github.com/facebook/jest/blob/5177236f31c705f1084822a9bb7b7da976a420f6/packages/jest-core/src/TestScheduler.ts#L311

It looks like they migrated to jest-preset.js in this PR: thymikee/jest-preset-angular#204. I assume Jest improved the handling for presets.

Still doesn't replace <rootDir>.

@gziolo might be suggesting to resolve it using something like require.resolve in JavaScript?

require.resolve( '@wordpress/jest-preset-default/scripts/travis-fold-passes-reporter' )

We should use <rootDir>/node_modules/@wordpress/jest-preset-default/scripts/... like with the other paths in this file, but first we need to update the package so that the new reporter file is present in the directory in node_modules.

I don't remember :) The way it's implemented at the moment, it won't work when installed from npm for all projects using wp-scripts test-unit-js.

aduth

In this branch, when I run tests locally, I see the "Pass" label, but no details of the specific tests.

This is without any TRAVIS environment variable.

 npm run test-unit packages/blocks/src/api/test/validation.js

> gutenberg@6.9.0 test-unit /Users/andrew/Documents/Code/gutenberg
> wp-scripts test-unit-js --config test/unit/jest.config.js "packages/blocks/src/api/test/validation.js"

 PASS  packages/blocks/src/api/test/validation.js

packages/jest-preset-default/scripts/travis-fold-passes-reporter.js

aduth · 2019-11-22T22:58:30Z

packages/jest-preset-default/jest-preset.json

@@ -23,5 +23,6 @@
 	"transform": {
 		"^.+\\.[jt]sx?$": "<rootDir>/node_modules/babel-jest"
 	},
-	"verbose": true
+	"verbose": true,
+	"reporters": [ "../../../@wordpress/jest-preset-default/scripts/travis-fold-passes-reporter.js" ]


@gziolo might be suggesting to resolve it using something like require.resolve in JavaScript?

require.resolve( '@wordpress/jest-preset-default/scripts/travis-fold-passes-reporter' )

epiqueras · 2019-11-23T01:07:05Z

In this branch, when I run tests locally, I see the "Pass" label, but no details of the specific tests.

Good catch. We use verbose: true so I should have extended and exported the VerboseReporter instead of the DefaultReporter, try it now.

gziolo · 2019-11-24T20:02:57Z

Can this be simplified to conditionally override the reported in this config file:
https://github.com/WordPress/gutenberg/blob/master/test/unit/jest.config.js?

This way this change will be isolated to Gutenberg. This will prevent enforcing the verbose reporter when running locally, make the issue with providing the path to the custom reporter no longer present.

epiqueras · 2019-11-24T20:06:52Z

This will prevent enforcing the verbose reporter when running locally

We already do this with verbose: true.

make the issue with providing the path to the custom reporter no longer valid

This is only temporary, until the file is in npm/node_modules.

aduth · 2019-12-02T22:35:43Z

packages/jest-preset-default/jest-preset.json

@@ -27,5 +27,6 @@
 	"transform": {
 		"^.+\\.[jt]sx?$": "<rootDir>/node_modules/babel-jest"
 	},
-	"verbose": true
+	"verbose": true,


With the latest changes, I wonder if this line is even necessary. And then, it gets me considering, if someone was to use this preset as a base, and then override "verbose": false, would it actually disable the VerboseRunner from being used?

It seems like something where the additional reporter shouldn't be making any assumptions outside its own custom behavior, which we could do if we included "default" in the reporters array:

https://jestjs.io/docs/en/configuration#reporters-arraymodulename--modulename-options

I suppose the problem then is: What would the export of travis-folder-passes.reporter.js be in the else condition of 'TRAVIS' in process.env ?

I could see a few options:

Move the condition into each of the onTestResult and onRunComplete to treat them as a noop.

Maybe it's enough to revert back to extending DefaultReporter if we include 'default' in the reporters configuration?

Or, we could convert this file to a JavaScript file, and conditionally include the custom reporter path based on the process.env condition.

Move the condition into each of the onTestResult and onRunComplete to treat them as a noop.

We still need to extend either the verbose or default reporter to get some output without running 2 reporters in Travis.

Maybe it's enough to revert back to extending DefaultReporter if we include 'default' in the reporters configuration?

Likewise, that would run the 2 reporters.

Or, we could convert this file to a JavaScript file, and conditionally include the custom reporter path based on the process.env condition.

That would work.

A potentially easier solution that respects the actual configuration would be to read the current Jest config, which reporters are constructed with:

https://github.com/facebook/jest/blob/ad5377333daf6716af3465bba39f86b7db485e2b/packages/jest-reporters/src/verbose_reporter.ts#L25

Seems to work: f7c72e7.

I think it can work. It still doesn't quite sit well with me, since it makes assumptions about what the default behavior would be from our custom code, rather than just letting Jest deal with the defaults. Converting jest-preset.json to a JavaScript file and manipulating the reporters array feels like a more durable long-term solution. I'm fine to explore that separately.

aduth · 2019-12-03T15:06:07Z

In the failing end-to-end test (which itself doesn't appear to be related), I see that the passes aren't collapsed. Do we use a separate configuration for this?

See: https://travis-ci.com/WordPress/gutenberg/jobs/262393864

epiqueras · 2019-12-03T17:20:59Z

In the failing end-to-end test (which itself doesn't appear to be related), I see that the passes aren't collapsed. Do we use a separate configuration for this?

It looks like E2E tests are using the jest-puppeteer preset instead of our @wordpress/jest-preset-default.

epiqueras · 2019-12-03T17:21:30Z

Can we update that in a follow-up PR?

aduth

Sure

epiqueras added [Type] Build Tooling Issues or PRs related to build tooling [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. labels Jul 25, 2019

epiqueras added this to the Future milestone Jul 25, 2019

epiqueras requested review from ajitbohra, gziolo, nerrad, ntwb, talldan and youknowriad as code owners July 25, 2019 16:58

epiqueras self-assigned this Jul 25, 2019

epiqueras force-pushed the try/collapsing-passed-tests-in-travis branch 3 times, most recently from 1191e78 to 7aaf678 Compare July 25, 2019 20:02

nerrad reviewed Jul 26, 2019

View reviewed changes

epiqueras force-pushed the try/collapsing-passed-tests-in-travis branch from 7aaf678 to be26416 Compare July 26, 2019 14:30

nerrad reviewed Jul 26, 2019

View reviewed changes

epiqueras requested a review from aduth November 20, 2019 22:38

aduth reviewed Nov 22, 2019

View reviewed changes

epiqueras added 2 commits November 22, 2019 16:43

Test: Collapse passed tests in Travis.

5907738

Jest Preset Default: Use verbose reporter and simplify code.

8d8024c

epiqueras force-pushed the try/collapsing-passed-tests-in-travis branch from be26416 to 8d8024c Compare November 23, 2019 01:03

aduth reviewed Dec 2, 2019

View reviewed changes

Jest Preset Default: Respect actual verbose setting.

f7c72e7

aduth approved these changes Dec 3, 2019

View reviewed changes

epiqueras merged commit ed8b4a5 into master Dec 3, 2019

epiqueras deleted the try/collapsing-passed-tests-in-travis branch December 3, 2019 17:24

epiqueras mentioned this pull request Dec 3, 2019

Follow-up: Test: Collapse passed tests in Travis. #18896

Merged

youknowriad modified the milestones: Future, Gutenberg 7.1 Dec 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test: Collapse passed tests in Travis. #16755

Test: Collapse passed tests in Travis. #16755

epiqueras commented Jul 25, 2019

epiqueras commented Jul 25, 2019

epiqueras commented Jul 25, 2019

nerrad left a comment

epiqueras commented Jul 26, 2019

youknowriad commented Jul 26, 2019

epiqueras commented Jul 26, 2019

nerrad Jul 26, 2019

epiqueras Jul 26, 2019

gziolo Jul 26, 2019

nerrad Jul 26, 2019

epiqueras Jul 26, 2019

gziolo Jul 26, 2019

epiqueras Jul 26, 2019

aduth Nov 22, 2019

epiqueras Nov 23, 2019

gziolo Nov 24, 2019

aduth left a comment

aduth Nov 22, 2019

epiqueras commented Nov 23, 2019

gziolo commented Nov 24, 2019 •

edited

Loading

epiqueras commented Nov 24, 2019

aduth Dec 2, 2019

epiqueras Dec 2, 2019

epiqueras Dec 2, 2019

aduth Dec 3, 2019

epiqueras Dec 3, 2019

epiqueras Dec 3, 2019

aduth commented Dec 3, 2019

epiqueras commented Dec 3, 2019

epiqueras commented Dec 3, 2019

aduth left a comment

Test: Collapse passed tests in Travis. #16755

Test: Collapse passed tests in Travis. #16755

Conversation

epiqueras commented Jul 25, 2019

Description

How has this been tested?

Types of Changes

Checklist:

epiqueras commented Jul 25, 2019

epiqueras commented Jul 25, 2019

nerrad left a comment

Choose a reason for hiding this comment

epiqueras commented Jul 26, 2019

youknowriad commented Jul 26, 2019

epiqueras commented Jul 26, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aduth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

epiqueras commented Nov 23, 2019

gziolo commented Nov 24, 2019 • edited Loading

epiqueras commented Nov 24, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aduth commented Dec 3, 2019

epiqueras commented Dec 3, 2019

epiqueras commented Dec 3, 2019

aduth left a comment

Choose a reason for hiding this comment

gziolo commented Nov 24, 2019 •

edited

Loading