Add combined Jest+Cypress code coverage reports #5262

cee-chen · 2021-10-12T20:46:03Z

Summary

While working on #5261, I wanted to calculate combined coverage to make sure I wasn't losing any uncovered lines when converting the focus behavior tests to Cypress. This PR adds a Cypress code coverage plugin and then a yarn combine-test-coverage script that combines our Cypress+Jest output into a single HTML report. It additionally adds wiki documentation for how to access our HTML code coverage reports.

I recommend following along by commit, as I added some misc. cleanup as part of this PR as well.

Example

I cherry-picked #5261 into this branch locally to get an example of how updating a subsection of Jest tests to Cypress would affect final combined coverage. Coverage remained approximately the same (with a 1% increase from adding Cypress!)

Before with just Jest coverage

After

Jest coverage:

Cypress coverage:

Combined coverage:

QAing locally

checkout this branch
yarn to install new dependencies
Run yarn test-unit --coverage to generate Jest coverage reports
Run yarn test-cypress which will automatically generate Cypress coverage reports
Run yarn combine-test-coverage, which should automatically open the main coverage report HTML page in your browser

Resources

Reference links to the resources I used while I was setting this up:

Cypress code coverage reports: [1], [2]
Jest+Cypress combined coverage reports: [1]
Storyboook+Jest+Cypress combined coverage, if we're ever interested in that someday: [1]

Checklist

N/A, internal/dev-only change

@see https://github.com/cypress-io/code-coverage#instrument-unit-tests

-into a single json & html report

- Ignore new `.nyc_output` dir generated by cypress/istanbul - Improve Jest config collectCoverageFrom !ignores: - DRY out component+services folder patterns with a {,} glob - Ignore new Cypress .spec files and .testenv files for jest coverage - eslintignore the cypress folder, since they're all .js anyway and it has a decent amount of idiosyncracies + cleanup unused eslint-disable + cleanup cypress comment boilerplate + remove unused paths from webpack config

package.json

scripts/combine-coverage.js

kibanamachine · 2021-10-12T21:13:55Z

Preview documentation changes for this PR: https://eui.elastic.co/pr_5262/

chandlerprall

Changes LGTM, looks like we want to pull the nyc config to a js file before merging?

thompsongl · 2021-10-20T16:00:22Z

I ran yarn combine-test-coverage locally and it errored out. So are yarn test-unit --coverage and yarn test-cypress prerequisites (it worked after running both)?
Any thought on having yarn combine-test-coverage run the prereqs also? Or adding another command that will run all three?

kibanamachine · 2021-10-20T16:08:03Z

Preview documentation changes for this PR: https://eui.elastic.co/pr_5262/

cee-chen · 2021-10-20T16:08:28Z

I ran yarn combine-test-coverage locally and it errored out. So are yarn test-unit --coverage and yarn test-cypress prerequisites (it worked after running both)?

Yes, that's documented in both in the new wiki docs I added and the script itself:

eui/scripts/combine-coverage.sh

Lines 4 to 5 in 52b1244

    
           # NOTE: `yarn test-cypress` and `yarn test-unit --coverage` must have already been run 
        
           # in order for the correct coverage JSON files to exist and be combined

Any thought on having yarn combine-test-coverage run the prereqs also?

Not realistically. That would literally require running every single jest test and cypress test which would result in the command potentially taking over 30 minutes. All the combine script does is combine the existing coverage reports that already exist in your local reports/ folder.

Additionally, making the command agnostic allows devs to only run coverage for specific tests that they're developing on (e.g. if you're working on just the focus_trap component, you'd do yarn test-unit focus_trap --coverage && yarn test-cypress --spec focus_trap.spec.tsx && yarn combine-coverage and then you would obtain coverage for only the component you care about without wasting extra time.

It's fairly manual, but this PR at least gets us in the right direction. IMO, I think the most helpful dev QOL step from here would be to add some kind of --watch flag using nodemon or other watch plugin, and dynamically respond/update the combined coverage whenever reports/jest-coverage/coverage-final.json and reports/cypress-coverage/coverage-final.json change.

thompsongl

Thanks for the clarification! It would help if I read the docs before trying to use the thing 😅

cee-chen · 2021-10-20T16:28:22Z

No worries!! Also whoops, I just realized I mashed auto merge without an actual approval, super sorry y'all 🙈 I'm totally out of wack this morning and somehow mistook Chandler's comment for an approval, my bad!

thompsongl · 2021-10-20T16:32:27Z

My last comment was an approval! All good ✅

* Install Cypress code coverage dependencies * Set up required babel+istanbul config * Set up Cypress code coverage config @see https://github.com/cypress-io/code-coverage#instrument-unit-tests * Add script for combining cypress & jest code coverage -into a single json & html report * Misc cleanup/ignores - Ignore new `.nyc_output` dir generated by cypress/istanbul - Improve Jest config collectCoverageFrom !ignores: - DRY out component+services folder patterns with a {,} glob - Ignore new Cypress .spec files and .testenv files for jest coverage - eslintignore the cypress folder, since they're all .js anyway and it has a decent amount of idiosyncracies + cleanup unused eslint-disable + cleanup cypress comment boilerplate + remove unused paths from webpack config * Add wiki documentation * [PR feedback] Move nyc config to own file * [PR feedback] Switch combine coverage script to .sh

* Provide rows for datagrid cells to be owned by * changelog * undoing things * working except virtualized container * working row wrapper implementation * Create datagrid row elements on-demand and render cells via portals * Provide rows for datagrid cells to be owned by * changelog * undoing things * working except virtualized container * working row wrapper implementation * Create datagrid row elements on-demand and render cells via portals * Small style cleanup * updated changelog * fixing a bad changelog merge * oh that style was important * Update src/components/datagrid/body/data_grid_row_manager.ts Co-authored-by: Constance <constancecchen@users.noreply.github.com> * Update src/components/datagrid/data_grid_types.ts Co-authored-by: Constance <constancecchen@users.noreply.github.com> * Ensure unique ID in combobox with prepended or appended labels (#5229) * Making an explicit check for ID prop in EUI Combobox * Checking for a user-passed ID prop * If no user-passed ID, we'll use the `rootId()` class method to set one * Prepend and append props need an ID to relate label to the input * Adding example for Combobox with prepended label * Confirmed the label `for` attribute is referencing input `id` * Added styles to round the edges of the child `<div>` when we prepend a label * Bumped package.json * Adding CHANGELOG entry and bumping version * Update combo_box.tsx * Updating styles to match other forms with prepend and apppend. * Finishing Combobox CSS and test updates * Simplified the Amsterdam overrides for prepended / appended labels * Added conditional classes to `combo_box.tsx` * Added unit tests for user passed ID, prepended and appended classes * Reordering props in child component. * Final adjustment to ID check in ComboBox * Final updates to styling and documentation * Added CSS rules for compressed, prepend and append border radius * Updated the Display Toggles to include append * Removed a duplicated instruction paragraph * Simplified the logic for a unique ID in combobox.ts * Adding Combobox unique ID to master changelog * Updated Form Control Layout SCSS * The borders were still rounded with prepend and append enabled * Added two more rules to make borders square for full height and compressed * Move to _combo_box.scss Co-authored-by: cchaos <caroline.horn@elastic.co> * update i18ntokens * 40.0.0 * Updated documentation. * Add combined Jest+Cypress code coverage reports (#5262) * Install Cypress code coverage dependencies * Set up required babel+istanbul config * Set up Cypress code coverage config @see https://github.com/cypress-io/code-coverage#instrument-unit-tests * Add script for combining cypress & jest code coverage -into a single json & html report * Misc cleanup/ignores - Ignore new `.nyc_output` dir generated by cypress/istanbul - Improve Jest config collectCoverageFrom !ignores: - DRY out component+services folder patterns with a {,} glob - Ignore new Cypress .spec files and .testenv files for jest coverage - eslintignore the cypress folder, since they're all .js anyway and it has a decent amount of idiosyncracies + cleanup unused eslint-disable + cleanup cypress comment boilerplate + remove unused paths from webpack config * Add wiki documentation * [PR feedback] Move nyc config to own file * [PR feedback] Switch combine coverage script to .sh * [EuiMarkdownEditor] Add `remark-breaks` and line break plugin (#5272) * Add remark-breaks and line break plugin to MarkdownEditor * Update changelog * [Docs] Separated out Borders to its own page & [EuiTableRowCell] fixes (#5283) * Moved Border tokens to its own page * Figured out a way to get subsections in the sidenav working * Added the new `_values` under `customizing/` * [EuiTableRowCell] Actually support `valign` property manually * [EuiTable*] Fixing some props and mobile options * Slightly better responsive table styles * Fix CL from #5272 * [Docs] Update EuiDatePicker types (#5318) * omit unsupported types * resolve console errors * CL * [Cypress] Add flakey test retries + harden intermittent context menu failures (#5317) * Add Cypress failed test retries to CI run mode * Increase cy.wait on flaky context menu tests (just in case) * More hardening * [Docs] Fix Colors guidelines (#5316) * use useJsonVars * use variable * Update dependency @elastic/charts to ^38.1.0 (#5321) * PR feedback * Added a cypress test for datagrid row rendering * Revert changes(?) to docs/ * revert changelog reformats * re-ignore a couple datagrid example pages when running a11y tests * Alternate way to know if the datagrid cells have rendered * PR feedback * changelog Co-authored-by: Constance <constancecchen@users.noreply.github.com> Co-authored-by: Trevor Pierce <1Copenut@users.noreply.github.com> Co-authored-by: cchaos <caroline.horn@elastic.co> Co-authored-by: ian moersen <ian@union.io> Co-authored-by: Caroline Horn <549577+cchaos@users.noreply.github.com> Co-authored-by: Greg Thompson <thompson.glowe@gmail.com> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

cee-chen added 5 commits October 12, 2021 11:52

Install Cypress code coverage dependencies

842bfa5

Set up required babel+istanbul config

07e8240

Set up Cypress code coverage config

c29be00

@see https://github.com/cypress-io/code-coverage#instrument-unit-tests

Add script for combining cypress & jest code coverage

2555559

-into a single json & html report

cee-chen added skip-changelog tech debt labels Oct 12, 2021

cee-chen force-pushed the cypress-code-coverage branch from 2eddfa3 to ff885b1 Compare October 12, 2021 20:48

Add wiki documentation

6716e63

cee-chen force-pushed the cypress-code-coverage branch from ff885b1 to 6716e63 Compare October 12, 2021 20:48

cee-chen commented Oct 12, 2021

View reviewed changes

package.json Outdated Show resolved Hide resolved

cee-chen commented Oct 12, 2021

View reviewed changes

scripts/combine-coverage.js Outdated Show resolved Hide resolved

chandlerprall reviewed Oct 19, 2021

View reviewed changes

cee-chen added 3 commits October 20, 2021 08:26

Merge branch 'master' into cypress-code-coverage

7f18e9a

[PR feedback] Move nyc config to own file

5072463

[PR feedback] Switch combine coverage script to .sh

52b1244

cee-chen enabled auto-merge (squash) October 20, 2021 15:39

thompsongl approved these changes Oct 20, 2021

View reviewed changes

cee-chen merged commit f69e88d into elastic:master Oct 20, 2021

cee-chen deleted the cypress-code-coverage branch October 20, 2021 16:21

thompsongl mentioned this pull request Dec 6, 2021

2021 Roadmap - First 6 months #4054

Closed

24 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add combined Jest+Cypress code coverage reports #5262

Add combined Jest+Cypress code coverage reports #5262

cee-chen commented Oct 12, 2021 •

edited

Loading

kibanamachine commented Oct 12, 2021

chandlerprall left a comment

thompsongl commented Oct 20, 2021

kibanamachine commented Oct 20, 2021

cee-chen commented Oct 20, 2021 •

edited

Loading

thompsongl left a comment

cee-chen commented Oct 20, 2021

thompsongl commented Oct 20, 2021

Add combined Jest+Cypress code coverage reports #5262

Add combined Jest+Cypress code coverage reports #5262

Conversation

cee-chen commented Oct 12, 2021 • edited Loading

Summary

Example

Before with just Jest coverage

After

QAing locally

Resources

Checklist

kibanamachine commented Oct 12, 2021

chandlerprall left a comment

Choose a reason for hiding this comment

thompsongl commented Oct 20, 2021

kibanamachine commented Oct 20, 2021

cee-chen commented Oct 20, 2021 • edited Loading

thompsongl left a comment

Choose a reason for hiding this comment

cee-chen commented Oct 20, 2021

thompsongl commented Oct 20, 2021

cee-chen commented Oct 12, 2021 •

edited

Loading

cee-chen commented Oct 20, 2021 •

edited

Loading