doc: instructions for generating coverage reports #15190

ssbrewster · 2017-09-04T22:37:00Z

This PR adds instructions for generating coverage reports. I only added the instructions to BUILDING.md but not CONTRIBUTING.md because that defers to BUILDING.md in the Test section (See the BUILDING.md for more details.), but I'm happy to add the instructions to that document too if people think they're needed there.

Checklist

make -j4 test
documentation is changed or added
commit message follows commit guidelines

Affected core subsystem(s)

doc

refack

LGTM % a wording suggestion & Windows fix

refack · 2017-09-04T23:47:18Z

BUILDING.md

@@ -188,6 +199,12 @@ To run the tests:
 > .\vcbuild test
 ```

+To run the tests and generate coverage reports:


Windows (i.e. vcbuild) does not have a counterpart, so you can remove this section

Thanks for your review @refack I will update the doc.

refack · 2017-09-04T23:50:29Z

BUILDING.md

+$ make coverage
+```
+
+This will generate coverage reports for both JavaScript and C++ tests (if you


Suggestion:

... (if you only want to generate a coverage report for the JavaScript tests then you do not need to run the first command, i.e. `configure`).

refack · 2017-09-04T23:51:40Z

@ssbrewster Thanks for this.

TimothyGu · 2017-09-05T00:16:40Z

BUILDING.md

@@ -126,6 +126,17 @@ To run the tests:
 $ make test
 ```

+To run the tests and generate code coverage reports:


Hey, thanks for writing it up!

From my limited experience with make coverage, it can have some quite surprising behavior. I'd suggest noting the following caveats in this section of the docs. Whether to note some or all of them is up to you!

~~It doesn't work on macOS (I think? /cc @Trott)~~ It does!

The lib/ directory is overwritten during make coverage, and the original copied to lib_/. I'd always suggest checking in the lib/ directory to Git before running coverage reports in case something awry happens. See

node/Makefile

Lines 151 to 152 in 86e7c61

if [ -d lib_ ]; then $(RM) -r lib; mv lib_ lib; fi

mv lib lib_

.

The initial make coverage run downloads some tools to the project root folder that are not .gitignored, which can be a nuisance.

All effects of make coverage (lib/ renaming, downloaded tools, raw and processed output) can be reverted with make coverage-clean

make coverage both builds and runs (i.e. make + make test); no way to split them properly unfortunately (one could coverage-build first, but when coverage-testing the JS files will get instrumented again, and the node binary rebuilt based on the instrumented JS files).

Thanks for your review @TimothyGu. I will document make coverage-clean. I'm also wondering if we should handle setting clean as an option on the command line i.e. ~~make coverage --clean=true~~ make coverage CLEAN=true so that the coverage reports can be generated and the clean-up done in one run.

I think this would be good but coverage-clean also removes the coverage/ dir with the reports on so if passing the option ~~--clean~~ CLEAN=true then we would want to leave the coverage/ dir intact. I don't think I see this as an issue as long as that dir is added to .gitignore. What do you think?

Also, I just noticed that on the install of istanbul-merge and nyc, npm generates a package-lock.json which isn't removed by make coverage-clean.

I'm also wondering if we should handle setting clean as an option on the command line i.e. make coverage --clean=true

That would be nice, but GNU Make doesn't support doing that AFAIK. But GNU Make does support environment variables so you can do make coverage CLEAN=true. PR welcome!

Also, I just noticed that on the install of istanbul-merge and nyc, npm generates a package-lock.json which isn't removed by make coverage-clean.

Good catch! You can submit another pull request to change that if you'd like to, or if not please file an issue about it so we don't forget :)

Great I will raise two other PRs and yes you're right my syntax was wrong on make coverage CLEAN=true :)

I'll wait to this to land before submitting the PR for handling passing clean as an option so that I can update the docs for that.

Trott · 2017-09-05T05:58:18Z

It doesn't work on macOS (I think? /cc @Trott)

Seems to work fine for me on macOS so no need for that caveat as far as I can tell.

Add instructions for generating code coverage reports to BUILDING.md

mhdawson

LGTM

Add instructions for generating code coverage reports to BUILDING.md PR-URL: #15190 Reviewed-By: Timothy Gu <timothygu99@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com> Reviewed-By: James M Snell <jasnell@gmail.com>

jasnell · 2017-09-07T05:15:57Z

Landed in 6e27fd7

Add instructions for generating code coverage reports to BUILDING.md PR-URL: #15190 Reviewed-By: Timothy Gu <timothygu99@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Add instructions for generating code coverage reports to BUILDING.md PR-URL: nodejs#15190 Reviewed-By: Timothy Gu <timothygu99@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com> Reviewed-By: James M Snell <jasnell@gmail.com>

Add instructions for generating code coverage reports to BUILDING.md PR-URL: #15190 Reviewed-By: Timothy Gu <timothygu99@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Michael Dawson <michael_dawson@ca.ibm.com> Reviewed-By: James M Snell <jasnell@gmail.com>

nodejs-github-bot added build Issues and PRs related to build files or the CI. doc Issues and PRs related to the documentations. labels Sep 4, 2017

refack reviewed Sep 4, 2017

View reviewed changes

TimothyGu reviewed Sep 5, 2017

View reviewed changes

ssbrewster force-pushed the doc-test-coverage branch from 040ec16 to 75d70bc Compare September 5, 2017 08:51

doc: instructions for generating coverage reports

0408aff

Add instructions for generating code coverage reports to BUILDING.md

ssbrewster force-pushed the doc-test-coverage branch from 75d70bc to 0408aff Compare September 5, 2017 08:53

doc: Document make coverage-clean

1a560f1

TimothyGu approved these changes Sep 5, 2017

View reviewed changes

lpinca approved these changes Sep 5, 2017

View reviewed changes

ssbrewster mentioned this pull request Sep 6, 2017

Cleaning up after make coverage #15214

Closed

mhdawson approved these changes Sep 6, 2017

View reviewed changes

jasnell approved these changes Sep 7, 2017

View reviewed changes

jasnell closed this Sep 7, 2017

ssbrewster deleted the doc-test-coverage branch September 8, 2017 06:13

MylesBorins mentioned this pull request Sep 10, 2017

v8.5.0 proposal #15308

Merged

MylesBorins added the land-on-v6.x label Sep 20, 2017

MylesBorins mentioned this pull request Sep 20, 2017

v6.11.4 proposal #15506

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

doc: instructions for generating coverage reports #15190

doc: instructions for generating coverage reports #15190

ssbrewster commented Sep 4, 2017

refack left a comment

refack Sep 4, 2017

ssbrewster Sep 5, 2017

refack Sep 4, 2017

refack commented Sep 4, 2017

TimothyGu Sep 5, 2017 •

edited

Loading

ssbrewster Sep 5, 2017 •

edited

Loading

ssbrewster Sep 5, 2017 •

edited

Loading

TimothyGu Sep 5, 2017 •

edited

Loading

ssbrewster Sep 5, 2017

ssbrewster Sep 5, 2017

Trott commented Sep 5, 2017

mhdawson left a comment

jasnell commented Sep 7, 2017

	if [ -d lib_ ]; then $(RM) -r lib; mv lib_ lib; fi
	mv lib lib_

doc: instructions for generating coverage reports #15190

doc: instructions for generating coverage reports #15190

Conversation

ssbrewster commented Sep 4, 2017

Checklist

Affected core subsystem(s)

refack left a comment

Choose a reason for hiding this comment

refack Sep 4, 2017

Choose a reason for hiding this comment

ssbrewster Sep 5, 2017

Choose a reason for hiding this comment

refack Sep 4, 2017

Choose a reason for hiding this comment

refack commented Sep 4, 2017

TimothyGu Sep 5, 2017 • edited Loading

Choose a reason for hiding this comment

ssbrewster Sep 5, 2017 • edited Loading

Choose a reason for hiding this comment

ssbrewster Sep 5, 2017 • edited Loading

Choose a reason for hiding this comment

TimothyGu Sep 5, 2017 • edited Loading

Choose a reason for hiding this comment

ssbrewster Sep 5, 2017

Choose a reason for hiding this comment

ssbrewster Sep 5, 2017

Choose a reason for hiding this comment

Trott commented Sep 5, 2017

mhdawson left a comment

Choose a reason for hiding this comment

jasnell commented Sep 7, 2017

TimothyGu Sep 5, 2017 •

edited

Loading

ssbrewster Sep 5, 2017 •

edited

Loading

ssbrewster Sep 5, 2017 •

edited

Loading

TimothyGu Sep 5, 2017 •

edited

Loading