Debugging coverage performance?

[mramato]

I'm not sure if an issue is the right way to ask for help, if it's not I apologize, but I couldn't find anyplace better.

I'm a maintainer on the Cesium project and we've recently switched our 200,000 line JavaScript code base over to use karma (over 7200 jasmine unit tests). The entire test suite runs in less than 40 seconds.

I've used istanbul in my other (smaller) projects, and love it. That being said, we are now trying to switch over code coverage in Cesium from an ancient version of JSCoverage to istanbul via karma-coverage and running into a major performance hurdle. Our old system takes about 90 seconds total to instrument and cover. However, running the same tests via istanbul takes over 7 minutes (and only if I comment a couple of tests that were taking upwards of 5 minutes each). I also had to jack up browserNoActivityTimeout and browserDisconnectTimeout in karma just to try and get it to run to completion (and that has only happened once so far and took 10 minutes, the results were not repeatable.

I can't say for certain, but it seems earlier tests run way faster and things just get slower as the tests go on. A few seemingly innocuous tests seem to take multiple minutes to complete on their own (but execute in seconds when not run under coverage). I'm not sure it's any particular test causing issues though.

I've tried multiple launchers and browsers in case that was the problem (we usually use Chrome by default), but all browsers exhibited the same performance issue.

It happens on both Windows and Linux.

While I don't expect anyone to magically solve my problem, I am hoping that someone can provide tips for tracking down whatever the root cause of the problem may be. Perhaps there are some options I can tune or extra logging commands I can run to figure out what is taking so long. I'd be more than happy to submit a pull request with performance improvements if it turns out that is the problem (vs some bug slowing things down)

Here's the relevant configuration we are using:

https://github.com/AnalyticalGraphicsInc/cesium/blob/coverage/Specs/karma-main.js
https://github.com/AnalyticalGraphicsInc/cesium/blob/coverage/Specs/karma.conf.js
https://github.com/AnalyticalGraphicsInc/cesium/blob/coverage/gulpfile.js#L281

Any help would be greatly appreciated, and thanks again for a great project.

[kirbysayshi]

@mramato I just ran into a performance problem, and I'm hoping my findings can help you.

Istanbul uses https://github.com/mklabs/node-fileset at startup, and inputs **/*.js as the include parameter by default. It also inputs **/node_modules/** as the exclude parameter by default.

In theory this seems perfectly fine. But exclude is actually a post-read filter!

Which means that at startup, even though node_modules is listed as an exclude, it still traverses the entire folder.

On our codebase at work, the time it takes to do so is quite extreme (these are the defaults istanbul uses https://github.com/istanbuljs/istanbul-api/blob/929abf6922e21eebcea694e9f8b0058fbfb207d7/lib/file-matcher.js#L25-L32):

$ time node -e 'require("fileset")("**/*.js", "**/node_modules/**", function () {})'

real  0m9.743s
user 0m7.807s
sys   0m2.651s

Removing the **/*.js and being just a bit more specific greatly helps:

$ time node -e 'require("fileset")("src/**/*.js", "**/node_modules/**", function () {})'

real  0m0.654s
user 0m0.529s
sys   0m0.192s

So I fixed this by changing my istanbul invocation:

instead of: istanbul cover _mocha

do: istanbul cover -i 'src/**/*.js' _mocha

Execution time dropped in half!

[mramato]

Thanks for the suggestion @kirbysayshi, unfortunately your issue doesn't appear to be the source of my problem. I'm not running the cover on the command line, but instead running programatically through karma. I was also already specifying my own include filter. I spent some time debugging and tweaking the code (I even hardcoded some changes in file-matcher) but nothing in that area had any affect. Thanks anyway.

[kirbysayshi]

Sorry to hear that @mramato ! Did you try hard coding a list of files for file-matcher to return? Just to remove any chance of this issue being the source of your performance woes?

[mramato]

Did you try hard coding a list of files for file-matcher to return?

Yes,I went into the node_modules directory and hardcoded values, but as far as I could tell running with karma doesn't even take this code path (because it uses the programmatic API to istanbul and file-matcher.js only appears to get called when using the command line (as far as I can tell).

Thanks again anyway.