Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Coverage reports wrong hit counter #25937

Closed
BridgeAR opened this issue Feb 5, 2019 · 20 comments
Closed

Coverage reports wrong hit counter #25937

BridgeAR opened this issue Feb 5, 2019 · 20 comments
Assignees
Labels
coverage Issues and PRs related to native coverage support. test Issues and PRs related to the tests.

Comments

@BridgeAR
Copy link
Member

BridgeAR commented Feb 5, 2019

I just looked at a coverage report and found the line hit counter to be wrong for all else cases. It seems to be an off by one error. The closing brackets from loops also seem to be counted wrong. I found a few more issues as well.

  1. An example for the first issue: https://coverage.nodejs.org/coverage-46af4c1d01b90ee6/lib/internal/util/comparisons.js.html#L522 and https://coverage.nodejs.org/coverage-46af4c1d01b90ee6/lib/internal/util/comparisons.js.html#L524. Line 522 should have the count from line 524 and line 524 should be 14 as line 525.

  2. Loops https://coverage.nodejs.org/coverage-46af4c1d01b90ee6/lib/internal/util/comparisons.js.html#L544 and https://coverage.nodejs.org/coverage-46af4c1d01b90ee6/lib/internal/util/comparisons.js.html#L549:

Here I would expect the hit counter to be 3186x instead of 6709x (the loop was reached 3186 times). It is also a kind of off by one error, just a different one.

  1. Something is also weird in this example: https://coverage.nodejs.org/coverage-46af4c1d01b90ee6/lib/internal/util/comparisons.js.html#L130. If the hit counter is indeed 46964 (which it should not be) than the next line should be zero but it is also counted as being hit 46964 times.

  2. Closing brackets in general do not seem to be counted properly: https://coverage.nodejs.org/coverage-46af4c1d01b90ee6/lib/internal/util/comparisons.js.html#L153 here the counter is 10 instead of 166 even though the if has been reached 166 times.

  3. Some numbers seem to increase magically, at least I did not find a logical correlation to them: https://coverage.nodejs.org/coverage-46af4c1d01b90ee6/lib/internal/util/comparisons.js.html#L134 reports 9134x which is logical due to the former 56098 - 46964. But the line afterwards reports 10263x?! How is it possible to increase the number of line hits in an if statement?
    The same with line 160 - 161.

  4. I am not sure about this one but I would expect function declarations to report the times evaluated instead of the times they are called. https://coverage.nodejs.org/coverage-46af4c1d01b90ee6/lib/internal/util/comparisons.js.html#L30. In that case the counter for line 30 should be 693x.

@Trott brought up another issue which is at least not ideal:

The line if (true && bar) would not complain that there's no branch coverage for the first condition to be falsy (which is off course impossible in my example).
Showing branch coverage like this would likely allow to identify more dead code.

// @bcoe @hashseed

@BridgeAR BridgeAR added test Issues and PRs related to the tests. coverage Issues and PRs related to native coverage support. labels Feb 5, 2019
@Trott
Copy link
Member

Trott commented Feb 5, 2019

@Trott brought up another issue which is at least not ideal:

The line if (true && bar) would not complain that there's no branch coverage for the first condition to be falsy (which is off course impossible in my example).
Showing branch coverage like this would likely allow to identify more dead code.

Code in file foo.js:

function testIt () {
 const thing = true && false;
 return true && false;
}

if (testIt())
  console.log('foo');

console.log('bar');

What I run from the command line:

 c8 node ./foo.js

Result I get:

bar
----------|----------|----------|----------|----------|-------------------|
File      |  % Stmts | % Branch |  % Funcs |  % Lines | Uncovered Line #s |
----------|----------|----------|----------|----------|-------------------|
All files |      100 |       60 |      100 |      100 |                   |
 foo.js   |      100 |       60 |      100 |      100 |               3,7 |
----------|----------|----------|----------|----------|-------------------|

Result I expect:

I expect line 2 (the assignment to thing) to also be reported as being uncovered.

@BridgeAR
Copy link
Member Author

BridgeAR commented Feb 7, 2019

I found more issues:

@bcoe
Copy link
Contributor

bcoe commented Feb 7, 2019

@BridgeAR I think there's a chance some of these edge-cases go away when we're able to completely upgrade the V8 version. I suggest that we run coverage with https://github.com/nodejs/node-v8 and remove from this enumerated list of issues and problems that go away on the latest V8.

With regards to issues related to return, I believe this will likely be a similar class of issue as addressed here https://chromium-review.googlesource.com/c/v8/v8/+/1339119 -- CC: @schuay, is there a chance we missed some edge cases around switch?

It would be helpful (even if you keep this open as a catch-all issue) if we could have individual reproductions and bug reports either in V8, within the code-coverage-block.js tests, or if the problem can be attributed to a remapping issue here, in the library that remaps from V8 format coverage to Istanbul format coverage.

☝️ I expect some of these tweaks to move coverage towards perfection will be in V8, and some will be in the library that attempts to translate from V8 to a format we can run reports on.

@hashseed
Copy link
Member

hashseed commented Feb 7, 2019

Yeah it would be nice to isolate some repros first.

@Trott
Copy link
Member

Trott commented Feb 9, 2019

Yeah it would be nice to isolate some repros first.

I put some short repros in bcoe/c8#61 and bcoe/c8#62.

@Trott
Copy link
Member

Trott commented Feb 15, 2019

Chromium bug where the fix will fix some or all of these: https://bugs.chromium.org/p/v8/issues/detail?id=8691
I have no idea if we should expect a fix soon or not so much. In the meantime, in order to get more accurate coverage reports at coverage.nodejs.org, there are a few options:

if (someCondition) {
  throw someError;
}

If that gets changed to this, then the coverage will be correct:

if (someCondition)
  throw someError;

Both of those options seem churn-y and hack-y and likely to meet with resistance. (I'm not sure I like either of them myself.)

Another possibility:

  • Find a way to automatically insert comments or strip braces in the CI job that generates coverage reports. (Seems hack-y and likely to be bug-prone to me.)

Which brings us at last to:

  • Do nothing and just wait for the fix in V8 (at which point we can float a patch until we upgrade V8).

Any other options I'm not thinking of?

@bcoe
Copy link
Contributor

bcoe commented Feb 16, 2019

@Trott I was going to start looking at what it would look like to support:

/* istanbul ignore next */

I was thinking, as a stop gap, we might be able to use similar logic to skip the statement after a throw no promises though, I might find myself going down a rabbit hole where I'm re-implementing the AST (this might be true of ignore functionality too).

@schuay
Copy link

schuay commented Feb 18, 2019

Chromium bug where the fix will fix some or all of these: https://bugs.chromium.org/p/v8/issues/detail?id=8691. I have no idea if we should expect a fix soon or not so much.

That V8 issue is a bit confusing because it describes two different things, both related to exceptions. For the code you describe above (where the closing '}' after a throw statement appears uncovered), the fix should be very similar to what we already have for other jump statements at the end of a block. @bcoe do you have time to look into a fix in V8?

@bcoe
Copy link
Contributor

bcoe commented Feb 18, 2019

@schuay I would happily look into a fix in V8, I would love to keep diving into that codebase until I'm feeling comfortable in it. Let's take the conversation to chat, and I'll update this thread once we have something in motion -- I was actually confused too, I didn't realize there was an easier fix for the closing } (was reading the thread as one solution for throws in general).

@bcoe
Copy link
Contributor

bcoe commented Feb 22, 2019

https://chromium-review.googlesource.com/c/v8/v8/+/1480632 I have a pull open that I believe addresses some of the throw issues.

@bcoe bcoe mentioned this issue Mar 3, 2019
2 tasks
@jaydenseric
Copy link
Contributor

I'm building a CLI code coverage reporter from scratch, using the NODE_V8_COVERAGE env var, targeting Node.js v10+.

In testing it's come to light Node.js v10 - 11 has a lot of false positives around closing braces. Will whatever the Node.js/V8 fixes were in v12+ ever be backported for v10 -11? If not, I'll have to figure out the best way to skip code coverage for unreliable Node.js versions, which is not ideal.

@jaydenseric
Copy link
Contributor

jaydenseric commented Dec 16, 2019

I've noticed that Node.js v12 (but not v10, or v13) needs semicolons in some situations to realize certain functions exist and are not covered.

Example:

- () => {}
+ () => {};

@BridgeAR
Copy link
Member Author

BridgeAR commented Jan 4, 2020

@bcoe are you still looking into this? The hit counters still seem off. I briefly checked https://coverage.nodejs.org/coverage-4bec6d13f9e9068f/lib/internal/util/comparisons.js.html and I found for example these:

image

Here the if is reached 115 times. If the next line is reached the same amount of time, the line afterwards should be hit zero times. But it says it's hit 49 times.

image

This is pretty weird as well: the if is reached less frequently than the actual content. That's impossible, since it's not a loop or something like that.

image

Here the switch case is reached 78 times. As soon as a case is hit, it returns, so the following hit counters of the cases should be lower by the number of former returns. But they are always 78 times, which can't be.


There are more issues but I guess it's best to address them one after the other and I can check from time to time

@BridgeAR
Copy link
Member Author

BridgeAR commented Jan 4, 2020

Finally blocks seem to get a branch not covered even though they will always be triggered:

image

https://coverage.nodejs.org/coverage-4bec6d13f9e9068f/lib/internal/util/inspector.js.html

@bcoe
Copy link
Contributor

bcoe commented Jan 5, 2020

@BridgeAR with regards to finally blocks: https://chromium-review.googlesource.com/c/v8/v8/+/1962265

@bcoe
Copy link
Contributor

bcoe commented Jan 5, 2020

@BridgeAR I don't believe there's a tracking issue for the execution counts in the v8 bug tracker it might be worth opening one.

bcoe added a commit that referenced this issue Jan 15, 2020
Original commit message:

    [coverage] Improve whitespace precision of coverage reporting

    This CL improves whitespace precision of coverage around try blocks;
    previously a small portion of whitespace could be reported as uncovered
    between try blocks and catch and/or finally blocks.

    Change-Id: I763ae3d15106c88f2278cf8893c12b0869a62528
    Fixed: v8:10030
    Bug: v8:10030
    Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1962265
    Reviewed-by: Toon Verwaest <verwaest@chromium.org>
    Reviewed-by: Jakob Gruber <jgruber@chromium.org>
    Commit-Queue: Sigurd Schneider <sigurds@chromium.org>
    Cr-Commit-Position: refs/heads/master@{#65593}

Refs: v8/v8@b9d3303

PR-URL: #31335
Refs: #25937
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Rich Trott <rtrott@gmail.com>
MylesBorins pushed a commit that referenced this issue Jan 16, 2020
Original commit message:

    [coverage] Improve whitespace precision of coverage reporting

    This CL improves whitespace precision of coverage around try blocks;
    previously a small portion of whitespace could be reported as uncovered
    between try blocks and catch and/or finally blocks.

    Change-Id: I763ae3d15106c88f2278cf8893c12b0869a62528
    Fixed: v8:10030
    Bug: v8:10030
    Reviewed-on: https://chromium-review.googlesource.com/c/v8/v8/+/1962265
    Reviewed-by: Toon Verwaest <verwaest@chromium.org>
    Reviewed-by: Jakob Gruber <jgruber@chromium.org>
    Commit-Queue: Sigurd Schneider <sigurds@chromium.org>
    Cr-Commit-Position: refs/heads/master@{#65593}

Refs: v8/v8@b9d3303

PR-URL: #31335
Refs: #25937
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>
Reviewed-By: Rich Trott <rtrott@gmail.com>
@bcoe
Copy link
Contributor

bcoe commented Jan 19, 2020

@BridgeAR I believe the issue with finally statements is now fixed, see:

https://coverage.nodejs.org/coverage-cc0748f509140b7b/lib/internal/util/inspector.js.html

Does that bring the outstanding issues down to the hit counters?

@jasnell
Copy link
Member

jasnell commented Jun 19, 2020

Ping @BridgeAR @bcoe ... is this fixed?

@bcoe
Copy link
Contributor

bcoe commented Jun 19, 2020

👋 it might be good to close this, and open detailed issues here, and I will make an effort to attach those issues to upstream v8 issues when appropriate.

@jasnell
Copy link
Member

jasnell commented Jun 19, 2020

Ok, will go ahead and close then. Thanks @bcoe!

@jasnell jasnell closed this as completed Jun 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
coverage Issues and PRs related to native coverage support. test Issues and PRs related to the tests.
Projects
None yet
Development

No branches or pull requests

7 participants