Coverage report is incorrect since 0.34.0

[nstepien]

Describe the bug

When multiple environments are used, it seems like this affects how coverage is computed.
I've also noticed that the coverage report is different on my machine vs CI.

Please check this branch: adazzle/react-data-grid#3298
The coverage dropped significantly

• CI:
  • 
• On my machine:
  • 
• Codecov:
  • 

In test/ssr.test.tsx, if I skip the test, the coverage INCREASES:

Again in test/ssr.test.tsx, if I skip the test AND remove // @vitest-environment node, the coverage INCREASES further, back to normal levels:

It seems like Vitest 0.34 does not like multiple environments. 🤔

Reproduction

• Clone and checkout this branch Update deps adazzle/react-data-grid#3298
• npm i
• npm t -- --coverage

System Info

System:
    OS: Windows 10 10.0.22621
    CPU: (64) x64 AMD Ryzen Threadripper 3970X 32-Core Processor
    Memory: 19.10 GB / 31.88 GB
  Binaries:
    Node: 20.5.0 - C:\Program Files\nodejs\node.EXE
    npm: 9.8.0 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: Spartan (44.22621.2070.0), Chromium (115.0.1901.188)
    Internet Explorer: 11.0.22621.1
  npmPackages:
    @vitejs/plugin-react: ^4.0.4 => 4.0.4
    @vitest/coverage-v8: ^0.34.1 => 0.34.1
    vite: ^4.4.8 => 4.4.8
    vitest: ^0.34.1 => 0.34.1

Used Package Manager

npm

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[AriPerkkio]

This sounds like #3251. In this case the transformMode differs.

A single file is tested in multiple environments. Vitest will pick source maps from the first result it finds.

[AriPerkkio]

Any ideas how to setup a minimal reproduction for this case? Using whole react-data-grid for debugging is too much.

[nstepien]

You can reduce vite.config.ts down to

import { defineConfig } from 'vite';

export default defineConfig({
  test: {
    environment: 'jsdom',
    globals: true,
    coverage: {
      provider: 'v8',
      enabled: true,
      include: ['src/**/*.{ts,tsx}', '!src/types.ts'],
      all: true
    },
    setupFiles: ['test/setup.ts']
  }
});

ssr.test.tsx is the only test file with the node environment, so you can delete most other test files, should be enough to generate different code coverage, for example:

Not sure what else to remove, we do need some code to cover after all 🙂

[AriPerkkio]

Minimal reproduction case now at AriPerkkio@b1301eb. The custom Vite plugin transforms the source file differently depending on SSR/CSR. This intentionally causes different padding to source maps and messes the V8 results. Same should happen with Istanbul as well.

On the left side is shown what coverage report looks like when same file is tested in Node and JSDOM. On the right side is what it looks like when it's tested only on one environment. The left one is clearly off.

We'll need to tell coverage providers the transformMode here:

vitest/packages/vitest/src/runtime/runners/index.ts

Lines 63 to 65 in f4e6e99

	testRunner.onAfterRun = async (files) => {
	const coverage = await takeCoverageInsideWorker(config.coverage, executor)
	rpc().onAfterSuiteRun({ coverage })

Then they can use it in provider when picking the correct source map:

vitest/packages/coverage-v8/src/provider.ts

Line 217 in f4e6e99

const transformResult = transformResults.get(filePath)

This requires vite-node to store the transformMode here:

vitest/packages/vite-node/src/server.ts

Lines 23 to 27 in f4e6e99

	fetchCache = new Map<string, {
	duration?: number
	timestamp: number
	result: FetchResult
	}>()

Currently vite-node's fetchCache is using filename as key. This won't work when same filename can have different transform result. Key should contain filename and transformMode.

[sheremet-va]

Then they can use it in provider when picking the correct source map:

I also want to point out that it might not have it if tests were running using browser. I remember that it would throw an error before, but currently, it just runs it with empty coverage.

[AriPerkkio]

I have now tested the idea described above and it works. However I'll need more time to finalize the implementation.

Here a file is transformed differently based on transform mode. On web it adds plenty of padding so that covered lines do not match with the SSR version. The coverage reports below are for individual test runs for each transform mode.

Here on left it's shown how the coverage is currently merged incorrectly, using the first source map Vitest finds. On right side is shown how the coverage report looks like when both coverage results are traced back using their own source maps, and then merged with Istanbul:

Happy to see that this actually works in practice. 😅