Coverage clears previous file results after each run

[asportnoy]

Describe the bug

With the new coverage reporting, the results for all previous files gets cleared when it runs again, even if the run is for a different file. This is especially noticeable when using watch mode.

Reproduction

1. Run the tests for one file including coverage and verify that the coverage percentage is shown in VSCode
2. Run the tests for a different file including coverage. The coverage percentage is now shown for that file, but the coverage for the initial file is now completely gone.

Output

[INFO 4:12:29 PM] [v0.5.15] Vitest extension is activated because Vitest is installed or there is a Vite/Vitest config file in the workspace.
[INFO 4:12:31 PM] [API] Running Vitest: v1.4.0 ([redacted]/vitest.config.ts)
[INFO 4:12:32 PM] [API] Vitest process 52649 created
[Worker] �[33mThe CJS build of Vite's Node API is deprecated. See https://vitejs.dev/guide/troubleshooting.html#vite-cjs-node-api-deprecated for more details.�[39m
[INFO 4:14:22 PM] [API] Vitest process 52649 closed successfully
[INFO 4:14:22 PM] [API] Running Vitest: v1.4.0 ([redacted]/vitest.config.ts)
[INFO 4:14:23 PM] [API] Vitest process 54726 created
[Worker] �[33mThe CJS build of Vite's Node API is deprecated. See https://vitejs.dev/guide/troubleshooting.html#vite-cjs-node-api-deprecated for more details.�[39m
[INFO 4:16:02 PM] Running 1 file(s): [redacted]
[INFO 4:16:37 PM] Running 1 file(s): [redacted]
[INFO 4:18:19 PM] Running 1 file(s) with name pattern: ^\s?[redacted]
[INFO 4:24:26 PM] Running 1 file(s) with name pattern: ^\s?[redacted]
[INFO 4:24:43 PM] Running 1 file(s) with name pattern: ^\s?[redacted]
[INFO 4:25:07 PM] [API] Vitest process 54726 closed successfully
[INFO 4:25:07 PM] [API] Running Vitest: v1.4.0 ([redacted]/vitest.config.ts)
[INFO 4:25:08 PM] [API] Vitest process 60430 created
[Worker] �[33mThe CJS build of Vite's Node API is deprecated. See https://vitejs.dev/guide/troubleshooting.html#vite-cjs-node-api-deprecated for more details.�[39m
[INFO 4:25:16 PM] Running 1 file(s): [redacted]
[INFO 4:29:37 PM] Running 1 file(s): [redacted]
[INFO 4:29:53 PM] Running 1 file(s): [redacted]
[INFO 4:32:01 PM] Running 1 file(s) with name pattern: ^\s?[redacted]
[INFO 4:32:12 PM] [API] Vitest process 60430 closed successfully
[INFO 4:32:12 PM] [API] Running Vitest: v1.4.0 ([redacted]/vitest.config.ts)
[INFO 4:32:13 PM] [API] Vitest process 64228 created
[Worker] �[33mThe CJS build of Vite's Node API is deprecated. See https://vitejs.dev/guide/troubleshooting.html#vite-cjs-node-api-deprecated for more details.�[39m
[INFO 4:32:16 PM] Running 1 file(s): [redacted]
[INFO 4:38:27 PM] Running 1 file(s) with name pattern: ^\s?[redacted]
[INFO 4:38:40 PM] Running 1 file(s): [redacted]

### Version

v0.5.15 (pre-release)

### Validations

- [X] Check that you are using the latest version of the extension
- [X] Check that there isn't [already an issue](https://github.com/vitest-dev/vscode/issues) that reports the same bug to avoid creating a duplicate.
- [X] Check that this is a concrete bug. For Q&A open a [GitHub Discussion](https://github.com/vitest-dev/vscode/discussions) or join our [Discord Chat Server](https://chat.vitest.dev).
- [X] The provided reproduction is a [minimal reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) of the bug.

[sheremet-va]

This is also how it works with Vitest CLI, no?

[asportnoy]

Seems to be inconsistent. If you make a change to a test file, it only shows you the results of the files covered by that test. If you make a change to a non-test file, it shows you the results of the previous coverage runs too. There is a cleanOnRerun option which I think would make it never clear, though for some reason that causes the CLI to crash in my repository (I created an issue for this in the main Vitest repo: vitest-dev/vitest#5521).

Regardless, it feels very weird that coverage information for files randomly disappears when I edit a completely unrelated file. The actual test results stay unchanged, so shouldn't the coverage stay unchanged as well?

[sheremet-va]

The change to a file triggers a new test run which has a different coverage information. You can still see the previous coverage data by clicking on the previous test run in “Test Results” window

[asportnoy]

Seems like the watching isn't actually updating coverage information at all for me, even though I have that enabled in the watcher options. It only updates the coverage when I manually press "run with coverage" on the test again.

Oddly, at some point during my testing, the coverage for a certain file completely disappeared and would not come back no matter what I did. I don't have consistent reproduction steps for this yet, but I'll reply back if I can figure some out.

It doesn't seem like there are any logs shown for anything related to coverage. Having those would definitely help narrow down what the issue is if you're able to add them.

[sheremet-va]

Does it work better with only the coverage profile selected?

Also, can you describe the workflow you expect? Starting from the very beginning ("click on continues run", I presume?)

The coverage is related to the test run. You can see all test runs on the right side in "Test Results" panel. Since you run only a single test on change, it shows only related coverage. I say this is expected behaviour. You can still see all coverage in the previous test run if you double-click on that run.

[asportnoy]

The coverage info is also shown in the file tree, which is where I'd be looking most of the time. The ideal workflow is this:

1. Run all the tests once to get coverage for all files
2. Enable watch mode
3. I make a change in some file, and the relevant tests for that file are rerun, outputting new coverage information. But importantly, the coverage information for all of the other files (which were not touched at all) should still be there.

I think it's ok that the test results panel only shows the coverage that was changed by that test, that makes sense to me. But the file tree should continue to show the coverage information for the unchanged files.

[sheremet-va]

I think it's ok that the test results panel only shows the coverage that was changed by that test, that makes sense to me. But the file tree should continue to show the coverage information for the unchanged files.

There is no such thing in the VSCode API as showing something in one place, but not in another. The whole interface shows the latest test run by default unless you select the test run on the right-side panel. So we either show the test coverage (like now), or all accumulated coverage.

[asportnoy]

I think the best option for the interim would be to add an option to control which behavior to use. Might be something to bring up to the VSCode team though, feels to me like there should be some way to persist coverage info in the file explorer while only showing the relevant files in the test results.

[sheremet-va]

Vitest doesn't support this at the moment: vitest-dev/vitest#5521 (comment)