Test Coverage API

[connor4312]

📣 2024-03 update: test coverage support has been finalized. You can find the finished, usable version of the API in vscode.d.ts and a guide in our docs.

Original discussion has been preserved below.

---

Notes researching of existing formats:

Clover

• Contains timestamp and total lines of code, files, classes, coverage numbers (see below)
• Also can include complexity information
• Lines are covered as XML simply <line num="24" count="5" type="stmt"/>
• Conditionals are represented as <line num="12" count="0" type="cond" truecount="0" falsecount="3"/>. Either truecount/falsecount being 0 indicates incomplete branch coverage.

Clover uses these measurements to produce a Total Coverage Percentage for each class, file, package and for the project as a whole. The Total Coverage Percentage allows entities to be ranked in reports. The Total Coverage Percentage (TPC) is calculated as follows:

TPC = (BT + BF + SC + MC)/(2*B + S + M) * 100%

where

BT - branches that evaluated to "true" at least once
BF - branches that evaluated to "false" at least once
SC - statements covered
MC - methods entered

B - total number of branches
S - total number of statements
M - total number of methods

gcov

Format info on the bottom of: http://ltp.sourceforge.net/coverage/lcov/geninfo.1.php

• Organized per test, which are associated with one or more files. Indicates functions and how many times each function was called.
• Lines have number of times they were hit, DA:<line number>,<execution count>[,<checksum>]
• Has similar branch coverage, but has indexed branches instead of true/false. A branch with 0 taken is uncovered.
  • BRDA:<line number>,<block number>,<branch number>,<taken>, declared multiple times for line, one for each branch
• No timestamp

cobertura

• General coverage information/count
• Number of times each line was hit.
• Has conditions in a slightly less-strict way, <line number="21" hits="58" branch="true" condition-coverage="50% (1/2)"/>
• Normal lines look like <line number="20" hits="5" branch="false"/>
• Has timestamp and metadata
• Has method call count (everything is organized in classes)

This results in the following API. The TestCoverageProvider is given as an optional provider on the managed TestRun object. This is similar to the SourceControl interface in vscode.d.ts. The provider is then pretty standard; it'll only be examined when the run finishes, so all its data is static.

It has a method to get general coverage information for all files involved in the run, and a method to provide detailed coverage for a URI.

I refer to the atomic unit of coverage as "statement coverage". The examined test formats only provide line-based coverage, but statement coverage is what these are actually trying convey approximate and most tooling (e.g. istanbul/nyc) is technically capable of per-statement rather than per-line coverage. Therefore, it's called statement coverage, but it can be used as line coverage.

The API is physically large due to the nature of data being provided, but is conceptually pretty simple (at least compared to the Test Controller API!). I don't like the method names on the TestCoverageProvider, but have not thought of better ones yet.

declare module 'vscode' {

	// https://github.com/microsoft/vscode/issues/123713

	export interface TestRun {
		/**
		 * Test coverage provider for this result. An extension can defer setting
		 * this until after a run is complete and coverage is available.
		 */
		coverageProvider?: TestCoverageProvider;
		// ...
	}

	/**
	 * Provides information about test coverage for a test result.
	 * Methods on the provider will not be called until the test run is complete
	 */
	export interface TestCoverageProvider<T extends FileCoverage = FileCoverage> {
		/**
		 * Returns coverage information for all files involved in the test run.
		 * @param token A cancellation token.
		 * @return Coverage metadata for all files involved in the test.
		 */
		// @API - pass something into the provide method:
		// (1) have TestController#coverageProvider: TestCoverageProvider
		// (2) pass TestRun into this method
		provideFileCoverage(token: CancellationToken): ProviderResult<T[]>;

		/**
		 * Give a FileCoverage to fill in more data, namely {@link FileCoverage.detailedCoverage}.
		 * The editor will only resolve a FileCoverage once, and only if detailedCoverage
		 * is undefined.
		 *
		 * @param coverage A coverage object obtained from {@link provideFileCoverage}
		 * @param token A cancellation token.
		 * @return The resolved file coverage, or a thenable that resolves to one. It
		 * is OK to return the given `coverage`. When no result is returned, the
		 * given `coverage` will be used.
		 */
		resolveFileCoverage?(coverage: T, token: CancellationToken): ProviderResult<T>;
	}

	/**
	 * A class that contains information about a covered resource. A count can
	 * be give for lines, branches, and functions in a file.
	 */
	export class CoveredCount {
		/**
		 * Number of items covered in the file.
		 */
		covered: number;
		/**
		 * Total number of covered items in the file.
		 */
		total: number;

		/**
		 * @param covered Value for {@link CovereredCount.covered}
		 * @param total Value for {@link CovereredCount.total}
		 */
		constructor(covered: number, total: number);
	}

	/**
	 * Contains coverage metadata for a file.
	 */
	export class FileCoverage {
		/**
		 * File URI.
		 */
		readonly uri: Uri;

		/**
		 * Statement coverage information. If the reporter does not provide statement
		 * coverage information, this can instead be used to represent line coverage.
		 */
		statementCoverage: CoveredCount;

		/**
		 * Branch coverage information.
		 */
		branchCoverage?: CoveredCount;

		/**
		 * Function coverage information.
		 */
		functionCoverage?: CoveredCount;

		/**
		 * Detailed, per-statement coverage. If this is undefined, the editor will
		 * call {@link TestCoverageProvider.resolveFileCoverage} when necessary.
		 */
		detailedCoverage?: DetailedCoverage[];

		/**
		 * Creates a {@link FileCoverage} instance with counts filled in from
		 * the coverage details.
		 * @param uri Covered file URI
		 * @param detailed Detailed coverage information
		 */
		static fromDetails(uri: Uri, details: readonly DetailedCoverage[]): FileCoverage;

		/**
		 * @param uri Covered file URI
		 * @param statementCoverage Statement coverage information. If the reporter
		 * does not provide statement coverage information, this can instead be
		 * used to represent line coverage.
		 * @param branchCoverage Branch coverage information
		 * @param functionCoverage Function coverage information
		 */
		constructor(
			uri: Uri,
			statementCoverage: CoveredCount,
			branchCoverage?: CoveredCount,
			functionCoverage?: CoveredCount,
		);
	}

	// @API are StatementCoverage and BranchCoverage etc really needed
	// or is a generic type with a kind-property enough

	/**
	 * Contains coverage information for a single statement or line.
	 */
	export class StatementCoverage {
		/**
		 * The number of times this statement was executed, or a boolean indicating
		 * whether it was executed if the exact count is unknown. If zero or false,
		 * the statement will be marked as un-covered.
		 */
		executed: number | boolean;

		/**
		 * Statement location.
		 */
		location: Position | Range;

		/**
		 * Coverage from branches of this line or statement. If it's not a
		 * conditional, this will be empty.
		 */
		branches: BranchCoverage[];

		/**
		 * @param location The statement position.
		 * @param executed The number of times this statement was executed, or a
		 * boolean indicating  whether it was executed if the exact count is
		 * unknown. If zero or false, the statement will be marked as un-covered.
		 * @param branches Coverage from branches of this line.  If it's not a
		 * conditional, this should be omitted.
		 */
		constructor(executed: number | boolean, location: Position | Range, branches?: BranchCoverage[]);
	}

	/**
	 * Contains coverage information for a branch of a {@link StatementCoverage}.
	 */
	export class BranchCoverage {
		/**
		 * The number of times this branch was executed, or a boolean indicating
		 * whether it was executed if the exact count is unknown. If zero or false,
		 * the branch will be marked as un-covered.
		 */
		executed: number | boolean;

		/**
		 * Branch location.
		 */
		location?: Position | Range;

		/**
		 * Label for the branch, used in the context of "the ${label} branch was
		 * not taken," for example.
		 */
		label?: string;

		/**
		 * @param executed The number of times this branch was executed, or a
		 * boolean indicating  whether it was executed if the exact count is
		 * unknown. If zero or false, the branch will be marked as un-covered.
		 * @param location The branch position.
		 */
		constructor(executed: number | boolean, location?: Position | Range, label?: string);
	}

	/**
	 * Contains coverage information for a function or method.
	 */
	export class FunctionCoverage {
		/**
		 * Name of the function or method.
		 */
		name: string;

		/**
		 * The number of times this function was executed, or a boolean indicating
		 * whether it was executed if the exact count is unknown. If zero or false,
		 * the function will be marked as un-covered.
		 */
		executed: number | boolean;

		/**
		 * Function location.
		 */
		location: Position | Range;

		/**
		 * @param executed The number of times this function was executed, or a
		 * boolean indicating  whether it was executed if the exact count is
		 * unknown. If zero or false, the function will be marked as un-covered.
		 * @param location The function position.
		 */
		constructor(name: string, executed: number | boolean, location: Position | Range);
	}

	export type DetailedCoverage = StatementCoverage | FunctionCoverage;

}

[connorshea]

Would it be possible to load the coverage data directly from a file on disk (or in the remote workspace, whatever we want to call it) rather than as a result of the output of the test run? It seems like it'd be pretty limiting to only have coverage if it were provided by a test runner extension, if I'm understanding your current API draft correctly.

In my codebase for work, the test suite takes well over 90 minutes to run in full. So it pretty much only gets run in CI (parallelized, obviously), which means that the coverage is only available as a CI artifact. It never exists locally (at least, not from a full run).

The more likely case where this might be useful would be if you ran your test suite locally (because it was fast enough), but not using a VS Code Test Explorer extension. So then you just had the coverage.xml (or whatever file/format your test framework outputs) on your machine, and VS Code would either know to look for that, or could be told to go into coverage mode using the data inside it.

[connor4312]

Would it be possible to load the coverage data directly from a file on disk (or in the remote workspace, whatever we want to call it) rather than as a result of the output of the test run? It seems like it'd be pretty limiting to only have coverage if it were provided by a test runner extension, if I'm understanding your current API draft correctly.

Yes, the APIs are designed for this 🙂 You can call createTestRun at any time, so you could do that when you load the file (passing in persist as false since VS Code does not need to remember the already-externally-stored results), parse them, then immediately .end() the run.

[connor4312]

also: I'm pretty interested in getting the CI story right, so I'm glad to have someone who has an existing use case and need for it! Please let me know if this or any other parts are confusing or seem to present roadblocks for you.

[connorshea]

I'd make sure to document the coverage-from-file use-case specifically (I know this is an API draft and actual docs are a while away, but I think it's important 😄).

Regarding CI, probably a crazy idea, but:

It'd be nice to point VS Code at a URL where it can get the coverage file (maybe it passes a param with the current branch name as well) so I wouldn't have to download the coverage.xml myself manually. But that'd have to be custom-built to redirect to the latest CI artifacts somehow since there's no stable URL that CircleCI (or any other CI system) provides for something like this, as far as I know.

And maybe that should be a capability handled by an extension rather than as something in core VS Code. 🤷‍♂️

[connor4312]

Yep, that's a good idea for an extension

[ashgti]

I have a question about the FileCoverage API. If I am looking at a coverage report, it looks like I only know the file URI. So, if someone starts a coverage test run (which may take a while to run) then makes any changes to the file the statements may not line up correctly with the current editor.

In the TextDocument API there is a version field that tracks revisions to a specific TextDocument.

Should we track FileCoverage to the specific version of the file the test run was started with? Maybe as an optional param?

[connor4312]

While we could store the version, there's not a lot of useful things we can do with it (and it gets reset if the file is opened and closed). The most reliable way to do this would be for the extension to use registerFileSystemProvider to create a custom uri scheme/readonly filesystem from which it can serve the version of the file that was used for the coverage run.

[jdneo]

The field executionCount in *Coverage classes shows the excution information, what about the missed part.

For example, a method or a line might have 4 branches, but only 2 of them are executed.

Take the Jacoco in Java as an example, the coverage will contains such information:

In editor each line can also have this metadata

---

Update

forget about it, I just realized, if the executionCount is zero, then it's uncovered.

[JustinGrote]

For reference, there is an updated API with migration guide that was posted: #122208 (comment)

[connor4312]

I implement the coverage API and data flow for it internally. There's no UI for it, but it's there to try out. At some point in the next few weeks UI will magically show up ✨

[ioquatix]

Oh, I figured it out, I needed to add the highlighted argument:

[connor4312]

Hi, just to update folks, we bumped this API to target finalization next iteration, rather than the this one (release next week.) This is a sizable API and there were several other exciting APIs entering proposed and finalization that took up our team's bandwidth.

[JustinGrote]

@connor4312 appreciate the hard work regardless, thanks!

[vogelsgesang]

I think I ran into a bug, trying to use the coverage API to integrate coverage data loaded from a file. I am trying to follow the instructions which were previously provided up-thread

Scenario

I am creating two test runs. Those test runs have no test cases attached, they only contain a name, some output (appendOutput) and the coverage data.

Minimal repro

import * as vscode from "vscode";

class BazelCoverageProvider implements vscode.TestCoverageProvider {
  provideFileCoverage(
    token: vscode.CancellationToken,
  ): vscode.ProviderResult<vscode.FileCoverage[]> {
    const detailedCoverage = [
      new vscode.DeclarationCoverage(
        "test_func",
        12,
        new vscode.Position(1, 12),
      ),
      new vscode.DeclarationCoverage(
        "test_func2",
        0,
        new vscode.Position(10, 0),
      ),
      new vscode.StatementCoverage(true, new vscode.Position(15, 3)),
      new vscode.StatementCoverage(false, new vscode.Position(16, 0)),
    ];
    return [
      vscode.FileCoverage.fromDetails(
        vscode.Uri.file("/Users/avogelsgesang/hyper/hyper-db/hyper/ir/IR.cpp"),
        detailedCoverage,
      ),
    ];
  }

  // Already fully resolved. Noting to resolve
  resolveFileCoverage?(
    coverage: vscode.FileCoverage,
    token: vscode.CancellationToken,
  ): vscode.ProviderResult<vscode.FileCoverage> {
    return coverage;
  }
}

export async function activate(context: vscode.ExtensionContext) {
  // Create the test controller
  const testController = vscode.tests.createTestController(
    "bazel-coverage",
    "Bazel Coverage",
  );
  context.subscriptions.push(testController);

  // Provide coverage info 1
  const run = testController.createTestRun(
    new vscode.TestRunRequest(),
    "/my/coverage.lcov",
    false,
  );
  run.appendOutput("Information about where we loaded the coverage from...");
  run.coverageProvider = new BazelCoverageProvider();
  run.end();

  // Provide coverage info 2
  const run2 = testController.createTestRun(
    new vscode.TestRunRequest(),
    "/my/second_coverage.lcov",
    false,
  );
  run2.appendOutput("Some other information...");
  run2.coverageProvider = new BazelCoverageProvider();
  run2.end();
}

Observed behavior

I see two coverage entries in the test result list, and I can switch between them. Also, I see the output from the 2nd test run.

However, I do not see the test run names "/my/coverage.lcov" and "/my/second_coverage.lcov" anywhere. Also, there seems to be no way to see inspect the appendOutput output from the first run.

Expected behavior

I guess the "Close test coverage"/"View test coverage" should be grouped under a test run? Even if there are no tests and only coverage data inside this test run? Also, it would be good to be able to set the overall test run into a "success"/"failed" state, even without listing explicit sub-tests

[jdneo]

@connor4312 Will the API change #207512 affect the coverage API stable release ETA? Do we still target on March release now?

[connor4312]

We're still targeting the March release, this API will be finalized in the next few hours.

@vogelsgesang I'll check that out, I've opened a new issue for tracking.

[ffMathy]

It would have been nice if I (like in Rider) could right-click a covered line and see all the tests that ran through that line, and then run a particular one.

This is not supported from this API, as it mostly brings total coverage across all tests.

[connor4312]

It would have been nice if I (like in Rider) could right-click a covered line and see all the tests that ran through that line, and then run a particular one.

This is not supported from this API, as it mostly brings total coverage across all tests.

Tracked in #212196