Test annotated with "@coversNothing" may lead to files missing from code coverage report

[sebastianbergmann]

phpunit.xml

<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/9.5/phpunit.xsd"
         bootstrap="vendor/autoload.php"
         cacheResultFile=".phpunit.cache/test-results"
         executionOrder="depends,defects"
         forceCoversAnnotation="true"
         beStrictAboutCoversAnnotation="true"
         beStrictAboutOutputDuringTests="true"
         beStrictAboutTodoAnnotatedTests="true"
         convertDeprecationsToExceptions="true"
         failOnRisky="true"
         failOnWarning="true"
         colors="true"
         verbose="true">
    <testsuites>
        <testsuite name="default">
            <directory>tests</directory>
        </testsuite>
    </testsuites>

    <coverage cacheDirectory=".phpunit.cache/code-coverage"
              processUncoveredFiles="true">
        <include>
            <directory suffix=".php">src</directory>
        </include>
    </coverage>
</phpunit>

src/Greeter.php

<?php declare(strict_types=1);
final class Greeter
{
    public function greet(string $name): string
    {
        return 'Hello, ' . $name;
    }
}

tests/GreeterTest.php

<?php declare(strict_types=1);
use PHPUnit\Framework\TestCase;

/**
 * @coversNothing
 */
final class GreeterTest extends TestCase
{
    public function testGreetsWithName(): void
    {
        $this->assertSame('Hello, Test', (new Greeter)->greet('Test'));
    }
}

$ ./vendor/bin/phpunit --coverage-html /tmp/coverage
PHPUnit 9.5.19 #StandWithUkraine

Runtime:       PHP 8.1.4 with PCOV 1.0.11
Configuration: /home/sb/bug/phpunit.xml

.                                                                   1 / 1 (100%)

Time: 00:00.005, Memory: 6.00 MB

OK (1 test, 1 assertion)

Generating code coverage report in HTML format ... done [00:00.006]

Actual

Expected

[sebastianbergmann]

@dvdoug @Slamdunk Do you guys have an idea why a sourcecode file does not show up in the report at all? Thanks!

[Slamdunk]

Do you guys have an idea why a sourcecode file does not show up in the report at all?

No, but I can assure and confirm that return 'Hello, ' . $name; line is reported as executable by ExecutableLinesFindingVisitor so the culprit is somewhere else.

[sebastianbergmann]

That much I was able to figure out already. So sorry for not mentioning that.

[dvdoug]

Haven't stepped through with a debugger to confirm, but my theory would be this:

• PHPUnit detects the @coversNothing annotation
• PHPUnit runs the test, but opts not to call start()/stop() inside php-code-coverage because it doesn't see the need (see logic starting at

  phpunit/src/Framework/TestResult.php

  Line 650 in 1ef13ce

  $isAnyCoverageRequired = TestUtil::requiresCodeCoverageDataCollection($test);

  )
• Therefore php-code-coverage does not obtain any data about the file during the actual execution of the test
• PHPUnit asks php-code-coverage for the report
• php-code-coverage runs processUncoveredFilesFromFilter() which correctly identifies that it does not have coverage data for Greeter.php
• It starts the coverage driver, does an include_once on the file. Then stops coverage. At this point, normally data about the file would be picked up and processed.
• However, unknown to php-code-coverage that file was already included/required earlier on during the test run. Therefore the include_once is actually a no-op, resulting in the coverage driver not reading/parsing/outputting any data about the file.

If any other test in a test suite (without @coversNothing) invoked that source code file in any way during execution I think it would show up in the report just fine, as the driver would then include data for it at that point

[sebastianbergmann]

That makes sense. Thank you, Doug!

This would mean, though, that the problem should not exist in master because there we no longer load uncovered files but instead use PHP-Parser to analyse them.

[sebastianbergmann]

This would mean, though, that the problem should not exist in master because there we no longer load uncovered files but instead use PHP-Parser to analyse them.

That is correct:

composer.json

{
    "autoload": {
        "classmap": [
            "src/"
        ]
    },
    "prefer-stable": true,
    "minimum-stability": "dev",
    "require-dev": {
        "phpunit/phpunit": "^10.0"
    }
}

phpunit.xml

<?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/10.0/phpunit.xsd"
         bootstrap="vendor/autoload.php"
         cacheDirectory=".phpunit.cache"
         executionOrder="depends,defects"
         requireCoverageMetadata="true"
         beStrictAboutCoverageMetadata="true"
         beStrictAboutOutputDuringTests="true"
         convertDeprecationsToExceptions="true"
         failOnRisky="true"
         failOnWarning="true"
         colors="true"
         verbose="true">
    <testsuites>
        <testsuite name="default">
            <directory>tests</directory>
        </testsuite>
    </testsuites>

    <coverage>
        <include>
            <directory suffix=".php">src</directory>
        </include>
    </coverage>
</phpunit>

Actual