miss-match between apex code coverage log vs. code-formatter cobertura report

[sambit-diff]

Summary

We are not using latest version of salesforce-cli, and the reason was already reported in past issue #2984. We haven't got a window from platform team to test / debug with latest updates. Meanwhile, we are still using salesforce-cli v2.51.6

In Azure pipelines, I am executing below SF command to validate a feature/* branch before merging to "develop" branch and deploying to non-prod sandboxes.

sf project deploy start --target-org dev --manifest source-dir-package.xml --pre-destructive-changes manifest/destructiveChanges/destructiveChanges.xml --ignore-warnings --test-level RunLocalTests --junit --coverage-formatters json-summary --coverage-formatters cobertura --dry-run --wait 60 --verbose --ignore-conflicts

Above command executes successfully, and generates 3 files:

• ./coverage/coverage/cobertura.xml
• ./coverage/coverage/coverage-summary.json
• ./coverage/junit/junit.xml

We also could see Apex code coverage in the logs, I have attached the logs here for review.

Please could you compare % coverage from the section "Apex Code Coverage" in logs against the generated coverage-summary.json file - they don't match, unfortunately!

Especially, could you please check the APEX class AMSIntegrationCtrl.cls? This class file in feature/* branch has 345 lines in total, and I believe code statements excluding comments and empty lines, total lines for run coverage should be less than 345. In coverage-summary.json, this class shows 410 total lines. I am not sure how are we getting count larger than the actual file.

Steps To Reproduce

We find this issue on everytime we run the command i.e., apex code coverage summary in logs does not match with coverage-summary.json, with an exception i.e., when test coverage for an APEX class is NOT 100%.

Expected result

Code coverage summary should be same, After deploy, in next pipeline steps, we validate overall code coverage from coverage-summary.json should be greater than or equals to 85% for success.

Actual result

The details in coverage-summary.json is not correct. At least we know the total lines for an APEX class is not correct.

Additional information

System Information

Using Azure DevOps

{
  "architecture": "linux-x64",
  "cliVersion": "@salesforce/cli/2.51.6",
  "nodeVersion": "node-v22.2.0",
  "osVersion": "Linux 4.18.0-553.16.1.el8_10.x86_64",
  "rootPath": "/azp/.nvm/versions/node/v22.2.0/lib/node_modules/@salesforce/cli",
  "shell": "bash",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.1.7 (core)",
    "@oclif/plugin-commands 4.0.7 (core)",
    "@oclif/plugin-help 6.2.6 (core)",
    "@oclif/plugin-not-found 3.2.11 (core)",
    "@oclif/plugin-plugins 5.3.7 (core)",
    "@oclif/plugin-search 1.2.3 (core)",
    "@oclif/plugin-update 4.4.9 (core)",
    "@oclif/plugin-version 2.2.8 (core)",
    "@oclif/plugin-warn-if-update-available 3.1.9 (core)",
    "@oclif/plugin-which 3.2.8 (core)",
    "@salesforce/cli 2.51.6 (core)",
    "apex 3.2.5 (core)",
    "auth 3.6.36 (core)",
    "data 3.5.5 (core)",
    "deploy-retrieve 3.9.17 (core)",
    "info 3.3.18 (core)",
    "limits 3.3.18 (core)",
    "marketplace 1.2.18 (core)",
    "org 4.3.5 (core)",
    "packaging 2.7.1 (core)",
    "schema 3.3.19 (core)",
    "settings 2.3.8 (core)",
    "sobject 1.4.19 (core)",
    "source 3.5.5 (core)",
    "telemetry 3.6.2 (core)",
    "templates 56.3.3 (core)",
    "trust 3.7.13 (core)",
    "user 3.5.18 (core)",
    "sfdmu 4.36.0 (user) published 15 days ago (Sat Sep 07 2024) (latest is 4.36.3)",
    "sfdx-git-delta 5.43.0 (user) published 19 days ago (Wed Sep 04 2024) (latest is 5.45.0)"
  ]
}

code_coverage_debug.zip

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[github-actions]

Hello @sambit-diff 👋 None of the versions of sf you shared match the latest release.

Shared: 2.51.6
Latest: 2.58.7

Update to the latest version of Salesforce CLI (docs) and confirm that you're still seeing your issue.
You can also try the rc and nightly releases! (docs)

After updating, share the full output of sf version --verbose --json

[WillieRuemmele]

Heya @sambit-diff -

Please could you compare % coverage from the section "Apex Code Coverage" in logs against the generated coverage-summary.json file - they don't match, unfortunately!

from the "Apex Code Coverage" in the command output? I don't see any information related to coverage % there, just passing/failing/total/time

once I lowered the coverage to <75% I got a warning,

 ›   Warning: Average test coverage across all Apex Classes and Triggers is 59%, at least 75% test coverage is required.

within the generated files, I see matching results in your logs, and personal testing in in the /coverage dir files, cobertura.xml and coverage-summary.xml

within coverage/junit/junit.xml I couldn't find any information about line coverage

so from my side, I see matching code coverage values and was unable to reproduce your scenario. Am I missing something?

[sambit-diff]

Hi @WillieRuemmele

If you could extract the test-coverage.log file from my attachment, and look for line no. 1964, the APEX class AMSIntegrationCtrl shows % Covered as 35%. Please refer the below screenshot for reference:

Then please search for this APEX class in coverage-summary.json and cobertura.xml in the same log file or in the attachment. You could see the difference in % Coverage.

In cobertura.xml file, the line <class name="AMSIntegrationCtrl" filename="no-map/AMSIntegrationCtrl" line-rate="0.6073" branch-rate="1"> shows the count of lines in the APEX class is 410 and line-rate="0.6073" is 60.73%. Where as AMSIntegrationCtrl in my feature branch has 345 lines in total.

coverage/junit/junit.xml does not contain information about line coverage, please ignore this file.

It's possible that you might not be able to replicate it, but in our case - we find this % Coverage miss-matched in almost all times.

[WillieRuemmele]

ok - I was able to reproduce this in dreamhouse-lwc by deleting the first two tests in FileUtilitiesTest => createFileSucceedsWhenCorrectInput() & createFileFailsWhenIncorrectRecordId()

sf project deploy start --source-dir force-app --test-level RunLocalTests --junit --coverage-formatters json-summary --coverage-formatters cobertura --results-dir take2 --verbose

.
.
.

Apex Code Coverage
 Name                 % Covered Uncovered Lines         
 ──────────────────── ───────── ─────────────────────── 
 FileUtilities        53%       15,18,22,23,24,25,26,28 

if you then look in take2/coverage/cobertura.xml and search for 'FileUtilities` You'll see

 <class name="FileUtilities" filename="no-map/FileUtilities" line-rate="0.68" branch-rate="1">

=> 68% code coverage, vs our displayed 53% in the command output

take2/coverage/coverage-summary.json =>

"no-map/FileUtilities": {"lines":{"total":25,"covered":17,"skipped":0,"pct":68}

=> 68% code coverage, vs our displayed 53% in the command output

[git2gus]

This issue has been linked to a new work item: W-16822712

[sambit-diff]

Hi @WillieRuemmele thanks for the update, and good to know you were able to reproduce the issue. Until we have this fixed, is it possible for you to confirm which % coverage is correct i.e., 1) apex code coverage in log, OR, 2) coverage-summary.json

[WillieRuemmele]

@sambit-diff - good question.

so I had this in the deployment

Apex Code Coverage
 Name                 % Covered Uncovered Lines         
 ──────────────────── ───────── ─────────────────────── 
 FileUtilities        53%       15,18,22,23,24,25,26,28 

and
<class name="FileUtilities" filename="no-map/FileUtilities" line-rate="0.68" branch-rate="1">

I was able to query for code coverage numbers
sf data query -q "SELECT NumLinesCovered, NumLinesUncovered, TestMethodName FROM ApexCodeCoverage" -t

which returned

 NUMLINESCOVERED NUMLINESUNCOVERED TESTMETHODNAME                         
 ─────────────── ───────────────── ──────────────────────────────────────              
 9               8                 createFileFailsWhenIncorrectFilename   
 9               8                 createFileFailsWhenIncorrectBase64Data 

so then I had 9/(9+8) = 52.9. I was expecting the command output to be wrong, not the code formatters results. This also matches the coverage reported in the dev console, with some differences in rounding

[sambit-diff]

thank you @WillieRuemmele for your help and support

Since coverage in command output is always less than code formatter results, we will now focus on improving code coverage based on command output.

[jshackell-sfdc]

This issue has been fixed in CLI version 2.62.6 (October 16, 2024).

[mcarvin8]

This sounds like the same issue I reported here with inaccuracies in the deployment coverage report.

forcedotcom/salesforcedx-vscode#5511

I made a Salesforce CLI plugin (https://github.com/mcarvin8/apex-code-coverage-transformer ) that could "work around" this by renumbering inaccurate lines. Like if an 100 line apex class is deployed, the coverage report may show 120 total lines and my plugin would have to renumber lines to make it work with SonarQube (not perfect but best I could come up with).

Is this now fixed? My issue hasn't been updated in a bit and I haven't had time to create a support case for it.

If this is fixed by a CLI build, I'd like to release a new version of my plugin which can remove the covered line adjustments I had to make.

[jfaderanga]

@jshackell-sfdc can you link the PR for the bugfix please?