Improve Codecov config for unexpected coverage changes #621
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| coverage: | ||
| status: | ||
| patch: | ||
| default: | ||
| target: 100% | ||
| threshold: 0.1% | ||
| project: | ||
| default: | ||
| target: auto | ||
| threshold: 0.1% | ||
|
There was a problem hiding this comment. I'd remove the threshhold here too. |
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we adding this threshold because the coverage might be irregular? Imo we shouldn't be affected by coverage irregularities and allowing a threshold of 0.1% is not the way to fix it.
We're always uploading our coverage reports, our tests should not be time sensitive etc. If we are having irregularities I'd like to investigate what the root cause of them is first before allowing for thresholds.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We are. Look at the report for the PR herein. I added a single non-code file and look how it changed.
It is. I made the change with 100% code coverage on patch preventing threshold from decreasing total coverage over time.
We are not. Example: https://app.codecov.io/gh/adobecom/milo/commit/8fa7b0aa06758cf0c65d7e43a078de7ee57bd347
I already have. That is the point of this PR after reading Unexpected Coverage Changes
and conducting my own investigation, with my findings outlined here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@hparra I've taken a deeper dive and it seems to me that this difference in code coverage is being caused by some of our tests not being idempotent. In my opinion we need to investigate which tests do not always give the same result and fix them instead of allowing for a threshhold of 0.1%. If your test coverage had gone down by 0.13% btw, instead of going up, then codecov would have failed. This threshold therefore seems like a bandage solution to a bigger problem that we have.
On the codecov link that you attached, they describe how to investigate this issue further, but do not describe setting thresholds as an acceptable solution. Reading this other github issue of a team having similar problems, they also had to investigate the root cause of their tests giving different results.
I'm still against setting these arbitrary thresholds.