Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Coverage reports are inconsistent. #279

Closed
samcunliffe opened this issue May 3, 2023 · 2 comments
Closed

Coverage reports are inconsistent. #279

samcunliffe opened this issue May 3, 2023 · 2 comments
Assignees
@samcunliffe
Labels
bug Something isn't working testing Adding or requesting more test coverage wontfix This will not be worked on

Comments

@samcunliffe
Copy link
Member

samcunliffe commented May 3, 2023 •
edited
Loading

Describe the bug
The coverage reports collected in ubuntu-latest and macos-latest differ by 50.0%.

To Reproduce
Happens on the build system (for example here on #245). A symptom of this is that one of the codecov tests can fail if the 20% report is received first and compared to the 40% report (it looks like a decrease in coverage of 20% which is unacceptable).

Expected behaviour
Coverage should be roughly consistent. Some differences are expected due to compiler optimisations and "the deep weeds of machine code". I don't expect such a precise discrepancy though.

Environment (please complete the following information):

  • OS: MacOS 12.6.5, and Ubuntu 22.04.2
  • Compiler: AppleClang 14.0.0.14000029 and GNU 11.3.0
  • MATLAB version (if relevant): R2022a
@samcunliffe samcunliffe added the bug Something isn't working label May 3, 2023
@samcunliffe samcunliffe self-assigned this May 3, 2023
@samcunliffe samcunliffe mentioned this issue May 3, 2023
Merged
@willGraham01 willGraham01 mentioned this issue May 4, 2023
Merged
samcunliffe added a commit that referenced this issue May 4, 2023
@samcunliffe
A kindof trivial test addition to help debug #279.
9d77494 
Doesn't hurt to test our utils but I'm mostly interested in the change
in coverage for the utils.cpp file.
@samcunliffe
Copy link
Member Author

Some things I'm starting to think we should anyway do:

samcunliffe added a commit that referenced this issue May 5, 2023
@samcunliffe
A kindof trivial test addition to help debug #279. (#283)
3ab1e2d 
Doesn't hurt to test our utils but I'm mostly interested in the change
in coverage for the utils.cpp file.
@samcunliffe samcunliffe mentioned this issue May 5, 2023
Merged
@samcunliffe samcunliffe mentioned this issue May 17, 2023
Closed
samcunliffe added a commit that referenced this issue May 19, 2023
@samcunliffe
Turn off coverage reporting in MacOS
3931e52 
Something is strange about the coverage reported by MacOS builds. Perhaps llvm-cov? Turning off whilst we investigate.

Relates to #279 and "solves" it by ignoring the problem.
@samcunliffe samcunliffe mentioned this issue May 19, 2023
Merged
samcunliffe added a commit that referenced this issue May 19, 2023
@samcunliffe
Turn off coverage reporting in MacOS (#305)
8ae70f0 
* Turn off coverage reporting in MacOS

Something is strange about the coverage reported by MacOS builds. Perhaps llvm-cov? Turning off whilst we investigate.

Relates to #279 and "solves" it by ignoring the problem.

* Of course.

https://github.com/orgs/community/discussions/27152
samcunliffe added a commit that referenced this issue May 24, 2023
@samcunliffe
Turn off coverage reporting in MacOS (#305)
f449d61 
* Turn off coverage reporting in MacOS

Something is strange about the coverage reported by MacOS builds. Perhaps llvm-cov? Turning off whilst we investigate.

Relates to #279 and "solves" it by ignoring the problem.

* Of course.

https://github.com/orgs/community/discussions/27152
@samcunliffe
Copy link
Member Author

Closing as superseded by #305.

@samcunliffe samcunliffe added wontfix This will not be worked on testing Adding or requesting more test coverage labels Jun 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@samcunliffe samcunliffe

Labels
bug Something isn't working testing Adding or requesting more test coverage wontfix This will not be worked on
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@samcunliffe