-
-
Notifications
You must be signed in to change notification settings - Fork 58
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
(Help required) Tests for nextest partitions #274
base: main
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR! I haven't looked at the details yet, but left a comment about the CI issue for now.
Co-authored-by: Taiki Endo <te316e89@gmail.com>
@taiki-e I've a question: when the tests are done on every partition, it runs By far I've managed to merge them as follows:
But then
|
a820cec
to
59cba33
Compare
2a48173
to
6f4700a
Compare
cc #271 (comment)
Here are two sample integration tests:
It shows that the coverage counted in partitions is not full. And can be a good example of how to do it.
The first step is to run tests and partition coverage (reports are used here only to illustrate that coverage on partitions should not be expected to be full). Then it archives the coverage data (@taiki-e please check if it's only the needed data, I left some debug output for that) to pass it to...
The second step. It merges and generates the coverage. I don't know if this is done correctly, please check.
It might also be a good idea to check this against some fixtures.
--archive-file
.This is to show that nextest archive when run with `--partition' and llvm-cov partial reports belong to the different stages of CI. And also serves as an example of how to do this.
The first step just runs nextest to create an archive and pass it to the partitions.
The second step is to partition the tests and run coverage like 2/2.
The third step just runs llvm-cov to merge the coverage from the partitions and create a report.
However, I couldn't get 2/3 to work. There's a lot of comments on my results.
I suspect that #265 should also address the
--extract-to
flag, maybe the problem is that (#271 (comment)) llvm-cov setstarget/llvm-cov-target
as a default and https://github.com/taiki-e/cargo-llvm-cov/pull/266/files#diff-36dcce9bb960d87a9efde38a0309b8e3e552a4193594bb2065ced1e749c9b743R251 uses it to extract nextest artifacts too. See the comments for details.Please feel free to contribute to this PR, it's open.
You can see the previous CI runs in TriplEight#2
When this is resolved, I'll write some documentation to https://nexte.st/book/test-coverage.html
cc @sunshowers, your help would be greatly appreciated.