Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Help required) Tests for nextest partitions #274

Open
wants to merge 39 commits into
base: main
Choose a base branch
from

Conversation

TriplEight
Copy link

@TriplEight TriplEight commented May 5, 2023

cc #271 (comment)

Here are two sample integration tests:

  • 1/2-2/2 how to use llvm-cov with nextest `-partition'.
    It shows that the coverage counted in partitions is not full. And can be a good example of how to do it.
    The first step is to run tests and partition coverage (reports are used here only to illustrate that coverage on partitions should not be expected to be full). Then it archives the coverage data (@taiki-e please check if it's only the needed data, I left some debug output for that) to pass it to...
    The second step. It merges and generates the coverage. I don't know if this is done correctly, please check.
    It might also be a good idea to check this against some fixtures.
  • 1/3-3/3 how to use llvm-cov with nextest --archive-file.
    This is to show that nextest archive when run with `--partition' and llvm-cov partial reports belong to the different stages of CI. And also serves as an example of how to do this.
    The first step just runs nextest to create an archive and pass it to the partitions.
    The second step is to partition the tests and run coverage like 2/2.
    The third step just runs llvm-cov to merge the coverage from the partitions and create a report.

However, I couldn't get 2/3 to work. There's a lot of comments on my results.
I suspect that #265 should also address the --extract-to flag, maybe the problem is that (#271 (comment)) llvm-cov sets target/llvm-cov-target as a default and https://github.com/taiki-e/cargo-llvm-cov/pull/266/files#diff-36dcce9bb960d87a9efde38a0309b8e3e552a4193594bb2065ced1e749c9b743R251 uses it to extract nextest artifacts too. See the comments for details.

Please feel free to contribute to this PR, it's open.
You can see the previous CI runs in TriplEight#2

When this is resolved, I'll write some documentation to https://nexte.st/book/test-coverage.html
cc @sunshowers, your help would be greatly appreciated.

Copy link
Owner

@taiki-e taiki-e left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I haven't looked at the details yet, but left a comment about the CI issue for now.

.github/workflows/ci.yml Outdated Show resolved Hide resolved
.github/workflows/ci.yml Outdated Show resolved Hide resolved
.github/workflows/ci.yml Outdated Show resolved Hide resolved
.github/workflows/ci.yml Outdated Show resolved Hide resolved
.github/workflows/ci.yml Outdated Show resolved Hide resolved
TriplEight and others added 2 commits May 6, 2023 14:23
@TriplEight
Copy link
Author

TriplEight commented May 19, 2023

@taiki-e I've a question: when the tests are done on every partition, it runs llvm-profdata merge -sparse -f [...]/target/llvm-cov-target/project-profraw-list -o [...]/target/llvm-cov-target/project.profdata.
On the larger project, running the setup similar to 1/2-2/2 the each target/llvm-cov-target with the instrumented code weighs in more than 70Gb, I just can't afford transferring 5 (from each partition) of those chunks to the next job.

By far I've managed to merge them as follows:

find . -name 'llvm-cov-target-*.tar.gz' -exec tar -xvzf {} \;
find . -name 'project-*.profdata' -exec llvm-profdata merge -o ./target/llvm-cov-target/project.profdata {} +

But then cargo llvm-cov report --summary-only gives me an error:

error: no input files specified. See llvm-profdata merge -help
error: failed to merge profile data: process didn't exit successfully: `/home/tripleight/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/llvm-profdata merge -sparse -f [...]/target/llvm-cov-target/project-profraw-list -o [...]/target/llvm-cov-target/project.profdata` (exit status: 1)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants