Speed up Time for Coverage Combine

[tanmayzengit]

Is your feature request related to a problem? Please describe.
I am trying to setup a code coverage representation for my pull requests for specific number of tests. Using coverage combine to combine all the data files takes a lot of time which is not very ideal. Is there a quick resolution for this. I tried moving to 5.0 but the combine performance degraded from previous versions which has pickled dictionary as the datastore instead of sqlite.

Describe the solution you'd like
My idea is to somehow bring down the combine step in multiprocessing cases where we parallely run the tests and combine them at a later stage

Describe alternatives you've considered
For doing a per PR coverage i did consider using collector itself to have the data but i would want to avoid any usage of database from my side and if possible i would prefer using that coverage data file as that is much more convenient to debug
Additional context
Add any other context about the feature request here.

[nedbat]

Can you provide some more details? Ideally, you'd link us to your repo so that we can see the behavior you are seeing. But even if you can't do that: how long is combine taking? How many data files do you have, and how large are they? Can you show us the output of coverage debug data before the combine step?

[tanmayzengit]

Linking the repo would not be possible but, i can adda little context here.
I have updated coverage version from 4.0a6 -> 5.0.0 and was expecting an improvement in combine time.
Combine time before-> ~45 mins
Combine time after -> ~95 mins
The data files are around ~8500 with parallel and multiprocessing configurations which gives the total data.
coverage debug data gives nothing because we dont have a .coverage file before combine step?

p.s My major goal is to report coverage on a PR so if combine cannot be improved is there a better way to do this?

[nedbat]

Why are you only updating to 5.0.0? Can you try 6.5.0?
You say you have 8500 data files? Are you running your test suite 8500 times? Maybe I'm misunderstanding something.

[tanmayzengit]

Eventual goal is to move to the latest version but currently need python 2.7 support thats why 5.0
So by data file i dont mean .coverage final data file, For every subprocess i am generating a separate coverage file (as i said multiprocessing and parallel flag check here) and combining them gives me .coverage file which has the data for the entire repo.

[nedbat]

Please try 5.5 (pip install coverage==5.5). It's the latest version that runs in 2.7.

[nedbat]

Two more ideas for you:

1. Instead of using parallel, try using append to put more data into each file. You have to coordinate to be sure that simultaneous runs (for example with pytest-xdist) don't try to use the same file.

2. I wonder how much unique data is in your data files? I used a small program to assess the uniqueness of my files, and found that 1111 files had 1028 unique fingerprints. That's not a huge savings, but you might have different results. If so, hashing the data files before combining could remove files that aren't adding anything new.

import collections
import glob
import hashlib

files = list(glob.glob(".coverage.*"))
print(f"{len(files)} files")

hashes = collections.defaultdict(list)
for file in files:
    with open(file, "rb") as f:
        h = hashlib.new("sha256")
        h.update(f.read())
        sha = h.hexdigest()
        hashes[sha].append(file)

print(f"{len(hashes)} hashes")
for sha, files in hashes.items():
    if len(files) > 1:
        print(sha)
        for file in files:
            print(f"    {file}")

[nedbat]

Oh, coverage.py writes a timestamp into the data file. If I remove that, then my 1111 files have only 761 unique hashes.

[tanmayzengit]

So i found a similar issue https://github.com/nedbat/coveragepy/pull/765/files
I could not acheive any better results though when i moved to version 5.5

Instead of using parallel, try using append to put more data into each file. You have to coordinate to be sure that simultaneous runs (for example with pytest-xdist) don't try to use the same file.

Not sure if this works as expected with multiprocessing though. So i my case the parent process creates a .coverage file and all the child process keep rewriting that file so i was having issues with it. switching to parallel helped solve the issue. I will try with append though but is the behaviour same with multiprocessing set as concurrency.

Oh, coverage.py writes a timestamp into the data file. If I remove that, then my 1111 files have only 761 unique hashes.

How do we do that?

[nedbat]

Oh, coverage.py writes a timestamp into the data file. If I remove that, then my 1111 files have only 761 unique hashes.

How do we do that?

You can install coverage.py from an experimental branch:

pip install git+https://github.com/nedbat/coveragepy@nedbat/dedup-combine

It will check the hashes of the files, and skip duplicates automatically. Let me know if you can try it, and what the results are. The final reports should all be exactly the same, but the combine time should be shorter.

[tanmayzengit]

Ahh Ok. I ll try and do it with coverage 5.5 as base for the experimental branch and update any improvement here

[nedbat]

This is now merged to coverage.py master, but it won't run on Python 2. I'd be interested if you could find a way to run it in your environment.

[ssbarnea]

Earlier today I discovered that I had ~68000 coverage files inside my .tox folder. We started to use coveragepy using multiprocessing and thread as we use xdist and we also use subprocesses.

It was so bad that I was not even able to use globbing to clean the files as I was reaching too many args error. I managed to clean using ls -a1 | grep .coverage | wc -l.

I have ~700 tests and a single run produces ~600 coverage files. I really wonder why so many as I clearly do not run so many subprocesses.

I am not sure when this happened but I suspect there is a regression in recent versions of coverage, maybe in 6.x? My version is 6.4.4 with C extensions.

• https://github.com/ansible/ansible-lint/blob/main/pyproject.toml#L17
• https://github.com/ansible/ansible-lint/blob/main/tox.ini#L57-L58

As combine is run only on a different tox environment I endup with a growing pile of files. Any ideas on how to limit its growing? Tx

[nedbat]

@ssbarnea I'd be interested to investigate this, but it sounds like it should be a new issue.

[nedbat]

I made some more improvements to the de-duplication during combining. If anyone wants to try it, install from master, or the specific commit:

pip install git+https://github.com/nedbat/coveragepy
pip install git+https://github.com/nedbat/coveragepy@1cd6c9bba0b4ba3018bf1b28fee645a7dd98fe68

Let me know how it works for you.

[nedbat]

I'm going to close this now as "improved" :) If people have further problems, feel free to open a new issue.

[nedbat]

This is now released as part of coverage 7.0.0b1.

[jbkkd]

I'm working on a pretty big project - our coverage report is 412mb big. The entire codebase is about 8 million lines.

We use combine to build the coverage database, and it's painfully slow - takes almost 3 hours to build. It combines 690 different coverage files together.

Are there any plans to introduce multithreading / multiprocessing into the process? This would be a huge win for us.

[nedbat]

We use combine to build the coverage database, and it's painfully slow - takes almost 3 hours to build. It combines 690 different coverage files together.

Three hours is a really long time, that's surprising. What version of coverage and Python are you using?

I haven't thought about multithreading or multiprocessing. It would be instructive to have a reproducible case so we can see where the time is being spent.

[nedbat]

@jbkkd (see above)

[nedbat]

@jbkkd BTW, I can consult hourly with an NDA if needed to work on private code.

[nedbat]

I fixed a few quadratic-time behaviors in the combining code. The 3+ hour combine now takes 7 minutes.
This is now released as part of coverage 7.5.3.