DONE% value when downloading a Zarr asset is not correct
#1407
Labels
Comments
yarikoptic
added a commit
that referenced
this issue
May 14, 2024
Since maxsize is dynamically computed as we go through the files. The idea, I guess, was that it would grow rapidly before actual downloads commense but it is not the case, so we endup with done% being always close to 100% since we get those reports on final downloads completed close to when individual files are downloaded. So this should close #1407 . But for total zarr file to be used, we needed to account also for skipped files. I added reporting of sizes for skipped files as well. It seems there is no negative side effect on regular files download. So now for the %done of zarr we might be getting to 100% of original size having downloaded nothing. But IMHO it is ok since user does not care as much of how many "subparts" are downloaded, but rather to have adequate progress report back. There also could be side effects if -e skip and we skip download of some updated files which would be smaller than the local ones so altogether we would get over 100% total at the end.
yarikoptic
added a commit
that referenced
this issue
May 15, 2024
Since maxsize is dynamically computed as we go through the files. The idea, I guess, was that it would grow rapidly before actual downloads commense but it is not the case, so we endup with done% being always close to 100% since we get those reports on final downloads completed close to when individual files are downloaded. So this should close #1407 . But for total zarr file to be used, we needed to account also for skipped files. I added reporting of sizes for skipped files as well. It seems there is no negative side effect on regular files download. So now for the %done of zarr we might be getting to 100% of original size having downloaded nothing. But IMHO it is ok since user does not care as much of how many "subparts" are downloaded, but rather to have adequate progress report back. There also could be side effects if -e skip and we skip download of some updated files which would be smaller than the local ones so altogether we would get over 100% total at the end.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
When downloading a single Zarr asset or downloading an entire Dandiset that contains a Zarr asset, I have found that the
DONE%value for the Zarr is constantly set as 98% or 99%. Whereas theSummarydisplays a seemingly more accurate progress (e.g. 1.95% in the screenshot below).Reproducibility
Thank you
The text was updated successfully, but these errors were encountered: