-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
zarr_content_read: include sizes for the "files" (and possibly "directories") #925
Comments
I am addressing this in #937 |
Reopening this to track the work required to update the dandischema version to |
Relatedly, if the endpoint also included "last modified" dates and S3 keys (and possibly also S3 version IDs) for Zarr entries, that would help reduce the number of requests needed by the backup script. |
🚀 Issue was released in |
@dchiquito I don't see how #991 resolves the original issue here. How does one get a file's size from a |
can't test ATM due to #1004 , will reopen just so it doesn't RiP while should be alive and chatty (feel welcome to re-close upon demonstration/confirmation of the effect achieved) ;) |
@yarikoptic, @jwodder, is this still important? How high of a priority is it at the moment? |
@waxlamp We removed the part of the upload code that checks local vs. remote file sizes (largely because this issue wasn't implemented). On the other hand, the download code still needs to makes a separate HEAD request before each Zarr entry download in order to get the size and modification time. I'd say it's no longer a priority, but I suppose it would be nice to have. |
motivation: dandi/dandi-cli#923 (comment) -- in dandi-cli, to possibly avoid relatively expensive checksumming of the content to decide either a (changed) file within zarr needs to be reuploaded, we first check the size(s) of the file. If that one changed -- we know that we need to reupload even without computing its checksum.
It would be helpful if that API endpoint also returned sizes of the files (if it can do it without extra runtime penalty)
The text was updated successfully, but these errors were encountered: