Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Monitor for AssetBlobs with undefined sha256 #471

Open
dchiquito opened this issue Aug 10, 2021 · 3 comments · Fixed by #1937
Open

Monitor for AssetBlobs with undefined sha256 #471

dchiquito opened this issue Aug 10, 2021 · 3 comments · Fixed by #1937
Assignees
Labels
enhancement New feature or request released This issue/pull request has been released.

Comments

@dchiquito
Copy link
Contributor

The sha256 checksum of newly uploaded blobs is calculated by an asynchronous task, which may fail. We should monitor the DB for older AssetBlobs which do not have a sh256 field, and have a way to conveniently recompute those checksums as necessary.

#254 is an instance of this happening.

@yarikoptic
Copy link
Member

yarikoptic commented Aug 11, 2021

FWIW, I have reverted not issuing a warning in our dataladification script: dandi/dandisets@3cd51b5 so now I think I should be getting those warnings in cron job report. Not a solution, but at least we might have some means (if I spot it) for detecting if smth like that happens again.

edit: note that the check would happen only if assets list (with metadata) got any changes.

so I tested if all assets currently as after a fix which fetched all updated and now have sha2-256 have it - and it seems to be all good
dandi@drogon:/mnt/backup/dandi/dandisets$ for d in 000*; do jq '.[].metadata.digest."dandi:sha2-256"' $d/.dandi/assets.json | grep -q '^null' && echo 'missing'; done
dandi@drogon:/mnt/backup/dandi/dandisets$ 

@waxlamp waxlamp added DX Affects developer experience enhancement New feature or request labels Dec 31, 2021
@waxlamp waxlamp removed the DX Affects developer experience label Mar 8, 2023
@waxlamp waxlamp added this to the Dandiset lifecycle milestone Mar 8, 2023
yarikoptic added a commit that referenced this issue May 10, 2024
…r a blob

Added ability to specify either blob id (where it applies) or asset id since
that is the id we typically know for which a blob lacks sha256.

Refs:
- #471
- #1578
yarikoptic added a commit that referenced this issue May 10, 2024
…r a blob

Added ability to specify either blob id (where it applies) or asset id since
that is the id we typically know for which a blob lacks sha256.

Refs:
- #471
- #1578
yarikoptic added a commit that referenced this issue May 10, 2024
…r a blob

Added ability to specify either blob id (where it applies) or asset id since
that is the id we typically know for which a blob lacks sha256.

Refs:
- #471
- #1578
yarikoptic added a commit that referenced this issue Jun 5, 2024
…r a blob

Added ability to specify either blob id (where it applies) or asset id since
that is the id we typically know for which a blob lacks sha256.

Refs:
- #471
- #1578
@dandibot
Copy link
Member

🚀 Issue was released in v0.3.92 🚀

@dandibot dandibot added the released This issue/pull request has been released. label Jul 30, 2024
@jjnesbitt jjnesbitt reopened this Jul 30, 2024
@jjnesbitt
Copy link
Member

#1937 Does not directly address this any longer (as I've reworked it), and I plan on addressing this fully in a future PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request released This issue/pull request has been released.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants