Remove --reset option from provenance.py (force to always write history from scratch) #311
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
also renames some variables
![sonatype-lift[bot]](https://avatars.githubusercontent.com/in/9843?s=60&v=4){kind=small}
nutjob4life
reviewed
Apr 11, 2023
| log.info(' aggregate lidvids into lid buckets') | ||
| aggregates = {lid: [] for lid in lids} | ||
| for lidvid in provenance: aggregates[lidvid.split('::')[0]].append(lidvid) | ||
| log.info(' ...binning LIDVIDs by LID...') |
There was a problem hiding this comment.
Me being pedantic: use an actual ellipsis character … instead of three periods ... 😅
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🗒️ Summary
Removes option to write iterative provenance updates, as this does not improve performance and may result in invalid provenance history in some edge cases.
Also involves extensive renaming and comments in an attempt to improve documentation of provenance script and speed up future changes.
⚙️ Test Data and/or Report
@jimmie do we have any tests that incorporate provenance? I've tested manually against a minimal dataset.
♻️ Related Issues
fixes #310
If one/some reviewers could sanity-check the comments/renaming, I'd appreciate it - I'm pretty confident that I've captured what's actually going on, but extra eyes never hurt. Most of the individual commits should be simple-ish to read, taken individually.