feat: Optimize performance for large-scale harvest/enrichment #83
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Implement folder grouping for extraction files to mitigate performance issues caused by large numbers of files in a single directory. - Previously, when enriching a collection of 20k items, performance degraded significantly as the extraction folder grew in size due to slow file operations. - Files are now grouped into subdirectories in batches of 100, significantly improving performance for large datasets. -⚠️ BREAKING CHANGE: Existing extractions must be recreated to be compatible with the new directory structure.
Code quality scoreUh oh! The code quality got worse for this PR! Better take a look!! 🚨
|
motizuki
force-pushed
the
gm/extraction-improv
branch
from
cfd6c65 to
141838e
Compare
motizuki
force-pushed
the
gm/extraction-improv
branch
from
1ab8a7a to
4c055d4
Compare
richardmatthewsdev
approved these changes
Aug 28, 2024
paul-mesnilgrente
approved these changes
Aug 28, 2024
motizuki
force-pushed
the
gm/extraction-improv
branch
14 times, most recently
from
6c90279 to
9756bc9
Compare
- loop through folders and files
motizuki
force-pushed
the
gm/extraction-improv
branch
from
9756bc9 to
371837a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement folder grouping for extraction files to mitigate performance issues caused by large numbers of files in a single directory.
Previously, when enriching a collection of 20k items, performance degraded significantly as the extraction folder grew in size due to slow file operations.
Files are now grouped into subdirectories in batches of 100, significantly improving performance for large datasets.