-
Notifications
You must be signed in to change notification settings - Fork 522
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
refactor: simplify dataset construction (#4437)
<!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Introduced a new function for dataset construction, enhancing data loading processes. - Added a method to improve pickling and unpickling capabilities for path handling classes. - **Bug Fixes** - Updated summary printing to prevent redundant output during distributed training. - **Refactor** - Simplified initialization of the BackgroundConsumer class. - Streamlined consumer thread and queue handling in the BufferedIterator class. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Chun Cai <amoycaic@gmail.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Han Wang <92130845+wanghan-iapcm@users.noreply.github.com>
- Loading branch information
1 parent
ce9aeb3
commit b4ade5c
Showing
2 changed files
with
34 additions
and
41 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters