Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize ZIP format inference #3404

Closed
albertvillanova opened this issue Dec 7, 2021 · 0 comments · Fixed by #3407
Closed

Optimize ZIP format inference #3404

albertvillanova opened this issue Dec 7, 2021 · 0 comments · Fixed by #3407
Assignees
@albertvillanova
Labels
enhancement New feature or request

Comments

@albertvillanova
Copy link
Member

Is your feature request related to a problem? Please describe.
When hundreds of ZIP files are present in a dataset, format inference takes too long.

See: bigscience-workshop/data_tooling#232 (comment)

Describe the solution you'd like
Iterate over a maximum number of files.

CC: @lhoestq
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@albertvillanova albertvillanova

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use max number of data files to infer module huggingface/datasets
1 participant
@albertvillanova