Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Polars backend can read only 1 csv #9230

Closed
1 task done
djouallah opened this issue May 22, 2024 · 1 comment · Fixed by #9232
Closed
1 task done

Polars backend can read only 1 csv #9230

djouallah opened this issue May 22, 2024 · 1 comment · Fixed by #9232
Assignees
@gforsyth
Labels
feature Features or general enhancements io Issues related to input and/or output polars The polars backend

Comments

@djouallah
Copy link

Is your feature request related to a problem?

Polars can read multiple files at the same time but in ibis, it can read only 1 file at the time which is not practical
https://ibis-project.org/backends/polars#ibis.backends.polars.Backend.read_csv

What is the motivation behind your request?

No response

Describe the solution you'd like

I would like ibis when using polars backend to be able to do this

ibis.read_csv(['file1,csv','file2.csv'])

What version of ibis are you running?

10.0.0.dev71

What backend(s) are you using, if any?

Polars

Code of Conduct

  • I agree to follow this project's Code of Conduct
@djouallah djouallah added the feature Features or general enhancements label May 22, 2024
@gforsyth gforsyth added polars The polars backend io Issues related to input and/or output labels May 22, 2024
@gforsyth
Copy link
Member

Thanks for the issue, @djouallah ! We use pl.scan_csv to keep things lazy and they didn't used to support multiple files, but now that they do, this should be relatively straightforward to add.

@gforsyth gforsyth self-assigned this May 22, 2024
@gforsyth gforsyth mentioned this issue May 22, 2024
Merged
gforsyth added a commit that referenced this issue May 22, 2024
@gforsyth
feat(polars): accept list of CSVs to read_csv (#9232)
7a272e3 
Polars now accepts a list of CSVs to `scan_csv` so we can expose this to
end-users.

A few caveats:
- it doesn't accept lists of globs, or lists of compressed CSVs, so we
  flatten the list of paths if it only has one element in case it is a
  glob or csv.gz

Fixes #9230
@github-project-automation github-project-automation bot moved this from backlog to done in Ibis planning and roadmap May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gforsyth gforsyth

Labels
feature Features or general enhancements io Issues related to input and/or output polars The polars backend
Projects
Ibis planning and roadmap
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(polars): accept list of CSVs to read_csv gforsyth/ibis
2 participants
@gforsyth @djouallah