Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(python) add arguments on read_csv to use all pyarrow options #6699

Closed
wants to merge 1 commit into from
Closed

feat(python) add arguments on read_csv to use all pyarrow options #6699

wants to merge 1 commit into from

Conversation

igmriegel
Copy link
Contributor

@igmriegel igmriegel commented Feb 6, 2023

This pull request gives full access to the pyarrow reading options, my team came across the necessity of parsing floats with comma as decimal separator wich is something we can't do on Polars right now, but pyarrow can and it also provides some other details for parsing, reading and converting.

The PR I'm referencing bellow raises an issue about comma as decimal separator wich can be solved with the alterations on my PR.
#6698

This is my first contribution, if it doesn't meet our standards, I will gladly rework on it.

@github-actions github-actions bot added enhancement New feature or an improvement of an existing feature python Related to Python Polars labels Feb 6, 2023
@alexander-beedie
Copy link
Collaborator

alexander-beedie commented Feb 6, 2023

Thanks for the PR ;) However, this isn't really the right way to go; ideally the polars-native Rust parser would be enhanced to handle both "," and "." decimals - if you want to use additional pyarrow features to create an Arrow from CSV (for now) then you can already do that by using pyarrow directly and passing the resulting table into polars at essentially zero cost.

I think I can see where the polars-native parser would need to be extended, and it doesn't look like a large job. Might take me a few days to get to it though, as I'm having a ridiculously busy week at work...

@igmriegel
Copy link
Contributor Author

@alexander-beedie
Thanks for the attention, I agree that it is better to have the features natively
I hope I can make some useful work in the near future 😃
I'm closing this PR.

Have a nice week

@igmriegel igmriegel closed this Feb 7, 2023
@igmriegel igmriegel deleted the aditional-options-pyarrow branch February 7, 2023 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or an improvement of an existing feature python Related to Python Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants