Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stream csv file directly to a fsttable object #7

Open
MarcusKlik opened this issue Mar 24, 2018 · 0 comments
Open

Stream csv file directly to a fsttable object #7

MarcusKlik opened this issue Mar 24, 2018 · 0 comments

Comments

@MarcusKlik
Copy link
Collaborator

We will need a method csv_to_fst in the fst package (planned). By using that method under the hood, we can define a fsttable object from a csv file:

ft <- fst_table_from_csv("somebigfile.csv", columns = c("A", "B", "C"))
print(ft[, p_sum(A)])

This would calculate the sum of column A in a csv file that is possibly too large to read with currently existing methods. With this code snippet, the csv file would be streamed (in blocks) to a fst file first, and then a fsttable reference is returned.

(and a new file somebigfile.fst would also be created simultaneously)

@MarcusKlik MarcusKlik added this to the Import csv milestone Mar 24, 2018
@MarcusKlik MarcusKlik changed the title Stream _csv_ file directly to a fsttable object Stream csv file directly to a fsttable object Mar 27, 2018
@MarcusKlik MarcusKlik modified the milestones: Import csv, Candidate Sep 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant