Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new function that reads data directly into Julia arrays (or static arrays) #624

Closed
HTofi opened this issue May 3, 2020 · 2 comments

Comments

@HTofi
Copy link

HTofi commented May 3, 2020

Hi,

I've been experimenting with the CSV package and reading through the documentation, and I was surprised to find no function that can read data directly into Julia arrays.

The CSV.File function reads data into a CSV.File data structure, which is a read-only data type not as flexible as arrays or even static arrays. For example:

julia> using StaticArrays, Tables

julia> x = SMatrix{3}(1, 2, 3) # returns a static array

julia> 2*x # returns a static array

julia> x = 2*x # executes normally

julia> CSV.write("data.csv", Tables.table(x, header=[:x]))

julia> f = CSV.File("data.csv")

julia> f.x # returns a CSV.column object

julia> 2 * f.x # returns an **array** object

julia> f.x = 2 * f.x # returns an error

The CSV.read function outputs a DataFrame object which can be pretty versatile, though only if the data is copied, which means more memory usage. Plus, sometimes one just needs to access and manipulate the data in form of simple arrays, without the need for something as sophisticated as a DataFrame.

Therefore, I suggest adding a new function to the package that can read data directly into arrays (and/or static arrays), so that users can have better versatility in controlling the data. I wrote a simple wrapper for CSV.read that converts the DataFrame into arrays, though I think this wrapper will still copy the data in the convert step, so it is not very ideal:

function readarray(source; unpack::Bool=false, kwargs...)
	df = CSV.read(source; kwargs...)
	if !unpack
		return convert(Matrix, df)   # returns 2-D array
	else
		return (convert(Vector, x) for x in eachcol(df))   # returns tuple of 1-D arrays
	end
end

I hope the developers would respond positively to my request!

Thank you.

@HTofi HTofi changed the title Add a new function that reads data durectly into Julia arrays (or static arrays) Add a new function that reads data directly into Julia arrays (or static arrays) May 3, 2020
@quinnj
Copy link
Member

quinnj commented Jun 18, 2020

This was implemented in #639

@quinnj quinnj closed this as completed Jun 18, 2020
@stevengj
Copy link

stevengj commented Dec 3, 2023

In particular, you can do e.g.

CSV.File("data.csv") |> CSV.Tables.matrix # returns a matrix

(I have to say that this was a bit buried in the manual. It would be nice if this were one of the very first examples.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants