vaex expression to pandas.Series support #456
Conversation
JovanVeljanoski
force-pushed
the
to_series
branch
from
fdb46f5 to
86e2494
Compare
| def to_pandas_series(self): | ||
| """Return a pandas.Series representation of the expression. | ||
|
|
||
| Note: a memory copy is created. |
There was a problem hiding this comment.
I don't think that is true. If a column is a non-virtual one, we just pass the ndarray, I doubt pandas will copy that.
There was a problem hiding this comment.
Well.. if a vaex dataframe has say 2 billion rows and it is reading them from an hdf5 file, the moment you do to_pandas_series i think it will materialize them to in memory right..?
There was a problem hiding this comment.
No, the columns in vaex, are memory mapped (usually), so it's a regular numpy array, pointing to a piece of memory mapped data, there is no 'materializing', there is no difference between a memory mapped array or a in memory array except when you start accessing it. Passing the numpy array around does not cause any copying, so as long as pandas does not copy it (I think it does not when you pass it to an Series but you can check).
No description provided.