Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add primitive arrays column wrappers #30

Open
altavir opened this issue Jun 28, 2021 · 4 comments · May be fixed by #712
Open

Add primitive arrays column wrappers #30

altavir opened this issue Jun 28, 2021 · 4 comments · May be fixed by #712
Labels
performance Something related to how fast the library can handle data research This requires a deeper dive to gather a better understanding
Milestone
Backlog

Comments

@altavir
Copy link

altavir commented Jun 28, 2021

Primitive array columns are required for optimized big-data applications. It is also possible to add numerical DataFrame integration with MultiK or KMath.
@nikitinas
Copy link
Contributor

nikitinas commented Dec 22, 2021 •
edited
Loading

Our current idea is to use Arrow as a backend for primitive types.
See #78

@altavir
Copy link
Author

altavir commented Dec 23, 2021

It is a great idea, but it will be worth it only in terms of interop with other platforms. For JVM-only, Arrow will give nothing new.

@nikitinas
Copy link
Contributor

Arrow should give significant performance increase for JVM due to nullable values types support. Current implementation generates quite a lot of boxing/unboxing. It can be solved without Arrow, but I expect Arrow implementation to be faster. We will do performance benchmarks before implementation.

And we need to support Arrow I/O anyway.

@Jolanrensen Jolanrensen mentioned this issue Apr 17, 2023
Open
@zaleslaw zaleslaw added the research This requires a deeper dive to gather a better understanding label Apr 25, 2023
@zaleslaw zaleslaw added this to the 0.11.0 milestone Apr 25, 2023
@zaleslaw zaleslaw added the performance Something related to how fast the library can handle data label Apr 25, 2023
@zaleslaw zaleslaw modified the milestones: 0.11.0, Backlog Jun 12, 2023
@Jolanrensen
Copy link
Collaborator

I was experimenting with asList() wrappers. Maybe this could solve this long-standing issue:
master...primitive-array-value-columns
but it needs more testing.

This was referenced Jun 3, 2024
Draft
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
performance Something related to how fast the library can handle data research This requires a deeper dive to gather a better understanding
Projects
None yet
Milestone
Backlog
Development

Successfully merging a pull request may close this issue.

Research: ColumnDataHolder/primitive arrays Kotlin/dataframe
4 participants
@zaleslaw @nikitinas @altavir @Jolanrensen