Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Native data iteration from python #5186

Closed
devinrsmith opened this issue Feb 22, 2024 · 3 comments · Fixed by #5595
Closed

Native data iteration from python #5186

devinrsmith opened this issue Feb 22, 2024 · 3 comments · Fixed by #5595
Assignees
Labels
core Core development tasks feature request New feature or request python python-server-side
Milestone

Comments

@devinrsmith
Copy link
Member

I'd like a native way to iterate over Table data from python. I think it can be efficient and batching behind the scenes. Feature wise, I think it needs to ability to iterate all the columns implicitly, or just a subset (without the need to create a view).

# iterate over all columns
for x, y, z in my_table:
  ...
  
# iterate over some columns
for foo, bar in my_table.iter(["Foo", "Bar"]):
   ...

Maybe the interface would need to be context manager based though? If so, then maybe something like:

with my_table.data() as data:
  for x, y, z in data:
    ...

with my_table.data(["Foo", "Bar"]) as data:
  for foo, bar in data:
    ...

I'm not picky about the naming.

As a workaround, I can currently do something like this:

for x, y, z in to_numpy(my_table):
    ...

for foo, bar in to_numpy(my_table, ["Foo", "Bar"]):
    ...

but this is an all-at-once approach that is only to be used with "small" tables.

@devinrsmith devinrsmith added feature request New feature or request triage labels Feb 22, 2024
@devinrsmith devinrsmith added this to the 3. Triage milestone Feb 22, 2024
@rcaudy rcaudy added core Core development tasks python python-server-side and removed triage labels Feb 22, 2024
@rcaudy rcaudy modified the milestones: 3. Triage, 4. Unscheduled Feb 22, 2024
@devinrsmith
Copy link
Member Author

Potential code in deephaven.table_listener._changes_to_numpy

@chipkent
Copy link
Member

How would this work with ticking tables? The iterator could change under you?

@jjbrosnan
Copy link
Contributor

It would be pretty awesome to enable the use of enumerate with something like this as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core development tasks feature request New feature or request python python-server-side
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants