Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tracking issue: Feature parity Arrow-based SDK logging #387

Closed
3 tasks done
Tracked by #376
jondo2010 opened this issue Nov 28, 2022 · 1 comment
Closed
3 tasks done
Tracked by #376

Tracking issue: Feature parity Arrow-based SDK logging #387

jondo2010 opened this issue Nov 28, 2022 · 1 comment
Labels
🏹 arrow concerning arrow 🐍 Python API Python logging API
Milestone
Arrow/Polars base...

Comments

@jondo2010
Copy link
Contributor

jondo2010 commented Nov 28, 2022 •
edited by nikolausWest
Loading

Expand the Python and Rust SDK:s to cover more data types.

Some things to figure out:

How should we store large components like Tensors and Meshes?

  • Structured data in Arrow
  • Blobs in Arrow
  • In a separate store which the Arrow table references with MeshId/TensorID
@jondo2010 jondo2010 mentioned this issue Nov 28, 2022
Closed
4 tasks
@emilk emilk mentioned this issue Nov 28, 2022
Closed
@emilk emilk added 🐍 Python API Python logging API 🏹 arrow concerning arrow labels Nov 28, 2022
@jondo2010 jondo2010 changed the title Solidify Arrow-based SDK layers Meta issue: Solidify Arrow-based SDK layers Dec 12, 2022
@jondo2010 jondo2010 changed the title Meta issue: Solidify Arrow-based SDK layers Meta issue: Additional Arrow-based SDK layers Dec 12, 2022
@jondo2010 jondo2010 changed the title Meta issue: Additional Arrow-based SDK layers Tracking issue: Additional Arrow-based SDK layers Dec 12, 2022
@teh-cmc
Copy link
Member

teh-cmc commented Dec 12, 2022

How should we store large components like Tensors and Meshes?

  • Structured data in Arrow
  • Blobs in Arrow
  • In a separate store which the Arrow table references with MeshId/TensorID

I guess we might as well use blobs and go straight for the generalized solution proposed by @emilk in #487.

@emilk emilk changed the title Tracking issue: Additional Arrow-based SDK layers Tracking issue: Feature parity Arrow-based SDK logging Dec 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🏹 arrow concerning arrow 🐍 Python API Python logging API
Projects
None yet
Milestone
Arrow/Polars based data engine 1.0
Development

No branches or pull requests
4 participants
@jondo2010 @emilk @nikolausWest @teh-cmc