-
Notifications
You must be signed in to change notification settings - Fork 369
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DataLoader
s 3: add 3D point cloud DataLoader
(.ply
)
#4519
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
teh-cmc
added
📺 re_viewer
affects re_viewer itself
do-not-merge
Do not merge this PR
include in changelog
CLI
Related to the Rerun CLI
labels
Dec 13, 2023
This was referenced Dec 14, 2023
teh-cmc
force-pushed
the
cmc/dataloaders_2_text_docs
branch
from
December 14, 2023 09:57
094adb7
to
cc9ae19
Compare
teh-cmc
force-pushed
the
cmc/dataloaders_3_ply
branch
from
December 14, 2023 10:55
6592efb
to
bc08cff
Compare
teh-cmc
force-pushed
the
cmc/dataloaders_3_ply
branch
2 times, most recently
from
December 14, 2023 11:48
81cca1a
to
b16db00
Compare
7 tasks
Size changes
|
Ha yes, the classic plot size optimization of adding point cloud support 👌 |
16 tasks
teh-cmc
force-pushed
the
cmc/dataloaders_2_text_docs
branch
from
December 14, 2023 17:27
7b1b30a
to
21a7fe2
Compare
teh-cmc
force-pushed
the
cmc/dataloaders_3_ply
branch
from
December 14, 2023 17:34
e9ac988
to
3df0c5f
Compare
I forgot to update the |
emilk
approved these changes
Dec 15, 2023
teh-cmc
added a commit
that referenced
this pull request
Dec 15, 2023
**COMMIT PER COMMIT!** _That's the nasty one... there's always a nasty one 😬_ Introduces the `DataLoader` trait, which can load any file, either from a filepath by reading from the local filesystem, or from a pre-loaded file's contents. `DataLoader`s are run in parallel where possible, and have complete say over what data they are interested in or not (i.e. they are not registered based on an extension, which is very limited in practice). I've decided to commit small example assets for the types we support, cause I feel we really need a fast way of checking whether everything still works from time to time. It's pretty light but might be controversial, I figure. Checks: - [x] `cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets/example.{glb,gltf,obj,jpg,png,rrd}` - [x] Native: `File > Open > examples/assets/*` - [x] Native: `Drag-n-drop > examples/assets/*` - [x] Web: `File > Open > examples/assets/*` - [x] Web: `Drag-n-drop > examples/assets/*` --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - TODO: register custom loaders - TODO: high level docs and guides for everything related to loading files
teh-cmc
force-pushed
the
cmc/dataloaders_2_text_docs
branch
from
December 15, 2023 13:33
21a7fe2
to
bae6752
Compare
teh-cmc
added a commit
that referenced
this pull request
Dec 15, 2023
What the title says. ![image](https://github.com/rerun-io/rerun/assets/2910679/68f2e499-f4df-4e75-95f6-0ed7f479c5e6) Checks: - [x] `cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets/example.{glb,gltf,obj,jpg,png,rrd,txt,md}` - [x] Native: `File > Open > examples/assets/*` - [x] Native: `Drag-n-drop > examples/assets/*` - [x] Web: `File > Open > examples/assets/*` - [x] Web: `Drag-n-drop > examples/assets/*` --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - TODO: register custom loaders - TODO: high level docs and guides for everything related to loading files
teh-cmc
force-pushed
the
cmc/dataloaders_3_ply
branch
from
December 15, 2023 14:03
3df0c5f
to
d397007
Compare
teh-cmc
added a commit
that referenced
this pull request
Dec 15, 2023
Load folders filled with whatever. Go nuts. The loader has no particular semantics (e.g. HIVE partitioning): it just loads everything in a folder using the appropriate loader(s) for each file. Checks: - [x] `cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets` - [x] Native: `File > Open > examples/assets/` - [x] Native: `Drag-n-drop > examples/assets/` --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - TODO: register custom loaders - TODO: high level docs and guides for everything related to loading files
teh-cmc
added a commit
that referenced
this pull request
Dec 15, 2023
…#4521) Turn any executable in your `PATH` into a `DataLoader` by logging data to stdout. Includes examples for all supported languages. ![image](https://github.com/rerun-io/rerun/assets/2910679/82391961-9ac1-4bd8-bdbe-a7378ed7a3fa) Checks: - [x] `rerun-loader-rust-file` just prints help message - [x] `rerun-loader-rust-file examples/rust/external_data_loader/src/main.rs` successfully does nothing - [x] `rerun-loader-rust-file examples/rust/external_data_loader/src/main.rs | rerun -` works - [x] `rerun-loader-python-file` just prints help message - [x] `rerun-loader-python-file examples/python/external_data_loader/main.py` successfully does nothing - [x] `rerun-loader-python-file examples/python/external_data_loader/main.py | rerun -` works - [x] `rerun-loader-cpp-file` just prints help message - [x] `rerun-loader-cpp-file examples/cpp/external_data_loader/main.cpp` successfully does nothing - [x] `rerun-loader-cpp-file examples/cpp/external_data_loader/main.cpp | rerun -` works - [x] `cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets` - [x] Native: `File > Open > examples/assets/` - [x] Native: `Drag-n-drop > examples/assets/` --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - TODO: register custom loaders - TODO: high level docs and guides for everything related to loading files
This was referenced Dec 17, 2023
teh-cmc
added a commit
that referenced
this pull request
Dec 19, 2023
Introduce a first-class protocol for `DataLoader`s -- whether they are builtin, custom, or external -- to announce that they don't support loading a given piece of data. This fixes one of the oldest issue when loading files in Rerun: loading unsupported data would always bottom out in either the image or websocket paths, leading to unrelated errors very confusing to users. The loading process is now a two-pass process: 1. Dispatch the data to be loaded to all loaders, which will respond ASAP whether they can do it or not. 1.1. If at least one compatible loader is found, go to 2 1.2. Otherwise, fail early with a nice error message for the end user 2. Dispatch the actual data loading. This has important/subtle ramifications on the threading model, but other than that is what you'd expect. Checks: - [x] Native: CLI examples/assets/* - [x] Native: CLI examples/assets - [x] Native: CLI examples/assets/* containing unsupported files - [x] Native: CLI examples/assets containing unsupported files - [x] Native: File>Open examples/assets/* - [x] Native: File>Open examples/assets - [x] Native: File>Open examples/assets/* containing unsupported files - [x] Native: File>Open examples/assets containing unsupported files - [x] Native: Drag-n-drop examples/assets/* - [x] Native: Drag-n-drop examples/assets - [x] Native: Drag-n-drop examples/assets/* containing unsupported files - [x] Native: Drag-n-drop examples/assets containing unsupported files - [x] Web: File>Open examples/assets/* - [x] Web: Drag-n-drop examples/assets/* - [x] Web: File>Open examples/assets/* containing unsupported files - [x] Web: Drag-n-drop examples/assets/* containing unsupported files --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - #4565 - #4566 - #4567
teh-cmc
added a commit
that referenced
this pull request
Dec 19, 2023
Add support for registering custom loaders, the same way we offer support for registering custom space views and store subscribers. Checks: - [x] cargo r -p custom_data_loader -- Cargo.toml --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - #4565 - #4566 - #4567
teh-cmc
added a commit
that referenced
this pull request
Dec 22, 2023
Some docs for the whole thing. It heavily relies on linking to the examples for exact details and to avoid rot. It's probably extremely bad. Help welcome. --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - #4565 - #4566 - #4567 Co-authored-by: Nikolaus West <niko@rerun.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
.ply
support, but point clouds only, at least for now.No feature flag: I had some, and tbh it only made thing needlessly hard to work with for very little gains.
.ply
are so common anyway, they always come up in user requests.Here's an example from there:
Checks:
cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets/example.{glb,gltf,obj,jpg,png,rrd,txt,md,ply}
File > Open > examples/assets/*
Drag-n-drop > examples/assets/*
File > Open > examples/assets/*
Drag-n-drop > examples/assets/*
Part of a series of PRs to make it possible to load any file from the local filesystem, by any means, on web and native:
DataLoader
s 0: utility for hierarchicalEntityPath
from file path #4516DataLoader
s 1: introduce, and migrate to,DataLoader
s #4517DataLoader
s 2: add text-basedDataLoader
(.txt
,.md
) #4518DataLoader
s 3: add 3D point cloudDataLoader
(.ply
) #4519DataLoader
s 4: add generic folderDataLoader
#4520DataLoader
s 5: add support for external binaryDataLoader
s (PATH) #4521Checklist