Dataframe queries 4: paginated dense range #7345
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
teh-cmc
added
⛃ re_datastore
6 tasks
|
uhoh, forgot one of the commits |
teh-cmc
force-pushed
the
cmc/dataframe_queries_5_range_pagination
branch
from
108a49f to
78d212a
Compare
|
there we go |
This was referenced Sep 3, 2024
teh-cmc
commented
Sep 3, 2024
teh-cmc
force-pushed
the
cmc/dataframe_queries_5_range_pagination
branch
from
78d212a to
2029369
Compare
abey79
approved these changes
Sep 3, 2024
jleibs
requested changes
Sep 3, 2024
jleibs
approved these changes
Sep 3, 2024
teh-cmc
added a commit
that referenced
this pull request
Sep 4, 2024
All the boilerplate for the new `re_dataframe`. Also introduces all the new types: * `QueryExpression`, `LatestAtQueryExpression`, `RangeQueryExpression` * `QueryHandle`, `LatestAtQueryHandle` (unimplemented), `RangeQueryHandle` (unimplemented) * `ColumnDescriptor`, `ControlColumnDescriptor`, `TimeColumnDescriptor`, `ComponentColumnDescriptor` No actual code logic, just definitions. * Part of #7284 --- Dataframe APIs PR series: - #7338 - #7339 - #7340 - #7341 - #7345
teh-cmc
added a commit
that referenced
this pull request
Sep 4, 2024
Implements the latest-api dataframe API.
Examples:
```
cargo r --all-features -p re_dataframe --example latest_at -- /tmp/helix.rrd
cargo r --all-features -p re_dataframe --example latest_at -- /tmp/helix.rrd /helix/structure/scaffolding/**
```
```rust
use itertools::Itertools as _;
use re_chunk::{TimeInt, Timeline};
use re_chunk_store::{ChunkStore, ChunkStoreConfig, LatestAtQueryExpression, VersionPolicy};
use re_dataframe::QueryEngine;
use re_log_types::StoreKind;
fn main() -> anyhow::Result<()> {
let args = std::env::args().collect_vec();
let get_arg = |i| {
let Some(value) = args.get(i) else {
eprintln!(
"Usage: {} <path_to_rrd> <entity_path_expr>",
args.first().map_or("$BIN", |s| s.as_str())
);
std::process::exit(1);
};
value
};
let path_to_rrd = get_arg(1);
let entity_path_expr = args.get(2).map_or("/**", |s| s.as_str());
let stores = ChunkStore::from_rrd_filepath(
&ChunkStoreConfig::DEFAULT,
path_to_rrd,
VersionPolicy::Warn,
)?;
for (store_id, store) in &stores {
if store_id.kind != StoreKind::Recording {
continue;
}
let cache = re_dataframe::external::re_query::Caches::new(store);
let engine = QueryEngine {
store,
cache: &cache,
};
let query = LatestAtQueryExpression {
entity_path_expr: entity_path_expr.into(),
timeline: Timeline::log_time(),
at: TimeInt::MAX,
};
let query_handle = engine.latest_at(&query, None /* columns */);
let batch = query_handle.get();
eprintln!("{query}:\n{batch}");
}
Ok(())
}
```
* Part of #7284
---
Dataframe APIs PR series:
- #7338
- #7339
- #7340
- #7341
- #7345
teh-cmc
force-pushed
the
cmc/dataframe_queries_4_range
branch
from
67df21f to
b1d1c51
Compare
teh-cmc
added a commit
that referenced
this pull request
Sep 4, 2024
Implements the dense range dataframe APIs.
Examples:
```
cargo r --all-features -p re_dataframe --example range -- /tmp/data.rrd /helix/structure/scaffolding/beads
cargo r --all-features -p re_dataframe --example range -- /tmp/data.rrd /helix/structure/scaffolding/beads /helix/structure/scaffolding/**
```
```rust
use itertools::Itertools as _;
use re_chunk_store::{
ChunkStore, ChunkStoreConfig, ComponentColumnDescriptor, RangeQueryExpression, Timeline,
VersionPolicy,
};
use re_dataframe::QueryEngine;
use re_log_types::{ResolvedTimeRange, StoreKind};
fn main() -> anyhow::Result<()> {
let args = std::env::args().collect_vec();
let get_arg = |i| {
let Some(value) = args.get(i) else {
eprintln!(
"Usage: {} <path_to_rrd_with_position3ds> <entity_path_pov> [entity_path_expr]",
args.first().map_or("$BIN", |s| s.as_str())
);
std::process::exit(1);
};
value
};
let path_to_rrd = get_arg(1);
let entity_path_pov = get_arg(2).as_str();
let entity_path_expr = args.get(3).map_or("/**", |s| s.as_str());
let stores = ChunkStore::from_rrd_filepath(
&ChunkStoreConfig::DEFAULT,
path_to_rrd,
VersionPolicy::Warn,
)?;
for (store_id, store) in &stores {
if store_id.kind != StoreKind::Recording {
continue;
}
let cache = re_dataframe::external::re_query::Caches::new(store);
let engine = QueryEngine {
store,
cache: &cache,
};
let query = RangeQueryExpression {
entity_path_expr: entity_path_expr.into(),
timeline: Timeline::log_tick(),
time_range: ResolvedTimeRange::new(0, 30),
pov: ComponentColumnDescriptor::new::<re_types::components::Position3D>(
entity_path_pov.into(),
),
};
let query_handle = engine.range(&query, None /* columns */);
eprintln!("{query}:");
for batch in query_handle.into_iter() {
eprintln!("{batch}");
}
}
Ok(())
}
```
* Fixes #7284
---
Dataframe APIs PR series:
- #7338
- #7339
- #7340
- #7341
- #7345
implement paginated dense range dataframe api
teh-cmc
force-pushed
the
cmc/dataframe_queries_5_range_pagination
branch
from
539309e to
1f16c71
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implements the paginated dense range dataframe APIs.
If there's no off-by-one anywhere in there, I will eat my hat.
Getting this in the hands of people is the highest prio though, I'll add tests later.
Dataframe APIs PR series:
Checklist
mainbuild: rerun.io/viewernightlybuild: rerun.io/viewerCHANGELOG.mdand the migration guideTo run all checks from
main, comment on the PR with@rerun-bot full-check.