Dataframe queries 1: schema resolution #7339
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
teh-cmc
added
⛃ re_datastore
This was referenced Sep 3, 2024
jleibs
approved these changes
Sep 3, 2024
6 tasks
teh-cmc
added a commit
that referenced
this pull request
Sep 4, 2024
All the boilerplate for the new `re_dataframe`. Also introduces all the new types: * `QueryExpression`, `LatestAtQueryExpression`, `RangeQueryExpression` * `QueryHandle`, `LatestAtQueryHandle` (unimplemented), `RangeQueryHandle` (unimplemented) * `ColumnDescriptor`, `ControlColumnDescriptor`, `TimeColumnDescriptor`, `ComponentColumnDescriptor` No actual code logic, just definitions. * Part of #7284 --- Dataframe APIs PR series: - #7338 - #7339 - #7340 - #7341 - #7345
teh-cmc
force-pushed
the
cmc/dataframe_queries_1_schemas
branch
from
da53000 to
3bc2a02
Compare
teh-cmc
added a commit
that referenced
this pull request
Sep 4, 2024
Implements the latest-api dataframe API.
Examples:
```
cargo r --all-features -p re_dataframe --example latest_at -- /tmp/helix.rrd
cargo r --all-features -p re_dataframe --example latest_at -- /tmp/helix.rrd /helix/structure/scaffolding/**
```
```rust
use itertools::Itertools as _;
use re_chunk::{TimeInt, Timeline};
use re_chunk_store::{ChunkStore, ChunkStoreConfig, LatestAtQueryExpression, VersionPolicy};
use re_dataframe::QueryEngine;
use re_log_types::StoreKind;
fn main() -> anyhow::Result<()> {
let args = std::env::args().collect_vec();
let get_arg = |i| {
let Some(value) = args.get(i) else {
eprintln!(
"Usage: {} <path_to_rrd> <entity_path_expr>",
args.first().map_or("$BIN", |s| s.as_str())
);
std::process::exit(1);
};
value
};
let path_to_rrd = get_arg(1);
let entity_path_expr = args.get(2).map_or("/**", |s| s.as_str());
let stores = ChunkStore::from_rrd_filepath(
&ChunkStoreConfig::DEFAULT,
path_to_rrd,
VersionPolicy::Warn,
)?;
for (store_id, store) in &stores {
if store_id.kind != StoreKind::Recording {
continue;
}
let cache = re_dataframe::external::re_query::Caches::new(store);
let engine = QueryEngine {
store,
cache: &cache,
};
let query = LatestAtQueryExpression {
entity_path_expr: entity_path_expr.into(),
timeline: Timeline::log_time(),
at: TimeInt::MAX,
};
let query_handle = engine.latest_at(&query, None /* columns */);
let batch = query_handle.get();
eprintln!("{query}:\n{batch}");
}
Ok(())
}
```
* Part of #7284
---
Dataframe APIs PR series:
- #7338
- #7339
- #7340
- #7341
- #7345
teh-cmc
added a commit
that referenced
this pull request
Sep 4, 2024
Implements the dense range dataframe APIs.
Examples:
```
cargo r --all-features -p re_dataframe --example range -- /tmp/data.rrd /helix/structure/scaffolding/beads
cargo r --all-features -p re_dataframe --example range -- /tmp/data.rrd /helix/structure/scaffolding/beads /helix/structure/scaffolding/**
```
```rust
use itertools::Itertools as _;
use re_chunk_store::{
ChunkStore, ChunkStoreConfig, ComponentColumnDescriptor, RangeQueryExpression, Timeline,
VersionPolicy,
};
use re_dataframe::QueryEngine;
use re_log_types::{ResolvedTimeRange, StoreKind};
fn main() -> anyhow::Result<()> {
let args = std::env::args().collect_vec();
let get_arg = |i| {
let Some(value) = args.get(i) else {
eprintln!(
"Usage: {} <path_to_rrd_with_position3ds> <entity_path_pov> [entity_path_expr]",
args.first().map_or("$BIN", |s| s.as_str())
);
std::process::exit(1);
};
value
};
let path_to_rrd = get_arg(1);
let entity_path_pov = get_arg(2).as_str();
let entity_path_expr = args.get(3).map_or("/**", |s| s.as_str());
let stores = ChunkStore::from_rrd_filepath(
&ChunkStoreConfig::DEFAULT,
path_to_rrd,
VersionPolicy::Warn,
)?;
for (store_id, store) in &stores {
if store_id.kind != StoreKind::Recording {
continue;
}
let cache = re_dataframe::external::re_query::Caches::new(store);
let engine = QueryEngine {
store,
cache: &cache,
};
let query = RangeQueryExpression {
entity_path_expr: entity_path_expr.into(),
timeline: Timeline::log_tick(),
time_range: ResolvedTimeRange::new(0, 30),
pov: ComponentColumnDescriptor::new::<re_types::components::Position3D>(
entity_path_pov.into(),
),
};
let query_handle = engine.range(&query, None /* columns */);
eprintln!("{query}:");
for batch in query_handle.into_iter() {
eprintln!("{batch}");
}
}
Ok(())
}
```
* Fixes #7284
---
Dataframe APIs PR series:
- #7338
- #7339
- #7340
- #7341
- #7345
teh-cmc
added a commit
that referenced
this pull request
Sep 4, 2024
Implements the paginated dense range dataframe APIs. If there's no off-by-one anywhere in there, I will eat my hat. Getting this in the hands of people is the highest prio though, I'll add tests later.   * Fixes #7284 --- Dataframe APIs PR series: - #7338 - #7339 - #7340 - #7341 - #7345
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The schema resolution logic.
Dataframe APIs PR series:
Checklist
mainbuild: rerun.io/viewernightlybuild: rerun.io/viewerCHANGELOG.mdand the migration guideTo run all checks from
main, comment on the PR with@rerun-bot full-check.