-
Notifications
You must be signed in to change notification settings - Fork 366
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Switch to eager asof join to speed up arrow queries #583
Conversation
ac112e7
to
7a07aa8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code LGTM, and if it passes the tests I guess it works :)
.iter() | ||
.zip(results) | ||
.filter_map(|(component, col)| col.map(|col| (component, col))) | ||
.map(|(&component, col)| Series::try_from((component, col))) | ||
.map(|(&component, col)| { | ||
// TODO(jleibs): Is it possible to have multiple rows here? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
multiple rows per component? I don't quite understand the context for the question.
adding a debug_assert!
is a good start to answering that question
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This call & the question both go away with #590
crates/re_query/src/query.rs
Outdated
if df.column(Instance::NAME).is_ok() { | ||
// If we have an InstanceKey column already, make sure that it's sorted. | ||
// TODO(jleibs): can remove this once we have a sort guarantee from the store | ||
Ok(df.sort([Instance::NAME], false)?) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok(df.sort([Instance::NAME], false)?) | |
let reverse = false; | |
Ok(df.sort([Instance::NAME], reverse)?) |
7a07aa8
to
f6e62fb
Compare
The previous join operation was quite expensive, asof_join is much less flexible, but significantly more performant.
Before:
After:
Checklist
CHANGELOG.md
(if this is a big enough change to warrant it)