Update the dataframe view to use `re_dataframe` and `egui_table` #7380

abey79 · 2024-09-09T15:42:11Z

What

Closes Dataframe view: add PoV to range query #7279

Major update to the dataframe view

display the data return by the new re_dataframe crate
- the PoV entity/component is now actually used
- entities are now always columns
  - see Add support for having entity path as rows #7379
use egui_table for the table
- hierarchical header
- sticky columns
- and much more...

TODO:

fix after merging Switch ArrowDict to signed and fix static datatype #7383

Checklist

I have read and agree to Contributor Guide and the Code of Conduct
I've included a screenshot or gif (if applicable)
I have tested the web demo (if applicable):
- Using examples from latest main build: rerun.io/viewer
- Using full set of examples from nightly build: rerun.io/viewer
The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG
If applicable, add a new check to the release checklist!
If have noted any breaking changes to the log API in CHANGELOG.md and the migration guide

To run all checks from main, comment on the PR with @rerun-bot full-check.

# Conflicts: # crates/store/re_chunk_store/src/dataframe.rs

crates/viewer/re_viewer/src/web.rs

crates/viewer/re_space_view_dataframe/src/space_view_class.rs

crates/viewer/re_space_view_dataframe/src/display_record_batch.rs

emilk · 2024-09-10T08:01:59Z

crates/viewer/re_space_view_dataframe/src/dataframe_ui.rs

+        }
+    }
+
+    fn get(&self, start: u64, num_rows: u64) -> Vec<RecordBatch> {


I think using rows: Range<u64> here (and elsewhere) instead would be more idiomatic and less error prone

Good call. I'll make a separate, follow-up PR for visibility, as it may affect @jleibs

crates/viewer/re_space_view_dataframe/src/dataframe_ui.rs

crates/viewer/re_space_view_dataframe/src/display_record_batch.rs

emilk · 2024-09-10T08:34:13Z

crates/viewer/re_space_view_dataframe/src/display_record_batch.rs

+                        row_id_times,
+                        row_id_counters,


minor nit: but "times" and "counters" only makes sense for RowId as TUID. If we switch to UUID these names becomes nonsense. We also don't treat them as times and counters anywhere in this file, so perhaps it would be better to name time row_id_hi and row_id_lo or something? 🤷

Also, why are we storing RowId as two arrays of u64 instead of a single array of [u8; 16] ? 🤔

I'm basically following what is being done in re_datastore. I expect this to be carved in stone with the sorbet spec, so let's adjust the implementation when we have that (fundamentally, the purpose of DisplayRecordBatch and friends is to decode a sorbet-compliant RecordBatch for the purpose of ui).

Related:

Document/publish our arrow schemas #6818

[skip_ci]

[skip ci]

### What - Closes #7279 Major update to the dataframe view - display the data return by the new `re_dataframe` crate - the PoV entity/component is now actually used - entities are now always columns - see #7379 - use [`egui_table`](https://github.com/rerun-io/egui_table) for the table - hierarchical header - sticky columns - and much more... TODO: - [x] fix after merging #7383 <img width="2670" alt="image" src="https://github.com/user-attachments/assets/cf09b69b-3c82-4ba9-9425-bf60622efae4"> ### Checklist * [x] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) * [x] I've included a screenshot or gif (if applicable) * [x] I have tested the web demo (if applicable): * Using examples from latest `main` build: [rerun.io/viewer](https://rerun.io/viewer/pr/7380?manifest_url=https://app.rerun.io/version/main/examples_manifest.json) * Using full set of examples from `nightly` build: [rerun.io/viewer](https://rerun.io/viewer/pr/7380?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json) * [x] The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG * [x] If applicable, add a new check to the [release checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)! * [x] If have noted any breaking changes to the log API in `CHANGELOG.md` and the migration guide - [PR Build Summary](https://build.rerun.io/pr/7380) - [Recent benchmark results](https://build.rerun.io/graphs/crates.html) - [Wasm size tracking](https://build.rerun.io/graphs/sizes.html) To run all checks from `main`, comment on the PR with `@rerun-bot full-check`. --------- Co-authored-by: Emil Ernerfeldt <emil.ernerfeldt@gmail.com> Co-authored-by: Clement Rey <cr.rey.clement@gmail.com>

abey79 and others added 30 commits September 5, 2024 09:22

WIP

d2c51ab

Now latest at is also using dataframe (but is broken)

78d06c9

Remove dead code

1a53b69

Remove max_scroll_height(f32::INFINITY)

eaf114b

Fix typos

d8a1b60

Start using egui_table

7372f9d

Handle multiple arrow types for component columns + some more clean-up

a8940c8

Merge branch 'refs/heads/main' into antoine/use-re-dataframe

5fdf38a

Make DisplayRecordBatch owned and add pagination

7ff2ed1

Merge branch 'refs/heads/main' into antoine/use-re-dataframe

531a963

remove dbg!

67b9f43

remove moar dbg!

517ee61

fix some warnings

5d0977f

Update egui_tables

edf6284

minor clean up

49c3eea

Restore egui style when resetting viewer

3998b6e

Better auto-sizing of table contents and other data uis

f39c74a

Update egui_table

08ad900

wip hierarchical header

4399953

Update egui_table with fix for row prefetching

dda52ed

Update egui_table with another off-by-one fix

796c2f8

simplified schema wrapping management

2b84d1b

Update egui with line-break fixes

a5c3b0d

Defensive programming against prefetch bug in egui_table

b8623ed

lints

2b7b023

Merge branch 'main' into antoine/use-re-dataframe

cdb6ee3

Merge branch 'main' into antoine/use-re-dataframe

293935e

# Conflicts: # crates/store/re_chunk_store/src/dataframe.rs

code cleanup

5807b4f

Update egui_table

44eb18b

future truth: adapt for re_dataframe using EntityPathFilter

8bfdb25

abey79 added 4 commits September 9, 2024 17:49

add docstring

e0be7c5

toml fmt

5c468a9

fix web build

183915a

change should propagate event to false (seems to be the default)

0a8630d

abey79 commented Sep 10, 2024

View reviewed changes

crates/viewer/re_viewer/src/web.rs Outdated Show resolved Hide resolved

abey79 mentioned this pull request Sep 10, 2024

Switch ArrowDict to signed and fix static datatype #7383

Merged

6 tasks

abey79 added 2 commits September 10, 2024 09:38

Merge branch 'main' into antoine/use-re-dataframe

bbff7dc

switch dictionary array to i32, as per #7383

9bffb2c

emilk approved these changes Sep 10, 2024

View reviewed changes

emilk and others added 14 commits September 10, 2024 10:36

Use ..Default::default in WebOptions

dd4f33e

struct BatchRef for better self-documentation

91794eb

Call it query_time_column_index for clarity

f09736f

"Latest at" -> "Latest-at"

e95bc6b

Use DesignTokens::table_header_height

cb5fb32

export arrow2 from re_chunk_store

2af5395

[skip_ci]

show timestamp and increment on row id's tooltip

fd10862

[skip ci]

add debug assert on get with latest-at

b627ef4

[skip ci]

dont hard-code rowid component name

dcb92a8

[skip ci]

fix anyhow context use

50f27cd

[skip ci]

more debug assert against egui_table

f532948

[skip ci]

fight some right drift

b325e7a

[skip ci]

clarify index argument

a32cd87

improve expect reason strings

190f43b

abey79 merged commit e9b02e3 into main Sep 10, 2024
35 checks passed

abey79 deleted the antoine/use-re-dataframe branch September 10, 2024 12:29

abey79 mentioned this pull request Sep 10, 2024

Use Range<u64> as argument for the RangeQueryHandle::get() #7395

Closed

6 tasks

abey79 mentioned this pull request Sep 19, 2024

Dataframe view: add support for PoV component for range queries #7072

Closed

emilk added exclude from changelog PRs with this won't show up in CHANGELOG.md and removed include in changelog labels Oct 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update the dataframe view to use `re_dataframe` and `egui_table` #7380

Update the dataframe view to use `re_dataframe` and `egui_table` #7380

abey79 commented Sep 9, 2024 •

edited

Loading

emilk Sep 10, 2024

abey79 Sep 10, 2024

emilk Sep 10, 2024

emilk Sep 10, 2024

abey79 Sep 10, 2024

abey79 Sep 10, 2024

Update the dataframe view to use re_dataframe and egui_table #7380

Update the dataframe view to use re_dataframe and egui_table #7380

Conversation

abey79 commented Sep 9, 2024 • edited Loading

What

Checklist

emilk Sep 10, 2024

Choose a reason for hiding this comment

abey79 Sep 10, 2024

Choose a reason for hiding this comment

emilk Sep 10, 2024

Choose a reason for hiding this comment

emilk Sep 10, 2024

Choose a reason for hiding this comment

abey79 Sep 10, 2024

Choose a reason for hiding this comment

abey79 Sep 10, 2024

Choose a reason for hiding this comment

Update the dataframe view to use `re_dataframe` and `egui_table` #7380

Update the dataframe view to use `re_dataframe` and `egui_table` #7380

abey79 commented Sep 9, 2024 •

edited

Loading