Recommended / best practice for virtual list of unlimited size

[bsenftner]

Version/Branch of Dear ImGui:

Version: 1.67
Branch: master

Visual Studio 2015, C++, OpenGL 2.0, Developing in Win10, targeting flavors of Windows

I have a video/security app that generates a “visual log” - timestamped lines where a “line” is either a line of text, or an image followed by text. Each “log line” is one application event, and over time there could be tens of thousands of log lines.

The goal is to have a vertically scrolling window providing users with a view of these “visual log” lines. The text is always the same height, but each image could be a different height, and there is not any order to when a “log line” is an image with text or just text.

It looks like there is a ListBox() signature that accepts an “items_getter” callback. It also has a comment in the code suggesting one implement their own ListBox() for cases where the height of each line may vary. I have not found any examples of using this ListBox() signature that accepts a callback; is there an example? In that ListBox() signature:

bool ImGui::ListBox(const char* label, int* current_item, bool (items_getter)(void, int, const char**), void* data, int items_count, int height_in_items)
It looks like the “void * data” parameter is my own data pointer that is ultimately passed to the callback. It also looks like the total number of items as well as the height of all those items is used by the implementation of this ListBox() signature. How would one go about implementing a similar ListBox() with items_getter callback, but supports variable height lines?

Reading the comments around ImGuiListClipper, that starts out saying the list should be evenly spaced items (meaning vertical spacing, right?) and random access to the list items. In my case, the list items are in a database, and only after retrieval from the database is any individual timestamp’s content known.

Sorry for the range of questions. They all surround this basic idea of how to implement a scrolling control for viewing a potentially very large number of items of unknown height. Any guide or suggestions are appreciated.

[ocornut]

Hello,

Sorrry I don't have an answer at the moment.

One aspect of the issue is handling virtual scrolling (where contents can grow on either side). I reckon it should be possible but I haven't tried making a demo for it and we'll probably run on subtle issue to get it right (e.g. keep the same item in view while everything is being offset, what happens when using on the scrollbar, etc.). As your items are streamed from the database your positions will change and you will want to keep the visible portion unchanged. This is an interesting problem and I'd like to try making a simple demo for it sometimes, but I can't promise anything now.

The second aspect is clipping large list of unevenly sized item. No magic here, for clipping you want to want to be able to "seek" at a given Y position so your data structure may be designed in a way that makes this possible with some simple precomputation. Even the simplest dumb approach would be to store the height in each item (or preferably in a dense array to be more cache-friendly) and quickly iterate through them to get to the visible portion of the screen. That's not optimal at all but it may be good enough if you only have 10k items. From there you can improve your data structure in many ways to optimize this Y->item lookup. ImGuiListClipper is not suited for that as is so you'd need your own code.

Note that your question doesn't have much to do with ListBox. You probably don't even want to use the ListBox() signature which is restrictive more than anything. Check the ListBoxHeader/Footer function or just use BeginChildFrame and output your contents there.

[bsenftner]

Thanks. I am glad you mentioned "or just use BeginChildFrame and output your contents there" because I was a bit confused trying to understand why I'd need more than them, as it looks like ListBox() is not really needed for this because the "list" is virtual.

[grandemk]

Hi, did you make a demo for this in the end ?

One aspect of the issue is handling virtual scrolling (where contents can grow on either side)

I've been implementing a logger which encounters the same kind of problems you talk about.
I still have one last problem with the scrolling:

I use an ImGui::BeginTable / ImGui::EndTable where each row is a log.

Each row is the same height, so I use ImGuiListClipper to only consider the visible area.

The latest logs are at the top of the table, so each frame, when new logs arrive, everything is offset by nb_new_logs * log_height

As new logs arrive all the time, I can't have a lag of one frame while dealing with the scrolling, because else I can't keep the same logs as the previous frame visible without jittering.

To handle the scrolling without a frame of delay, I use ImGui::SetNextWindowScroll before ImGui::BeginTable (which creates a Window internally when ScrollY is enabled) to offset the scroll value to keep the visible area on the same logs than the previous frame if I'm not looking at the top of the table.

The only problem I have remaining is if I want to see the first logs, because the value I set using SetNextWindowScroll is clamped by MaxScrollY, which is the value from the previous frame.

So when new logs are arriving, I can't see nb_new_logs of the oldest logs, because I can't set the next window scroll to a value that would make them visible.

I guess I might need a SetNextWindowMaxScroll, I'm not sure what the implication would be. (first time using this library for something less trivial than a debug window). Don't know how to handle the scrollbar though.

Here is what it looks like, I could make the imgui code available if it is interesting to someone, thank you for this great library ;)