[data grid] Lazy load data with virtualization

[DanailH]

Make lazy loader work together with the rowCount prop.

• I have searched the issues of this repository and believe that this is not a duplicate.

Summary 💡

In order for the lazy loader feature to be complete, we need to make it work in combination with the rowCount prop. This means that a developer can set the total number of rows beforehand. The grid will then allocate space for all the rows and size this scrollbar correctly.

The loadRows callback will be called requestion the rows within an area of [a, y] with the right sorting, filtering, etc. parameters and load them in their appropriate place.

Examples 🌈

<DataGridPro
  rows={[]}
  rowCount={500}
  loadRows={handleLoadRows}
/>

Motivation 🔦

Allow products that would benefit from fewer pages for UX reasons to have pages of up to 100,000 rows.

#2038 is a step in this direction. It allows developers knowing what are the currently visible rows in the viewport.

Benchmark

• https://bvaughn.github.io/react-virtualized/#/components/InfiniteLoader
• https://ag-grid.com/react-grid/infinite-scrolling/
• https://ag-grid.com/react-data-grid/server-side-model/

[sebastianfrey]

Any plans when this feature will be released?

[oliviertassinari]

Any plans when this feature will be released?

@sebastianfrey Unclear, but we would love to hear more about the problem you wish to solve with this feature. Could you expand? :)

[sebastianfrey]

@oliviertassinari Sure :).

I use the grid in a currently new developed GIS application to visualize the attributes of various heterogeneous spatial data sources with support for paginated queries. The amount of data this sources have stored goes from a few hundred records to a few hundred thousand and somtimes up to a million or more.

So while having a few hundered to thousand records the current approach by just loading the next batch of rows when the user reaches the bottom end of the scroll area is acceptable. With an increasing amount of records this becomes unwieldy.

Sure users can always filter the records by attributes or spatial conditions, but since in the current version of our application, the here proposed feature is already supported and there are likely some users, which will miss that feature, it would be nice, to have this feature supported by XGrid as well.

FYI:

In the current version of our GIS application we use the Dojo Toolkit library dgrid in conjunction with a custom dstore implementation for data fetching.

[erwelch]

I'd like to see this land as well. We have a lot of tables in our application and are using graphql w/relay as our data client. Relay's pagination appends the loaded data, so infinite scroll tends to be more favorable with our queries over traditional pagination. When the grid displays the total # of rows as a different number than the true # of records it is a confusing user experience. Would love to display the total count so users aren't guessing how far they can scroll until they reach the end of the list.

[oliviertassinari]

Interesting feedback in https://mui.zendesk.com/agent/tickets/2164. AWS Lambda limits the payload of responses to 6 MB, so typically, developers can't fetch more than ~1,000 rows per request. I would argue that they don't want to fetch more than 100 rows in practice, that they should lazy load the rest as the end-user scrolls the page.

When using the data grid for internal apps, I would expect it to be frequent to have millions of records. So I assume that these people have to 1. enable pages of about 1,000 rows or 2. use the infinite loading feature: https://mui.com/components/data-grid/rows/#infinite-loading but when you know the exact number of records and you can fetch the rows on-demand, it's not great (crippled scroll, no pagination)

[oliviertassinari]

I have found https://glideapps.github.io/glide-data-grid/?path=/story/glide-data-grid-dataeditor-demos--server-side-data that could be a great source of inspiration for the solution. The source of the demo. For example, it relies on the onVisibleRegionChanged prop to implement the lazy data fetching. I assume it the same concept that AG Grid uses internally for https://ag-grid.com/javascript-data-grid/viewport/.

What's the plan for this issue? I assume that we need to move carefully, the problem is complex, and has a lot of interdependencies. I assume that this issue is only the tip of the iceberg. Should we start with the high-level RFC on the whole picture, with each step along the way?

[DanailH]

Should we close this one? The PR linked adds that functionality.

[oliviertassinari]

@DanailH do we have an issue about the next steps for improving the server-side integration? cc @joserodolfofreitas

I'm asking because I was answering this proposal https://www.notion.so/mui-org/Low-code-directions-available-10aeeb10596d41ac97593c6928c361aa#8e34e9925de9442195c9251915e52bd5 which also resonates with https://www.notion.so/mui-org/Sparklabs-Mike-Willbanks-432e780944c3429e9f2b6eea3a5ac6bb#8a82d9ec0ce641208e2e8ff6b237eb2e

[DanailH]

@oliviertassinari this is the one that makes the most sense to be picked up #3377. @MBilalShafi has expressed some interest to check it but we can start thinking about this once we have the beta for v6 out.