[data grid] Handle 1m records

[oliviertassinari]

I have been wondering about this aspect. I believe the limitation for displaying, say 1m records is the number of pixels the browser accepts. This is discussed in bvaughn/react-virtualized#396 and WICG/display-locking#49.

How about we automatically trigger the pagination if a user exceeds this limit? It would also be great to document it.

Another thought, in such a case, the server might not even know how many rows they are. With PostgreSQL a simple count(*) on a table with many rows can be too time-consuming to be used in production. We would need to handle this case, the case where the total number of rows is unknown.

Proof: https://codesandbox.io/s/beautiful-silence-1yifo?file=/src/App.js

We document the limitation at: https://material-ui.com/components/data-grid/virtualization/.

Benchmark

• https://glideapps.github.io/glide-data-grid/?path=/story/glide-data-grid-dataeditor-demos--one-million-rows
• https://glideapps.github.io/glide-data-grid/?path=/story/glide-data-grid-dataeditor-demos--silly-numbers
• https://ag-grid.com/javascript-data-grid/massive-row-count/

[mak-elena]

I'm currently testing DataGrid component and huge data sets (without pagination) are exacly the case I'm interested in.
In this regard: Do you consider to implement a behavior similar to react-virtualized/InfiniteLoader?

Or alternatively:
The ColumnApi has the method getVisibleColumns (BTW, it returns always all the columns. Is it still under implementation? )
Is it possible to have a similar method for rows? So that prefetching and caching of data requred for rendering can be implemented on top of it.

Thanks.

[oliviertassinari]

@mak-elena We definitely plan to introduce lazy loading for the rows. Right now the only alternative I'm aware of is to set pagination={true}. This new data fetch API will support sorting, filtering, search, grouping, etc.

[oliviertassinari]

It would also be great to document it.

Done in https://material-ui.com/components/data-grid/rendering/#virtualization

Browsers set a limit on the number of pixels a scroll container can host: 17.5 million pixels on Firefox, 33.5 million pixels on Chrome, Edge, and Safari. A reproduction.

[oliviertassinari]

@mak-elena You can track and upvote #404.