Table with lazy-loaded pagination

[AlyShmahell]

Question

while trying to implement a custom list to pass to the ui.table element, the list had to be explicitly iterable (maybe I'm wrong here, but going through the code it seemed the only way), and each element needed to be json serializable.
Going off of the above assumptions, nicegui does not support lazyloaded pagination.
further tests of large-ish numbers of simple records (10million+ rows) supports this idea, as it takes a hot second or two for first page rendering to start.
So my question is, does it support lazy loaded pagination by some mean? or am I right in my assumptions that it doesn't?

Thank you

[falkoschindler]

Hi @AlyShmahell,

I think this is currently not possible. We might need to use server-side pagination to request the rows for the current page from the server instead of sending them all at once. I'll convert this question into a feature request.

[Kolterdyx]

For the meantime, the workaround is to generate a list with the size as big as the whole dataset, but only fill in the data in the correct place. Here's how I do this using Peewee ORM to load only the page I'm at:

import os.path
from typing import List

from nicegui import ui

from app.backend.db import Object
from app.ui.tabs.base_tab import BaseTab


"""
Object is a DB model that for the purposes of this example just has a "path" attribute which is a simple utf-8 string
"""


class Home(BaseTab):
    def __init__(self):
        super().__init__()
        self.table = None
        self.page_size = 10
        self.rows: List = [None] * Object.select().count()

    def run(self):
        super().run()

        columns = [
            {
                'name': 'path',
                'label': 'Name',
                'field': 'name',
                'align': 'left',
                'sortable': True
            },
            {
                'name': 'checksum',
                'label': 'Checksum',
                'field': 'checksum',
                'align': 'left'
            },
        ]

        with ui.card():
            ui.label('Objects')
            self.table = ui.table(columns, self.rows, on_pagination_change=lambda x: self.get_objects(x.value), pagination=self.page_size)

        self.get_objects({
            'page': 1,
            'rowsPerPage': self.page_size,
            'sortBy': None,
            'descending': False
        })

    def get_objects(self, pagination_event: dict):
        page: int = pagination_event['page']
        rowsPerPage: int = pagination_event['rowsPerPage']
        sortBy = pagination_event['sortBy']
        descending: bool = pagination_event['descending']

        if not rowsPerPage:
            rowsPerPage = len(self.rows)

        objects = Object.select()
        if sortBy:
            objects = objects.order_by(getattr(Object, sortBy).desc() if descending else getattr(Object, sortBy))

        # To preserve the sorting functionality, we need to arrange the dummy data so that when nicegui sorts it, it displays our loaded data in the current page.
        for i in range((page - 1) * rowsPerPage):
            self.rows[i] = {
                'name': 'a' if not descending else 'z' * 100
            }
        for i in range(page * rowsPerPage, len(self.rows)):
            self.rows[i] = {
                'name': ('z' * 100) if not descending else 'a'
            }
        for i, obj in enumerate(objects.paginate(page, rowsPerPage)):
            index = ((page - 1) * rowsPerPage + i)
            self.rows[index] = {
                'name': os.path.basename(obj.path),
                'checksum': obj.checksum
            }
        if self.table:
            self.table.update()

This properly preserves the sorting and pagination functionality while only loading the necessary data from the DB. I don't like my solution for preserving the proper sorting behavior, but that's all I could figure out.

This approach will create a massive array with a lot of dummy data, and this will cause trouble if the record count of the table (or dataset) gets too big, since it wastes a lot of memory. This also will only work properly if you know exactly how much data you have in total, even if you don't retrieve it all at once

[Kolterdyx]

There is the rowsNumber attribute for the pagination object, but setting that to any value completely breaks the table. When that attribute is defined, Quasar expects you to handle all changes to the pagination object. This means that the client stops sending signals to the backend, so the pagination callback in Python won't be executed. I tried to follow the Quasar docs and listen to the @request event, but I couldn't figure out how the vue components communicate to their server counterparts, so I'm a bit lost.

This is probably as simple as just adding the event handler on the client to tell the server to execute the pagination handler and setting the pagination attribute, but I don't have too much experience with Vue and this repo is a bit too complex for me to figure out on my own, I hope this is useful to someone more knowledgeable

[AlyShmahell]

Hello @Kolterdyx

thank you for the feedback, I initially tossed and turned as well trying to figure it out but eventually I settled on something like this:

import random
from nicegui import ui

class LazyLoaded(list):
    def __init__(self, *args, **kwargs):
        self.pagination = kwargs.pop('pagination', 1)
        super().__init__(*args, **kwargs)
        self()
    def __call__(self):
        for _ in range(self.pagination):
            self.append({"number": random.random()})
    @property
    def len(self):
        return len(self)

class PaginationChange:
    def __init__(self):
        self.maxpage = {}
    def __call__(self, e, data):
        prev = self.maxpage.get(str(e.sender), 0)
        curr = e.value['page']
        if curr > prev:
            data()
        e.sender.update()
        self.maxpage[str(e.sender)] = max(curr, self.maxpage.get(str(e.sender), 0))

@ui.page("/")
def index():
    pagination_change = PaginationChange()
    data = LazyLoaded(pagination=11)
    ui.label().bind_text_from(data, 'len', lambda x: f"number of rows stored on the backend: {x}")
    table = ui.table(
        columns=[{'name': 'number', 'label': 'number', 'field': 'number'}],
        rows=data,
        pagination=10
    )
    table.on_pagination_change(lambda e, data=data: pagination_change(e, data))

ui.run()

This LazyLoaded function simulates getting infinite data while still being incrementally iterable and some variation of PaginationChange will handle the hot loading part (the next if you will).
As long as you can guarantee that the rows object itself is iterable (in my example I inherited from list) then you're golden.

[zceemja]

Found more elegant solution

import asyncio

from nicegui.events import GenericEventArguments
from nicegui import ui



@ui.page("/")
async def index():
    columns = [{'name': 'path', 'label': 'Name', 'field': 'name'}]
    pagination = {'rowsPerPage': 10, 'rowsNumber': 100, 'page': 1}

    async def get_rows():
        page = pagination['page']
        rpp = pagination['rowsPerPage']
        print(f'Fetching {rpp} rows')
        await asyncio.sleep(2)
        return [{'name': f'Row-{i:02d}'} for i in range((page - 1) * rpp, page * rpp)]

    async def on_request(e: GenericEventArguments) -> None:
        nonlocal pagination
        pagination = e.args['pagination']
        table.props('loading')
        rows = await get_rows()
        table.pagination.update(pagination)
        table.props(remove='loading')
        table.update_rows(rows)

    table = ui.table(columns=columns, rows=await get_rows(), row_key='name', pagination=pagination)
    table.on('request', on_request)

ui.run()