Allow pandas DataFrame df and rows as two sources of rows, make columns optional, and introduce default column config

[tmlmt]

Motivation

While it's relatively easy to edit an ui.table's column options after creating it from a pandas DataFrame, similar to how it can be done for ui.aggrid options (#947), it is a bit verbose and requires an update of the element.

table = ui.table.from_pandas(df)
# Making all columns sortable
for col in table.columns:
    col["sortable"] = True
    if col["name"] == "age":
        col[":format"] = "(val, _row) => 2*val"
table.update()

More over, that method returns a table that cannot be easily updated as the dataframe is changed, unless something like PR is implemented.

Besides, the logic would have to be duplicated to support polars dataframes.

Finally, there is an opportunity to simplify the call for creating a new table from any type of source, while leveraging the embedded feature in Quasar's table element to infer columns from the first row of data.

Proposal

This PR proposes to partially rewrite the table element to:

• make columns optional and infer columns names with the first row of data
• simplify the creation of tables from Pandas dataframes
• enable providing a default configuration for columns (default: {'sortable': True})

rows = [{'id': 1, 'name': 'Alice', 'age': 20}, {'id': 2, 'name': 'Bob', 'age': 19}]

columns = [{'name': 'name', 'field': 'name', 'label': 'Name'}, {'name': 'age', 'field': 'age', 'label': 'Age'}]

df = pd.DataFrame(data={'id': [3, 4], 'name': ['Patrick', 'Pamella'], 'age': [25, 26]})

# Quick and dirty from rows:
ui.table(rows=rows)

# Rows and columns to customize labels and columns to show for example
ui.table(rows=rows, columns=columns)

# Quick and dirty from Pandas
ui.table(df=df)

# From Pandas with columns
ui.table(df=df, columns=columns)

# When columns are provided, applying defaults 
ui.table(rows=rows, columns=columns, default_column={'headerClasses': 'text-bold', 'sortable': True})
ui.table(df=df, columns=columns, default_column={'headerClasses': 'text-bold', 'sortable': True})

[falkoschindler]

Thanks for this pull request, @tmlmt!

I've been thinking about it again and again this week. Somehow I'm not a fan of adding even more functionality to the from_pandas method. Manipulating columns seems a bit unrelated, and could even be useful when creating tables without pandas.

What do you think about extracting a dedicated update_columns method? It could be called like this:

ui.table.from_pandas(df).update_columns(
    {'sortable': True},
    age={':format': '(val, _row) => `${val} years`'},
)

A single positional argument controls "default" settings, and keyword arguments control settings per column.

[tmlmt]

Thanks for the comment @falkoschindler and sorry to have caused you trouble the entire week 😅

update_columns suggests a transformation, while I think what I'm trying to suggest is that there should be full control of at time of initialization of the table, for dataframes:

• from_pandas arbitrarily decides not to apply any of the cool features you may want if you just want to quickly display a dataframe (like sortable columns)
• on the other hand, when calling ui.table, you pass on a columns parameter that you can fully customize.

I think it would actually be even greater to be able to either pass columns and rows to ui.table, or a pandas dataframe (with customizable column configs), or a polars dataframe (with customizable column configs). Instead of having to call from_pandas or a potential from_polars. The constructor would then figure it out.

As you suggest though, we could also introduce a update_columns or simply columns, that can be called on any ui.table instance. And it could also be used after from_pandas if you want to maintain this logic. After all, that's what we do on elements, e.g ui.element().props("...").classes("...")

You're the maintainer, so I'll let you decide and I can update this PR accordingly 😊 Let me know what you think.

[falkoschindler]

Ok, you're starting to convince me that support for pandas (or polars) dataframes should be improved. Calling some column transformation every time the data is updated from a dataframe is tedious.

I think it would actually be even greater to be able to either pass columns and rows to ui.table, or a pandas dataframe (with customizable column configs), or a polars dataframe (with customizable column configs).

Let's draft an API for this idea:

class Table(FilterElement, component='table.js'):
    def __init__(self,
    	         *,
                 columns: Optional[List[Dict]] = None,
                 rows: Optional[List[Dict]] = None,
                 df: Union['pd.DataFrame', 'pl.DataFrame'],
                 ...)

# 1. "normal" table from row and column dictionaries
ui.table(rows=[...], columns=[...], ...)

# 2. table from pandas dataframe
ui.table(df=df)

# 3. table from pandas dataframe, extending given colums dictionaries
ui.table(columns=[...], df=df)

# 4. table from pandas dataframe _and_ row dictionaries raises an exception
ui.table(rows=[...], df=df)

This would remove some code (and documentation) duplication by merging different ways to create tables into one. Same holds for ui.aggrid, by the way.

With this API you can define some columns using dictionaries and still load data with additional columns from a dataframe. But how to define default attributes that apply to all columns like "sortable"? Maybe we just need an additional default_column:

# 5. table from pandas dataframe, extending given columns and default column definition
ui.table(default_column={'sortable': True}, columns=[...], df=df)

And like table.rows and table.columns we should add a table.df property that can be read and written. Then we can update table data with a simple assignment and all column definitions are preserved automatically:

table.df = my_updated_df

[tmlmt]

I like the API, more or less what I had in mind. A few things:

# 3. table from pandas dataframe, extending given colums dictionaries

ui.table(columns=[...], df=df)

What's the behavior in this case? Do you merge columns from columns and df? Meaning the table will have columns in common and additional columns? Cells would be empty if df doesn't contain values for columns that are in columns but not in df. I would either throw an exception is not all field values in columns are not df columns, or silently not handle those are that not.

Maybe we just need an additional default_column:

I agree.

What should be the behavior when:

• columns is later updated manually (table.columns = new_columns)? Should default_column be reset or preserved and applies to the new columns (my preference)?
• df is later updated manually (table.df = updated_df)? Should columns be set to those of updated_df (my preference)? Or do the new columns become those of table.columns merged with those of updated_df?

[falkoschindler]

Very good questions...

What's the behavior in this case? Do you merge columns from columns and df?

My intuition was that we copy columns first, and then iterate over columns in df and merge each one with its possibly existing counterpart in columns.

What should be the behavior when: columns is later updated manually [...] df is later updated manually [...]

Actually, I don't know. And if there isn't an obvious, intuitive behavior and the user can't be sure what will happen when updating this or that, we might be on the wrong track. Maybe we need to treat columns and rows/df separately. So we would only use df to fill our rows, but not for manipulating columns. And I just noticed that Quasar doesn't require you to provide columns at all, as it can infer the header from the first row. Therefore I'd propose the following:

• Introduce df as an alternative parameter for filling rows.
• Make columns optional, since they can be inferred from rows or df.
• Introduce a new optional default_column parameter.
• Keep default_column and columns as separate properties. Whenever one of them changes, merge them before writing to _props['columns'].
• Same for rows and df: Whenever one of them changes, merge them before writing to _props['rows'].
• We should also add df to the initializer of ui.aggrid. But because AG Grid natively supports the "defaultColDef" option, we won't need a default_column parameter.

Note that these are breaking changes, thus need to wait for the upcoming 2.0.0 release.

[tmlmt]

Sounds really good that Quasar can infer columns. I can read here:

You can omit specifying the columns. QTable will infer the columns from the properties of the first row of the data. Note that labels are uppercased and sorting is enabled.

I therefore also propose that default_column defaults to {sortable: True} (while we're introducing breaking changes), and that it only takes effect if columns is provided. To preserve the inference behavior.

All in all that would really fit well: calling ui.table(df) would do straight what we'd expect it to do i.e. quickly display a dataframe as a table and be able to sort it on any column straight away.

I'll start working on it 🙂. I suggest that I introduce all this logic first and then I can work on extending it to polars dataframe in a separate PR.

[tmlmt]

Hi @falkoschindler, I've now implemented things on ui.table. See expanded set of tests.

Due to the merging step, we now lose the ability for the table to update automatically when the initially provided lists are changed, i.e:

rows = <some list>
columns = <some list>
t = ui.table(columns=columns, rows=rows)

columns = [{'name': 'name', 'field': 'name', 'label': 'name'}]
t.update()
# would add the 'name' column before as a sortable column, but it does not add the sortable property now.

Do we need to keep this ability to update a table using update() after changing the initially provided input variables? Or is it fine to have to reassign the columns property (t.columns += [{'name': 'name', 'field': 'name', 'label': 'name'}]) instead?

If we need to:

• would it work to use ObservableList (dunno if it would work in this case)? (edit: not on nicegui's side)
• alternatively, as for rows, there already exists the functions add_rows, remove_rows and update_rows; we could create add_columns, remove_columns and update_columns and update_df.

Let me know what you think about what the user experience should be for updating rows, columns and df

[tmlmt]

New implementation of ui.table ready for review, including tests and docs.

• Updating rows can be done either via setting table.rows, or by using one of the helper methods (add_rows, remove_rows, update_rows)
• Updating columns can be done via setting table.columns (or setting table.default_column to change the defaults applied to all)

I've also updated the first post of this PR with a more detailed motivation and description of the proposal

[falkoschindler]

Thanks @tmlmt! I finally found the time for a thorough code review:

• Even though I like type annotations and see the benefits of TypedDict, I removed ColumnsConfigOptions for now:

  • It seemed to be using an "old" syntax (https://peps.python.org/pep-0589/#alternative-syntax) and could be rewritten for Python 3.8.
  • There was an unused ColumnsConfig.
  • When using ui.table, the IDE's type hint popup will only show "ColumnsConfigOptions", which I find less useful than "Dict". So even a static type checker will be more helpful with typed dicts, I don't like type information for the user being "hidden" behind another definition.
  • Most importantly: This PR is already quite complex. And because we don't really need the typed dict, I chose to remove it in favor for a smaller, more manageable diff.

• I found the "none" string passed to the frontend rather confusing. Instead I'm inferring columns from the first row in Python, which is just one line and allows to keep the frontend unchanged. Sure, we miss the chance to use Quasar's default behavior. But it's easily reproduced and gives full power to the backend.

There are a few open questions / issue we'll need to address:

• Do we really need to wait for 2.0 and simply break the API? Or can we provide backward compatibility?

  • For now I only made df and default_column keyword arguments. This way the old API with positional arguments still works.
  • The from_pandas method could be added back in and marked as deprecated. It should print a deprecation warning like we do, e.g., in get_slot_stack().
  • The new behavior of all columns being sortable by default might be the only remaining breaking change. But we could initialize default_columns with an empty dictionary for now and change it to {sortable: True} in 2.0.

• For consistency we should update ui.aggrid accordingly. We might want to do it in a separate PR, but if we break anything, we should break both APIs at once in 2.0.

• What bothers me the most is that updating columns (and rows!) doesn't work like before. I think your example from Allow pandas DataFrame df and rows as two sources of rows, make columns optional, and introduce default column config #3263 (comment) didn't work before, but table.column['...'] = ...; table.update() is broken now. I find it rather critical, because it is hard to spot. So even if our users are aware of all breaking changes, they might miss this one in their project. Using observable collections for _rows, _columns (and default_column?) might be a great solution though. We just need to make sure to instantiate it once and update it in the property setters so that we don't loose their reference.

What do you think? There are still some things to do, but it feels like we are on the home stretch. 🙂

[tmlmt]

Hi @falkoschindler, sorry for my late reply. Thanks for the thorough review, I had in mind to also take care of aggrid and polars, but didn't want to go too far before you had a look and make sure that coding style and so forth was OK. Looks like that was a good idea!

I agree with your comments, or at least I understand the motivation of not wanting to introduce any breaking change for the moment. It seems you are already addressing the points you mentioned. Let me know if you'd like me to take care of any piece.

[falkoschindler]

I just experimented with observable collections, but noticed a fundamental problem: If you pass a list rows to ui.table where it is wrapped in an ObservableList, any changes to table.rows will be noticed and the table can update automatically. But if you modify the original list, the table has no chance to recognize the change.

Worked before this PR:

table = ui.table(columns=columns, rows=rows)
columns[:] = [{'name': 'name', 'field': 'name', 'label': 'name'}]
table.update()

Works after this PR:

table = ui.table(columns=columns, rows=rows)
table.columns[:] = [{'name': 'name', 'field': 'name', 'label': 'name'}]
# no update() call

This is a subtle but ugly breaking change. A possible solution might be to call both merge functions with every update(). But I'm not sure about the efficiency of this approach.

I'll drop the non-working implementation here for reference.

```py from typing import Any, Callable, Dict, List, Literal, Optional, Union

from typing_extensions import Self

from .. import helpers, observables, optional_features
from ..element import Element
from ..events import GenericEventArguments, TableSelectionEventArguments, ValueChangeEventArguments, handle_event
from .mixins.filter_element import FilterElement

try:
import pandas as pd
optional_features.register('pandas')
except ImportError:
pass

class Table(FilterElement, component='table.js'):

def __init__(self,
             columns: Optional[List[Dict]] = None,
             rows: Optional[List[Dict]] = None,
             row_key: str = 'id',
             title: Optional[str] = None,
             selection: Optional[Literal['single', 'multiple']] = None,
             pagination: Optional[Union[int, dict]] = None,
             on_select: Optional[Callable[..., Any]] = None,
             on_pagination_change: Optional[Callable[..., Any]] = None,
             *,  # DEPRECATED: all arguments will be keyword-only in the future
             df: Optional['pd.DataFrame'] = None,
             default_column: Optional[Dict[str, Any]] = None,
             ) -> None:
    """Table

    A table based on Quasar's `QTable <https://quasar.dev/vue-components/table>`_ component.

    :param columns: optional list of column dictionaries (inferred from the first row if not provided)
    :param rows: optional list of row dictionaries (will be added to those from ``df``, default: ``[]``)
    :param df: optional Pandas DataFrame (rows will be added to those from ``rows``, default: ``None``)
    :param default_column: default options which apply to all columns (default: ``{}``)
    :param row_key: name of the column containing unique data identifying the row (default: "id")
    :param title: title of the table
    :param selection: selection type ("single" or "multiple"; default: ``None``)
    :param pagination: a dictionary correlating to a pagination object or number of rows per page (``None`` hides the pagination, 0 means "infinite"; default: ``None``).
    :param on_select: callback which is invoked when the selection changes
    :param on_pagination_change: callback which is invoked when the pagination changes

    If selection is "single" or "multiple", then a ``selected`` property is accessible containing the selected rows.
    """
    super().__init__()

    self._df = self._clean_pandas_dataframe(df)
    self._rows = observables.ObservableList(rows or [], on_change=self._handle_rows_change)
    self._props['rows'] = self._merge_rows_with_dataframe()

    self._default_column = observables.ObservableDict(default_column or {},
                                                      on_change=self._handle_default_column_change)

    first_row = self._props['rows'][0] if self._props['rows'] else {}
    columns = columns or [{'name': key, 'label': str(key), 'field': key} for key in first_row]
    self._columns = observables.ObservableList(columns, on_change=self._handle_columns_change)
    self._props['columns'] = self._merge_columns_with_default()

    self._props['row-key'] = row_key
    self._props['title'] = title
    self._props['hide-pagination'] = pagination is None
    self._props['pagination'] = pagination if isinstance(pagination, dict) else {'rowsPerPage': pagination or 0}
    self._props['selection'] = selection or 'none'
    self._props['selected'] = []
    self._props['fullscreen'] = False
    self._selection_handlers = [on_select] if on_select else []
    self._pagination_change_handlers = [on_pagination_change] if on_pagination_change else []

    def handle_selection(e: GenericEventArguments) -> None:
        if e.args['added']:
            if selection == 'single':
                self.selected.clear()
            self.selected.extend(e.args['rows'])
        else:
            self.selected = [row for row in self.selected if row[row_key] not in e.args['keys']]
        self.update()
        arguments = TableSelectionEventArguments(sender=self, client=self.client, selection=self.selected)
        for handler in self._selection_handlers:
            handle_event(handler, arguments)
    self.on('selection', handle_selection, ['added', 'rows', 'keys'])

    def handle_pagination_change(e: GenericEventArguments) -> None:
        self.pagination = e.args
        self.update()
        arguments = ValueChangeEventArguments(sender=self, client=self.client, value=self.pagination)
        for handler in self._pagination_change_handlers:
            handle_event(handler, arguments)
    self.on('update:pagination', handle_pagination_change)

def on_select(self, callback: Callable[..., Any]) -> Self:
    """Add a callback to be invoked when the selection changes."""
    self._selection_handlers.append(callback)
    return self

def on_pagination_change(self, callback: Callable[..., Any]) -> Self:
    """Add a callback to be invoked when the pagination changes."""
    self._pagination_change_handlers.append(callback)
    return self

@classmethod
def from_pandas(cls,
                df: 'pd.DataFrame',
                row_key: str = 'id',
                title: Optional[str] = None,
                selection: Optional[Literal['single', 'multiple']] = None,
                pagination: Optional[Union[int, dict]] = None,
                on_select: Optional[Callable[..., Any]] = None,
                ) -> Self:  # DEPRECATED
    """Create a Table from a Pandas DataFrame."""
    helpers.warn_once('ui.table.from_pandas is deprecated. Use ui.table with the df argument instead.')
    return cls(df=df, row_key=row_key, title=title, selection=selection, pagination=pagination, on_select=on_select)

@property
def rows(self) -> List[Dict]:
    """List of rows, which are added to those from ``df``."""
    return self._rows

@rows.setter
def rows(self, value: List[Dict]) -> None:
    self._rows[:] = value

def _handle_rows_change(self) -> None:
    self._props['rows'] = self._merge_rows_with_dataframe()
    self.update()

@property
def df(self) -> Optional['pd.DataFrame']:
    """Optional Pandas DataFrame, which is added to those from ``rows``."""
    return self._df

@df.setter
def df(self, value: 'pd.DataFrame') -> None:
    self._df = self._clean_pandas_dataframe(value)
    self._props['rows'] = self._merge_rows_with_dataframe()
    self.update()

@property
def columns(self) -> List[Dict]:
    """List of column dictionaries."""
    return self._columns

@columns.setter
def columns(self, value: List[Dict]) -> None:
    self._columns[:] = value

def _handle_columns_change(self) -> None:
    self._props['columns'] = self._merge_columns_with_default()
    self.update()

@property
def default_column(self) -> Dict[str, Any]:
    """Default column dictionary."""
    return self._default_column

@default_column.setter
def default_column(self, value: Dict[str, Any]) -> None:
    self._default_column.update(value)

def _handle_default_column_change(self) -> None:
    self._props['columns'] = self._merge_columns_with_default()
    self.update()

def _clean_pandas_dataframe(self, df: Optional['pd.DataFrame']) -> Optional['pd.DataFrame']:
    if df is None:
        return None

    def is_special_dtype(dtype):
        return (pd.api.types.is_datetime64_any_dtype(dtype) or
                pd.api.types.is_timedelta64_dtype(dtype) or
                pd.api.types.is_complex_dtype(dtype) or
                isinstance(dtype, pd.PeriodDtype))
    special_cols = df.columns[df.dtypes.apply(is_special_dtype)]
    if not special_cols.empty:
        df = df.copy()
        df[special_cols] = df[special_cols].astype(str)

    if isinstance(df.columns, pd.MultiIndex):
        raise ValueError('MultiIndex columns are not supported. '
                         'You can convert them to strings using something like '
                         '`df.columns = ["_".join(col) for col in df.columns.values]`.')
    return df

def _merge_rows_with_dataframe(self) -> List[Dict]:
    assert self._rows is not None or self._df is not None, 'Either rows or df must be set.'
    return [
        *(self._rows or []),
        *(self._df.to_dict('records') if self._df is not None else []),
    ]

def _merge_columns_with_default(self) -> List[Dict]:
    """Merge columns with default column configuration."""
    return [{**self._default_column, **column} for column in self._columns]

@property
def row_key(self) -> str:
    """Name of the column containing unique data identifying the row."""
    return self._props['row-key']

@row_key.setter
def row_key(self, value: str) -> None:
    self._props['row-key'] = value
    self.update()

@property
def selected(self) -> List[Dict]:
    """List of selected rows."""
    return self._props['selected']

@selected.setter
def selected(self, value: List[Dict]) -> None:
    self._props['selected'][:] = value
    self.update()

@property
def pagination(self) -> dict:
    """Pagination object."""
    return self._props['pagination']

@pagination.setter
def pagination(self, value: dict) -> None:
    self._props['pagination'] = value
    self.update()

@property
def is_fullscreen(self) -> bool:
    """Whether the table is in fullscreen mode."""
    return self._props['fullscreen']

@is_fullscreen.setter
def is_fullscreen(self, value: bool) -> None:
    """Set fullscreen mode."""
    self._props['fullscreen'] = value
    self.update()

def set_fullscreen(self, value: bool) -> None:
    """Set fullscreen mode."""
    self.is_fullscreen = value

def toggle_fullscreen(self) -> None:
    """Toggle fullscreen mode."""
    self.is_fullscreen = not self.is_fullscreen

def add_rows(self, *rows: Dict) -> None:
    """Add rows to the table."""
    self.rows += rows

def remove_rows(self, *rows: Dict) -> None:
    """Remove rows from the table."""
    keys = [row[self.row_key] for row in rows]
    self.rows[:] = [row for row in self.rows if row[self.row_key] not in keys]
    self.selected[:] = [row for row in self.selected if row[self.row_key] not in keys]
    self.update()

def update_rows(self, rows: List[Dict], *, clear_selection: bool = True) -> None:
    """Update rows in the table.

    :param rows: list of rows to update
    :param clear_selection: whether to clear the selection (default: True)
    """
    self.rows[:] = rows
    if clear_selection:
        self.selected.clear()
    self.update()

async def get_filtered_sorted_rows(self, *, timeout: float = 1) -> List[Dict]:
    """Asynchronously return the filtered and sorted rows of the table."""
    return await self.get_computed_prop('filteredSortedRows', timeout=timeout)

async def get_computed_rows(self, *, timeout: float = 1) -> List[Dict]:
    """Asynchronously return the computed rows of the table."""
    return await self.get_computed_prop('computedRows', timeout=timeout)

async def get_computed_rows_number(self, *, timeout: float = 1) -> int:
    """Asynchronously return the number of computed rows of the table."""
    return await self.get_computed_prop('computedRowsNumber', timeout=timeout)

class row(Element):

    def __init__(self) -> None:
        """Row Element

        This element is based on Quasar's `QTr <https://quasar.dev/vue-components/table#qtr-api>`_ component.
        """
        super().__init__('q-tr')

class header(Element):

    def __init__(self) -> None:
        """Header Element

        This element is based on Quasar's `QTh <https://quasar.dev/vue-components/table#qth-api>`_ component.
        """
        super().__init__('q-th')

class cell(Element):

    def __init__(self) -> None:
        """Cell Element

        This element is based on Quasar's `QTd <https://quasar.dev/vue-components/table#qtd-api>`_ component.
        """
        super().__init__('q-td')

</p>
</details> 

[falkoschindler]

Thanks again for this pull request, @tmlmt!
I kept thinking about it, but couldn't find a way merge rows and columns from different sources while still allowing to modify the original data and calling table.update(). So I went one or two steps back and thought about the objectives again:

1. Allow updating pandas dataframes.
2. Allow inferring columns from the first row.
3. Support default parameters for columns.

The idea of moving from_pandas() into the initializer was kind of a dead end, because it complicates things without adding much value. Keeping a separate method to create tables from pandas seems to be the better choice.

Because I felt like starting over again, based on all the insights from your PR, implementing these three features one by one without breaking existing code, I created a new PR #3525. It also implements #2633 to increase consistency of the methods for adding, removing and updating rows. It turned out pretty well and got reviewed and merged today.

Therefore I'm closing this PR in favor of #3525. It always feels a bit weird to discard valuable contributions like yours, but sometimes we need to try things out to spark new ideas and to see what works best. I hope you understand. Thanks again for your work!

[tmlmt]

Hey @falkoschindler thanks a lot for the follow-up. And sorry I haven't continued the work before you did, things have been busy at home and I wouldn't have been able to resume work before October.

I still think that a unified way to generate tables regardless of the sources would offer a better DX but I fully understand your intentions here to avoid breaking changes and keep consistency across elements. Thanks a lot for reusing up the ideas and thought process from this PR to create the new one! With columns_defaults and update_from_pandas introduced by the new PR, the original objectives are fulfilled 😊