feat(pandas): support memtable in pandas backend

[ogrisel]

What happened?

While working on #5466 I discovered that ibis.memtable does not work with the pandas or dask backends:

>>> import ibis
>>> expr = ibis.memtable({"column": [0, 1]})
>>> ibis.get_backend(expr)
<ibis.backends.duckdb.Backend object at 0x1288d7e90>
>>> expr.execute()
   column
0       0
1       1
>>> ibis.set_backend("pandas")
>>> expr.execute()
Traceback (most recent call last):
  Cell In[6], line 2
    expr.execute()
  File ~/code/ibis/ibis/expr/types/core.py:300 in execute
    return self._find_backend(use_default=True).execute(
  File ~/code/ibis/ibis/backends/pandas/__init__.py:240 in execute
    return execute_and_reset(node, params=params, **kwargs)
  File ~/code/ibis/ibis/backends/pandas/core.py:489 in execute_and_reset
    result = execute(
  File ~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/multipledispatch/dispatcher.py:278 in __call__
    return func(*args, **kwargs)
  File ~/code/ibis/ibis/backends/pandas/trace.py:136 in traced_func
    return func(*args, **kwargs)
  File ~/code/ibis/ibis/backends/pandas/core.py:435 in main_execute
    return execute_with_scope(
  File ~/code/ibis/ibis/backends/pandas/core.py:222 in execute_with_scope
    result = execute_until_in_scope(
  File ~/code/ibis/ibis/backends/pandas/trace.py:136 in traced_func
    return func(*args, **kwargs)
  File ~/code/ibis/ibis/backends/pandas/core.py:354 in execute_until_in_scope
    result = execute_node(
  File ~/mambaforge/envs/ibisdev/lib/python3.11/site-packages/multipledispatch/dispatcher.py:278 in __call__
    return func(*args, **kwargs)
  File ~/code/ibis/ibis/backends/pandas/trace.py:136 in traced_func
    return func(*args, **kwargs)
  File ~/code/ibis/ibis/backends/pandas/dispatch.py:32 in raise_unknown_table_node
    raise com.UnboundExpressionError(
UnboundExpressionError: Node of type 'PandasInMemoryTable' has no data bound to it. You probably tried to execute an expression without a data source.

dask has a similar problem, probably because it delegates the operation to pandas.

I have also tried with sqlite and polars which work as expected. For backends that do not support the operation we get a more explicit exception, for instance for datafusion:

Traceback (most recent call last):
  Cell In[13], line 1
    expr.execute()
  File ~/code/ibis/ibis/expr/types/core.py:300 in execute
    return self._find_backend(use_default=True).execute(
  File ~/code/ibis/ibis/backends/datafusion/__init__.py:254 in execute
    output = self.to_pyarrow(expr, params=params, limit=limit, **kwargs)
  File ~/code/ibis/ibis/backends/base/__init__.py:266 in to_pyarrow
    self.to_pyarrow_batches(expr, params=params, limit=limit, **kwargs)
  File ~/code/ibis/ibis/backends/datafusion/__init__.py:244 in to_pyarrow_batches
    frame = self._get_frame(expr, params, limit, **kwargs)
  File ~/code/ibis/ibis/backends/datafusion/__init__.py:213 in _get_frame
    return self.compile(expr, params, **kwargs)
  File ~/code/ibis/ibis/backends/datafusion/__init__.py:272 in compile
    return translate(expr.op())
  File ~/mambaforge/envs/ibisdev/lib/python3.11/functools.py:909 in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File ~/code/ibis/ibis/backends/datafusion/compiler.py:24 in operation
    raise com.OperationNotDefinedError(f'No translation rule for {type(op)}')
OperationNotDefinedError: No translation rule for <class 'ibis.backends.pandas.client.PandasInMemoryTable'>

What version of ibis are you using?

ibis master

What backend(s) are you using, if any?

pandas
dask

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[jcrist]

Yeah, the pandas backend doesn't support memtable currently. I agree the error message could be better here.

Do you have a particular need for the pandas backend, or are you more exploring the API?

[ogrisel]

No, I was just trying to inspect how the default backend config impacts some of the ibis top level function to document it #5466 and I would have expected that pandas would naturally work as a memtable backend. Raising an informative exception would do.

[anjakefala]

/take