fix: use PyCapsule Interface instead of Dataframe Interchange Protocol

[MarcoGorelli]

closes #3756
closes #3533
I'm hoping that this can supersede #3534

This means that you get support for quite a lot more, e.g.:

• DuckDB:

• cuDF (their interchange protocol implementation is currently broken anyway [BUG] pd.api.interchange.from_dataframe fails with simple cuDF dataframe rapidsai/cudf#17282)

• Polars: it fixes the issue reported in Try to_pandas rather than erroring if interchanging to pandas doesn't work? #3533, because the PyCapsule interface actually supports nested data types:

In addition, this has no effect on existing pandas users, as there's already an early return for pandas https://github.com/MarcoGorelli/seaborn/blob/0bd85071284d45f38cbf419b8cf1efb2179eda24/seaborn/_core/data.py#L284-L285

---

I'm sorry for having introduced the Interchange Protocol in the first place. It's turned out to be fairly problematic, see pandas-dev/pandas#56732 (comment) as the associated discussion for more context

---

cc @WillAyd for comments

[WillAyd]

Implementation wise I think this looks great. Nice work @MarcoGorelli

[mwaskom]

Cool, thanks for this. I think my one question is about how compatible this will be for users that are currently benefitting from the (seemingly more-or-less built-in) interchange protocol. Do we need to provide backwards compatibility for them?

[mwaskom]

Is this generally a dependency of non-pandas dataframe libraries now? Or could this change introduce a regression for e.g. polars users who are currently leveraging the dataframe interchange protocol?

[MarcoGorelli]

Thanks for your review!

Polars doesn't depend on PyArrow, but polars.DataFrame.to_pandas always requires PyArrow. So, in practice, anyone working with both dataframe libraries may well already have PyArrow already installed

To avoid requiring PyArrow for the cases when it's not necessary, one way could be to do something like:

• try using the interchange protocol
• if it raises, then fall back to the PyCapsule Interface (which currently requires PyArrow)

This has the upside of not requiring PyArrow in some cases, but the downside of hiding issues where the interchange protocol silently produces invalid results

It may be possible to do this PyCapsule Interface conversion in the future without PyArrow but with something lighter instead, like arro3 by @kylebarron (who I'm ccing in case he has comments too)

What would be your preference?

[kylebarron]

Some polars users may not have pyarrow installed. If seaborn needs to get pandas data, the only production-ready way to do Arrow -> pandas that I know of is using pyarrow.

As Marco mentions I'm working on arro3, which is a minimal library for Arrow in Python, but Pandas interop is not a primary concern, and it's not production-ready today.

[WillAyd]

FWIW pandas 3.x is going to strongly incentivize users to install PyArrow, although it stops short of outright requiring it. In theory, the only people that shouldn't have PyArrow installed are those that operate in space/resource constrained environments, probably in headless environments like AWS Lambda where seaborn won't be used

Of course up to you how much you want to support non-PyArrow configurations, but the dataframe interchange protocol is relatively buggy and gets very little support, so you may find it easier altogether to force users towards PyArrow

[MarcoGorelli]

cuDF have said that they will deprecate the interchange format: rapidsai/cudf#17282

Plotly have stopped using it, so Seaborn is the only library left using it

At this point, I think there's a greater risk in keeping it - I don't want to force anything here of course, just making sure you're aware

[MarcoGorelli]

Could you please clarify what you mean by "if the infra isn't there yet"?

The Arrow C Interface already has quite widespread adoption and I'm not aware of edge cases in its implementations. @WillAyd wrote about switching his Pantab project over to it in Leveraging the Arrow C Data Interface, and noted

Almost immediately my issues went away [...] I felt more confident in the implementation and had to deal with less memory corruption / crashes than before. And, perhaps most importantly, I saved a lot of time.

That was nearly a year ago, and given that he's now suggesting it here in Plotly, I'd say that his experience has stayed just as positive

---

Regarding PyArrow dependency, I'll also note that polars.DataFrame.to_pandas also requires PyArrow, so any Polars user (such as myself) would already have needed PyArrow installed if they were converting to pandas via the Polars official method

[mwaskom]

Basically my threshold is "do I need to think about it at all". I'm just not interested in the minutia of competing Python dataframe libraries or the various attempts to make them work better together. The previous approach was sold as a simple protocol that always works, but it turns out that wasn't the case. Maybe this new way is better, the problem is I have no real way to say for sure without spending a lot of time learning about something that doesn't interest me.

[MarcoGorelli]

Shall I close and leave you to remove cross-dataframe compatibility altogether?

[mwaskom]

The problem is then I get issues bugging me about Polars, so I have to think about it anyway :D

[MarcoGorelli]

😄 that's understandable

I'm aware that you said that using Narwhals was a complete non-starter, but just to showcase that as a possibility:

import narwhals.stable.v1 as nw
from narwhals.stable.v1.typing import IntoDataFrame
import polars as pl
import pandas as pd

def convert_dataframe_to_pandas(data: IntoDataFrame) -> pd.DataFrame:
    return nw.from_native(data).to_pandas()

and then leave it up to Narwhals to convert to pandas in the best way for each input library

Altair, Plotly, and Vegafusion are using it as required dependency now, and Bokeh have a PR in progress to do the same

---

For completeness: the way the other libraries are using Narwhals is by making the whole logic dataframe-agnostic. In Plotly this resulted in 2-3x better performance for many plots involving group-bys (compared with converting all inputs to pandas), but I understand that you may not be interested in that

[mwaskom]

Nice, TIL