REF: use public pandas API in dataframe.empty

[jbrockmendel]

The current usage of non-public API is breaking the docbuild on a pandas PR: https://github.com/pandas-dev/pandas/pull/40149/checks?check_run_id=2102847175#step:5:152

See also: pandas-dev/pandas#40226

A slightly more performant implementation may become possible following pandas-dev/pandas#39776

[martindurant]

Thanks for taking an interest and helping out here!

Unfortunately, the existence of this function and the convoluted code it contains, is because of pandas' poor performance in forming dataframes to be assigned into. Benchmarking, I find:

# main branch
In [2]: %timeit out = fastparquet.dataframe.empty('i4,u2,f4,f2,f4,M8,category', 1000000, cats={6: 5})
903 µs ± 86.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

# this branch
In [3]: %timeit out = fastparquet.dataframe.empty('i4,u2,f4,f2,f4,M8,category', 1000000, cats={6: 5})
43.6 ms ± 668 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Fastparquet does not want this factor of 50 slowdown! I really do hope that Pandas itself creates this functionality, so that we don't have to.

[jbrockmendel]

Yep, thats a pretty big slowdown. I'll see what I can do upstream, and take another try at this in the interim.

[martindurant]

Thanks, @jbrockmendel !

Interestingly, I wrote about this all the way back in 2017. @jreback helped write this initially, and @TomAugspurger has helped keep this module in sync with pandas changes.

[jbrockmendel]

Updated with an implementation that slightly outperforms the status quo

In [3]: %timeit out = fastparquet.dataframe.empty('i4,u2,f4,f2,f4,M8,category', 1000000, cats={6: 5})
741 µs ± 11.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <-- main
712 µs ± 4.04 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <-- PR

This still accesses internals in a way that is not officially supported, but should be at least future-proof to pandas-dev/pandas#40149 (though not ArrayManager)

[martindurant]

Perfect - that persuades me. I'll leave this a day or so to see if there are other ocmments.

[martindurant]

Thanks @jbrockmendel !

[jbrockmendel]

Some of what I'm working on in pandas would be simplified if this made it into a released version of fastparquet. Is that on the horizon?

[martindurant]

Not in the next day or two, but yes.

[jbrockmendel]

i think the fix to pandas-dev/pandas#41366 (comment) is to add here

            if not isinstance(bvalues, np.ndarray):
                # e.g. DatetimeLikeBlock backed by DatetimeArray/TimedeltaArray
                values = type(bvalues)._from_sequence(values)

i'll try to reproduce the problem locally after some caffeine

[martindurant]

Thanks for looking into it. Happy to push a point release to fix pandas' CI, if needed.

I hope the line above doesn't make a copy!

[jbrockmendel]

if its a 3rd-party EA then there's no telling, but for the dt64 case this should be copy-free

[jbrockmendel]

reproduced the failure locally and the edit above fixes 2 of the 8 tests. the others look pytz-related.

[martindurant]

It it's a copy, fastparquet will be filling out an array that's no longer connected to the actual dataframe storage?

[jbrockmendel]

It it's a copy, fastparquet will be filling out an array that's no longer connected to the actual dataframe storage?

No, correctness wouldn't be affected.