PERF: regression in DataFrame reduction ops performance #37081

[ukarroum]

• closes PERF: regression in DataFrame reduction ops performance #37081
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff

Made the change proposed by @jorisvandenbossche in #35881 (comment)

Did a very quick comparison :

With self.dtypes (old version) :

In [8]: values = np.random.randn(100000, 4)   
   ...: df = pd.DataFrame(values).astype("int") 
   ...: %timeit df.sum() 
714 µs ± 21 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

With self._iter_column_arrays() (new version) :

In [4]: values = np.random.randn(100000, 4)   
   ...: df = pd.DataFrame(values).astype("int") 
   ...: %timeit df.sum() 
477 µs ± 8.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

[jreback]

do we have an asv that covers this case, if not can you add one?

this doesn't need a note as its on master.

[ukarroum]

do we have an asv that covers this case, if not can you add one?

this doesn't need a note as its on master.

I believe we do :

https://pandas.pydata.org/speed/pandas/#stat_ops.FrameOps.time_op?p-op='sum'&p-dtype='int'

[jorisvandenbossche]

Indeed. Jeff, see the issue, it was actually catched thanks to our asv suite

[jbrockmendel]

let's only find the dtypes once (i.e. share with dtype_is_dt above

own_dtypes = [arr.dtype for arr in self._iter_column_arrays()]
# or 
own_dtypes = [blk.dtype for blk in self._mgr.blocks]

[ukarroum]

done in 41827fb

[jorisvandenbossche]

Thanks @ukarroum, looks good!

[ukarroum]

Should i do something about the : 2 failed azure pipelines ?
Looks like they're failing on master too and the ./ci/code_checks.sh localy return no error.

[jreback]

thanks @ukarroum