Rollback pandas-1.5

[bendnorman]

Since updating to pandas-1.5, our nightly builds have doubled in memory use. I'm opening this PR to test the memory use with pandas 1.4.x.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[codecov]

Codecov Report

Base: 82.8% // Head: 82.8% // Increases project coverage by +0.0% 🎉

Coverage data is based on head (6397698) compared to base (32caab0).
Patch has no changes to coverable lines.

Additional details and impacted files

@@          Coverage Diff          @@
##             dev   #1945   +/-   ##
=====================================
  Coverage   82.8%   82.8%           
=====================================
  Files         65      65           
  Lines       7436    7436           
=====================================
+ Hits        6158    6159    +1     
+ Misses      1278    1277    -1     

Impacted Files	Coverage Δ
src/pudl/helpers.py	87.7% <0.0%> (+0.2%)	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[bendnorman]

The nightly build for this branch passed on a 32 GB machine and has a memory profile similar to the builds before upgrading to pandas 1.5. Memory profiles:

Some potential pandas-1.5 bugs:

• Maybe we haven't found all of places we are downcast dates. See BUG: In 1.5rc0 casting Series to datetime64 with specific but non-nanosecond units has no effect pandas-dev/pandas#48574
• Maybe our downcasting workaround isn't reducing memory? Ensure PUDL works with Pandas 1.5.0 #1902
• We use .drop(inplace=True) a lot which can blow up memory but was also present in 1.4.4. See PERF: Memory duplication spotted when dropping columns/axis from a dataframe with inplace=True pandas-dev/pandas#48682

[zaneselvans]

I'd be surprised if the downcasting solution wasn't reducing memory usage in the places where it's been applied. The blowup was happening because e.g. running astype("datetime64[Y]") on an hourly dataframe in pandas 1.5 results in... an hourly dataframe. So there were still a huge number of records, and then they got merged with other dataframes, leading to tens of millions of rows where there should only have been thousands. The workaround that turns a Series into a Numpy array, downcasts, and then rebuilds a new Series definitely seemed to result in the right number of records.

I searched the codebase for any instance of datetime64 with day/month/year resolutions and the only one I found outside the report_date formatting function was in the FERC-714 hourly demand table transform, which I fixed.

Maybe there are other ways that one might try and invoke the same behavior without explicitly mentioning the data type? Should we run a memory profiler and see where it explodes?

[bendnorman]

I'll profile the EIA ETL because its using 2x memory with pandas-1.5.

[zaneselvans]

Welp, it's annoying but I guess we should merge this into dev to get the nightly builds working again and re-open #1901 until we've hunted down the memory issue, or it's been fixed upstream.