Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Make PPL process more memory efficient #1951

Closed
wants to merge 1 commit into from
Closed

Conversation

katie-lamb
Copy link
Member

After some changes made to the plant part list inputs the plant part list creation process uses too much memory for the RMI CI GitHub runners. This PR makes some changes to the plant part list process so that the RMI CI can pass.

@katie-lamb katie-lamb self-assigned this Sep 25, 2022
@katie-lamb katie-lamb added the ppe Plant Parts EIA (formerly the EIA plant parts list) label Sep 25, 2022
@codecov
Copy link

codecov bot commented Sep 25, 2022

Codecov Report

Base: 82.8% // Head: 82.5% // Decreases project coverage by -0.2% ⚠️

Coverage data is based on head (5be4135) compared to base (32caab0).
Patch coverage: 100.0% of modified lines in pull request are covered.

Additional details and impacted files
@@           Coverage Diff           @@
##             dev   #1951     +/-   ##
=======================================
- Coverage   82.8%   82.5%   -0.3%     
=======================================
  Files         65      64      -1     
  Lines       7436    7087    -349     
=======================================
- Hits        6158    5853    -305     
+ Misses      1278    1234     -44     
Impacted Files Coverage Δ
src/pudl/metadata/fields.py 100.0% <ø> (ø)
src/pudl/metadata/resources/eia.py 100.0% <ø> (ø)
src/pudl/analysis/plant_parts_eia.py 96.6% <100.0%> (+<0.1%) ⬆️
src/pudl/helpers.py

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@zaneselvans
Copy link
Member

Did the memory usage increase happen to correspond with using pandas 1.5.0? Or is this just coincidental and independent from #1945 ?

@katie-lamb
Copy link
Member Author

The increase in memory usage was independent of the pandas update. I'm not entirely sure what happened but maybe a change with the MCOE or EIA 860 tables?

@katie-lamb
Copy link
Member Author

This was resolved in the RMI repo. The CI no longer uses too much memory and passes so I'm closing this PR.

@katie-lamb katie-lamb closed this Nov 10, 2022
@zaneselvans
Copy link
Member

@katie-lamb can we also delete the rmi-ppl-mem-fix branch if it's no longer valuable?

@katie-lamb katie-lamb deleted the rmi-ppl-mem-fix branch November 11, 2022 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ppe Plant Parts EIA (formerly the EIA plant parts list)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants