Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid realizing data in preprocessor function concatenate when cubes overlap #2109

Merged
merged 1 commit into from
Jun 22, 2023

Conversation

bouweandela
Copy link
Member

@bouweandela bouweandela commented Jun 22, 2023

Description

Avoid realizing data in preprocessor function concatenate when cubes overlap.


Before you get started

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.


To help with the number pull requests:

@bouweandela bouweandela added preprocessor Related to the preprocessor dask related to improvements using Dask labels Jun 22, 2023
@codecov
Copy link

codecov bot commented Jun 22, 2023

Codecov Report

Merging #2109 (6ff7304) into main (cf3824e) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #2109   +/-   ##
=======================================
  Coverage   93.08%   93.08%           
=======================================
  Files         237      237           
  Lines       12793    12793           
=======================================
  Hits        11909    11909           
  Misses        884      884           
Impacted Files Coverage Δ
esmvalcore/preprocessor/_io.py 86.74% <100.00%> (ø)

@bouweandela bouweandela marked this pull request as ready for review June 22, 2023 08:06
Copy link
Contributor

@valeriupredoi valeriupredoi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, bud! I am a bit rusty when it comes to scalar cubes - are we 100% sure a scalar cube is always of shape () or if it got an AuxCoord could be (1, )? A test would prob be needed, but let's just get this in

@bouweandela
Copy link
Member Author

are we 100% sure a scalar cube is always of shape () or if it got an AuxCoord could be (1, )?

I would be really surprised if Cube.shape was not the same as Cube.data.shape.

A test would prob be needed

I assume you wrote some very thorough tests when you implemented this, no? Codecov reports 100% coverage, so I would expect the functional tests to work. I could add a test that lazy input results in lazy output?

@valeriupredoi
Copy link
Contributor

am gonna merge since am 99.9% sure my use case above is a non-issue. Do you want to include this in the 2.9 release @bouweandela ? It's a small fix that may bear a hefty weight

@valeriupredoi valeriupredoi merged commit c52a1fa into main Jun 22, 2023
@valeriupredoi valeriupredoi deleted the more-lazy-concatenate branch June 22, 2023 12:16
@bouweandela bouweandela added this to the v2.9.0 milestone Jun 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
dask related to improvements using Dask preprocessor Related to the preprocessor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants