Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release GIL in gradient search resampling #445

Closed
djhoese opened this issue Jul 14, 2022 · 1 comment · Fixed by #455
Closed

Release GIL in gradient search resampling #445

djhoese opened this issue Jul 14, 2022 · 1 comment · Fixed by #455
Assignees
Labels
enhancement performance improves speed or decreases memory consumption, but does not otherwise change functionality

Comments

@djhoese
Copy link
Member

djhoese commented Jul 14, 2022

Problem description

It was noted in recent EWA profiling that the Python Global Interpreter Lock (GIL) was not being released during processing which caused CPU usage to hover around 100% even though multiple dask workers were being used. Adding with nogil: to the proper spots in the Cython code improved execution time and increased CPU usage to ~700% usage for 8 workers. This same lack of releasing the GIL was noticed in the gradient search code. Specifically here:

one_step_gradient_search_no_gil(data,
src_x, src_y,
xl, xp, yl, yp,
dst_x, dst_y,
x_size, y_size,
fun, image,
elements)
# return the output image
return image

and here:

one_step_gradient_search_no_gil(fake_data,
src_x, src_y,
xl, xp, yl, yp,
dst_x, dst_y,
x_size, y_size,
indices_xy, indices,
elements)
return indices

My initial tests at naively adding with nogil: around these calls resulted in a segmentation fault. My hope is that someone like @mraspaud who is familiar with the algorithm can figure out why this might be happening.

@djhoese djhoese added enhancement performance improves speed or decreases memory consumption, but does not otherwise change functionality labels Jul 14, 2022
@mraspaud
Copy link
Member

For reference: I looked a this last week and was unable to pinpoint where the segfault was coming from. Cygdb wasn't very cooperative.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement performance improves speed or decreases memory consumption, but does not otherwise change functionality
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants